Seeing 5XXs When Configuring a Kubernetes API Gateway for the First Time?

DZone 's Guide to

Seeing 5XXs When Configuring a Kubernetes API Gateway for the First Time?

I wanted to share some work we've been doing to create the K8s Initializer tool, which automatically generates Kubernetes Ingress configuration.

· DevOps Zone ·
Free Resource

Kubernetes is a fantastic foundation for an application platform, but it is just that: a foundational component. In order for K8s to be useful for application developers the following components must be added to Kubernetes: ingress, an API gateway, and observability; you need to get user traffic into your applications, and you need to be able to understand what is going on. 

Getting K8s Ingress up and running for the first time can be challenging due to the various cloud vendor load balancer implementations. I've seen my fair share of 5XX HTTP errors, and have not been able to identify where the problem lies...

Often it has been a result of bad K8s config on my part. I wanted to share some work we've been doing to create the K8s Initializer tool, which automatically generates Kubernetes Ingress configuration that follows best practices and integrates correctly cloud networking and security config.

Debugging Kubernetes API Gateway Issues

When you are first setting up your cluster it can be infuriating to make requests to your services and see 404s, 503s, or some other error and not know where the problem is. I’ve found myself curling various endpoints, often with the verbose flag set, with a hope of getting a clue to the actual issue.

I frequently find myself running “kubectl get svc” and looking at the IP of the Ingress. I then switch to my cloud vendor CLI and attempt to figure out if I’ve configured the underlying load balancer and networking correctly. I’m often making requests against the ingress and running “kubectl logs -f” on the ingress pod in the hope of seeing my request show up here. Sometimes these tactics work, sometimes they don’t. 

Another tool I regularly use in this situation is ksniff, which provides a great Kubernetes-focused UX on top of Wireshark. I've written about this more in a previous DZone article "Verifying Service Mesh TLS in Kubernetes, Using Ksniff and Wireshark

When it comes to identifying the issue with Ingress, more often than not I find out that I have made a silly Kubernetes config error that is causing everything to break. My attempts at composing my Ingress setup and platform config by copying and pasting StackOverflow YAML snippets never ends well...

One tactic I learned early in my software development career was to isolate components and reduce the variables involved when trying to debug an issue. With the ingress problems described here, the easiest variable to eliminate is bad Kubernetes config.

Start With Firm (Kubernetes) Foundations

I learned in college that computer networking is not easy. Virtualized networking in the cloud and with containers is next-level challenging. I’ve frequently struggled with selecting the correct load balancers, e.g. in AWS should I use an ALB, NLB, or ELB?. And I get stumped when configuring the edge stack: should I reject insecure HTTP traffic, how do I terminate TLS at the gateway, and can I preserve the client IP?

At Ambassador Labs we've seen and fixed these problems for customers repeatedly over the last few years, and so we have decided to encode our learnings into a free tool: K8s Initializer. With this tool you simply answer a few simple questions about your Kubernetes cluster setup (or intended setup) and a series of battle-hardened YAML config files are then automatically generated for you to download and apply to your cluster. 


You can still choose what cloud load balancer you are running and also specify observability configurations and security options, but the logic under the hood of the K8s Initializer makes sure that the YAML generated integrates all of your chosen options using the current best practices. This means no more “curl -v http://mycluster” or grepping ingress logs. At least not as a result of bad Kubernetes config that you wrote!

You can find the K8s Initializer at https://app.getambassador.io/ We're looking to add support for more applications and configuration over the coming months. We're very keen to hear your feedback, and also to understand where any more challenging config issues remain!

api, api gateway, debugging, devops, ingress controller, kubernetes, networking

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}