DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Last call! Secure your stack and shape the future! Help dev teams across the globe navigate their software supply chain security challenges.

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Releasing software shouldn't be stressful or risky. Learn how to leverage progressive delivery techniques to ensure safer deployments.

Avoid machine learning mistakes and boost model performance! Discover key ML patterns, anti-patterns, data strategies, and more.

Related

  • A Guide to Microservices Deployment: Elastic Beanstalk vs Manual Setup
  • Automate Application Load Balancers With AWS Load Balancer Controller and Ingress
  • Adding a Custom Domain and SSL to AWS EC2
  • Zero Trust for AWS NLBs: Why It Matters and How to Do It

Trending

  • MySQL to PostgreSQL Database Migration: A Practical Case Study
  • Integration Isn’t a Task — It’s an Architectural Discipline
  • Beyond ChatGPT, AI Reasoning 2.0: Engineering AI Models With Human-Like Reasoning
  • A Deep Dive Into Firmware Over the Air for IoT Devices
  1. DZone
  2. Software Design and Architecture
  3. Cloud Architecture
  4. Troubleshooting HTTP 502 Bad Gateway in AWS EBS

Troubleshooting HTTP 502 Bad Gateway in AWS EBS

Learn how to troubleshoot HTTP 502 bad gateway in AWS EBS.

By 
Ram Lakshmanan user avatar
Ram Lakshmanan
DZone Core CORE ·
Updated Jun. 25, 22 · Tutorial
Likes (1)
Comment
Save
Tweet
Share
17.9K Views

Join the DZone community and get the full member experience.

Join For Free

The application that we are going to discuss in this post was running on Elastic Beanstalk (EBS) service in Amazon Web Services (AWS). Intermittently this application was throwing an HTTP 502 Bad Gateway error. In this post, let’s discuss how we troubleshoot and resolve this HTTP 502 bad gateway error in the Elastic Beanstalk service.

AWS Elastic Beanstalk Architecture

This application was running on AWS Elastic Load Balancer, Nginx 1.18.0, Java 8, Tomcat 8, and Amazon Linux in AWS Elastic BeanStalk service (EBS). For the folks who are not that familiar with EBS, below is its high-level architecture. 

Fig: AWS Elastic Beanstalk architecture

AWS Elastic Beanstalk architecture

There is an AWS elastic load balancer in the forefront. This Load Balancer will distribute the traffic to a set of EC2 instances (which can be auto-scaled). Each EC2 instance will have an Nginx web server and a Tomcat application server. Requests sent by the Elastic Load Balancer are first handled by the Nginx server. Then the Nginx server forwards the request to the tomcat server. 

HTTP 502 Bad Gateway Error

Intermittently (not always), this application was throwing HTTP 502 bad gateway errors. A few seconds later once again service will resume and things will start to function normally. It wasn’t clear what was causing this HTTP 502 bad gateway error in the AWS Elastic Beanstalk environment.

We first need to understand what this HTTP 502 bad gateway error means. This error is thrown by a web server/gateway/proxy server when it gets an invalid response from the backend end server to which it’s talking.

HTTP 502 Thrown by Nginx in AWS EBS

Now the question is: There are three primary components in the EBS stack:

  1. Elastic Load Balancer
  2. Nginx web server
  3. Tomcat Application server

In these 3 components which one is throwing HTTP 502 bad gateway error? 

Screenshot of the HTTP 502 Bad gateway error thrown by Nginx server

Screenshot of the HTTP 502 Bad gateway error thrown by Nginx server

Above is the screenshot of the HTTP 502 bad gateway error that we were receiving. There is a clue in this screenshot to indicate who is throwing this HTTP 502 error. If you notice the highlighted part of the screen, you will see this HTTP 502 bad gateway error to be thrown by the Nginx server. 

As per the HTTP 502 error definition, Nginx should be throwing this error only if it would have got an invalid response from the tomcat server. Thus, this clue helped to narrow down that the Tomcat server is the source of the problem.

Out of Memory: Kill Process or Sacrifice Child

In order to identify the source of the problem, we executed the open-source yCrash script on the EC2 instance in which the tomcat server was running. yCrash script captures 16 different artifacts from the technology stack, which includes: Garbage Collection log, thread dump, heap dump, ps, top, top -H, vmstat, netstat, ….. We uploaded the captured artifacts into the yCrash server for analysis. 

One of the artifacts that yCrash script captures is a kernel log file. In this log file, all the events that happened in the Linux kernel can be found. yCrash pulls out critical errors and warnings from the log file & presents them. Below is the analysis report of the kernel log generated by the yCrash.

yCrash’s Kernel log analysis reporting ‘Out of memory: kill process or sacrifice child’

yCrash’s Kernel log analysis reporting ‘Out of memory: kill the process or sacrifice child’

Please see the highlighted error message in the kernel log:

[Sat May 21 17:31:00 2022] Out of memory: Kill process 24339 (java) score 874 or sacrifice child

It indicates that the Tomcat server which is a Java process was terminated. Linux kernel will terminate processes if their memory consumption exceeds the device’s RAM capacity limit. This is the exact scenario happening in this application as well. Whenever the application’s memory consumption goes beyond the capacity limits, the Linux kernel was terminating the tomcat server. 

Root Cause – Lack of RAM

Now the question is: How is Linux terminating the tomcat server that can result in intermittent HTTP 502 bad gateway error? Shouldn’t a complete outage need to happen? It’s a fair question. 

If you recall, this application is running on AWS Elastic Beanstalk (EBS) service. EBS service will automatically restart the tomcat server whenever it gets terminated. Thus it’s hilarious. Linux is terminating and EBS is restarting the tomcat server. During this intermittent period, customers were experiencing HTTP 502 bad gateway errors.

Solution – Upgrading EC2 Instance RAM capacity

Apparently, the application was running on EC2 instances which had only 1GB RAM capacity. It wasn’t sufficient memory to run the tomcat server, Nginx server, and other kernel processes. Thus, when the application was upgraded to run on a 2GB RAM capacity EC2 instance, the problem got resolved. 

Note: Here is a similar problem faced by another application that was running in a non-AWS environment. It might be an interesting read as well.


AWS AWS Elastic Beanstalk Apache Tomcat Linux kernel application Load balancing (computing)

Opinions expressed by DZone contributors are their own.

Related

  • A Guide to Microservices Deployment: Elastic Beanstalk vs Manual Setup
  • Automate Application Load Balancers With AWS Load Balancer Controller and Ingress
  • Adding a Custom Domain and SSL to AWS EC2
  • Zero Trust for AWS NLBs: Why It Matters and How to Do It

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!