Recently, we transported a B2B web services application from a traditional hosting provider to AWS cloud. The application’s traffic was originally load balanced using Apache Load Balancer. In the AWS cloud environment, we replaced Apache Load Balancer with AWS Elastic Load Balancer. During this migration, we ran into an interesting hiccup. In this article, I will share the problem and solution, as it may benefit others, too.

Problem 

The application was a B2B (web services) application. Its services were consumed by multiple clients (who are geographically distributed all throughout the world). The application maintained the white list of IP addresses of its clients. Only the clients in the white list can invoke the services. The team had developed a Java ServletFilter in the application layer, and using this filter, it was authenticating the incoming requests IP address. (Even though it’s questionable whether it’s a secure solution, this is outside the focus of this article.)

The requests which were serviced properly under Apache Load Balancer, but started to fail when we moved to AWS Elastic Load Balancer. In fact, it was failing because of IP address authentication. How can the requests fail IP address authentication when IP filtering was happening in the application layer code? The application layer code remained unchanged in both environments. Isn’t that puzzling?

Root Cause

Figure 1: Difference in the originator's IP address.

Note: Before reading further, please review the above diagram for few seconds. It will help you to understand the below content easily.

Apache Load Balancer Preserved Originator’s IP Address

Assume that the client from IP address 1.1.1.1 fires an HTTP request to the application. The request first hits the Apache Load Balancer, then the Load Balancer send the request to the Application Server. When the request finally hits the application server, the client’s IP address was preserved in the HTTP request. For example, the Server would see the originator’s IP address as 1.1.1.1, thus, authentication worked correctly without any issues under the Apache Load Balancer.

AWS Elastic Load Balancer Not Preserving Originator’s IP Address

Under AWS Elastic Load Balancer, when the client’s HTTP request hits the application server, the client’s IP address was not preserved. For example, suppose the client sent the request from the IP 1.1.1.1. When the request hits the Server, the Server sees the AWS Elastic Load Balancer’s IP address only (i.e., 20.20.20.20) and not the client’s IP address (i.e., 1.1.1.1). Since AWS Elastic Load Balancer’s IP address isn’t white-listed, the application started to reject the requests.

Because of this discrepancy in the load balancer behavior, the application started to reject all incoming requests!

Solutions

There are a couple of solutions to address this problem.

Web ACL

AWS provides a simple but yet powerful Web Access Control List (Web ACL) service. Using this service, one can do the IP filtering in the AWS Elastic Load Balancer. Thus, the team moved the IP Filtering logic from the application layer to the Load Balancer using the Web ACL service.

The application layer also maintained a white list of Load Balancer’s IP addresses only so that no one besides Load Balancers can directly shoot the requests to the application server.

X-Forwarded-For

AWS Elastic balancer can be configured to pass client's IP address in the X-Forward-For header element. The X-Forwarded-For request header takes the following form:

X-Forwarded-For: clientIPAddress (for example, X-Forwarded-For: 203.0.113.7).

The team went with the first solution, Web ACL, as it's better to knock down the rogue request at the Load Balancer level itself instead of letting it come all the way to the Application server.