Over a million developers have joined DZone.

Load Balancing Microservices at Web Scale

Architecting microservices is great, but load-balancing microservices isn't as prevalent. Here's how how major players like Netflix and Uber are accomplishing microservices load balancing at web scale.

· Performance Zone

Download Forrester’s “Vendor Landscape, Application Performance Management” report that examines the evolving role of APM as a key driver of customer satisfaction and business success, brought to you in partnership with BMC.

There’s no shortage of guides and blog posts available to provide you with best practices in architecting microservices. While all this information is helpful, what doesn’t seem to be available in such a great number are hands-on guidelines regarding how microservices can be scaled. Following a little research and sifting through lots of theoretical discussion, here is how load balancing microservices is done in practice by the big players.

Living on the Edge

When a web application frontend client communicates with a microservices-based backend server, does the frontend need to know about all the microservice instances that are available to it? For example, does a client really need to be aware of all five services that deliver web page data? The answer is a resounding NO!

Sudhir Tonse, who worked with Netflix previously and now works for Uber, talks about the concept of edge services in his talk on Scalable Microservices at Netflix. An edge service serves as a gateway to a microservices infrastructure. So, in regards to the question of which microservices a frontend client needs to know about, following Sudhir’s approach, each client only communicates directly with just a single edge service. There can be one dedicated edge service per client. For example, Netflix serves more than a thousand device types—and each device type has its own dedicated edge service that serves as its single entry point.

Image title

Load balanced edge services act as gateways to microservice environments

Big players like Netflix and Riot Games, both of which run on Amazon AWS, utilize Elastic Load Balancers (ELB) to ensure that their edge services are available at all times.

Beyond the Edge Service

Each incoming request is analyzed. Multiple fan-out requests are then issued to the microservices that form the ecosystem. A single inbound request results in an average of about ten fan-out requests. The nearly two-billion requests that Netflix receives each day result in roughly 20 billion internal API calls.

Image title

Fan-out requests in a microservice environment

How does Netflix ensure that its microservices can handle such load and remain available 24/7? Again, load balancing is the solution. But this time, it’s not by means of ELBs. With 500 different microservices, you’d need to configure about 500 ELBs! This is why Netflix’s tools come with built-in load balancing capabilities. Netflix has created numerous libraries and tools that integrate easily with one another. By integrating the required libraries directly into each microservice, it is able to register itself with all managing services.

Fear Not the Edge Service

With edge services being so important, you’re screwed if your edge service fails, right? Actually, no. For one, edge services absolutely must be load balanced. This means that your visitors likely won’t even notice edge service outages. And besides, what’s the alternative? With monolithic-application environments, each service is like an edge service, so an outage of any central service—in the absence of a load balancer—implies total outage, right?

Nonetheless, it’s true that edge services are amongst the most delicate of services and therefore do require special attention.

The Takeaway

You should seriously consider running edge services to handle your inbound traffic. And definitely load-balance your edge services with whatever mechanism is provided by your cloud provider. All internal traffic should be handled by your own tools as this allows you to run your environment with minimal configuration overhead. So, ultimately, the most important tool required for effective scaling in microservices is, not surprisingly, load balancing.

Stay Tuned

One of my next posts deals with the question of containerization. And be aware, Docker isn’t the only solution that builds on the concepts of containers.

See Forrester’s Report, “Vendor Landscape, Application Performance Management” to identify the right vendor to help IT deliver better service at a lower cost, brought to you in partnership with BMC.

performance,load balancing,web-scale

Published at DZone with permission of Martin Goodwell, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}