How to Implement Client-Side Load Balancing With Spring Cloud

It is common for microservice systems to run more than one instance of each service. This is needed to enforce resiliency. It is therefore important to distribute the load between those instances. The component that does this is the load balancer. Spring provides a Spring Cloud Load Balancer library. In this article, you will learn how to use it to implement client-side load balancing in a Spring Boot project.

Client and Server Side Load Balancing

We talk about client-side load balancing when one microservice calls another service deployed with multiple instances and distributes the load on those instances without relying on external servers to do the job. Conversely, in the server-side mode, the balancing feature is delegated to a separate server, that dispatches the incoming requests. In this article, we will discuss an example based on the client-side scenario.

Load Balancing Algorithms

There are several ways to implement load balancing.  We list here some of the possible algorithms:

• Round robin: The instances are chosen one after the other sequentially, in a circular way (after having called the last instance in the sequence, we restart from the first).
• Random choice: The instance is chosen randomly.
• Weighted: The choice is made by a weight assigned to each node, based on some quantity (CPU or memory load, for example). 
• Same instance: The same instance previously called is chosen if it's available.

Spring Cloud provides easily configurable implementations for all of the above scenarios. 

Spring Cloud Load Balancer Starter

Supposing you work with Maven, to integrate Spring Cloud Load Balancer in your Spring Boot project, you should first define the release train in the dependency management section:

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-dependencies</artifactId>
      <version>2023.0.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

Then you should include the starter named spring-cloud-starter-loadbalancer in the list of dependencies:

<dependencies>
  <dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-loadbalancer</artifactId>
  </dependency>        
  ...     
</dependencies>

Load Balancing Configuration

We can configure our component using the application.yaml file. The @LoadBalancerClients annotation activates the load balancer feature and defines a configuration class by the defaultConfiguration parameter.

@SpringBootApplication 
@EnableFeignClients(defaultConfiguration = BookClientConfiguration.class) 
@LoadBalancerClients(defaultConfiguration = LoadBalancerConfiguration.class) 
public class AppMain { 

    public static void main(String[] args) { 	
		SpringApplication.run(AppMain.class, args);     
	} 
}

The configuration class defines a bean of type ServiceInstanceListSupplier and allows us to set the specific balancing algorithm we want to use. In the example below we use the weighted algorithm. This algorithm chooses the service based on a weight assigned to each node.

@Configuration
public class LoadBalancerConfiguration {

    @Bean
    public ServiceInstanceListSupplier discoveryClientServiceInstanceListSupplier(ConfigurableApplicationContext context) {
        return ServiceInstanceListSupplier.builder()
            .withBlockingDiscoveryClient()
            .withWeighted()
            .build(context);
    }
}

Testing Client-Side Load Balancing

We will show an example using two simple microservices, one that acts as a server and the other as a client. We imagine the client as a book service of a library application that calls an author service. We will implement this demonstration using a JUnit Test. You can find the example in the link at the bottom of this article.

The client will call the server through OpenFeign. We will implement a test case simulating the calls on two server instances using Hoverfly, an API simulation tool. The example uses the following versions of Java, Spring Boot, and Spring Cloud.

• Spring Boot: 3.2.1
• Spring Cloud: 2023.0.0 
• Java 17

To use Hoverfly in our JUnit test, we have to include the following dependency:

<dependencies>		
  <!-- Hoverfly -->
  <dependency>
    <groupId>io.specto</groupId>
    <artifactId>hoverfly-java-junit5</artifactId>
    <scope>test</scope>
  </dependency>
</dependencies>

We will configure the load balancer in the client application with the withSameInstancePreference algorithm. That means that it will always prefer the previously selected instance if available. You can implement that behavior with a configuration class like the following:

@Configuration
public class LoadBalancerConfiguration {

    @Bean
    public ServiceInstanceListSupplier discoveryClientServiceInstanceListSupplier(ConfigurableApplicationContext context) {
        return ServiceInstanceListSupplier.builder()
            .withBlockingDiscoveryClient()
            .withSameInstancePreference()
            .build(context);
    }
}

We want to test the client component independently from the external environment. To do so we disable the discovery server client feature in the application.yaml file by setting the eureka.client.enabled property to be false. We then statically define two author service instances, on ports 8091 and 8092:

spring:
   application:
      name: book-service
   cloud:
      discovery:
         client:
            simple:
               instances:
                  author-service:
                    - service-id: author-service
                      uri: http://author-service:8091
                    - service-id: author-service
                      uri: http://author-service:8092                          
eureka:
   client: 
       enabled: false

We annotate our test class with @SpringBootTest, which will start the client's application context. To use the port configured in the application.yaml file, we set the webEnvironment parameter to the value of SpringBootTest.WebEnvironment.DEFINED_PORT. We also annotate it with @ExtendWith(HoverflyExtension.class), to integrate Hoverfly into the running environment.

Using the Hoverfly Domain-Specific Language, we simulate two instances of the server application, exposing the endpoint /authors/getInstanceLB. We set a different latency for the two, by the endDelay method. On the client, we define a /library/getAuthorServiceInstanceLB endpoint, that forwards the call through the load balancer and directs it to one instance or the other of the getInstanceLB REST service. 

We will perform 10 calls to /library/getAuthorServiceInstanceLB in a for loop. Since we have configured the two instances with very different delays we expect most of the calls to land on the service with the least delay. We can see the implementation of the test in the code below:

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.DEFINED_PORT)
@ExtendWith(HoverflyExtension.class)
class LoadBalancingTest {

    private static Logger logger = LoggerFactory.getLogger(LoadBalancingTest.class);

    @Autowired
    private TestRestTemplate restTemplate;

    @Test
    public void testLoadBalancing(Hoverfly hoverfly) {
        hoverfly.simulate(dsl(
            service("http://author-service:8091").andDelay(10000, TimeUnit.MILLISECONDS)
                .forAll()
                .get(startsWith("/authors/getInstanceLB"))
                .willReturn(success("author-service:8091", "application/json")),
    
            service("http://author-service:8092").andDelay(50, TimeUnit.MILLISECONDS)
                .forAll()
                .get(startsWith("/authors/getInstanceLB"))
                .willReturn(success("author-service:8092", "application/json"))));

        int a = 0, b = 0;
        for (int i = 0; i < 10; i++) {
            String result = restTemplate.getForObject("http://localhost:8080/library/getAuthorServiceInstanceLB", String.class);
            if (result.contains("8091")) {
                ++a;
            } else if (result.contains("8092")) {
                ++b;
            }
            logger.info("Result: ", result);
        }
        logger.info("Result: author-service:8091={}, author-service:8092={}", a, b);
    }
}

If we run the test we can see all the calls targeting the instance with 20 milliseconds delay. You can change the values by setting a lower range between the two delays to see how the outcome changes.

Conclusion

Client load balancing is an important part of microservices systems. It guarantees system resilience when one or more of the nodes serving the application are down. In this article, we have shown how it can be implemented by using Spring Cloud Load Balancer.

You can find the source code of the example of this article on GitHub.