Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Microservices Communication: Hystrix as the Jon Snow

DZone's Guide to

Microservices Communication: Hystrix as the Jon Snow

Learn how to set up a Hystrix circuit breaker in your microservices in Java, to enable early detection of resource leaks.

· Integration Zone ·
Free Resource

The new Gartner Critical Capabilities report explains how APIs and microservices enable digital leaders to deliver better B2B, open banking and mobile projects.

In the previous microservices tutorial, we learned about how to use Zuul API gateway. In this tutorial, we will learn about Hystrix, which acts as a circuit breaker of the services. Circuit breaker - is the term new to you in terms of software architecture? Don’t worry, I will discuss it in detail.

But before that, let's discuss a well-known incident in a support project (monolith).

Birth of the Night King

Folks who are in on-call support: how many times does it happen- you got a call in the night saying, the system is not responding, it is a priority 1 issue. You wake up and open your laptop, check the health check pages, and find that some servers are down, some servers have a huge memory spike. So immediately you take a thread dump and all the necessary details then restart all the servers in the pool. After restarting, you find things are quite normal and go to sleep. If you are lucky enough, then you get a good sleep, but if you are unlucky, in the morning you may face the same scenario.

So, the next day, you and your team are researching why this happened: what is the root cause of the birth of White Walkers, which ate up all the precious resources and eventually made the server become unresponsive.

You may find there is a resource leak somewhere, maybe in the code level- someone forgot to close a precious resource, like a connection. Or there were unnecessary open threads. Or there is a blocking session in the database, etc.

But hold on, why we can’t find this resource leak/birth of Night King the first time? Why does the Night King grow up silently and when he is in action, then we get notified?

It opens our eyes that there is a problem in our architecture (King's Landing), there are no techniques for early detection of a resource leak (no Jon Snow to watch the wall!).

A Practical Scenario

Let examine a simple scenario which may cause this type of scenario. Say we have an architecture where Service A and Service B are dependent on Service C. Both Service A and B query the Service C API to get some result. Now Service C is used by the underlying database to fetch a result, but unfortunately, the programmer does not close the connection in the finally block, he does it in the try block.

Now in production, if any error occurs in Service C regarding a database connection/query, it does not release the connection, so connections are not back in connection pools (the connection pool has finite resources).  But Service A and B are not aware of this scenario; they query Service C as a request comes and Service C eats up one by one free connections from the connections pool. So after a certain time, all connections are eaten up by Service C and there is no connection available in the connection pool and White Walkers (Service C) have eaten up your system. After restarting all the servers, its gives you relief for some time, but if the Service C error continues (programming fault), then again you might have to wake up in the morning (the Night King is back).

It all happens due to Service A and B, they are not aware Service C is not responding the way it should be. If they are aware, they just simply stop the querying, then we would not have faced this situation. Here the concept of the circuit breaker (in GOT, Night's Watch) comes up.

Resourse Leak--Birth of Night King

Circuit Breaker Pattern

The circuit breaker concept is same as an electrical circuit. When the circuit is closed, electrons flow through the circuit, but if any unusual thing happens, it trips the circuit, and the circuit is opened up so there is no flow of electrons through the circuit. It provides the circuit time to recover itself, and after a certain amount of time, the circuit closes and the flow of electrons continues.

Netflix Hystrix is such a framework, which works on the same principle.

It is always monitoring the calls, so if any dependent service response is greater than the threshold limit, it trips the circuit, so no further calls will flow to the dependent service. It gives the dependent service time to recover itself. In that time, there is a fallback policy; all the requests go to that fallback path. After a certain amount of time, the circuit is closed again and requests flow as is.

Please note that we can enable Hystrix (Jon Snow- King of the North) in Spring cloud. Previously, it supported only the Service and Component level, @Service or @Component. With the latest version, it supports @Controller also.

Hystrinx As JonSnow

Coding Time

Let's recap the EmployeeDashBoardService. It calls EmployeeSearchService to find employees based on the id. Currently, if EmployeeSearchService is unavailable, then EmployeeDashBoardService does not get the result and shows an error. But we want to show a Default Employee Value if EmployeeSearchService is not available, so to incorporate the change in EmployeeDashboardService, we have to do the following changes.

Step 1: Add Hystrix plugin into pom.xml.

<dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-starter-hystrix</artifactId>
</dependency>

Step 2: Add @EnableCircuitBreaker on top of  EmployeeDashBoardService, to enable Hystrix for this service.

package com.example.EmployeeDashBoardService;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.web.client.RestTemplateBuilder;
import org.springframework.cloud.client.circuitbreaker.EnableCircuitBreaker;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.cloud.netflix.feign.EnableFeignClients;
import org.springframework.context.annotation.Bean;
import org.springframework.web.client.RestTemplate;

@EnableDiscoveryClient
@EnableCircuitBreaker
@EnableFeignClients
@SpringBootApplication
public class EmployeeDashBoardService {

   public static void main(String[] args) {
      SpringApplication.run(EmployeeDashBoardService.class, args);
   }

   @Bean
   public RestTemplate restTemplate(RestTemplateBuilder builder) {
      return builder.build();
   }
}

Step 3:  Now we will change the EmployeeInfoController.java so it can be Hystrix enabled.

package com.example.EmployeeDashBoardService.controller;

import java.util.Collection;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.cloud.context.config.annotation.RefreshScope;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.client.RestTemplate;

import com.example.EmployeeDashBoardService.domain.model.EmployeeInfo;
import com.netflix.appinfo.InstanceInfo;
import com.netflix.discovery.EurekaClient;
import com.netflix.discovery.shared.Application;
import com.netflix.hystrix.contrib.javanica.annotation.HystrixCommand;

@RefreshScope
@RestController
public class EmployeeInfoController {
   
    @Autowired
    private RestTemplate restTemplate;
   
    @Autowired
    private EurekaClient eurekaClient;
   
    @Value("${service.employyesearch.serviceId}")
    private String employeeSearchServiceId;


   @RequestMapping("/dashboard/{myself}")
   @HystrixCommand(fallbackMethod="defaultMe")
   public EmployeeInfo findme(@PathVariable Long myself){
      Application application = eurekaClient.getApplication(employeeSearchServiceId);
       InstanceInfo instanceInfo = application.getInstances().get(0);
       String url = "http://"+instanceInfo.getIPAddr()+ ":"+instanceInfo.getPort()+"/"+"employee/find/"+myself;
       System.out.println("URL" + url);
       EmployeeInfo emp = restTemplate.getForObject(url, EmployeeInfo.class);
       System.out.println("RESPONSE " + emp);
       return emp;
   }
   
   private EmployeeInfo defaultMe(Long id){
      EmployeeInfo info = new EmployeeInfo();
      info.setEmployeeId(id);
      info.setName("Hystrix fallback");
      info.setCompanyInfo("Netfilx");
      info.setDesignation("Fallback");
      return info;
   }
   
   
   @RequestMapping("/dashboard/peers")
   public  Collection<EmployeeInfo> findPeers(){
      Application application = eurekaClient.getApplication(employeeSearchServiceId);
       InstanceInfo instanceInfo = application.getInstances().get(0);
       String url = "http://"+instanceInfo.getIPAddr()+ ":"+instanceInfo.getPort()+"/"+"employee/findall";
       System.out.println("URL" + url);
       Collection<EmployeeInfo> list= restTemplate.getForObject(url, Collection.class);
        System.out.println("RESPONSE " + list);
       return list;
   }
}

Carefully note the method named findme. It actually calls the EmployeeService, so I use a
@HystrixCommand (fallbackMethod="defaultMe") annotation on top of this method. By doing som we instruct Spring to proxy this method, so that if any error occurs or EmployeeService is not available, it goes through the fallback method and calls it, and shows the default value rather than showing an error.

For that, we add the attribute fallbackmethod=defaultMe, where "defaultMe" is the default method. Please note that method signature and return type must be the same as the findme method. Otherwise you face an error, "no such method found." It internally uses Spring AOP, which intercepts the method call.

If the EmployeeService is not available, then it calls the defaultMe method and returns the default employee.

Let's check it. Start Config server, Eureka server, and EmployeeDashBoardService. Intentionally, I did not start the EmployeeSearchService, so it is unavailable when we call the findme method

If you hit the URL http://localhost:8081/dashboard/2, you will see the following response, as the actual EmployeeSearchService is down.

{
  "employeeId":2,
  "name":"Hystrix fallback",
  "practiceArea":null,
  "designation":"Fallback",
  "companyInfo":"Netfilx"
}

The new Gartner Critical Capabilities for Full Lifecycle API Management report shows how CA Technologies helps digital leaders with their B2B, open banking, and mobile initiatives. Get your copy from CA Technologies.

Topics:
integration ,java ,microservices ,hystrix ,software architecture ,circuit breaker

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}