Circuit Breaker Design Pattern Using Netflix Hystrix

You may also like: Design Patterns in Java: Singleton

1. Introduction

A circuit breaker is an electronic/electrical component that makes a circuit open so that no current can flow through it. It is used to prevent the damage caused by excess current from an overload or short circuit.

Generally, fuses are considered to be a circuit breaker, but there is a subtle difference between circuit breakers and fuses. Fuses operate once and must be replaced but the circuit breaker can be reset to resume the normal operation.

This is exactly the role of the circuit breaker in a microservices architecture. In a distributed microservices architecture multiple services can be dependent on each other. Now, what will happen if some services fail or any exception is thrown by some services? Then this exception is propagated to upstream services and finally comes to the end-user. Circuit Breaker pattern prevents failure cascading and gives a default behavior when services fail.

Netflix Hystrix allows us to introduce fault tolerance and latency tolerance by isolating failure and by preventing them from cascading into the other part of the system building a more robust distributed application. Netflix also provides a feature to monitor all of the services within the distributed application boundary.

It can be unmanageable to monitor the health and state of all the services as more and more service will be stood up and some services may be taken down or simply break down. Hystrix comes with help by providing a user-friendly dashboard.

Article Overview

1. Create a Hystrix application for circuit breaking

2. Configure the Hystrix application.

3. Hystrix Dashboards for Monitoring service state and health.

4. Learn to recover from failover issue. 

In this article, we will create two applications. One is a very simple microservice that will just return a list of String and the other is going to be a microservice that calls the first services. Next, we will set up a Hystrix command that is going to provide us with the circuit breaking capabilities.

A circuit breaker has two state-Closed and Open. When it is in the closed state, we take the normal route and when the circuit breaker is in open state or tripped, we then take an alternative route. When there is no failure, we will continue to route the traffic in the normal set up, but if a fault occurs and we have an issue, then we are going to divert the traffic to some other microservice.

The classic example given for circuit breaking is the Netflix example. They have a microservice that provides a tailored list of movies back to the user. If the service fails, they reroute the traffic to circumvent the failure to another vanilla microservice that simply returns the top 10 movies that are family-friendly. So they have this safe failover that they can go to and that is the classic example of first circuit breaking.

Circuit Breaking is all about circumventing failures in a graceful fault-tolerant manner. Once you set up a command for circuit breaking you can specify properties to find to when circuit breaking occurs. You can set up different latency rules that can be helpful to prevent long-running commands.

We will go ahead and put those in place within our application, then we will route some traffic to perform some demonstration and get those properties we have set up.

The other thing we will do is we will take a look at the Hystrix Dashboard within our Hystrix application. We are going to be provided a stream and this is a stream of information about how our microservice is performing, whether there is a failure, how many requests are coming into the microservice.

We can see the information about service endpoints within that dashboard that can be very handy. If we start to notice service degradation within the system, it provides us with a very convenient way to monitor and identify potential failure within our distributed system.

Finally, throughout this chapter, we are going to demonstrate several failure recovery scenarios. We are going to be removing microservice and watching the failover occur as well as recovery. When a microservice goes down and Hystrix starts rerouting the traffic via circuit breaker, you will find that when service comes back- Hystrix is smart enough to close the circuit and go back to the normal method of traffic. So we will be taking a look at several scenarios where we fail just to test out some of the demonstrations of Hystrix.

2. Hystrix Application

Create a Spring Boot Application named simple-product-service-application using STS 4 and add web dependency to the project. Now edit the main class as follows.

package com.example.demo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.bind.annotation.RequestMapping;

@RestController
@SpringBootApplication
public class SimpleProductServiceApplication {

  @GetMapping(value = "/products")
  public List<String> getAllproduct(){
    return Arrays.asList(“[“Eureka”,“Hystrix”,“Zuul”,“Ribbon””)
  }

  public static void main(String[] args) {
    SpringApplication.run(SimpleProductServiceApplication.class, args);
  }
}

Next set server.port=8090  in application.properties and run the application as Spring Boot Application and hit the URL as http://localhost:8090/products, you will see the response [“Eureka”, “Hystrix”, “Zuul”, “Ribbon”] in the browser.

Next, create another spring boot application simple-client-application with web and Hystrix dependency on clsaapath that will act as client microservice as follows.

package com.example.demo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.client.RestTemplate;
import java.net.URI;

@RestController
@SpringBootApplication
public class SimpleClientApplication {

  @GetMapping
  public List<String> cloudProductList() {

RestTemplate restTemplate = new RestTemplate();
    URI uri = URI.create("http://localhost:8090/products");

    return restTemplate.getForObject(uri, List.class);
  }

  public static void main(String[] args) {
    SpringApplication.run(SimpleClientApplication.class, args);
  }
}

Add server.port = 8080 in application.properties.

Next in browser type the URL http://localhost:8080 and see a list of cloud components as [“Eureka”, “Hystrix”, “Zuul”, “Ribbon”] in the browser.

RestTemplate is used to consume other spring rest services. Its getForObject() will call the SimpleServiceApplication /products endpoint and return the List of String.

4. Circuit Breaking With Hystrix

Now, what will happen if SimpleProductService fails or down? Stop the SimpleProductService and hit the browser with http://localhost:8080 and you will see an error page.

That’s where the Hystrix comes in. It provides us a feature that will allow the traffic to the other route so that at least end-user do not need to see this type of error page. This type of exception propagates through the application in a distributed environment and the worst case makes the whole application broken down.

In that case, you can display a tailored list of the product either based on user preference or simply a static list. To do that you need two annotations.

At the top of the class add  @EnableCircuitBreaker. It makes this class be able to redirect the traffic in case of any failure. Another annotation you need is  @HystrixCommand. It takes a parameter  fallbackMethod which is the name of the method to which the request will be redirected. The resulting code will be the following.

package com.example.demo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.client.RestTemplate;
import java.net.URI;

@RestController
@SpringBootApplication
@EnableCircuitBreaker
public class SimpleClientApplication {

  @GetMapping
  @HystrixCommand(fallbackMethod = "defaultProductList")
  public String cloudProductList() {
    RestTemplate restTemplate = new RestTemplate();
    URI uri = URI.create("http://localhost:8090/products");

    return restTemplate.getForObject(uri, String.class);
  }
public List<String> defaultProductList() {
    return Arrays.asList("spring cloud");
  }


  public static void main(String[] args) {
    SpringApplication.run(SimpleClientApplication.class, args);
  }
}

Now if the  SimpleProductService  is down then request will be redirected to  defaultProductList  method and it will return the response [“spring cloud”].

Now again start the SimpleProductservice and hit the URL in the browser http://localhost:8080. You will see the response as [“Eureka”, “Hystrix”, “Zuul”, “Ribbon”]. Hystrix is smart enough to detect that the SimpleProductServiceis up again so it directs the traffic to its normal route again.

5. Configuring Hystrix

In the previous example, how does Hystrix know that  SimpleProductService  is down? The answer is, well pretty simple, it calls the SimpleProductService endpoints and waits for its response for a default period. If any response does not come from this service then within that time, Hystrix assumes it a failure. Now, what if you want to change this default period? Well, you can do it. You can also configure different other properties of Hystrix using commandProperties attribute of @HystrixCommand annotation.

WithcommandProperties, the attribute takes a value of an array of HystrixProperty which is, in turn, a key-value pair that are used to configure different behavior. Modify the previous example as follows

import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.client.RestTemplate;
import java.net.URI;

@RestController
@SpringBootApplication
@EnableCircuitBreaker
public class SimpleClientApplication {

  @GetMapping
  @HystricCommand(fallbackMethod=”defaultProductList”, commandProperties =    { @HystrixProperty(name=”execution.isolation.thread.timeoutInMilliSeconds”, value=”500”)
}) 
public String cloudProductList(@PathVariable long timeout) {

   Thread.sleep(timeout);
   RestTemplate restTemplate = new RestTemplate();
   URI uri = URI.create("http://localhost:8090/products");
   return restTemplate.getForObject(uri, String.class);

  }
public List<String> defaultProductList(long timeout) {
    return Arrays.asList("spring cloud");
  }


  public static void main(String[] args) {
    SpringApplication.run(SimpleClientApplication.class, args);
  }
}

@HystricCommand(fallbackMethod=”defaultProductList”, commandProperties = { @HystrixProperty(name=”execution.isolation.thread.timeoutInMilliSeconds”, value=”500”)})

Thread.sleep() will induce a delay that will be given by the user. One thing you must remember that the signature of the normal method and fallbackMethod must be the same.

Now restart both the application and hit the URL in browser with http://localhost:8080/400.

Since here timeout given by the user is less than timeout given in the  @HystrixPriperty ,  Hystrix will direct the traffic to normal route and response will be [“Eureka”, “Hystrix”, “Zuul”, “Ribbon”]. Now hit the browser with URL  http://localhost:8080/600 .

 Here timeout is more than timeout is given in  @HystrixPriperty  and Hystrix will consider it as a failure and redirect traffic to fallbackMethod and response will come from this  fallbackMethod  and you will see the response as [“spring cloud”].

6. Fallback Method Chaining

Now, what happens if the fallback method itself fails? You can have a fallback method of a fallback method and in turn, this fallback method can also have its fallback method and so on.

For that, you only need to mark the fallback method by @HystrixCommand as below;

@HystrixCommand(fallbackMethod = "zuul")
public String product() {
return getAllProductAsString();
}

@HystrixCommand(fallbackMethod = "eureka")
private String zuul() {
    return “Zuul”;
}

@HystrixCommand
private String eureka (){
return “Eureka”;
}

7. Default Fallback

If you need a common behavior whenever a failure occurs then you need not declare a separate fallback method for separate HysrixCommand.Instead, you could have only one default fallback method at class level as the following example.

@DefaultProperties(defaultFallback = "fallback")
    public class SimpleProductService {
        @RequestMapping(value = "/zuul")
        @HystrixCommand
        public String zuul(){
            // some codes here
            return 
        }

        @RequestMapping(value = "/eureka")
        @HystrixCommand
        public String eureka() {
            // some codes here
            return “Eureka”
        }
        @RequestMapping(value = "/hystrix")
        @HystrixCommand
        public String hystrix() {
            // some codes here
            return “Hystrix”
        }
private String fallback() {
            return “Defalut”;
        }
    }

5. Hystrix Dashboard

Hystrix dashboard allows us to monitor all of the HystrixCommand we have set up a circuit breaker.

 To enable the Hystrix dashboard you have to add HysrixDashboard dependency in the classpath and you have to replace the @EnableCircuitBreaker annotation with two annotation

 @EnableHystrix and @EnableHysrixDashBoard.

Next restart both the application and go to http://localhost:8080/hystrix. You will see a page like this

Enter the URL on this page as  http://localhost:8080/hystrix.stream  and click the button  MonitorStreamYou  will get a visual representation of metrics of all the  HystrixCommand in your application.

You can generate traffic by refreshing the  http://localhost:8080/{timeout }and make it open or closed by increasing the timeout value less or more than the threshold value (500) and check the status of the HystrixCommand.

Further Reading

Decorator Design Pattern in Java