How Netflix's Circuit Breaker Works

When working with microservice remote calls, execution in different software is one of the most important and common aspects of an environment. These systems will probably be on different machines distributed over a network. But what happens when a call to one of these systems fails or doesn't respond appropriately? The system may fail as a whole, and the problem can cascade to other systems that depend on that request. 

To avoid this kind of problem, we can use a software pattern called Circuit Breaker. The idea is simple — exactly how a circuit breaker works in a house. You design a remote call for a circuit breaker object, which monitors failures. When these failures reach a certain limit, the circuit breaker trips, and when that happens, your system can disarm the service and provide a warning that the circuit breaker has been tripped.

Netflix has created a tool that implements Circuit Breaker called Hystrix — and Spring Cloud helped ease this standard into place.

The first component to this sample is eureka, though I talk about eureka in these posts. In this post, I will focus on Hystrix and Turbine. Just for information, here's the component code for setting up the eureka server.

And this component should be the first to run. When it's up, you should see this screen on http://localhost:8761:

The second component is a service with the eureka client called recommend-service.

This service has two operations — buying and recommending books. You can see it on http://localhost:8090.

The third component is the Service with Hystrix/ Circuit Breaker.

This service is called read-service, in this service will be check if service dependency (recommended-service) is available, and check if circuit is Open or Close.

There are two operations: /to-read  and /to-buy

This service has configuration to Hystrix Dashboard and eureka client as well.

Available on http://localhost:8080

Third component code:

In this picture the annotation @HystrixCommand is the configuration to circuit breaker, if something wrong happen the method reliable will be called, the same in the second @HystrixCommand, if something wrong happen the method realiableBuy will be called.

The services are called through Eureka Client, the concept is, ask for eureka server the address to recommend-service for example.

The last component is Turbine, which is a tool for aggregating events produced on Hystrix.

Let's imagine that we have a cluster with almost 10 read-services, each one with Hystrix. It's difficult monitoring all the circuits. That's where Turbine comes in, providing aggregation for the circuit breakers. It is available on http://localhost:8989.

In this picture, the cluster config means the service that has @HystrixCommand should have the service name registered on Eureka as READ.

After all these components start on the eureka server, http://localhost:8761, it's possible to see three instances registered.

Through the link http://localhost:8080/hystrix, we can see the Hystrix dashboard. Inform the Single Hystrix with http://localhost:8080/hystrix.stream.

This screen below is waiting for a request to build the circuit.

Fist request: http://localhost:8080/to-read.
Second request: http://localhost:8080/to-buy.

In the above picture, the circuit is closed.

To open Turbine, it's the same steps as Hystrix, but you should inform the cluster via Turbine: http://localhost:8989//turbine.stream?cluster=READ.

That will open the same screen that Hystrix, but if I have more services, they will appear in an aggregate way.

Now I switched off recommend-service. The default configuration is 20 requests to consider the circuit open. I made 20 to-buy requests, http://localhost:8080/to-buy, and here was the result:

All my project samples are on GitHub.