Microservices - Part 4: Spring Cloud Circuit Breaker Using Netflix Hystrix

In the microservices world, to fulfill a client request, one microservice may need to talk to other microservices. We should minimize this kind of direct dependencies on other microservices, but in some cases, it is unavoidable. If a microservice is down or not functioning properly then the issue may cascade up to the upstream services. Netflix created Hystrix library implementing the Circuit Breaker pattern to address these kinds of issues. We can use Spring Cloud Netflix Hystrix Circuit Breaker to protect microservices from cascading failures.

• Implementing Circuit Breaker pattern using @HystrixCommand
• How to propagate ThreadLocal variables
• Monitoring Circuit Breakers using Hystrix Dashboard

From catalog-service, we are invoking a REST endpoint on inventory-service to get the inventory level of a product. What if inventory-service is down? What if inventory-service is taking too long to respond, thereby slowing down all the services depending on it? We would like to have some timeouts and implement a fallback mechanism.

Add the Hystrix starter to catalog-service.

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
</dependency>

To enable the Circuit Breaker, add an @EnableCircuitBreaker annotation on the catalog-service entry-point class.

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.circuitbreaker.EnableCircuitBreaker;
import org.springframework.cloud.client.loadbalancer.LoadBalanced;
import org.springframework.context.annotation.Bean;
import org.springframework.web.client.RestTemplate;

@EnableCircuitBreaker
@SpringBootApplication
public class CatalogServiceApplication {

    @Bean
    @LoadBalanced
    public RestTemplate restTemplate() {
        return new RestTemplate();
    }

    public static void main(String[] args) {
        SpringApplication.run(CatalogServiceApplication.class, args);
    }
}

Now we can use @HystrixCommand annotation on any method we want to apply timeout and fallback method.

Let us create InventoryServiceClient.java which will invoke inventory-service REST endpoint and apply @HystrixCommand with a fallback implementation.

import com.netflix.hystrix.contrib.javanica.annotation.HystrixCommand;
import com.sivalabs.catalogservice.web.models.ProductInventoryResponse;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Service;
import org.springframework.web.client.RestTemplate;

import java.util.Optional;

@Service
@Slf4j
public class InventoryServiceClient {
    private final RestTemplate restTemplate;

    @Autowired
    public InventoryServiceClient(RestTemplate restTemplate) {
        this.restTemplate = restTemplate;
    }

    @HystrixCommand(fallbackMethod = "getDefaultProductInventoryByCode")
    public Optional<ProductInventoryResponse> getProductInventoryByCode(String productCode)
    {
        ResponseEntity<ProductInventoryResponse> itemResponseEntity =
                restTemplate.getForEntity("http://inventory-service/api/inventory/{code}",
                        ProductInventoryResponse.class,
                        productCode);
        if (itemResponseEntity.getStatusCode() == HttpStatus.OK) {
            return Optional.ofNullable(itemResponseEntity.getBody());
        } else {
            log.error("Unable to get inventory level for product_code: " + productCode + ", StatusCode: " + itemResponseEntity.getStatusCode());
            return Optional.empty();
        }
    }

    @SuppressWarnings("unused")
    Optional<ProductInventoryResponse> getDefaultProductInventoryByCode(String productCode) {
        log.info("Returning default ProductInventoryByCode for productCode: "+productCode);
        ProductInventoryResponse response = new ProductInventoryResponse();
        response.setProductCode(productCode);
        response.setAvailableQuantity(50);
        return Optional.ofNullable(response);
    }
}

import lombok.Data;

@Data
public class ProductInventoryResponse {
    private String productCode;
    private int availableQuantity;
}

We have annotated the method from where we are making a REST call with @HystrixCommand(fallbackMethod = "getDefaultProductInventoryByCode") so that if it doesn't receive the response within the certain time limit the call gets timed out and invoke the configured fallback method. The fallback method should be defined in the same class and should have the same signature. In the fallback method getDefaultProductInventoryByCode() we are setting the availableQuantity to 50, obviously, this behavior depends on what business wants.

We can customize the @HystrixCommand default behavior by configuring properties using @HystrixProperty annotations.

@HystrixCommand(fallbackMethod = "getDefaultProductInventoryByCode",
    commandProperties = {
       @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "3000"),
       @HystrixProperty(name = "circuitBreaker.errorThresholdPercentage", value="60")
    }
)
public Optional<ProductInventoryResponse> getProductInventoryByCode(String productCode)
{
    ....
}

Instead of configuring these parameter values in the code, we can configure them in bootstrap.properties/yml files as follows.

hystrix.command.getProductInventoryByCode.execution.isolation.thread.timeoutInMilliseconds=2000
hystrix.command.getProductInventoryByCode.circuitBreaker.errorThresholdPercentage=60

Note that we are using the method name as the commandKey which is the default behavior. We can customize the commandKey name as follows:

@HystrixCommand(commandKey = "inventory-by-productcode", fallbackMethod = "getDefaultProductInventoryByCode")
public Optional<ProductInventoryResponse> getProductInventoryByCode(String productCode)
{
    ...
}

hystrix.command.inventory-by-productcode.execution.isolation.thread.timeoutInMilliseconds=2000
hystrix.command.inventory-by-productcode.circuitBreaker.errorThresholdPercentage=60

You can find all the configuration options available here.

By default, the methods with @HystrixCommand will be executed on a different thread because the default execution.isolation.strategy is ExecutionIsolationStrategy.THREAD. So, the ThreadLocal variables we set before invoking @HystrixCommand methods won't be available within @HystrixCommand methods.

One option to make the ThreadLocal variables available is using execution.isolation.strategy=SEMAPHORE.

@HystrixCommand(fallbackMethod = "getDefaultProductInventoryByCode",
    commandProperties = {
        @HystrixProperty(name="execution.isolation.strategy", value="SEMAPHORE")
    }
)
public Optional<ProductInventoryResponse> getProductInventoryByCode(String productCode)
{
    ...
}

If you set the property execution.isolation.strategy to SEMAPHORE then Hystrix will use semaphores instead of threads to limit the number of concurrent parent threads that invoke the command. You can read more about How Isolation Works here.

Another option to make ThreadLocal variables available within Hystrix command methods is implementing our own HystrixConcurrencyStrategy.

Suppose you want to propagate some CorrelationId set as a ThreadLocal variable.

public class MyThreadLocalsHolder {
    private static final ThreadLocal<String> CORRELATION_ID = new ThreadLocal();

    public static void setCorrelationId(String correlationId) {
        CORRELATION_ID.set(correlationId);
    }

    public static String getCorrelationId() {
        return CORRELATION_ID.get();
    }
}

Let us implement our own HystrixConcurrencyStrategy.

@Component
@Slf4j
public class ContextCopyHystrixConcurrencyStrategy extends HystrixConcurrencyStrategy {

    public ContextCopyHystrixConcurrencyStrategy() {
        HystrixPlugins.getInstance().registerConcurrencyStrategy(this);
    }

    @Override
    public <T> Callable<T> wrapCallable(Callable<T> callable) {
        return new MyCallable(callable, MyThreadLocalsHolder.getCorrelationId());
    }

    public static class MyCallable<T> implements Callable<T> {

        private final Callable<T> actual;
        private final String correlationId;

        public MyCallable(Callable<T> callable, String correlationId) {
            this.actual = callable;
            this.correlationId = correlationId;
        }

        @Override
        public T call() throws Exception {
            MyThreadLocalsHolder.setCorrelationId(correlationId);
            try {
                return actual.call();
            } finally {
                MyThreadLocalsHolder.setCorrelationId(null);
            }
        }
    }
}

Now you can set the CorrelationId before calling Hystrix command and access CorrelationId within Hystrix command.

public Optional<Product> findProductByCode(String code) 
{
    ....
    String correlationId = UUID.randomUUID().toString();
    MyThreadLocalsHolder.setCorrelationId(correlationId);
    log.info("Before CorrelationID: "+ MyThreadLocalsHolder.getCorrelationId());
    Optional<ProductInventoryResponse> responseEntity = inventoryServiceClient.getProductInventoryByCode(code);
    ...
    log.info("After CorrelationID: "+ MyThreadLocalsHolder.getCorrelationId());
    ....

}

@HystrixCommand(fallbackMethod = "getDefaultProductInventoryByCode")
public Optional<ProductInventoryResponse> getProductInventoryByCode(String productCode)
{
    ...
    log.info("CorrelationID: "+ MyThreadLocalsHolder.getCorrelationId());
}

This is just one example of how to propagate the data to the Hystrix command. Similarly, we can pass any data available in current HTTP Request, let's say by using Spring components like RequestContextHolder etc.

Jakub Narloch wrote a nice article on how to propagate Request Context and even created a Spring Boot starter too. Please look at his blog and GitHub Repo.

Once we add Hystrix starter to catalog-service we can get the circuits status as a stream of events using Actuator endpoint http://localhost:8181/actuator/hystrix.stream, assuming catalog-service is running on 8181 port.

Spring Cloud also provides a nice dashboard to monitor the status of Hystrix commands.
Create a Spring Boot application with Hystrix Dashboard starter and annotate the main entry-point class with @EnableHystrixDashboard.

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-netflix-hystrix-dashboard</artifactId>
</dependency>

Let us say we are running Hystrix Dashboard on 8788 port, then go to http://localhost:8788/hystrix to view the dashboard.

Now, in the Hystrix Dashboard home page, enter http://localhost:8181/actuator/hystrix.stream as the stream URL and give Catalog Service as the Title and click on the Monitor Stream button.

Now invoke the catalog-service REST endpoint which internally invokes inventory-service REST endpoint and you can see the Circuit status along with how many calls succeed and how many failures occurred etc.

Instead of having a separate dashboard for every service we can use Turbine to provide a unified view of all services in a single dashboard. For more details see http://cloud.spring.io/spring-cloud-static/Finchley.M7/single/spring-cloud.html#_turbine.

You can find the source code for this article at https://github.com/sivaprasadreddy/spring-boot-microservices-series.