Spring Retry — How to Handle Failures

In the microservice world, we have services talking to each other. One method of communication is synchronous. However, in the cloud computing world, the fact is that we cannot avoid network glitches, temporary service downtime (due to a restart or crash; not more than a few seconds). When clients need real-time data and your downstream service is not responding momentarily, it may impact the users, so you should create a retry mechanism. There are many solution options available in Java. I am going to talk about Spring Retry in this blog. We will build a small application and see how Spring Retry works. Before we start that, let's first understand a few basics about the Retry pattern:

• Retry should be tried only if you think that it may meet your requirements. You should not use it for each use case. This is something you don't build right from the beginning but based on what you learn during development or testing. For example, you might find while testing that when you hit a resource, it works one time, but the next time, it gives a timeout error, then works fine when hit again. After checking with the downstream system, you're not able to find out any root cause or solution. You might want to build a Retry feature to handle your application's side, but the first attempt should be to fix the downstream side. Don't jump quickly to building a solution on your end.

• Retry may cause resource clogging and make things even worse, preventing the application from recovering; therefore, the number of retries has to be limited. You should try to start with a minimum count like 3 and not going beyond 5 or so.

• Retry should not be done for each exception. It should be coded only for a particular type of exception. For example, instead of putting code around Exception.class, do it for SQLException.class.

• Retry can cause multiple threads trying to access the same shared resource and locking can be a big issue. An exponential backoff algorithm has to be applied to continually increase the delay between retries until you reach the maximum limit.

• While applying Retry, idempotency has to be handled. Triggering the same request again should not trigger a duplicate transaction in the system.

Now, let's build a simple service showcasing how Spring Retry helps to address Retry.

Pre-Requisites

• Spring Boot 2.1.x

• Gradle

• Visual Studio Code/Eclipse

Gradle Dependencies

dependencies {
 implementation('org.springframework.boot:spring-boot-starter-web')
 implementation('org.springframework.retry:spring-retry')
 implementation('org.springframework.boot:spring-boot-starter-aop')

 testImplementation('org.springframework.boot:spring-boot-starter-test')
}

Spring Retry uses Spring AOP internally to workm, so it must be added as a dependency.

Enable Retry

Put the @EnableRetry annotation in the SpringBoot main class.

@EnableRetry
@SpringBootApplication
public class DemoApplication {

 public static void main(String[] args) {
  SpringApplication.run(DemoApplication.class, args);
 }
}

Put Retryable Logic in the Service

The @Retryable annotation has to be applied to a method which needs to have Retry logic. In this code, I have put a counter variable to show in the logs how many times it is trying the retry logic.

• You can configure on what exception it should trigger the retry.

• It can also define how many retry attempts it can do. The default is 3 if you don't define it. 

• The @Recover method will be called once all the retry attempts are exhausted and service still throws the Exception (SQLException, in this case). The recover method should handle the fallback mechanism for those requests.

@Service
public class BillingService {
    private static final Logger LOGGER = LoggerFactory.getLogger(BillingService.class);
    int counter =0;

    @Retryable(value = { SQLException.class }, maxAttempts = 3)
    public String simpleRetry() throws SQLException {
        counter++;
        LOGGER.info("Billing Service Failed "+ counter);
        throw new SQLException();

    }

    @Recover
    public String recover(SQLException t){
        LOGGER.info("Service recovering");
        return "Service recovered from billing service failure.";
    }
}   

Create a REST Endpoint to Test

This client is created just to hit the BillingService simpleRetry() method. 

@RestController
@RequestMapping(value="/billing")
public class BillingClientService {

    @Autowired
    private BillingService billingService;
    @GetMapping
    public String callRetryService() throws SQLException {
        return billingService.simpleRetry();
    }
}

Launch the URL and See the Logs

http://localhost:8080/billing

2018-11-16 09:59:51.399  INFO 17288 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService       : Billing Service Failed 1
2018-11-16 09:59:52.401  INFO 17288 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService       : Billing Service Failed 2
2018-11-16 09:59:53.401  INFO 17288 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService       : Billing Service Failed 3
2018-11-16 09:59:53.402  INFO 17288 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService       : Service recovering

The logs show it has tried the simpleRetry method 3 times and then routed to the recover method.

Apply BackOff Policy

As we discussed above, having back to back retries can cause locking of the resources, so we should add a BackOff policy to create a gap between retries. Change the BillingService simpleRetry method code as below:

  @Retryable(value = { SQLException.class }, maxAttempts = 3, backoff = @Backoff(delay = 5000))
    public String simpleRetry() throws SQLException {
        counter++;
        LOGGER.info("Billing Service Failed "+ counter);
        throw new SQLException();

    }

Capture the logs showing a 5 seconds gap in each retry.

2018-11-17 23:02:12.491  INFO 53392 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService       : Billing Service Failed 1
2018-11-17 23:02:17.494  INFO 53392 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService       : Billing Service Failed 2
2018-11-17 23:02:22.497  INFO 53392 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService       : Billing Service Failed 3
2018-11-17 23:02:22.497  INFO 53392 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService       : Service recovering

There are many other options for applying the Retry pattern in Java. Some are as following:

• AWS SDK can be used only if you are using an AWS-related service and AWS API Gateway through AWS SDK APIs.

• Failsafe is a lightweight, zero-dependency library for handling failures. It was designed to be as easy to use as possible, with a concise API for handling everyday use cases and the flexibility to handle everything else. 

• Java 8 Function Interface.

That's all for this blog. Let me know what all libraries you are using in microservices to handle failures and retry.