How To Create Asynchronous and Retryable Methods With Failover Support

While developing an application, we need to make some processing more robust and less fault-tolerant, especially when requesting remote services that may remain down for a long duration.

In this article, we will introduce a new framework that aims to provide a declarative non-blocking retry support for methods in Spring-based applications using annotations.

This framework has two main implementations:

1. Thread pool task-based implementation: This implementation is based on ThreadPoolTaskExecutor without keeping the task executor thread busy during the whole retry processing, unlike the combination of @Async (see "Spring Boot - Async methods") and @Retryable (see "Retry Handling With Spring-Retry"). Indeed, when using the Spring traditional retry annotation, the thread that runs the method performs the whole retry policy, including waiting periods, and remains busy until the end. For example, if the ThreadPoolTaskExecutor has 10 threads with a retry policy that may take 5 minutes, and the application receives 30 requests, only 10 requests can be processed simultaneously and the others will be blocked for the whole 5 minutes. So the execution of the 30 requests may take 15 minutes.
2. Quartz job-based implementation: This implementation is based on the Quartz library. It supports load-balancing, failover, and persistence if configured with JDBC JobStore. This means that even if a node in the cluster is down, the others can take over the operation and perform the retries.

Basic Concepts

Annotation

In order to make a method retryable, you can annotate it with @AsyncRetryable and specify the following attributes:

• retryPolicy (mandatory): The policy bean name that defines the next retry time if an exception has been thrown during the method execution
• retryFor (optional): List of the exceptions for which the retry should be performed
• noRetryFor (optional): List of the exceptions for which the retry should not be performed
• retryListener (optional): The listener bean name that triggers events related to the annotated method execution

The following example shows how to use @AsyncRetryable in a declarative style:

@Bean
public class Service {
    
    @AsyncRetryable(retryPolicy = "fixedWindowRetryableSchedulingPolicy", 
                    retryFor = IOException.class, 
                    noRetryFor={ArithmeticException.class}, 
                    retryListener = "retryListener")
    public void method(String arg1, Object arg2){
        // ...do something
    }
}

In this example, if an exception of type IOException is thrown during the method perform() execution, the retry will be made according to the policy fixedWindowRetryableSchedulingPolicy. If an exception of type ArithmeticException is thrown, no retry will be made.

All the events that happened during the method call are reported to the bean listener retryListener.

Retry Policy

The retry policy defines the next execution time for each failing execution. Indeed, when the annotated method throws a retryable exception, the retry policy bean is called in order to get the period to wait before calling the method again.

This framework provides three basic implementations:

1. FixedWindowRetryableSchedulingPolicy: This policy is used for retries with a fixed waiting period and a max attempt limit.
2. StaticAsyncRetryableSchedulingPolicy: This policy accepts an array of waiting periods for each attempt.
3. LinearAsyncRetryableSchedulingPolicy: This policy multiplies each time the previous waiting period by a coefficient in order to increase the duration of the next one. The coefficient default value is 2.

The following example shows how to configure a FixedWindowRetryableSchedulingPolicy that will trigger the annotated method for the first time in 10 seconds, then make 3 retries within a waiting period of 20 seconds each.

@Configuration
public class AsyncRetryConfiguration {
    
    ...

    @Bean
    public FixedWindowRetryableSchedulingPolicy fixedWindowRetryableSchedulingPolicy() {
        return new FixedWindowRetryableSchedulingPolicy(10000,3,20000);
    }
}

Remark: It is possible to customize the retry policy by implementing the interface AsyncRetryableSchedulingPolicy. 

Retry Listener

The retry listener is used to detect events during the retry processing life cycle. The AsyncRetryableListener interface is defined as below:

public interface AsyncRetryableListener<T> {

    void beforeRetry(Integer retryCount, Object[] args);

    void afterRetry(Integer retryCount,T result, Object[] args, Throwable e);

    void onRetryEnd(Object[] args, Throwable e);

}

The methods beforeRetry() and afterRetry() are triggered respectively before and after the call of the annotated method. The method onRetryEnd() is triggered at the end of the retry process. The methods defined above are called nether the annotated method succeeds or fails.

The methods' attributes are:

• retryCount: The current retry number
• result: The value returned by the annotated method in case of success
• args: The annotated method argument values in the same order
• e: The thrown exception during the execution of the annotated method; this value is null if the method is executed with success.

How To Use It

Thread Pool Task-Based Implementation

In order to use the asynchronous retry feature based on the Spring Thread pool task scheduler, all you have to do is to add the following dependency:

<dependency>
     <artifactId>async-retry-spring-scheduler</artifactId>
     <groupId>org.digibooster.retryable</groupId>
     <version>1.0.2</version>
</dependency>

Add the annotation EnableThreadPoolBasedAsyncRetry to a configuration class, and finally, define the retry policy bean as follow:

@Configuration
@EnableThreadPoolBasedAsyncRetry
public class AsyncRetryConfiguration {

    /**
    * the annotated method will be triggered the first time after 1 second and will
    * perform 2 retries eatch 20 seconds in case of failure
    */
    @Bean
    public FixedWindowRetryableSchedulingPolicy fixedWindowRetryableSchedulingPolicy() {
        return new FixedWindowRetryableSchedulingPolicy(10000,3,20000);
    }
}

Quartz-Based Implementation (In Memory)

This configuration uses the RAM in order to store the retry jobs. It is not persistent and doesn't support load-balancing and failover. So retries will be lost if the server restarts.

In order to use this implementation, add the following dependencies:

<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-quartz</artifactId>
</dependency>
<dependency>
	<groupId>org.quartz-scheduler</groupId>
	<artifactId>quartz</artifactId>
</dependency>
<dependency>
	<artifactId>async-retry-quartz-scheduler</artifactId>
	<groupId>org.digibooster.retryable</groupId>
	<version>1.0.2</version>
</dependency>

Add a configuration class that extends DefaultQuartzBasedAsyncRetryableConfigAdapter: 

@Configuration
public class RetryAsyncQuartzInMemoryConfiguration extends DefaultQuartzBasedAsyncRetryableConfigAdapter {

    @Bean
    public FixedWindowRetryableSchedulingPolicy fixedWindowRetryableSchedulingPolicy() {
        return new FixedWindowRetryableSchedulingPolicy(10000,3,20000);
    }

    @Bean("schedulerFactoryBean")
    public SchedulerFactoryBean schedulerFactoryBean(@Autowired QuartzSchedulerJobFactory quartzSchedulerJobFactory,
                                                     @Autowired QuartzProperties quartzProperties) {
        SchedulerFactoryBean factory = new SchedulerFactoryBean();
        Properties properties = new Properties();
        properties.putAll(quartzProperties.getProperties());
        factory.setQuartzProperties(properties);
        factory.setJobFactory(quartzSchedulerJobFactory);
        return factory;
    }

}

Finally, add the following lines to the application.yml file:

spring:
  quartz:
    auto-startup: true
    job-store-type: memory

Quartz-Based Implementation (JDBC)

This configuration uses the database in order to store the retry jobs. It is persistent and supports load-balancing and failover, so retries will not be lost if the server restarts.

In order to use this implementation, add the following dependencies:

<dependency>
	<groupId>org.springframework</groupId>
	<artifactId>spring-tx</artifactId>
</dependency>
<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-quartz</artifactId>
</dependency>
<dependency>
	<groupId>org.quartz-scheduler</groupId>
	<artifactId>quartz</artifactId>
</dependency>
<dependency>
	<artifactId>async-retry-quartz-scheduler</artifactId>
	<groupId>org.digibooster.retryable</groupId>
	<version>1.0.2</version>
</dependency>

Add a configuration class that extends QuartzDBBasedAsyncRetryableConfigAdapter as follows:

@Configuration
public class ConfigurationClass extends QuartzDBBasedAsyncRetryableConfigAdapter {

    @Autowired
    PlatformTransactionManager transactionManager;

    @Override
    public PlatformTransactionManager getTransactionManager() {
        return transactionManager;
    }

    @Bean
    public FixedWindowRetryableSchedulingPolicy fixedWindowRetryableSchedulingPolicy() {
        return new FixedWindowRetryableSchedulingPolicy(10000,3,20000);
    }

    @Bean("schedulerFactoryBean")
    public SchedulerFactoryBean schedulerFactoryBean(@Autowired QuartzSchedulerJobFactory quartzSchedulerJobFactory,
                                                     @Autowired QuartzProperties quartzProperties,
                                                     @Autowired DataSource dataSource) {
        SchedulerFactoryBean factory = new SchedulerFactoryBean();
        Properties properties = new Properties();
        properties.putAll(quartzProperties.getProperties());
        factory.setQuartzProperties(properties);
        factory.setJobFactory(quartzSchedulerJobFactory);
        factory.setDataSource(dataSource);
        return factory;
    }

}

Finally, add the following configuration to the application.yml file:

spring:
  quartz:
    auto-startup: true
    job-store-type: jdbc
    properties:
      org.quartz.jobStore.isClustered: true
      org.quartz.scheduler.instanceName: RetryInstance # optional
      org.quartz.scheduler.instanceId: AUTO # optional
    jdbc:
      initialize-schema: always # optional

Remark: When using Quartz with a database, the retry will be executed with a delay due to Quartz implementation. To decrease the delay, you can change the value of the property org.quartz.jobStore.clusterCheckinInterval. The framework source code is published on GitHub.