Asynchronous Task Execution Using Redis and Spring Boot

In this article, we are going to learn how to use Spring Boot 2.x and Redis to execute asynchronous tasks, with the final code demonstrating the steps described in this post.

You may also like: Spring and Threads: Async

Spring/Spring Boot

Spring is the most popular framework available for Java application development. As such, Spring has one of the largest open source communities. Besides that, Spring provides extensive and up-to-date documentation that covers the inner workings of the framework and sample projects on their blog — there are 100K+ questions on StackOverflow. 

In the early days, Spring only supported XML-based configuration, and, because of that, it was prone to many criticisms. Later, Spring introduced an annotation-based configuration that changed everything. Spring 3.0 was the first version that supported the annotation-based configuration. In 2014, Spring Boot 1.0 was released, completely changing how we look at the Spring framework ecosystem. A more detailed timeline can be found here. 

Redis

Redis is one of the most popular NoSQL in-memory databases. Redis supports different types of data structures. Redis supports different types of data structures, e.g. Sets, Hash tables, Lists, simple key-value pairs, just to name a few. The latency of Redis calls and operations are bot sub-milliseconds, which makes it even more attractive across the developer community.

Why Asynchronous Task Execution

A typical API call consists of five things:

1. Execute one or more database (RDBMS/NoSQL) queries.
2. One or more operations on some cache systems (In-Memory, Distributed, etc.).
3. Some computations (it could be some data crunching doing some math operations).
4. Calling some other service(s) (internal/external).
5. Schedule one or more tasks to be executed at a later time or immediately, but in the background.

A task can be scheduled at a later time for many reasons. For example, an invoice must be generated seven days after the order creation or shipment. Similarly, email notifications do not need to be sent immediately, so we can delay them. 

With these real-world examples in mind, sometimes, we need to execute tasks asynchronously to reduce API response time. For example, we get a request to delete 1K+ records at once, if we delete all of these records in the same API call, then the API response time would be increased for sure. To reduce API response time, we can run a task in the background that would delete those records. 

Delayed Queue

Whenever we schedule a task to run at a given time or a certain interval, then we use cron jobs that are scheduled at a specific time or interval. We can run schedule tasks using different tools like UNIX style crontabs, Chronos; if we’re using Spring frameworks then we can use an out-of-the-box Scheduled annotation. 

Most cron jobs find the records for when a particular action has to be taken, e.g. finding all shipments after seven days have elapsed and for which invoices were not generated. Most of these scheduling mechanisms suffer scaling problems, where we do scan database(s) to find the relevant rows/records. In many situations, this leads to a full table scan which performs very poorly. Imagine the case where the same database is used by a real-time application and this batch processing system. As it's not scalable, we would need some scalable system that can execute tasks at a given time or interval without any performance problems. There are many ways to scale in this way, like running tasks in a batched fashion or operating tasks on a particular subset of users/regions. Another way could be to run a specific task at a given time without depending on other tasks, like serverless functions. A delayed queue can be used in cases where as soon as the timer reaches the scheduled time a job would be triggered. There’re many queuing systems/software available, but very few of them provide this feature, like SQS which provides a delay of 15 minutes, not an arbitrary delay like 7 hours or 7 days.

Rqueue

Rqueue is a message broker built for the Spring framework that stores data in Redis and provides a mechanism to execute a task at any arbitrary delay. Rqueue is backed by Redis since Redis has some advantages over other widely used queuing systems, like Kafka or SQS. In most web applications' backends, Redis is used to store either cached data or other purposes. In today's world, 8.4% of web applications are using the Redis database.

Generally, for a queue, we use either Kafka, SQS, or some other systems. These systems bring an additional overhead in different dimensions, e.g. money that can be reduced to zero using Rqueue and Redis.

Apart from the cost, if we use Kafka then we need to do infrastructure setup, maintenance, i.e. more ops, as most of the applications are already using Redis so we won’t have ops overhead. In fact, the same Redis server/cluster can be used with Rqueue, as Rqueue supports an arbitrary delay.

Message Delivery

Rqueue guarantees at-least-once message delivery, as long data is not lost in the database. You can read more about this here: Introducing Rqueue.

Tools we will need:

1. Any IDE
2. Gradle 
3. Java
4. Redis 

We're going to use Spring Boot for simplicity. We'll create a Gradle project from the Spring Boot initializer at https://start.spring.io/.

For dependencies, we will need:  

1. Spring Data Redis
2. Spring Web
3. Lombok 

The directory/folder structure is shown below:

We’re going to use the Rqueue library to execute any tasks with any arbitrary delay. Rqueue is a Spring-based asynchronous task executor that can execute tasks at any delay. It’s built with the Spring messaging library and backed by Redis.

We’ll add the Rqueue Spring Boot starter 2.7.0 dependency:

com.github.sonus21:rqueue-spring-boot-starter:2.7.0-RELEASE

dependencies {  

  implementation 'org.springframework.boot:spring-boot-starter-data-redis'

  implementation 'org.springframework.boot:spring-boot-starter-web'

  implementation 'com.github.sonus21:rqueue-spring-boot-starter:2.0.0-RELEASE'

  compileOnly 'org.projectlombok:lombok'   

  annotationProcessor 'org.projectlombok:lombok'

  providedRuntime 'org.springframework.boot:spring-boot-starter-tomcat'

  testImplementation('org.springframework.boot:spring-boot-starter-test') {

    exclude group: 'org.junit.vintage', module: 'junit-vintage-engine'  

  }

}

For testing purposes, we will enable the Spring Web MVC feature so that we can send test requests. 

Update the application file as shown below:

xxxxxxxxxx

@SpringBootApplication

@EnableRedisRepositories

@EnableWebMvc

public class AsynchronousTaskExecutorApplication { 

  public static void main(String[] args) { 

    SpringApplication.run(AsynchronousTaskExecutorApplication.class, args);

  }

}

Adding listeners using Rqueue is very simple. This is as simple as annotating a method with RqueueListener. The RqueuListener annotation has many fields that can be set based on the use case. For dead letter queues, set deadLetterQueue to push tasks to another queue, otherwise, the task will be discarded on failure. We can also set how many times a task should be retried using the numRetries field.

Create a Java file named MessageListener and add some methods to execute message in the background:

xxxxxxxxxx

@Component

@Slf4j

public class MessageListener {

  @RqueueListener(value = "${email.queue.name}")

  public void sendEmail(Email email) {

    log.info("Email {}", email);

  }

  @RqueueListener(value = "${invoice.queue.name}")

  public void generateInvoice(Invoice invoice) {

    log.info("Invoice {}", invoice);

  }

}

We will need Email and Invoice classes to store email and invoice data respectively. For simplicity, classes would only have a handful of fields.

Invoice.java:

xxxxxxxxxx

import lombok.Data;

@Data

@AllArgsConstructor

@NoArgsConstructor

public class Invoice {

  private String id;

  private String type;

}

Email.java:

xxxxxxxxxx

import lombok.Data;

@Data

@AllArgsConstructor

@NoArgsConstructor

public class Email {

  private String email;

  private String subject;

  private String content;

}

Task Submissions

A task can be submitted using the RqueueMessageEnqueuer bean. RqueueMessageEnqueuer has multiple methods to enqueue tasks depending on the use case use one of the available methods. For simple tasks use enqueue for delayed tasks use enqueueIn.

We need to auto-wire RqueueMessageEnqueuer or use the constructor to inject this bean.

Here's how to create a Controller for testing purposes. 

We're going to schedule invoice generation that can be done in 30 seconds. For this, we'll submit a task with 30000 (milliseconds) delay on the invoice queue. Also, we'll try to send an email that can be executed in the background. For this purpose, we'll add two GET methods, sendEmail and generateInvoice; we can use POST as well.

xxxxxxxxxx

@RestController

@RequiredArgsConstructor(onConstructor = @__(@Autowired))

@Slf4j

public class Controller {

  private @NonNull RqueueMessageEnqueuer rqueueMessageEnqueuer;

  @Value("${email.queue.name}")

  private String emailQueueName;

  @Value("${invoice.queue.name}")

  private String invoiceQueueName;

  @Value("${invoice.queue.delay}")

  private Long invoiceDelay;

  @GetMapping("email")

  public String sendEmail(

      @RequestParam String email, @RequestParam String subject, @RequestParam String content) {

    log.info("Sending email");

    rqueueMessageEnqueuer.enqueue(emailQueueName, new Email(email, subject, content));

    return "Please check your inbox!";

  }

  @GetMapping("invoice")

  public String generateInvoice(@RequestParam String id, @RequestParam String type) {

    log.info("Generate invoice");

    rqueueMessageEnqueuer.enqueueIn(invoiceQueueName, new Invoice(id, type), invoiceDelay);

    return "Invoice would be generated in " + invoiceDelay + " milliseconds";

  }

}

Add the following in the application.properties file:

xxxxxxxxxx

email.queue.name=email-queue

invoice.queue.name=invoice-queue

# 30 seconds delay for invoice

invoice.queue.delay=300000

It's time to fire up the Spring Boot app. Once the application successfully starts, hit the below link to send an email.

http://localhost:8080/email?email=xample@exampl.com&subject=%22test%20email%22&content=%22testing%20email%22

In the log, we can see the email task is being executed in the background:

Below is our invoice scheduling after 30 seconds:

http://localhost:8080/invoice?id=INV-1234&type=PROFORMA

Conclusion

We can now schedule tasks using Rqueue without much boiler plate code! We made important considerations when configuring and using the Rqueue library. One important thing to keep in mind is that whether a task is a delayed task or not, by default, it's assumed that tasks need to be executed as soon as possible.

The complete code for this post can be found in my GitHub repo.  

If you found this post helpful, please share it with your friends and colleagues, and don't forget to give it a thumbs up!

Further Reading

Spring Boot: Creating Asynchronous Methods Using @Async Annotation

Spring and Threads: Async

Distributed Tasks Execution and Scheduling in Java, Powered by Redis