{{announcement.body}}
{{announcement.title}}

Asynchronous Task Execution Using Redis and Spring Boot

DZone 's Guide to

Asynchronous Task Execution Using Redis and Spring Boot

In this article, we are going to learn how to use Spring Boot 2.x and Redis to execute asynchronous tasks.

· Java Zone ·
Free Resource
Asynchronous tasksLearn more about executing asynchronous tasks using Redis and Spring Boot.

In this article, we are going to learn how to use Spring Boot 2.x and Redis to execute asynchronous tasks, with the final code demonstrating the steps described in this post.

You may also like:  Spring and Threads: Async

Spring/Spring Boot

Spring is the most popular framework available for Java application development. As such, Spring has one of the largest open-source communities. Besides that, Spring provides extensive and up-to-date documentation that covers the inner workings of the framework and sample projects on their blog — there are 100K+ questions on StackOverflow

In the early days, Spring only supported XML-based configuration, and because of that, it was prone to many criticisms. Later, Spring introduced an annotation-based configuration that changed everything. Spring 3.0 was the first version that supported the annotation-based configuration. In 2014, Spring Boot 1.0 was released, completely changing how we look at the Spring framework ecosystem. A more detailed timeline can be found here

Redis

Redis is one of the most popular NoSQL in-memory databases. Redis supports different types of data structures.  Redis supports different types of data structures e.g. Set, Hash table, List, simple key-value pair just name a few. The latency of Redis call is sub-milliseconds, support of a replica set, etc. The latency of the Redis operation is sub-milliseconds that makes it even more attractive across the developer community.

Why Asynchronous task execution

A typical API call consists of five things

  1. Execute one or more database(RDBMS/NoSQL) queries
  2. One or more operations on some cache systems (In-Memory, Distributed, etc )
  3. Some computations (it could be some data crunching doing some math operations)
  4. Calling some other service(s) (internal/external)
  5. Schedule one or more tasks to be executed at a later time or immediately but in the background.


A task can be scheduled at a later time for many reasons. For example, an invoice must be generated 7 days after the order creation or shipment. Similarly, email notifications need not be sent immediately, so we can make them delayed. 

With these real-world examples in mind, sometimes, we need to execute tasks asynchronously to reduce API response time. For example, if we delete 1K+ records at once, and if we delete all of these records in the same API call, then the API response time would be increased for sure. To reduce API response time, we can run a task in the background that would delete those records. 

Delayed Queue

Whenever we schedule a task to run at a given time or at a certain interval, then we use cron jobs that are scheduled at a specific time or interval. We can run schedule tasks using different tools like UNIX style crontabs, Chronos, if we’re using Spring frameworks then it’s out of box Scheduled annotation ❤️.

Most of the cron jobs find the records for when a particular action has to be taken, e.g. finding all shipments after seven days have elapsed and for which invoices were not generated. Most of these scheduling mechanisms suffer scaling problems, where we do scan database(s) to find the relevant rows/records. In many of the situations, this leads to a full table scan which performs very poor. Imagine the case where the same database is used by a real-time application and this batch processing system. As it's not scalable, we would need some scalable system that can execute tasks at a given time or interval without any performance problems. There are many ways to scale in this way, like run tasks in a batched fashion or operate tasks on a particular subset of users/regions. Another way could be to run a specific task at a given time without depending on other tasks, like serverless function. A delayed queue can be used in such cases where as soon as the timer reaches the scheduled time a job would be triggered. There’re many queuing systems/software available but very few of them provides this feature, like SQS which provides a delay of 15 minutes, not an arbitrary delay like 7 hours or 7 days, etc.


Rqueue

Rqueue is a message broker built for the spring framework that stores data in Redis and provides a mechanism to execute a task at any arbitrary delay. Rqueue is backed by Redis since Redis has some advantages over the widely used queuing systems like Kafka, SQS. In most of the web applications backend, Redis is used to store either cache data or for some other purpose. In today's world, 8.4% web applications are using the Redis database.

Generally, for a queue, we use either Kafka/SQS or some other systems these systems bring an additional overhead in different dimensions e.g money which can be reduced to zero using Rqueue and Redis.

Apart from the cost if we use Kafka then we need to do infrastructure setup, maintenance i.e. more ops, as most of the applications are already using Redis so we won’t have ops overhead, in fact, same Redis server/cluster can be used with Rqueue. Rqueue supports an arbitrary delay


Image title


Message Delivery

Rqueue guarantee at-least-once message delivery as long data is not lost in the database. Read about it more at Introducing Rqueue

Tools we will need:

1. Any IDE

2. Gradle 

3. Java

4. Redis 

We're going to use Spring Boot for simplicity. We'll create a Gradle project from the Spring Boot initializer at https://start.spring.io/.

For dependencies, we will need:  

1. Spring Data Redis 2. Spring Web 3. Lombok and any others. 

The directory/folder structure is shown below:

folder structure

We’re going to use the Rqueue library to execute any tasks with any arbitrary delay. Rqueue is a Spring-based asynchronous task executor, that can execute tasks at any delay, it’s built upon the Spring messaging library and backed by Redis.

We’ll add the dependency of Rqueue spring boot starter with com.github.sonus21:rqueue-spring-boot-starter:1.2-RELEASE


dependencies {  

  implementation 'org.springframework.boot:spring-boot-starter-data-redis'
  implementation 'org.springframework.boot:spring-boot-starter-web'
  implementation 'com.github.sonus21:rqueue-spring-boot-starter:1.2-RELEASE'
  compileOnly 'org.projectlombok:lombok'   
  annotationProcessor 'org.projectlombok:lombok'
  providedRuntime 'org.springframework.boot:spring-boot-starter-tomcat'
  testImplementation('org.springframework.boot:spring-boot-starter-test') {
    exclude group: 'org.junit.vintage', module: 'junit-vintage-engine'  
  }

}


We need to enable Redis Spring Boot features. For testing purposes, we will enable WEB MVC as well.

Update the application file as:

@SpringBootApplication
@EnableRedisRepositories
@EnableWebMvc
public class AsynchronousTaskExecutorApplication { 
  public static void main(String[] args) { 
    SpringApplication.run(AsynchronousTaskExecutorApplication.class, args);
  }
}


Adding tasks using Rqueue is very simple. We need to annotate a method with RqueueListener.  The RqueuListener annotation has a few fields that can be set based on the use case.  For delayed tasks, we need to set delayedQueue="true" and value must be provided; otherwise, it'll be ignored. The value is the name of a given queue. Set deadLetterQueue to push tasks to another queue. Otherwise, the task will be discarded on failure. We can also set how many times a task should be retried using the numRetries field.

Create a Java file named MessageListener and add some methods to execute tasks:

@Component
@Slf4j
public class MessageListener {

  @RqueueListener(value = "${email.queue.name}") (1)
  public void sendEmail(Email email) {
    log.info("Email {}", email);
  }

  @RqueueListener(delayedQueue = "true", value = "${invoice.queue.name}") (2)
  public void generateInvoice(Invoice invoice) {
    log.info("Invoice {}", invoice);
  }
}


We will need Email and Invoice classes to store email and invoice data respectively.  For simplicity, classes would only have a handful number of fields.

Invoice.java:

import lombok.Data;

@Data
@AllArgsConstructor
@NoArgsConstructor
public class Invoice {
  private String id;
  private String type;
}


Email.java:

import lombok.Data;

@Data
@AllArgsConstructor
@NoArgsConstructor
public class Email {
  private String email;
  private String subject;
  private String content;
}


Task Submissions

A task can be submitted using the  RqueueMessageSender bean. This has multiple methods to put a task depending on the use case. For example, for retry, use a method retry count, and then use delay for delayed tasks.

We need to auto-wire RqueueMessageSender or use the constructor-based injection to inject these beans.

Here's how to create a Controller for testing purposes. 

We're going to schedule invoice generation that can be done in 30 seconds. For this, we'll submit a task with 30000 (milliseconds) delay on the invoice queue. Also, we'll try to send an email that can be executed in the background. For this purpose, we'll add two GET methods, sendEmail and generateInvoice, we can use POST as well.

@RestController
@RequiredArgsConstructor(onConstructor = @__(@Autowired))
@Slf4j
public class Controller {
  private @NonNull RqueueMessageSender rqueueMessageSender;

  @Value("${email.queue.name}")
  private String emailQueueName;

  @Value("${invoice.queue.name}")
  private String invoiceQueueName;

  @Value("${invoice.queue.delay}")
  private Long invoiceDelay;

  @GetMapping("email")
  public String sendEmail(
      @RequestParam String email, @RequestParam String subject, @RequestParam String content) {
    log.info("Sending email");
    rqueueMessageSender.put(emailQueueName, new Email(email, subject, content));
    return "Please check your inbox!";
  }

  @GetMapping("invoice")
  public String generateInvoice(@RequestParam String id, @RequestParam String type) {
    log.info("Generate invoice");
    rqueueMessageSender.put(invoiceQueueName, new Invoice(id, type), invoiceDelay);
    return "Invoice would be generated in " + invoiceDelay + " milliseconds";
  }
}


Add the following in application.properties file:

email.queue.name=email-queue
invoice.queue.name=invoice-queue
# 30 seconds delay for invoice
invoice.queue.delay=300000


Now, we can run this application. Once the application starts successfully, you can browse this link.

In the log, we can see the email task is being executed in the background:

Image title



Below is invoice scheduling after 30 seconds:

http://localhost:8080/invoice?id=INV-1234&type=PROFORMA

Image title

Conclusion

We can now schedule tasks using Rqueue without much boiler code! We made important considerations when configuring and using the Rqueue library. One important thing to keep in mind is: Whether a task is a delayed task or not, by default, it's assumed that tasks need to be executed as soon as possible.

The complete code for this post can be found at my GitHub repo.  

If you found this post helpful, please share it with your friends and colleagues, and don't forget to give it a thumbs up!

Further Reading

Spring Boot: Creating Asynchronous Methods Using @Async Annotation

Spring and Threads: Async

Distributed Tasks Execution and Scheduling in Java, Powered by Redis

Topics:
spring ,spring-boot ,java ,redis ,asynchronous processing ,task execution

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}