Spring Batch

by: Chris Schaefer

Spring Batch is a lightweight, open-source Java framework for batch processing built on top of the popular Spring Framework. In addition to the core framework, Spring Batch also provides developers with components to build robust batch applications that are used in everyday enterprise environments. The most current project documentation can be found at:

http://static.springsource.org/spring-batch/reference/html/

And the source code can be found on GitHub at:

https://github.com/SpringSource/spring-batch

Spring Batch is built upon common architecture and patterns that batch applications have been using for decades. The following components are central to the Spring Batch architecture:

A Job encapsulates one or more Steps. Steps contain the necessary configuration that is used to execute the batch job. Steps can use custom or out of the box Spring Batch components. Spring Batch provides a large number of infrastructure components, making it possible to create a Job with little to no custom code needed.

As with most projects in the Spring Portfolio, Spring Batch provides XML namespace support providing a rich configuration interface. To use the Spring Batch namespace support, create an XML configuration file similar to the template below. This will serve as your Job configuration file.

​

<?xml version="1.0" encoding="UTF-8"?>

​

<beans:beans xmlns="http://www.springframework.org/schema/batch"

  xmlns:beans="http://www.springframework.org/schema/beans"

  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

  xsi:schemaLocation="

  http://www.springframework.org/schema/beans

  http://www.springframework.org/schema/beans/spring-beans.xsd

  http://www.springframework.org/schema/batch

  http://www.springframework.org/schema/batch/spring-batch.xsd">

​

  <!-- Job configuration goes here -->

​

</beans:bean>

​

Job

A Job in Spring Batch is composed of a list of Steps and transitions that make up the batch job. The first Step in the Job definition is always the first Step to be executed.

Step

A Step represents the current stage of processing in the batch job. Steps can inherit from a parent; provide transitional direction and define restartability settings.

Step Components

A Step is made up of various components that define how that specific Step will execute.

Tasklet

A Tasklet defines the implementation strategy that will be used for this Step.

Chunk

A chunk is a child element of the Tasklet definition and declares the current Step as being one that performs chunk oriented processing. At the minimum, an ItemReader, ItemWriter and commit-interval are specified. An ItemProcessor is an optional component. Once the chunk CompletionPolicy is satisfied, the transaction is committed.

The Tasklet interface represents a strategy for processing in a Step. To create a custom Tasklet, implement the Tasklet interface, and use the bean as the tasklet reference:

​

public class MyTasklet implements Tasklet {

    @Override

    public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {

        // Custom processing logic here

        return RepeatStatus.FINISHED;

    }

}

​

<beans:bean id="myTasklet" class="com.example.MyTasklet"/>

​

<tasklet ref="myTasklet" .../>

​

Spring Batch provides the following Tasklet implementations:

ItemReader is a strategy interface for providing data in a chunk oriented processing Step. Custom ItemReader's are created by implementing this interface and have their read method called to obtain more data until exhausted at which point null is returned.

To create a custom ItemReader, implement the ItemReader interface, wire it as a standard Spring bean and provide a reference to it as the reader attribute in your Tasklet chunk configuration:

​

public class MyItemReader implements ItemReader<Person> {

    @Override

    public Person read() throws Exception {

        // read item, then return null when results are exhausted

    }

}

​

<beans:bean id="myItemReader" class="com.example.MyItemReader"/>

​

<tasklet ...>

    <chunk reader="myItemReader" .../>

</tasklet>

​

Spring Batch provides the following ItemReader implementations:

The ItemProcessor interface provides an interception point for custom logic to transform an item in a chunk oriented processing Step between the reading and writing. An ItemProcessor implementation can return the same or different object that it has received. Returning null from the process method in the implementation indicates to the framework that no processing should occur on the current item.

To create a custom ItemProcessor, implement the ItemProcessor interface, wire it as a standard Spring bean and provide a reference to it as the processor attribute in your Tasklet chunk configuration:

​

public class MyItemProcessor 

             implements ItemProcessor<Person, Person> {

    @Override

    public Person process(Person item) throws Exception {

        // process the current item and return it or return null 

        // to indicate no processing should be performed

    }

}

​

<beans:bean id="myItemProcessor"

            class="com.example.MyItemProcessor"/>

​

<tasklet ...>

    <chunk processor="myItemProcessor" .../>

</tasklet>

​

Spring Batch provides the following ItemProcessor implementations:

The ItemWriter interface provides a basic interface for writing a list of items. Items are generally expected to be batch processed as a chunk and delivered to the implementing method as a List of items.

To create a custom ItemWriter, implement the ItemWriter interface, wire it as a standard Spring bean and provide a reference to it as the writer attribute in your Tasklet chunk configuration:

​

public class MyItemWriter implements ItemWriter<Person> {

    @Override

    public void write(List<? extends Person> items) 

                                       throws Exception {

        // write items

    }

}

​

<beans:bean id="myItemWriter" class="com.example.MyItemWriter"/>

​

<tasklet ...>

    <chunk writer="myItemWriter" .../>

</tasklet>

​

Spring Batch provides the following ItemWriter implementations:

The Job configuration is an important part of your batch application, but it needs to be started to do anything meaningful. A Job can be started in a variety of ways: most commonly programmatically, manually from the command line, through scripts, or even from an HTTP controller in a web container.

JobLauncher

At the most basic level, Spring Batch provides the JobLauncher interface with a run method as a way to programmatically launch a Job. The run method accepts a Job instance representing the Job to launch and the JobParameters that are used to define the instance of the Job that is to be launched. The implementation of the JobLauncher provided by Spring Batch is the SimpleJobLauncher and by default a Job is launched synchronously. With your Job configuration ready to go, a programmatic invocation of the Job via a simple main method could look something like:

​

public class JobRunner {

  private static final String CONFIG 

                        = "classpath:launch-context.xml";

​

  public static void main(final String[] args) throws Exception {

        ApplicationContext ctx = new   

                            ClassPathXmlApplicationContext(CONFIG);

​

        Job job = ctx.getBean("personJob", Job.class);

​

        JobLauncher jobLauncher = 

                      ctx.getBean("jobLauncher", JobLauncher.class);

​

        JobParameters jobParameters = new JobParametersBuilder()

                     .addString("lastName","Smith")

                     .toJobParameters();

​

        jobLauncher.run(job, jobParameters);

    }

}

​

Command line execution

Jobs can be launched manually via the command line or kicked off via a scheduler of your choice such as a Cron, Quartz, etc. Spring Batch does not provide nor dictate any specific scheduler. Out of the box, Spring Batch provides a CommandLineJobRunner class with a main method that can be executed from a script or manually on the command line. The CommandLineJobRunner arguments are as follows:

It is also worth noting that the CommandLineJobRunner provides support for optionally passing Job control options.

Execution in a Web Container

Jobs can also be started within a web container using the same programmatic method of calling the run method of the JobLauncher. A simple SpringMVC controller used to launch a Job could look like the following:

​

public class JobController {

    private Job job;

    private JobLauncher jobLauncher;

​

    public JobController(Job job, JobLauncher jobLauncher) {

        this.job = job;

        this.jobLauncher = jobLauncher;

    }

​

    @RequestMapping("/runJob")

    public void runJob() throws Exception {

        JobParameters jobParameters = new JobParametersBuilder()

                      .addString("lastName", "Smith")

                      .toJobParameters();

​

        jobLauncher.run(job, jobParameters);

    }

}

​

One key difference when running in a web container is that launching a Job should most likely be executed in an asynchronous fashion to avoid having the HTTP request wait until the batch job is complete. This can be accomplished by providing a TaskExecutor implementation of your choice to the configured JobLauncher bean that will be used to execute the job.

The JobOperator interface provides methods to inspect and control jobs. To stop a job, the stop method can be called with the provided JobExecution ID of the running job. The JobExecution ID must be known and can be retrieved by obtaining the running executions of the Job:

​

Set<Long> executions =

         jobOperator.getRunningExecutions("personJob");

​

As this method returns back a Set of JobExecution IDs, it's up to the developer to obtain the proper JobExecution ID to stop:

​

jobOperator.stop(JOB_EXECUTION_ID);

​

Stopping a Job might not be immediate, as the job may be processing user-defined code. Once execution is returned back to the framework, the Job will be stopped and the current StepExecution and corresponding JobExecution status will be marked as BatchStatus.STOPPED.

Restarting a Job provides the ability to pick up where the Job last left off, for example in a situation where a Step failed or stopped. A Job that already has an existing non-COMPLETED JobInstance is a candidate for being restarted. By default, all Jobs are restartable and will restart on the next Step to be executed. A Job can be restarted programmatically by way of the restart method provided by the JobOperator interface, for example:

​

jobOperator.restart(JOB_EXECUTION_ID);

​

The Job will be restarted or an exception will be thrown if the provided JobExecution ID does not exist or it has already completed successfully.

When your Job needs to have the ability to scale out, Spring Batch makes it easy and, out of the box, provides two modes of parallel processing: Single Process (multi-threaded), and Multi-Process.

Using a Multi-Threaded Step in Spring Batch is a simple way to add parallel processing to your Job. Adding this capability only requires the addition of a TaskExecutor reference as an attribute on the intended Step's Tasklet, for example:

​

<step id="multiThreadedStep">

    <tasklet task-executor="taskExecutor" …>

    …

    </tasklet>

</step>

​

<beans:bean id="taskExecutor" class="org.springframework.core.task.SimpleAsyncTaskExecutor"/>

​

Any TaskExecutor implementation can be used and the Spring Framework itself provides many out of the box implementations. In a chunk oriented processing mode, a Multi-threaded Step will result in reading, processing, and writing each chunk of items in a separate thread. All components must be thread safe and limited by any configuration applied to the TaskExecutor as well as the Tasklet concurrency configuration options (notably the throttle-limit attribute).

Parallel Steps allow you to configure your Job to allow multiple Steps to execute in Parallel with another Step. Parallel Steps are composed in a Split element along with a TaskExecutor and the Flow definition. The Flow definitions encapsulate the Steps that will be parallelized.

​

<job id="personJob">

    <split id="splitFlow1" task-executor="taskExecutor" 

           next="step4">

        <flow>

            <step id="step1" next="step2">

            ...

            </step>

            <step id="step2">

            ...

            </step>

        </flow>

        <flow>

            <step id="step3">

            ...

            </step>

        </flow>

    </split>

    <step id="step4">

    ...

    </step>

</job>

​

<beans:bean id="taskExecutor" class="org.springframework.core.task.SimpleAsyncTaskExecutor"/>

​

In the above configuration, "step1" and "step2" will be executed in parallel with "step3", then after every flow in the split has completed the Job will transition to "step4".

Remote Chunking allows Step processing to be split across multiple processes via durable and guaranteed delivery middleware. In Remote Chunking, the master is a single process and slaves are remote processes. This pattern is typically used when reading of the items is less expensive than processing of the items. Reading is performed on the "master", processing is done on the "slave" nodes.

To achieve a Remote Chunking setup, Spring Batch can be used in conjunction with Spring Integration for middleware integration and the Spring Batch Integration module of Spring Batch Admin for chunk handling and writing. See the API documentation of Spring Batch Admin for more details, specifically the RemoteChunkHandlerFactoryBean and ChunkMessageChannelItemWriter classes.

Spring Batch provides another scalability option, which is "partitioning" a Step and executing it remotely or in local threads. Unlike Remote Chunking, slaves do not need to be durable nor provide guaranteed delivery. Partitioning divides responsibilities of data partitioning and StepExecution handling. A simple example of configuring partitioning would be similar to the following.

​

<job id="partitionJob">

    <step id="partitionedStep">

        <partition step="readWriteStep" partitioner="partitioner">

            <handler grid-size="2" task-executor="taskExecutor" />

        </partition>

    </step>

</job>

​

<step id="readWriteStep">

    <tasklet>

        <chunk reader="personReader" writer="personWriter" 

               commit-interval="10"/>

    </tasklet>

</step>

​

With this configuration a Partitioner implementation that dictates how data will be partitioned is required. The Partitioner has access to the ExecutionContext and can be used to store information that will be used by the reader to determine what data to process, for example a range of database ID's that can be injected into the reader via late binding.