Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Using Spring Integration for the First Time, Again.

DZone's Guide to

Using Spring Integration for the First Time, Again.

Check out this awesome breakdown of trying Spring integration on a Big Data pipeline.

· Integration Zone ·
Free Resource

The new Gartner Critical Capabilities report explains how APIs and microservices enable digital leaders to deliver better B2B, open banking and mobile projects.

I work on a Big Data pipeline, that is a bunch of components that ingest large amounts of advertising data from external sites like Google, Twitter, and Facebook, into Hadoop where the data is ETL'd and eventually uploaded into a columnar database where they can be analysed by others.

The pipeline components were written (in a hurry) in Java using core Spring, and components communicate passing messages over ActiveMQ. Due to some other development, one existing aspect of the pipeline needs to be reworked, the configurable download retry mechanism, so I thought that using Spring Integration, specifically the Java DSL, would be appropriate. 

Retry attempts are configured in the database per task and error type: the maximum number of retries and the delay between retries can be set.

Currently the retry mechanism is spread across 3 components, so it is hard to understand, retry attempts are saved in a database table from which a quartz scheduler triggers the attempts, and the count of attempts is recorded in the database. 

What I want to achieve is to move as much of the retry logic into one new component, the Retrier, to make use of ActiveMQ header 'AMQ_SCHEDULED_DELAY' to delay the retries, and to use JMS headers to count the number of retries rather than updating a database table. 

The new Gartner Critical Capabilities for Full Lifecycle API Management report shows how CA Technologies helps digital leaders with their B2B, open banking, and mobile initiatives. Get your copy from CA Technologies.

Topics:
spring integration ,java ,dsl

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}