Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Using Spring Integration for the First Time, Again.

DZone's Guide to

Using Spring Integration for the First Time, Again.

Check out this awesome breakdown of trying Spring integration on a Big Data pipeline.

· Integration Zone
Free Resource

Build APIs from SQL and NoSQL or Salesforce data sources in seconds. Read the Creating REST APIs white paper, brought to you in partnership with CA Technologies.

I work on a Big Data pipeline, that is a bunch of components that ingest large amounts of advertising data from external sites like Google, Twitter, and Facebook, into Hadoop where the data is ETL'd and eventually uploaded into a columnar database where they can be analysed by others.

The pipeline components were written (in a hurry) in Java using core Spring, and components communicate passing messages over ActiveMQ. Due to some other development, one existing aspect of the pipeline needs to be reworked, the configurable download retry mechanism, so I thought that using Spring Integration, specifically the Java DSL, would be appropriate. 

Retry attempts are configured in the database per task and error type: the maximum number of retries and the delay between retries can be set.

Currently the retry mechanism is spread across 3 components, so it is hard to understand, retry attempts are saved in a database table from which a quartz scheduler triggers the attempts, and the count of attempts is recorded in the database. 

What I want to achieve is to move as much of the retry logic into one new component, the Retrier, to make use of ActiveMQ header 'AMQ_SCHEDULED_DELAY' to delay the retries, and to use JMS headers to count the number of retries rather than updating a database table. 

The Integration Zone is brought to you in partnership with CA Technologies.  Use CA Live API Creator to quickly create complete application backends, with secure APIs and robust application logic, in an easy to use interface.

Topics:
spring integration ,java ,dsl

Opinions expressed by DZone contributors are their own.

THE DZONE NEWSLETTER

Dev Resources & Solutions Straight to Your Inbox

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

X

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}