Over a million developers have joined DZone.

Using Spring Integration for the First Time, Again.

Check out this awesome breakdown of trying Spring integration on a Big Data pipeline.

· Integration Zone

Is iPaaS solving the right problems? Not knowing the fundamental difference between iPaaS and dPaaS could cost you down the road. Brought to you in partnership with Liaison Technologies.

I work on a Big Data pipeline, that is a bunch of components that ingest large amounts of advertising data from external sites like Google, Twitter, and Facebook, into Hadoop where the data is ETL'd and eventually uploaded into a columnar database where they can be analysed by others.

The pipeline components were written (in a hurry) in Java using core Spring, and components communicate passing messages over ActiveMQ. Due to some other development, one existing aspect of the pipeline needs to be reworked, the configurable download retry mechanism, so I thought that using Spring Integration, specifically the Java DSL, would be appropriate. 

Retry attempts are configured in the database per task and error type: the maximum number of retries and the delay between retries can be set.

Currently the retry mechanism is spread across 3 components, so it is hard to understand, retry attempts are saved in a database table from which a quartz scheduler triggers the attempts, and the count of attempts is recorded in the database. 

What I want to achieve is to move as much of the retry logic into one new component, the Retrier, to make use of ActiveMQ header 'AMQ_SCHEDULED_DELAY' to delay the retries, and to use JMS headers to count the number of retries rather than updating a database table. 

Discover the unprecedented possibilities and challenges, created by today’s fast paced data climate and why your current integration solution is not enough, brought to you in partnership with Liaison Technologies.

Topics:
spring integration ,java ,dsl

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

SEE AN EXAMPLE
Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.
Subscribe

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}