Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Apache Camel 2.12 - Backoff Support for Less Aggressive Polling Routes

DZone's Guide to

Apache Camel 2.12 - Backoff Support for Less Aggressive Polling Routes

· Integration Zone ·
Free Resource

SnapLogic is the leading self-service enterprise-grade integration platform. Download the 2018 GartnerMagic Quadrant for Enterprise iPaaS or play around on the platform, risk free, for 30 days.

This is another blog post about some upcoming improvements and new features in the Apache Camel 2.12 release.

In the last blog I spoke about cron expressions directly supported in the routes. This blog post is related as we have added another new feature to the polling routes (using scheduled polling consumers).

The scheduled polling routes will, by default, poll for new messages at a fixed rate (usually every half second). Though in some use-cases new messages do not arrive evenly spread over the day but may tend to arrive in batches. So during the times where there are no new messages arriving, the scheduled polling consumer would not pickup new messages but still keep looking for new messages in the same pace.

So what we have introduced is a backoff that allows you to let the scheduled polling consumer check less aggressively when there are no messages.

To support that we have introduce 3 new options

  1. backoffMultiplier = To let the scheduled polling consumer backoff if there has been a number of subsequent idles/errors in a row. The multiplier is then the number of polls that will be skipped before the next actual attempt is happening again. When this option is in use then backoffIdleThreshold and/or backoffErrorThreshold must also be configured.
  2. backoffIdleThreshold = The number of subsequent idle polls that should happen before the backoffMultipler should kick-in.
  3. backoffErrorThreshold = The number of subsequent error polls (failed due some error) that should happen before the backoffMultipler should kick-in.

The scheduled Polling Consumer is by default static by using the same poll frequency whether or not there is messages to pickup or not. From Camel 2.12 onwards you can configure the scheduled Polling Consumer to be more dynamic by using backoff. This allows the scheduler to skip N number of polls when it becomes idle, or if there has been X number of errors in a row. See more details in the table above for the backoffXXX options.
For example, to let a FTP consumer backoff if it becomes idle for a while, you can do:
from("ftp://myserver?username=foo&passowrd=secret
      ?delete=true&delay=5s
      &backoffMultiplier=6&backoffIdleThreshold=5")
  .to("bean:processFile");

In this example, the FTP consumer will poll for new FTP files every 5th second. But if it has been idle for 5 attempts in a row, then it will backoff using a multiplier of 6, which means it will now poll every 5 x 6 = 30th second instead. When the consumer eventually picks up a file, then the backoff will reset, and the consumer will go back and poll every 5th second again.
In JMX you can see whether the backoff has kicked-in, as the consumer would have the attribute backoffCounter > 0.
You can find more details at


With SnapLogic’s integration platform you can save millions of dollars, increase integrator productivity by 5X, and reduce integration time to value by 90%. Sign up for our risk-free 30-day trial!

Topics:

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}