Image Loader Powered by Apache Camel

(The application has been improved and moved to www.livephotostream.com read about the move here)
This is a very short post about a very simple application way back during the London 2012 Olympics and based on Apache Camel examples, I've created an application that displays twitter images related to the Olympics in real time. It listens for tweets containing images, filters out duplicates, and sends the images to the browser using websockets every 5 seconds. See it live running on amazon micro instance or check it out from my github account.

This is an end-to-end push application: once a user pushes an image to twitter, twitter pushes the image to the camel application, and then the camel application pushes the image to all the clients. It is built using only "free stuff": Twitter's free streaming API, Apache Camel framework, Amazon's free micro instance and done during my free time. Here is the essence:

public void configure() throws Exception {

    from("twitter://streaming/filter?type=event&keywords=" + searchTerm)

            .to("log:tweetStream?level=INFO&groupInterval=60000&groupDelay=60000&groupActiveOnly=false")

            .process(new ImageExtractor())

            .process(new Statistics())

            .filter(body().isInstanceOf(Tweet.class))

            .idempotentConsumer(header(UNIQUE_IMAGE_URL), MemoryIdempotentRepository.memoryIdempotentRepository(10000))

            .to("log:imageStream?level=INFO&groupInterval=60000&groupDelay=60000&groupActiveOnly=false")

            .throttle(1).timePeriodMillis(5000).asyncDelayed().callerRunsWhenRejected(false)

            .marshal().json(JsonLibrary.Jackson)

            .to("websocket:camelympics?sendToAll=true");
}

Twitter pushes tweets containing the #Olympics and #London2012 tags to the app:

from("twitter://streaming/filter?type=event&keywords=" + searchTerm)

Log statistical information about the number of messages every minute. Not visible the users:

.to("log:tweetStream?level=INFO&groupInterval=60000&groupDelay=60000&groupActiveOnly=false")

Extract images from the tweets which has media associated with:

.process(new ImageExtractor())

Put the current number of tweets and images in the message:

.process(new Statistics())

Filter out all the tweets which doesn't contain images:

.filter(body().isInstanceOf(Tweet.class))

Filter out duplicated images, identified by their url:

.idempotentConsumer(header(UNIQUE_IMAGE_URL), MemoryIdempotentRepository.memoryIdempotentRepository(10000))

Log again, the messages which reached this far in the route:

.to("log:imageStream?level=INFO&groupInterval=60000&groupDelay=60000&groupActiveOnly=false")

Let images go with 5 seconds difference, so the user can enjoy it. Also important - don't block the twitter listener by using callerRunsWhenRejected if the image buffer fills up, Twitter will block you:

.throttle(1).timePeriodMillis(5000).asyncDelayed().callerRunsWhenRejected(false)

Serialize into json:

.marshal().json(JsonLibrary.Jackson)

Push it to the users:

.to("websocket:camelympics?sendToAll=true");

The application can be run also locally and allows filtering images not only for the Olympics but for any keywords passed as argument. Don't forget to use your own twitter oath tokens when using locally though.

Enjoy the Olympics.