Over a million developers have joined DZone.

Real-Time Search and Analytics in Minutes with ElasticSearch and Mule ESB

DZone 's Guide to

Real-Time Search and Analytics in Minutes with ElasticSearch and Mule ESB

· Integration Zone ·
Free Resource

Search, (near) real-time analytics and data visualization are powerful tools driving the Business 2.0 economy. These often come with a slew of new complex tools,  complicated deploy and steep learning curve .However with Mule ESB and ElasticSearch you can be up and running processing large amounts of data in literally minutes not weeks.

ElasticSearch is mostly known for enterprise search, powering many well-known websites like SoundCloud and GitHub. It does this by being a very powerful, scalable JSON document store with sophisticated storage and retrieval options. ElasticSearch has a web based visualization tool called Kibana.

Mule ESB is one of the leading tools to secure, move, control and manage enterprise data from on premise or the cloud.

To integrate these two powerful tools:

  • Download ElasticSearch server from http://www.elasticsearch.com/ , extract and run ElasticSearch, and decide whether you want to change its default node name to prevent it forming a cluster with other default installs.
  • Download MuleStudio, the Eclipse development environment, from http://www.mulesoft.org/

Step 1: Migrate CSV Data to Elastic Search

The 5 simple steps in the flow are:

  • Create a file folder watcher to let you drop in CSV files
  • The enterprise component, DataMapper, can map from a variety of sources: CSV, XML, Excel etc. to JSON, the ElasticSearch native format. In my example, I’m mapping an application log mapfile to ElasticSearch for analysis.

  • The JsonObject is passed through the Mule flow and can be converted to a String for logging and posting to the ElasticSearch server.
  • The final endpoint performs a simple post of the payload to ElasticSearch 

Step2: Drop some test CSV data into the file folder being watched by Mule

... and hey presto each line of the data will be converted and inserted into ElasticSearch.

Confirm the JSON data has been inserted into ElasticSearch with

curl -XGET 'http://localhost:9200/test2/_search?pretty=true' -d '{
 "query": {
     "bool": {
         "must": [ {"match_all": {}} ]


You can view the CSV data being converted to JSON with the logger in the flow

Visualizing and Searching the Data with Mule and Kibana3

ElasticSearch also provide a data search and visualization tool called Kibana. This powerful webapp can be hosted in Mule’s own Jetty Global component:

Simply create a new jetty component and point it to an extracted Kibana

Kibana, out of the box, provides simple dashboards that will vew all your ElasticSearch indexs, a few clicks and you can have graphs and tables and a search box for your ElasticSearch data.

With a little more tweaking, you can have access to a  powerful analytics and insight dashboards including geo-location searches:

As you can see on the Kibana demo page http://demo.kibana.org/

Just the Tip of the (data) Iceberg 

This simple integration is just the tip of a very promising iceberg, with Mule's vast arrange of  connectors (http://www.mulesoft.org/connectors)  from legacy to the latest cloud offerings the enterprise is able to query and mashup data to all new forms of analysis and products. 

java ,enterprise-integration ,architecture ,integration ,search ,mule esb ,elasticsearch ,#java ,big data analytics ,application integration

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}