Why Jetwick moved from Solr to ElasticSearch

i like both technologies solr and elasticsearch and a lot work is going into both. so, let me explain why i choose to migrate from solr to elasticsearch (es).

what is elastic?

• es lets you add and remove nodes [video] and the requests will be handled from the correct node. nodes will even do ‘zero config’ discovery.
  to scale if the load increases you can use replicas. elasticsearch will automatically play the loadbalancer and choose the appropriated node.
• es lets you scale if data amount increase, because then you can easily use sharding: it’s just a number in es (either via api or via configuration).

with that features es is well prepared for the century of the cloud [blog] !

what’s the difference to solr?

solr wasn’t designed from the ground up with the ‘cloud’ in mind, but of course you can do sharding, use replication and use multiple cores with solr. it’s just a bit more complicated.

when using solr cloud and zookeeper this gets better. you’ll also need to invest some time to make solr near real time to be comparable with es. this all seemed to be a bit too tricky to me (in dec 2010) and i don’t have any time for administration work in my free time e.g. to set up backups/replicas, add shards/indices, …

other options?

what are my other options? there is xapian , sphinx etc . but  only the following two projects fullfilled my requirements:

1. using solandra or
2. moving from solr to elasticsearch

i wanted a lucene based solution and a solution where it works out of the box to shard and create indices. i simply wanted more data from twitter available in jetwick.

the first option is very nice, no changes to your code are required – only a minor change in your solrconfig.xml and you will get a distributed and real time solr! so, i tried solandra and after a lot support from jake (thanks!) i got it running with jetwick! but at the end i still had performance issues with my indexing strategy, so i tried – in parallel – the second step.

what are the advantages of elasticsearch?

to be honest jetwick doesn’t really need to be elastic – i’m only using the sharding feature at the moment as i don’t own capacity on a cloud. but elasticsearch is also elastic in a different area: es lets you manage indices very very easy! a clever thing in es is that you don’t define the document structure in an index like you do in solr – no, you define types and then create documents of a specific type in a specific index. and documents in es don’t need to be flat – they can be nested as they are pure json.

that and the ‘elasticity’ could make es suitable as a hip nosql storage

another advantage over solr is the near real time behaviour, which you’ll get at no costs when switching to es.

the move!

moving to elasticsearch with jetwick wasn’t that easy as i hoped. although i’m sure one can make a normal migration in one day with my experience now . it took a lot of time to understand the new technology and more importantly to migrate my ui code where i made too much use to construct a solrquery object. at the end i created a custom solr2elastichelper utility to avoid this clumsy work at the beginning. and at some day i will fully migrate even this code.

when moving to elasticsearch be sure that it supports all feature solr has. although shay works really hard to integrate new features into es he cannot do all the work alone! e.g. i had to integrate solrs’ worddelimiterfilter , but this wasn’t that difficult – just copy & paste; plus some configuration.

es uses netty under the hood – no other webserver is necessary. just start the node either via api or in directly via bin/elasticsearch and then query the node via curl or the browser. for example you can use the nice elasticsearch head project :

or elasticsearch-js which are equivalents to the solr admin page. to add a node simply start another es instance and they will automagically discover each other. you can also use curl on the command line to query and feed the index as documented in the rest api documentation .

no technology is perfect so keep in mind the following disadvantages which will disappear over time in my opinion:

• solr has more analyzers, filters, etc., but it is relative easy to use them in es as well.
• solr has a larger community, a larger user base and more companies offering professional support
• solr has better documentation and more books. regarding the docs of es: they are moving now to the github wiki and the docs will now improve imo.
• solr has more tooling e.g. solrmonitor, lucidgaze and newrelic, but you still have yourkit and jvisualvm

but keep in mind also the following unmentioned notes:

• shay fixes bugs very quickly!
• elasticsearch has a more recent lucene version and releases more frequently
• it is very easy to contribute via github (just a pull request away )

to get introduced into elasticsearch you can read this article .

from http://karussell.wordpress.com/2011/02/07/why-jetwick-moved-from-solr-to-elasticsearch/