Elasticsearch Cluster Deployment: Tips

background

elasticsearch is a search server based on lucene. it provides a distributed, multitenant-capable full-text search engine with a restful web interface and schema-free json documents. elasticsearch is developed in java and is released as open source under the terms of the apache license. (source: wikipedia)

things to consider

based on my experiences with elasticsearch cluster deployments, i think the following considerations should be kept in mind.

keep the jvm version consistent

you will start seeing serialization exceptions in the logs at the serverside or in the elasticsearch clients, if there is a mismatch in the jvm versions.

the typical exceptions have patterns similar to the following:

org.elasticsearch.transport.remotetransportexception: failed to deserialize exception response from stream

it will usually be a classnotfoundexception with one of the serializationexception classes.

according to shay banon, the creator of elasticsearch,

( source : http://jontai.me/blog/2013/06/elasticsearch-remotetransportexception-failed-to-deserialize-exception-response-from-stream/ )

tip :  try to keep the jvm version consistent across the cluster.

virtual memory map settings

apache lucene which is the underlying indexing project driving elasticsearch, may throw an exception that looks something like:

caused by: java.lang.outofmemoryerror: map failed
at sun.nio.ch.filechannelimpl.map0(native method)
at sun.nio.ch.filechannelimpl.map(filechannelimpl.java:885)
... 20 more

you will notice that this is choking in native code of the jvm. so we have to look for the os settings.

based on the issue identified in https://github.com/elasticsearch/elasticsearch/issues/4547

it is important to check your virtual memory settings.

check if your vm.max_map_count is at the default of 65530. if yes, raise it higher.

refer to

http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_file_descriptors_and_mmap.html for additional guidance.

tip : check your virtual memory settings for the os.

understand the 3 modes of an elasticsearch node

an elasticsearch node can act in 3 modes:

1. elasticsearch master node
2. elasticsearch data node
3. elasticsearch client node

understand the heap settings and sizing

i consider the following page in the elasticsearch document to be very important in designing/sizing your cluster.

http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html

tip : for a data node, never allocate more than 32gb to the java heap. give half your memory to your os (which will be used by lucene).

threadpool settings

elasticsearch documentation on the threadpools http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-threadpool.html

gives information on the queue sizes.

for performance, it makes sense to use bulk requests and set your bulk queue size appropriately.

tip 9 from https://www.loggly.com/blog/nine-tips-configuring-elasticsearch-for-high-performance/ gives a hint on one possible case.

it is very important to manage the bulk queue because the clients can start seeing exceptions and there can be data loss.

tip : configure the threadpool.bulk.queue_size property properly.

split brains

this should be a no-brainer for serious elasticsearch administrators and developers. you will have to configure the minimum number of master nodes setting correctly.

refer to http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_important_configuration_changes.html for details on this.

tip : set minimum_master_nodes to (number of master-eligible nodes / 2) + 1

disable multicast and use unicast

refer to the discussion in

http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_important_configuration_changes.html

you will need to disable multicast in production and use unicast. this will prevent nodes from accidentally joining your cluster in production.

tip : ensure that the "discovery.zen.ping.multicast.enabled" property is set to "false".

conclusion

it is very important to understand the elasticsearch architecture, concepts and configuration before deploying it in production. your devops will be grateful when you do the right things.

i hope you find this article useful. please inform me if i have made any mistakes or if there are any additional tips that you would like me to add.