DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations

Trending

  • Database Integration Tests With Spring Boot and Testcontainers
  • Transactional Outbox Patterns Step by Step With Spring and Kotlin
  • Reactive Programming
  • Chaining API Requests With API Gateway

Trending

  • Database Integration Tests With Spring Boot and Testcontainers
  • Transactional Outbox Patterns Step by Step With Spring and Kotlin
  • Reactive Programming
  • Chaining API Requests With API Gateway
  1. DZone
  2. Data Engineering
  3. Big Data
  4. Elasticsearch Cluster Deployment: Tips

Elasticsearch Cluster Deployment: Tips

Anil Saldanha user avatar by
Anil Saldanha
·
Feb. 05, 15 · Interview
Like (0)
Save
Tweet
Share
1.51K Views

Join the DZone community and get the full member experience.

Join For Free

background

elasticsearch is a search server based on lucene. it provides a distributed, multitenant-capable full-text search engine with a restful web interface and schema-free json documents. elasticsearch is developed in java and is released as open source under the terms of the apache license. (source: wikipedia)

things to consider

based on my experiences with elasticsearch cluster deployments, i think the following considerations should be kept in mind.

keep the jvm version consistent

you will start seeing serialization exceptions in the logs at the serverside or in the elasticsearch clients, if there is a mismatch in the jvm versions.

the typical exceptions have patterns similar to the following:

org.elasticsearch.transport.remotetransportexception: failed to deserialize exception response from stream

it will usually be a classnotfoundexception with one of the serializationexception classes.

according to shay banon, the creator of elasticsearch,


( source : http://jontai.me/blog/2013/06/elasticsearch-remotetransportexception-failed-to-deserialize-exception-response-from-stream/ )

tip :  try to keep the jvm version consistent across the cluster.

virtual memory map settings

apache lucene which is the underlying indexing project driving elasticsearch, may throw an exception that looks something like:

caused by: java.lang.outofmemoryerror: map failed
at sun.nio.ch.filechannelimpl.map0(native method)
at sun.nio.ch.filechannelimpl.map(filechannelimpl.java:885)
... 20 more

you will notice that this is choking in native code of the jvm. so we have to look for the os settings.

based on the issue identified in https://github.com/elasticsearch/elasticsearch/issues/4547

it is important to check your virtual memory settings.


check if your vm.max_map_count is at the default of 65530. if yes, raise it higher.

refer to

http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_file_descriptors_and_mmap.html for additional guidance.

tip : check your virtual memory settings for the os.

understand the 3 modes of an elasticsearch node

an elasticsearch node can act in 3 modes:

  1. elasticsearch master node
  2. elasticsearch data node
  3. elasticsearch client node
by default, a node operates in all 3 modes.
ensure that you operate in the right mode for each node.
you can separate out the master nodes, client nodes and data nodes in your cluster to maintain separation of concerns . but if you are running short on capital investment, then you have to run the nodes in a combination of these 3 modes.

understand the heap settings and sizing

i consider the following page in the elasticsearch document to be very important in designing/sizing your cluster.

http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html

tip : for a data node, never allocate more than 32gb to the java heap. give half your memory to your os (which will be used by lucene).

threadpool settings

elasticsearch documentation on the threadpools http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-threadpool.html

gives information on the queue sizes.

for performance, it makes sense to use bulk requests and set your bulk queue size appropriately.

tip 9 from https://www.loggly.com/blog/nine-tips-configuring-elasticsearch-for-high-performance/ gives a hint on one possible case.

it is very important to manage the bulk queue because the clients can start seeing exceptions and there can be data loss.

tip : configure the threadpool.bulk.queue_size property properly.

split brains

this should be a no-brainer for serious elasticsearch administrators and developers. you will have to configure the minimum number of master nodes setting correctly.

refer to http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_important_configuration_changes.html for details on this.

tip : set minimum_master_nodes to (number of master-eligible nodes / 2) + 1

disable multicast and use unicast

refer to the discussion in

http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_important_configuration_changes.html

you will need to disable multicast in production and use unicast. this will prevent nodes from accidentally joining your cluster in production.

tip : ensure that the "discovery.zen.ping.multicast.enabled" property is set to "false".

conclusion

it is very important to understand the elasticsearch architecture, concepts and configuration before deploying it in production. your devops will be grateful when you do the right things.

i hope you find this article useful. please inform me if i have made any mistakes or if there are any additional tips that you would like me to add.

Elasticsearch cluster

Opinions expressed by DZone contributors are their own.

Trending

  • Database Integration Tests With Spring Boot and Testcontainers
  • Transactional Outbox Patterns Step by Step With Spring and Kotlin
  • Reactive Programming
  • Chaining API Requests With API Gateway

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com

Let's be friends: