The Loggly service utilizes ElasticSearch as the search engine underneath a lot of our core functionality. As Jon Gifford explained in his recent post on ElasticSearch vs Solr, log management imposes some tough requirements on search technology. To boil it down, it must be able to:
- Reliably perform near real-time indexing at huge scale – in our case, more than 100,000 log events per second
- Simultaneously handle high search volumes on the same index with solid performance and efficiency
When we were building our Gen2 log management service, we wanted to be sure that we were setting all of the ElasticSearch configurations in the way that would deliver maximum performance for both indexing and search. Unfortunately, we found it very difficult to find this information in the ElasticSearch documentation because it’s not located in one place. This post summarizes our learnings and can serve as a checklist of configuration properties you can reference to optimize ES for your application.
Tip #1: Know Your Deployment Topology Before You Set Configs
Loggly is running ES 0.90.13 with separate master and data nodes. We won’t be going into too much detail about that right now (look out for a subsequent post), other than to say that you need to determine your deployment topology in order to make the right configuration decisions.
In addition, we use the ES node client to talk to data nodes. This makes the client transparent to data nodes; all it cares about is talking to node client. You establish your ES nodes as data and master using two properties that are set as true or false. For example, to make an Elasticsearch a data node, you set:
node.master: false and node.data: true
So that was the easy part. Now we’ll talk about some advanced ES properties that deserve your attention. Their default settings are sufficient for most deployments, but if your ES use cases are anywhere near as tough as those we see with log management, you’ll get a lot of benefit from the advice below.
Tip #2: mlockall Offers the Biggest Bang for the Performance Efficiency Buck
Linux divides its physical RAM into chunks of memory called pages. Swapping is the process whereby a page of memory is copied to the preconfigured space on the hard disk, called swap space, to free up that page of memory. The combined sizes of the physical memory and the swap space is the amount of virtual memory available.
Swapping does have a downside. Compared to memory, disks are very slow. Memory speeds can be measured in nanoseconds, while disks are measured in milliseconds; so accessing the disk can be tens of thousands times slower than accessing physical memory. The more swapping that occurs, the slower your process will be, so you should avoid swapping at all cost.
The mlockall property in ES allows the ES node not to swap its memory. (Note that it is available only for Linux/Unix systems.) This property can be set in the yaml file by doing the following.
mlockall is set to false by default, meaning that the ES node will allow swapping. Once you add this value to the property file, you need to restart your ES node. You can verify if the value is set properly or not by doing the following:
If you are setting this property, make sure you are giving enough memory to the ES node using the -DXmx option or ES_HEAP_SIZE.
Tip #3: discovery.zen Properties Control the Discovery Protocol for ElasticSearch
Zen discovery is the protocol is used by ElasticSearch to discover and communicate between the nodes in the cluster. The zen discovery setting is controlled by the discovery.zen.* properties. Both unicast and multicast are available as part of discovery protocol:
- Multicast is when nodes in a cluster are discovered by sending one or more multicast requests to all the nodes.
- Unicast is a one-to-one connection between the nodes and the list of IP addresses specified under the discovery.zen.ping.unicast.hosts.
In order to enable unicast, you set discovery.zen.ping.multicast.enabled to false.
For unicast, you also must specify the group of hosts that is used to communicate for discovery. This is done using the property discovery.zen.ping.unicast.hosts, which contains the master hosts which unicast can use to communicate.
The discovery.zen.minimum_master_nodes control the minimum number of eligible master nodes that a node should “see” in order to operate within the cluster. It’s recommended that you set it to a higher value than 1 when running more than 2 nodes in the cluster. One way to calculate value for this will be N/2 + 1 where N is the number of master nodes.
Data and master nodes detect each other in two different ways:
- By the master pinging all other nodes in the cluster and verify they are up and running
- By all other nodes pinging the master nodes to verify if they are up and running or if an election process needs to be initiated
The node detection process is controlled by discover.zen.fd.ping_timeout property. The default value is 30s, which determines how long the node will wait for a response. This property should be adjusted if you are operating on a slow or congested network. If you are on a slow network, set the value higher. The higher the value, the smaller the chance of discovery failure.
Loggly has configured our discovery.zen properties as follows:
discovery.zen.fd.ping_timeout: 30s discovery.zen.minimum_master_nodes: 2 discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: [“esmaster01″,”esmaster02″,”esmaster03″]
The above properties say that node detection should happen within 30 seconds; this is done by setting discovery.zen.fd.ping_timeout. In addition, two minimum master nodes should be detected by other nodes (we have 3 masters). This discovery method used is unicast and the unicast hosts are esmaster01, esmaster02, esmaster03.
Tip #4: Watch Out for delete_all_indices!
It’s really important to know that the curl API in ES does not have very good authentication built into it. A simple curl API can cause all the indices to delete themselves and lose all data. This is just one example of a command that could cause a mistaken deletion:
curl -XDELETE ‘http://localhost:9200/*/’
To avoid this type of grief, you can set the following property:
This will make sure when above command is given, it will not delete the index and will instead result in an error.
Tip #5: Field Data Caching Can Cause Extremely Slow Facet Searches
This is how the ElasticSearch help describes the field data cache:
The field data cache is used mainly when sorting on or faceting on a field. It loads all the field values to memory in order to provide fast document based access to those values. The field data cache can be expensive to build for a field, so its recommended to have enough memory to allocate it, and to keep it loaded.
You need to keep in mind that not setting this value properly can cause:
- Facet searches and sorting to have very poor performance
- The ES node to run out of memory if you run the facet query against a large index
In setting this value, the key consideration is what kind of facet searches your application performs.
To Be Continued...
Be sure to catch the last four tips in this two-part series.