5 Easy Ways to Crash Elasticsearch

the bad news is that as awesome as the elk stack is for centralized logging and monitoring, it can also be a tricky beast to handle. sometimes, all it takes is one simple search queried against a big pool of data to bring the whole stack tumbling down on your head.

the good news is that it’s easy to avoid these crashes by applying some best practices.

learning from the mistakes made by our users and those made by ourselves as well, we have compiled a concise list of five things to avoid doing in your elk deployment. as a company with an elk cloud service , we have blocked these dangerous user behaviors in our ui. but if you’re managing your own elk deployment, you should take special note of this list.

1. leading wildcard searches

querying elasticsearch from kibana is an art because many different types of searches are available. from free-text searches to field-level and regex searches, there are many options, and this variety is one of reasons that people opt for the elk stack in the first place. as implied in the opening statement above, some kibana searches are going to crash elasticsearch in certain circumstances.

for example, using a leading wildcard search on a large data set has the potential of stalling the system and should therefore be avoided.

best practice: avoid using wildcard queries if possible, especially when performed against very large data sets.

2. term aggregation on analyzed fields

in elasticsearch, analyzed fields are separated using a tokenizer. the default tokenizer used by elasticsearch separates fields using white spaces.

this means that if we were analyzing the following fields:

“this is a sentence a b c”
“this is a sentence a b c”
“this is a sentence a b c d”

our terms output would look as follows:

“a” – 6
“this” – 3
“is” – 3
“sentence” – 3
“b” – 3
“c” – 3
“d” – 1

instead of having only two results:

“this is a sentence a b c” – 2
“this is a sentence a b c d” – 1

the end-result of this elasticsearch behavior is that using terms in large data sets can consume a very large amount of memory, which could potentially result in the crashing of your elasticsearch cluster.

best practice: if you need to use term aggregation on a text field, you most likely need the field to be not analyzed. this can be configured in the mapping when creating an index or via templates.

3. cardinality

cardinality aggregation is used to count distinct values in a data set. for example, if you want to know the number of ips used in your system, you can use this aggregation on an ip field and then count the results.

despite the usefulness, cardinality can also be a touchy elasticsearch feature to use. performing a unique count on a field with a multitude of possible values when configuring a visualization, for example, can bring elasticsearch to a halt.

best practice: only use cardinality (unique count) when you are sure that the field cardinality is not too big. if you are not sure, there are usually different ways in elasticsearch to achieve the same purpose.

4. mappings

since elasticsearch 2.x, any type of mapping changes the “locks” of the cluster for indexing, and the elected master node is responsible for all mapping changes.

if, for example, i send a new field, the master node halts all indexing to the cluster, syncs the changes to the data nodes, and then resumes.

this means that if we apply frequent mapping changes, elasticsearch indexing can come to a halt. at logz.io, we see this occurring when our customers, for example, ignore the distinction between keys and values or when an arbitrary url is used as a field name.

best practice: take a careful look at your documents and restructure them so that they have fixed keys, if you identify keys that change according to values.

5. kibana advanced settings

some kibana-specific configurations can cause your browser to crash. for example, depending on your browser and system settings, changing the value of the discover:samplesize setting to a high number can easily cause kibana to freeze.

that is why the good folks at elastic have placed a warning at the top of the page that is supposed to convince us to be extra careful. anyone with a guess on how successful this warning is?

best practice: pay heed to the warnings in kibana, and be extremely careful when making changes to advanced settings.

summary

at logz.io, we’ve applied safeguards to make sure that users cannot bring down the stack. if you’ve got your own stack deployed, just be sure to be aware of these loopholes.

we’d be happy to update the list above with other examples, so if you have a war story to share, please feel free to comment below.