Custom Field Handling in Elasticsearch: Managing Total Number of Fields Limit
Join the DZone community and get the full member experience.
Join For Free
While using Elasticsearch to handle custom fields in your product, you soon hit the limit of the total number of fields in an index. Now, there are two ways to go.
1. Increase this limit just by updating index.mapping.total_fields.limit with a higher number. This will require additional memory for your Elasticsearch cluster.
PUT my_index_name/_settings
{
"index.mapping.total_fields.limit": 5000
}
2. Better manage your custom fields so that you use this limit in an optimum way and your Elasticsearch cluster requires no additional memory.
Let's assume that you have 10 clients for which you're using the same Elasticsearch index, and they have their own custom fields. So, client 1 has "model name," "Driver name," "vehicle brand," "vehicle number," "date of joining," etc. custom fields, while client two has "shade," "light," "texture," "brightness," etc. custom fields in their account.
If we count, each one of them has 200 of these unique custom fields. To support 10 clients, we need 10 * 200 = 2000 fields in Elasticsearch. Now, to handle a cluster with 2,000 fields of support in an index is costly. How can we manage it better? Let's see.
Let's assume we secured some custom fields of each type. For example:
50 custom fields of type date
cf_1_date=> custom field 1 of type date.cf_2_date.cf_3_date.cf_4_dateup tocf_50_date.
Similarly, we secured other types like text, boolean, long, etc. (and any data type that you support).
cf_1_textup tocf_50_text.cf_1_booleanup tocf_50_boolean.cf_1_longup tocf_50_long.
Now, we have a list of secured fields with us, so let's map client 1's custom fields with it.
- "model name" is of type text, so assign
cf_1_textto it. - "Driver name"is of type text. So let's pick the next field for text from the secured list.
cf_1_textis taken, so assigncf_2_textto it - "vehicle brand" is of type text, so assign
cf_3_textto it. - "vehicle number" is of type long, so assign
cf_1_longto it. - "date of joining" is of type date, so assign
cf_1_dateto it.
And, we keep the mapping like:
xxxxxxxxxx
{
"cf_1_text": "model name",
"cf_2_text": "Driver name",
"cf_3_text": "vehicle brand",
"cf_1_long": "vehicle number",
"cf_1_date": "date of joining"
}
Let's map client 2's custom fields.
"shade" is of type text, so assign
cf_1_textto it."light" is of type text, so assign
cf_2_textto it."texture" is of type text, so assign
cf_3_textto it."brightness" is of type text so assign
cf_4_textto it.
So our final mapping looks like this...
xxxxxxxxxx
{
"client_1":
{
"cf_1_text": "model name",
"cf_2_text": "Driver name",
"cf_3_text": "vehicle brand",
"cf_1_long": "vehicle number",
"cf_1_date": "date of joining"
},
"client_2":
{
"cf_1_text": "shade",
"cf_2_text": "light",
"cf_3_text": "texture",
"cf_4_text": "brightness"
}
}
Now, let's talk about how to index the data:
As soon as you're indexing the data for client 1, replace the client's field with secured fields using the above mapping. Ex:
xxxxxxxxxx
{
"model name": "BT 383",
"Driver name": "Arik Ykam",
"vehicle brand": "Bitol",
"vehicle number": "IUA 74 29744",
"date of joining": "2017-02-10"
}
- Replace it with:
xxxxxxxxxx
{
"cf_1_text": "BT 383",
"cf_2_text": "Arik Ykam",
"cf_3_text": "Bitol",
"cf_1_long": "IUA 74 29744",
"cf_1_date": "2017-02-10"
}
This is about indexing it. To ensure its data types, dynamic templates can be used with Elaticsearch. This is an example mapping:
{
"your_type_name":
{
"dynamic_templates": [
{
"dt_text":
{
"match": "*_text",
"mapping":
{
"type": "text",
}
}
},
{
"dt_long":
{
"match": "*_long",
"mapping":
{
"type": "double"
}
}
},
{
"dt_date":
{
"match": "*_date",
"mapping":
{
"type": "date"
}
}
},
{
"dt_boolean":
{
"match": "*_boolean",
"mapping":
{
"type": "boolean"
}
}
}],
"properties":
{}
}
}
Using a dynamic template and match like *_text, you can ensure that all field names matching this pattern will be of type text. (You can customize this.)
As soon as you index the data, any field name matching to
*_textwill be indexed as text.Field name matching to
*_longwill be indexed as double.Field name matching to
*_datewill be indexed as date data type.Field name matching to
*_booleanwill be indexed as boolean type.
Now, let's come to the presentation part where you fetch the data from index and show the actual fields to the client
You can simply add a wrapper to the data fetch class, which does this conversion:
xxxxxxxxxx
{
"cf_1_text": "BT 383",
"cf_2_text": "Arik Ykam",
"cf_3_text": "Bitol",
"cf_1_long": "IUA 74 29744",
"cf_1_date": "2017-02-10"
}
This is what Elasticseach will return. You'll check the mapping available for this client (client 1 in this case) which is:
xxxxxxxxxx
{
"cf_1_text": "model name",
"cf_2_text": "Driver name",
"cf_3_text": "vehicle brand",
"cf_1_long": "vehicle number",
"cf_1_date": "date of joining"
}
And then, simply replace the cf_1_text with "model name" and cf_2_text with "Driver name."
So, the final data will look like:
xxxxxxxxxx
{
"model name": "BT 383",
"Driver name": "Arik Ykam",
"vehicle brand": "Bitol",
"vehicle number": "IUA 74 29744",
"date of joining": "2017-02-10"
}
And that's it. Now, you can onboard any number of clients without having a need to update the field limit on the Elastiseach side.
xxxxxxxxxx
{
"your_type_name":
{
"dynamic_templates": [
{
"dt_text":
{
"match": "*_text",
"mapping":
{
"type": "text",
}
}
},
{
"dt_long":
{
"match": "*_long",
"mapping":
{
"type": "double"
}
}
},
{
"dt_date":
{
"match": "*_date",
"mapping":
{
"type": "date"
}
}
},
{
"dt_boolean":
{
"match": "*_boolean",
"mapping":
{
"type": "boolean"
}
}
}],
"properties":
{}
}
}
xxxxxxxxxx
{
"your_type_name":
{
"dynamic_templates": [
{
"dt_text":
{
"match": "*_text",
"mapping":
{
"type": "text",
}
}
},
{
"dt_long":
{
"match": "*_long",
"mapping":
{
"type": "double"
}
}
},
{
"dt_date":
{
"match": "*_date",
"mapping":
{
"type": "date"
}
}
},
{
"dt_boolean":
{
"match": "*_boolean",
"mapping":
{
"type": "boolean"
}
}
}],
"properties":
{}
}
}
xxxxxxxxxx
{
"your_type_name":
{
"dynamic_templates": [
{
"dt_text":
{
"match": "*_text",
"mapping":
{
"type": "text",
}
}
},
{
"dt_long":
{
"match": "*_long",
"mapping":
{
"type": "double"
}
}
},
{
"dt_date":
{
"match": "*_date",
"mapping":
{
"type": "date"
}
}
},
{
"dt_boolean":
{
"match": "*_boolean",
"mapping":
{
"type": "boolean"
}
}
}],
"properties":
{}
}
}
Further Reading
Opinions expressed by DZone contributors are their own.
Comments