{{announcement.body}}
{{announcement.title}}

Custom Field Handling in Elasticsearch: Managing Total Number of Fields Limit

DZone 's Guide to

Custom Field Handling in Elasticsearch: Managing Total Number of Fields Limit

In this article, we discuss how to implement custom field handling in Elastisearch in order to manage total number of fields and their limits.

· Big Data Zone ·
Free Resource

While using Elasticsearch to handle custom fields in your product, you soon hit the limit of the total number of fields in an index. Now, there are two ways to go.

1. Increase this limit just by updating index.mapping.total_fields.limit with a higher number. This will require additional memory for your Elasticsearch cluster.

Java




x


 
1
PUT my_index_name/_settings
2
{
3
  "index.mapping.total_fields.limit": 5000
4
}



2. Better manage your custom fields so that you use this limit in an optimum way and your Elasticsearch cluster requires no additional memory.

Let's assume that you have 10 clients for which you're using the same Elasticsearch index, and they have their own custom fields. So, client 1 has "model name," "Driver name," "vehicle brand," "vehicle number," "date of joining," etc. custom fields, while client two has "shade," "light," "texture," "brightness," etc. custom fields in their account.

You may also like: What Is Elasticsearch? (And Why You Need to Be Using It).

If we count, each one of them has 200 of these unique custom fields. To support 10 clients, we need 10 * 200 = 2000 fields in Elasticsearch. Now, to handle a cluster with 2,000 fields of support in an index is costly. How can we manage it better? Let's see.

Let's assume we secured some custom fields of each type. For example:

  • 50 custom fields of type date

    • cf_1_date => custom field 1 of type date.

    • cf_2_date.

    • cf_3_date.

    • cf_4_date up to cf_50_date.

  • Similarly, we secured other types like text, boolean, long, etc. (and any data type that you support).

    • cf_1_text up to cf_50_text.

    • cf_1_boolean up to cf_50_boolean.

    • cf_1_long up to cf_50_long.

Now, we have a list of secured fields with us, so let's map client 1's custom fields with it.

  • "model name" is of type text, so assign cf_1_text to it.
  • "Driver name"is of type text. So let's pick the next field for text from the secured list. cf_1_text  is taken, so assign cf_2_text to it
  • "vehicle brand" is of type text, so assign cf_3_text to it.
  • "vehicle number" is of type long, so assign cf_1_long to it.
  • "date of joining" is of type date, so assign cf_1_date to it.

And, we keep the mapping like:

JSON




xxxxxxxxxx
1


 
1
{
2
    "cf_1_text": "model name",
3
    "cf_2_text": "Driver name",
4
    "cf_3_text": "vehicle brand",
5
    "cf_1_long": "vehicle number",
6
    "cf_1_date": "date of joining"
7
}



Let's map client 2's custom fields.

  • "shade" is of type text, so assign cf_1_text to it.

  • "light" is of type text, so assign cf_2_text to it.

  • "texture" is of type text, so assign cf_3_text to it.

  • "brightness" is of type text so assign cf_4_text to it.

So our final mapping looks like this...

JSON




xxxxxxxxxx
1
17


 
1
{
2
    "client_1":
3
    {
4
        "cf_1_text": "model name",
5
        "cf_2_text": "Driver name",
6
        "cf_3_text": "vehicle brand",
7
        "cf_1_long": "vehicle number",
8
        "cf_1_date": "date of joining"
9
    },
10
    "client_2":
11
    {
12
        "cf_1_text": "shade",
13
        "cf_2_text": "light",
14
        "cf_3_text": "texture",
15
        "cf_4_text": "brightness"
16
    }
17
}



Now, let's talk about how to index the data:

As soon as you're indexing the data for client 1, replace the client's field with secured fields using the above mapping. Ex:

JSON




xxxxxxxxxx
1


 
1
{
2
    "model name": "BT 383",
3
    "Driver name": "Arik Ykam",
4
    "vehicle brand": "Bitol",
5
    "vehicle number": "IUA 74 29744",
6
    "date of joining": "2017-02-10"
7
}



  • Replace it with:
JSON




xxxxxxxxxx
1


 
1
{
2
    "cf_1_text": "BT 383",
3
    "cf_2_text": "Arik Ykam",
4
    "cf_3_text": "Bitol",
5
    "cf_1_long": "IUA 74 29744",
6
    "cf_1_date": "2017-02-10"
7
}



This is about indexing it. To ensure its data types, dynamic templates can be used with Elaticsearch. This is an example mapping:

JSON




x
48


 
1
{
2
    "your_type_name":
3
    {
4
        "dynamic_templates": [
5
        {
6
            "dt_text":
7
            {
8
                "match": "*_text",
9
                "mapping":
10
                {
11
                    "type": "text",
12
                }
13
            }
14
        },
15
        {
16
            "dt_long":
17
            {
18
                "match": "*_long",
19
                "mapping":
20
                {
21
                    "type": "double"
22
                }
23
            }
24
        },
25
        {
26
            "dt_date":
27
            {
28
                "match": "*_date",
29
                "mapping":
30
                {
31
                    "type": "date"
32
                }
33
            }
34
        },
35
        {
36
            "dt_boolean":
37
            {
38
                "match": "*_boolean",
39
                "mapping":
40
                {
41
                    "type": "boolean"
42
                }
43
            }
44
        }],
45
        "properties":
46
        {}
47
    }
48
}



Using a dynamic template and match like *_text, you can ensure that all field names matching this pattern will be of type text. (You can customize this.)

  • As soon as you index the data, any field name matching to *_text will be indexed as text.

  • Field name matching to *_long will be indexed as double.

  • Field name matching to *_date will be indexed as date data type.

  • Field name matching to *_boolean will be indexed as boolean type.

Now, let's come to the presentation part where you fetch the data from index and show the actual fields to the client

  • You can simply add a wrapper to the data fetch class, which does this conversion:

JSON




xxxxxxxxxx
1


 
1
{
2
    "cf_1_text": "BT 383",
3
    "cf_2_text": "Arik Ykam",
4
    "cf_3_text": "Bitol",
5
    "cf_1_long": "IUA 74 29744",
6
    "cf_1_date": "2017-02-10"
7
}



This is what Elasticseach will return. You'll check the mapping available for this client (client 1 in this case) which is: 

JSON




xxxxxxxxxx
1


 
1
{
2
    "cf_1_text": "model name",
3
    "cf_2_text": "Driver name",
4
    "cf_3_text": "vehicle brand",
5
    "cf_1_long": "vehicle number",
6
    "cf_1_date": "date of joining"
7
}



And then, simply replace the cf_1_text with "model name" and  cf_2_text with "Driver name."

So, the final data will look like: 

JSON




xxxxxxxxxx
1


 
1
{
2
    "model name": "BT 383",
3
    "Driver name": "Arik Ykam",
4
    "vehicle brand": "Bitol",
5
    "vehicle number": "IUA 74 29744",
6
    "date of joining": "2017-02-10"
7
}



And that's it. Now, you can onboard any number of clients without having a need to update the field limit on the Elastiseach side.

JSON




xxxxxxxxxx
1
48


 
1
{
2
    "your_type_name":
3
    {
4
        "dynamic_templates": [
5
        {
6
            "dt_text":
7
            {
8
                "match": "*_text",
9
                "mapping":
10
                {
11
                    "type": "text",
12
                }
13
            }
14
        },
15
        {
16
            "dt_long":
17
            {
18
                "match": "*_long",
19
                "mapping":
20
                {
21
                    "type": "double"
22
                }
23
            }
24
        },
25
        {
26
            "dt_date":
27
            {
28
                "match": "*_date",
29
                "mapping":
30
                {
31
                    "type": "date"
32
                }
33
            }
34
        },
35
        {
36
            "dt_boolean":
37
            {
38
                "match": "*_boolean",
39
                "mapping":
40
                {
41
                    "type": "boolean"
42
                }
43
            }
44
        }],
45
        "properties":
46
        {}
47
    }
48
}


JSON




xxxxxxxxxx
1
48


 
1
{
2
    "your_type_name":
3
    {
4
        "dynamic_templates": [
5
        {
6
            "dt_text":
7
            {
8
                "match": "*_text",
9
                "mapping":
10
                {
11
                    "type": "text",
12
                }
13
            }
14
        },
15
        {
16
            "dt_long":
17
            {
18
                "match": "*_long",
19
                "mapping":
20
                {
21
                    "type": "double"
22
                }
23
            }
24
        },
25
        {
26
            "dt_date":
27
            {
28
                "match": "*_date",
29
                "mapping":
30
                {
31
                    "type": "date"
32
                }
33
            }
34
        },
35
        {
36
            "dt_boolean":
37
            {
38
                "match": "*_boolean",
39
                "mapping":
40
                {
41
                    "type": "boolean"
42
                }
43
            }
44
        }],
45
        "properties":
46
        {}
47
    }
48
}


JSON




xxxxxxxxxx
1
48


 
1
{
2
    "your_type_name":
3
    {
4
        "dynamic_templates": [
5
        {
6
            "dt_text":
7
            {
8
                "match": "*_text",
9
                "mapping":
10
                {
11
                    "type": "text",
12
                }
13
            }
14
        },
15
        {
16
            "dt_long":
17
            {
18
                "match": "*_long",
19
                "mapping":
20
                {
21
                    "type": "double"
22
                }
23
            }
24
        },
25
        {
26
            "dt_date":
27
            {
28
                "match": "*_date",
29
                "mapping":
30
                {
31
                    "type": "date"
32
                }
33
            }
34
        },
35
        {
36
            "dt_boolean":
37
            {
38
                "match": "*_boolean",
39
                "mapping":
40
                {
41
                    "type": "boolean"
42
                }
43
            }
44
        }],
45
        "properties":
46
        {}
47
    }
48
}



Further Reading

Topics:
big data, elastisearch, field handling, filed limit, tutorial

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}