ES[7.6.x] dynamic mapping of learning notes

Posted by madan koshti on Tue, 05 May 2020 05:24:33 +0200

In general, we use es to build an index by creating an index, defining the fields in the index and the mapped types, and then importing data into the index. Dynamic mapping is a very important concept in ES. You can directly import a piece of data into the document. At the same time, index, field and field type will be created automatically without any other operation. This is the magic of dynamic mapping.

Dynamic field mapping

The dynamic mapping of ES is on by default, and the default rules of dynamic mapping are as follows:

Data type of JSON Data types in ES
null Fields will not be mapped
true or false boolean type
Floating point number float
Integer number long
JSON object Object
array First non null value type
String 1. If the format of date type is satisfied, map to date type
2. If the format of number type is satisfied, map to long or float
3. If it is a string, it will be mapped to a text type and a keyword type

Next, let's look at an example of dynamic mapping. We store a piece of data directly into the dynamic index index. Note that we haven't created the dynamic index. If we store the data directly, the index will be created automatically. Next, let's look at the specific request:

PUT /dynamic-index/_doc/1
{
  "my_null": null,
  "my_boolean": false,
  "my_float": 1.56,
  "my_long": 3,
  "my_object": {
      "my_first": "first value",
      "my_second": "second_value"
  },
  "my_array": [1,2,3],
  "my_date_1": "2020-05-01",
  "my_date_2": "2020/05/01 12:03:03",
  "my_date_3": "05/01/2020",
  "my_string_long": "1",
  "my_string_float": "4.6",
  "my_string": "The People's Republic of China"
}

After the request is executed successfully, let's first look at the type of index:

GET /dynamic-index/_mapping

The results returned are as follows:

{
    "dynamic-index": {
        "mappings": {
            "properties": {
                "my_array": {
                    "type": "long"
                },
                "my_boolean": {
                    "type": "boolean"
                },
                "my_date_1": {
                    "type": "date"
                },
                "my_date_2": {
                    "type": "date",
                    "format": "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis"
                },
                "my_date_3": {
                    "type": "text",
                    "fields": {
                        "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                        }
                    }
                },
                "my_float": {
                    "type": "float"
                },
                "my_long": {
                    "type": "long"
                },
                "my_object": {
                    "properties": {
                        "my_first": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 256
                                }
                            }
                        },
                        "my_second": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 256
                                }
                            }
                        }
                    }
                },
                "my_string": {
                    "type": "text",
                    "fields": {
                        "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                        }
                    }
                },
                "my_string_float": {
                    "type": "text",
                    "fields": {
                        "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                        }
                    }
                },
                "my_string_long": {
                    "type": "text",
                    "fields": {
                        "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                        }
                    }
                }
            }
        }
    }
}

The returned result is relatively long. Let's take a look at each field to see if the dynamically mapped field meets our expectation:

field Mapping results Whether it meets the expectation Reason
my_null No mapping yes null value does not map
my_boolean boolean yes
my_float float yes
my_long long yes
my_object object yes The mapping of two fields is automatically generated in my object
my_array long yes The numbers in the array are long
my_date_1 date yes
my_date_2 date yes
my_date_3 text no This date format is not specified, so it is mapped to text
my_string_long text no Digital detection is off by default, not on
my_string_float text no Digital detection is off by default, not on
my_string text yes Normal string, mapped to text

Next, we turn on the digital detection and execute the following request:

PUT /dynamic-index
{
  "mappings": {
    "numeric_detection": true
  }
}

Since there is a mapping relationship in our index dynamic index, setting it again will result in an error. Therefore, we need to delete the index and execute the following request:

DELETE /dynamic-index

After the index is deleted successfully, the previous settings will be executed. The execution is successful and the digital probe has been opened. Then add a date format MM/dd/yyyy. The request is as follows:

PUT /dynamic-index
{
  "mappings": {
    "dynamic_date_formats": ["MM/dd/yyyy"]
  }
}

The error message is the same as before. It seems that the setting of the date and the number probe should be together. Then we delete the index and send the request. The two settings are together:

PUT /dynamic-index
{
  "mappings": {
    "numeric_detection": true,
    "dynamic_date_formats": ["MM/dd/yyyy"]
  }
}

Execution successful, we will send the request to create index data before

PUT /dynamic-index/_doc/1
{
  "my_null": null,
  "my_boolean": false,
  "my_float": 1.56,
  "my_long": 3,
  "my_object": {
      "my_first": "first value",
      "my_second": "second_value"
  },
  "my_array": [1,2,3],
  "my_date_1": "2020-05-01",
  "my_date_2": "2020/05/01 12:03:03",
  "my_date_3": "05/01/2020",
  "my_string_long": "1",
  "my_string_float": "4.6",
  "my_string": "The People's Republic of China"
}

The execution is successful. Let's look at the index mapping again,

"my_string_float": {
    "type": "float"
},
"my_string_long": {
    "type": "long"
}
"my_date_1": {
    "type": "text",
    "fields": {
        "keyword": {
            "type": "keyword",
            "ignore_above": 256
        }
    }
},
"my_date_2": {
    "type": "text",
    "fields": {
        "keyword": {
            "type": "keyword",
            "ignore_above": 256
        }
    }
},
"my_date_3": {
    "type": "date",
    "format": "MM/dd/yyyy"
},

Let's take a look at the above fields. My string? Float and my string? Long map to the type we want, because we have turned on number detection. Let's take a look at the three date types we map, eh? Only my date 3 maps the date type, and the other two map to text type. This is because we only specify one format when setting dynamic date formats. We just need to add the other two types of date formats.

{
  "mappings": {
    "numeric_detection": true,
    "dynamic_date_formats": ["MM/dd/yyyy","yyyy/MM/dd HH:mm:ss","yyyy-MM-dd"]
  }
}

I won't give you a specific demonstration here. I'd like to have a try.

Dynamic field is a very important function in ES, which brings us great convenience and saves us the time of creating index field in development. It's a multiplier with half the effort, so we need to master it well~~

Topics: Java JSON