brief introduction
Elasticsearch is a Lucene based search server. It provides a distributed multi-user full-text search engine based on RESTful web interface. Elasticsearch is developed in the Java language and released as an open source under the Apache license terms. It is a popular enterprise search engine. Elasticsearch is used in cloud computing. It can achieve real-time search, stable, reliable, fast and easy to install and use. The official client is in Java NET (C#), PHP, Python, Apache, groovy, Ruby, and many other languages are available. According to the ranking of DB engines, elasticsearch is the most popular enterprise search engine, followed by Apache Solr, which is also based on Lucene.
1. ELASTICSEARCH
1. Install elastic search
Installing elastic search in dokcer
(1) Download easy search and kibana
docker pull elasticsearch:7.6.2 docker pull kibana:7.6.2
(2) Disposition
mkdir -p /mydata/elasticsearch/config Create directory mkdir -p /mydata/elasticsearch/data echo "http.host: 0.0.0.0" >/mydata/elasticsearch/config/elasticsearch.yml //Make the files in the mydata/elasticsearch / folder readable and writable chmod -R 777 /mydata/elasticsearch/
(3) Launch Elastic search
docker run --name elasticsearch -p 9200:9200 -p 9300:9300 \ -e "discovery.type=single-node" \ -e ES_JAVA_OPTS="-Xms64m -Xmx512m" \ -v /mydata/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \ -v /mydata/elasticsearch/data:/usr/share/elasticsearch/data \ -v /mydata/elasticsearch/plugins:/usr/share/elasticsearch/plugins \ -d elasticsearch:7.6.2
Set startup to start elasticsearch
docker update elasticsearch --restart=always
(4) Start kibana:
docker run --name kibana -e ELASTICSEARCH_HOSTS=http://192.168.6.128:9200 -p 5601:5601 -d kibana:7.6.2
Set boot up kibana
docker update kibana --restart=always
(5) Testing
To view elasticsearch version information: http://192.168.6.128:9200/
{
"name": "0adeb7852e00",
"cluster_name": "elasticsearch",
"cluster_uuid": "9gglpP0HTfyOTRAaSe2rIg",
"version": {
"number": "7.6.2",
"build_flavor": "default",
"build_type": "docker",
"build_hash": "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
"build_date": "2020-03-26T06:34:37.794943Z",
"build_snapshot": false,
"lucene_version": "8.4.0",
"minimum_wire_compatibility_version": "6.8.0",
"minimum_index_compatibility_version": "6.0.0-beta1"
},
"tagline": "You Know, for Search"
}
Displays the elasticsearch node information http://192.168.6.128:9200/_cat/nodes ，
127.0.0.1 76 95 1 0.26 1.40 1.22 dilm * 0adeb7852e00
Visit Kibana: http://192.168.6.128:5601/app/kibana
[the external chain image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-UDyxhFZt-1624414458545)(images/image-20200501192629304.png)]
2. Preliminary search
1)_CAT
(1) GET/cat/nodes: view all nodes
For example: http://192.168.6.128:9200/_cat/nodes :
127.0.0.1 61 91 11 0.08 0.49 0.87 dilm * 0adeb7852e00
Note: * indicates the primary node in the cluster
(2) GET/cat/health: View es health
For example: http://192.168.6.128:9200/_cat/health
1588332616 11:30:16 elasticsearch green 1 1 3 3 0 0 0 0 - 100.0%
Note: green indicates that the health value is normal
(3)GET/_cat/master: View master node_ information
For example: http://192.168.6.128:9200/_cat/master
vfpgxbusTC6-W3C2Np31EQ 127.0.0.1 127.0.0.1 0adeb7852e00
(4)GET/_ Cat / indexes: view all indexes, which is equivalent to show databases of mysql database;
For example: http://192.168.6.128:9200/_cat/indices
green open .kibana_task_manager_1 KWLtjcKRRuaV9so_v15WYg 1 0 2 0 39.8kb 39.8kb green open .apm-agent-configuration cuwCpJ5ER0OYsSgAJ7bVYA 1 0 0 0 283b 283b green open .kibana_1 PqK_LdUYRpWMy4fK0tMSPw 1 0 7 0 31.2kb 31.2kb
2) Index a document
Save a data, and specify the unique identifier under which index and type it is saved
PUT customer/external/1; Save the No. 1 data under the external type under the customer index as
PUT customer/external/1
{
"name":"John Doe"
}
Both PUT and POST are OK
POST added. If you do not specify an id, an id is automatically generated. Specifying the id will modify the data and add a version number;
PUT can be added or modified. PUT must specify id; Because PUT needs to specify an id, it is generally used for modification. If the id is not specified, an error will be reported.
Here are the test data in postman:
[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-1i353kex-1624414458551)(images/image-20200501194449944.png)]
After the data is created successfully, 201 created is displayed, indicating that the record is inserted successfully.
{
"_index": "customer",
"_type": "external",
"_id": "1",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"_seq_no": 0,
"_primary_term": 1
}
The meaning of these returned JSON strings; Those that begin with an underscore are called metadata and reflect the current basic information.
"_index": "customer" indicates which database the data is in;
"_type": "external" indicates the type of the data;
"_id": "1" indicates the ID of the saved data;
'_version': 1, the version of the saved data
"result": "created" here is a piece of data created. If you put another piece of data, the status will change to updated and the version number will also change.
POST mode is selected as follows:
When adding data, the ID will be automatically generated without specifying the ID, and the type is new:
Using POST to insert data again is still new:
When adding data, specify an ID, which will be used, and the type is new:
Use POST again to insert data of type updated
3) View document
GET /customer/external/1
http://192.168.6.128:9200/customer/external/1
{
"_index": "customer",//At which index
"_type": "external",//In which type
"_id": "1",//Record id
"_version": 3,//Version number
"_seq_no": 6,//The concurrency control field will be + 1 every time it is updated, which is used as an optimistic lock
"_primary_term": 1,//As above, the main partition will be reassigned. If it is restarted, it will change
"found": true,
"_source": {
"name": "John Doe"
}
}
Through "if_seq_no = 1 & if_primary_term = 1", it can be modified only when the serial numbers match, otherwise it will not be modified.
Example: update the data with id=1 to name=1, and then update it to name=2 again, starting from_ seq_no=6，_ primary_term=1
(1) Update name to 1
http://192.168.6.128:9200/customer/external/1?if_seq_no=1&if_primary_term=1
(2) Update the name to 2 and use SEQ during the update process_ no=6
http://192.168.6.128:9200/customer/external/1?if_seq_no=6&if_primary_term=1
An update error occurred.
(3) Query new data
http://192.168.6.128:9200/customer/external/1
[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-a2cxxfc2-1624414458555) (images / image-202005012924094. PNG)]
Can see_ seq_no becomes 7.
(4) Update again. The update is successful
http://192.168.6.128:9200/customer/external/1?if_seq_no=7&if_primary_term=1
4) Update document
[the external chain image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-itzwgioz-1624414458560) (images / image-2020050124522818. PNG)]
[external link image transfer failed. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-AbFF4rUB-1624414458563)(images/image-20200501215746139.png)]
(1) POST update document with_ update
http://192.168.6.128:9200/customer/external/1/_update
[the external chain image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-hpkyox8l-1624414458564) (images / image-2020050124141810741. PNG)]
If the update is performed again, no action is performed and the serial number does not change
[the external chain image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-gfetiuyr-1624414458567) (images / image-2020050124912607. PNG)]
The POST update method will compare the original data. If it is the same as the original data, no operation (version and _seq_no) will be performed.
(2) POST update document without_ update
[the external chain image transfer fails, and the source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-unlcrojl-1624414458569) (images / image-2020050125358666. PNG)]
During the update process, the update operation is repeated, and the data can be updated successfully without comparison with the original data.
5) Delete document or index
DELETE customer/external/1 DELETE customer
Note: elastic search does not provide delete operations, but only delete indexes and documents.
Example: delete the data with id=1, and continue to query after deletion
Example: delete the entire costomer index data
All indexes before deletion
green open .kibana_task_manager_1 KWLtjcKRRuaV9so_v15WYg 1 0 2 0 39.8kb 39.8kb green open .apm-agent-configuration cuwCpJ5ER0OYsSgAJ7bVYA 1 0 0 0 283b 283b green open .kibana_1 PqK_LdUYRpWMy4fK0tMSPw 1 0 7 0 31.2kb 31.2kb yellow open customer nzDYCdnvQjSsapJrAIT8Zw 1 1 4 0 4.4kb 4.4kb
Delete "customer" index
[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-u6ndszk7-1624414458571) (images / image-2020050121105476. PNG)]
After deletion, all indexes
green open .kibana_task_manager_1 KWLtjcKRRuaV9so_v15WYg 1 0 2 0 39.8kb 39.8kb green open .apm-agent-configuration cuwCpJ5ER0OYsSgAJ7bVYA 1 0 0 0 283b 283b green open .kibana_1 PqK_LdUYRpWMy4fK0tMSPw 1 0 7 0 31.2kb 31.2kb
6) bulk operation of electronic search
Syntax format:
{action:{metadata}}\n //For example, index saves records and update updates
{request body }\n
{action:{metadata}}\n
{request body }\n
For batch operations here, when an execution fails, other data can still be executed, that is, they are independent of each other.
The bulk api executes all actions in this order. If a single action fails for any reason, it will continue to process the remaining actions after it. When the bulk api returns, it will provide the status of each action (in the same order as sending), so you can check whether a specified action has failed.
Example 1: execute multiple pieces of data (postman reports an error, which is executed in Dev_tools in kibana)
POST /customer/external/_bulk
{"index":{"_id":"1"}}
{"name":"John Doe"}
{"index":{"_id":"2"}}
{"name":"John Doe"}
results of enforcement
#! Deprecation: [types removal] Specifying types in bulk requests is deprecated.
{
"took" : 491,
"errors" : false,
"items" : [
{
"index" : {
"_index" : "customer",
"_type" : "external",
"_id" : "1",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 0,
"_primary_term" : 1,
"status" : 201
}
},
{
"index" : {
"_index" : "customer",
"_type" : "external",
"_id" : "2",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 1,
"_primary_term" : 1,
"status" : 201
}
}
]
}
Example 2: batch operation for the entire index
POST /_bulk
{"delete":{"_index":"website","_type":"blog","_id":"123"}}
{"create":{"_index":"website","_type":"blog","_id":"123"}}
{"title":"my first blog post"}
{"index":{"_index":"website","_type":"blog"}}
{"title":"my second blog post"}
{"update":{"_index":"website","_type":"blog","_id":"123"}}
{"doc":{"title":"my updated blog post"}}
Operation results:
#! Deprecation: [types removal] Specifying types in bulk requests is deprecated.
{
"took" : 608,
"errors" : false,
"items" : [
{
"delete" : {
"_index" : "website",
"_type" : "blog",
"_id" : "123",
"_version" : 1,
"result" : "not_found",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 0,
"_primary_term" : 1,
"status" : 404
}
},
{
"create" : {
"_index" : "website",
"_type" : "blog",
"_id" : "123",
"_version" : 2,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 1,
"_primary_term" : 1,
"status" : 201
}
},
{
"index" : {
"_index" : "website",
"_type" : "blog",
"_id" : "MCOs0HEBHYK_MJXUyYIz",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 2,
"_primary_term" : 1,
"status" : 201
}
},
{
"update" : {
"_index" : "website",
"_type" : "blog",
"_id" : "123",
"_version" : 3,
"result" : "updated",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 3,
"_primary_term" : 1,
"status" : 200
}
}
]
}
7) Sample test data
A fictional JSON document sample of customer bank account information is prepared. Each document has the following schema.
{
"account_number": 1,
"balance": 39225,
"firstname": "Amber",
"lastname": "Duke",
"age": 32,
"gender": "M",
"address": "880 Holmes Lane",
"employer": "Pyrami",
"email": "amberduke@pyrami.com",
"city": "Brogan",
"state": "IL"
}
https://github.com/elastic/elasticsearch/blob/master/docs/src/test/resources/accounts.json?raw=true , import test data,
POST bank/account/_bulk
3. Search
1)search Api
ES supports two basic retrieval methods;
- Send search parameters (uri + search parameters) through REST request uri;
- Send them through REST request body (uri + request body);
Information retrieval
[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-hBV2DBg4-1624414458574)(images/image-20200501233938697.png)]
[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-Kn826E90-1624414458575)(images/image-20200502000221422.png)]
[the external chain image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-gOYRR2DR-1624414458578)(images/image-20200502000733594.png)]
uri + request body for retrieval
GET /bank/_search
{
"query": { "match_all": {} },
"sort": [
{ "account_number": "asc" },
{"balance":"desc"}
]
}
HTTP client tool (), the get request cannot carry the request body,
GET bank/_search?q=*&sort=account_number:asc //q = * query all, sort=account_number:asc according to account_number in asc ascending sort
Return result:
{
"took" : 235,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1000,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "bank",
"_type" : "account",
"_id" : "0",
"_score" : null,
"_source" : {
"account_number" : 0,
"balance" : 16623,
"firstname" : "Bradshaw",
"lastname" : "Mckenzie",
"age" : 29,
"gender" : "F",
"address" : "244 Columbus Place",
"employer" : "Euron",
"email" : "bradshawmckenzie@euron.com",
"city" : "Hobucken",
"state" : "CO"
},
"sort" : [
0
]
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "1",
"_score" : null,
"_source" : {
"account_number" : 1,
"balance" : 39225,
"firstname" : "Amber",
"lastname" : "Duke",
"age" : 32,
"gender" : "M",
"address" : "880 Holmes Lane",
"employer" : "Pyrami",
"email" : "amberduke@pyrami.com",
"city" : "Brogan",
"state" : "IL"
},
"sort" : [
1
]
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "2",
"_score" : null,
"_source" : {
"account_number" : 2,
"balance" : 28838,
"firstname" : "Roberta",
"lastname" : "Bender",
"age" : 22,
"gender" : "F",
"address" : "560 Kingsway Place",
"employer" : "Chillium",
"email" : "robertabender@chillium.com",
"city" : "Bennett",
"state" : "LA"
},
"sort" : [
2
]
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "3",
"_score" : null,
"_source" : {
"account_number" : 3,
"balance" : 44947,
"firstname" : "Levine",
"lastname" : "Burks",
"age" : 26,
"gender" : "F",
"address" : "328 Wilson Avenue",
"employer" : "Amtap",
"email" : "levineburks@amtap.com",
"city" : "Cochranville",
"state" : "HI"
},
"sort" : [
3
]
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "4",
"_score" : null,
"_source" : {
"account_number" : 4,
"balance" : 27658,
"firstname" : "Rodriquez",
"lastname" : "Flores",
"age" : 31,
"gender" : "F",
"address" : "986 Wyckoff Avenue",
"employer" : "Tourmania",
"email" : "rodriquezflores@tourmania.com",
"city" : "Eastvale",
"state" : "HI"
},
"sort" : [
4
]
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "5",
"_score" : null,
"_source" : {
"account_number" : 5,
"balance" : 29342,
"firstname" : "Leola",
"lastname" : "Stewart",
"age" : 30,
"gender" : "F",
"address" : "311 Elm Place",
"employer" : "Diginetic",
"email" : "leolastewart@diginetic.com",
"city" : "Fairview",
"state" : "NJ"
},
"sort" : [
5
]
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "6",
"_score" : null,
"_source" : {
"account_number" : 6,
"balance" : 5686,
"firstname" : "Hattie",
"lastname" : "Bond",
"age" : 36,
"gender" : "M",
"address" : "671 Bristol Street",
"employer" : "Netagy",
"email" : "hattiebond@netagy.com",
"city" : "Dante",
"state" : "TN"
},
"sort" : [
6
]
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "7",
"_score" : null,
"_source" : {
"account_number" : 7,
"balance" : 39121,
"firstname" : "Levy",
"lastname" : "Richard",
"age" : 22,
"gender" : "M",
"address" : "820 Logan Street",
"employer" : "Teraprene",
"email" : "levyrichard@teraprene.com",
"city" : "Shrewsbury",
"state" : "MO"
},
"sort" : [
7
]
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "8",
"_score" : null,
"_source" : {
"account_number" : 8,
"balance" : 48868,
"firstname" : "Jan",
"lastname" : "Burns",
"age" : 35,
"gender" : "M",
"address" : "699 Visitation Place",
"employer" : "Glasstep",
"email" : "janburns@glasstep.com",
"city" : "Wakulla",
"state" : "AZ"
},
"sort" : [
8
]
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "9",
"_score" : null,
"_source" : {
"account_number" : 9,
"balance" : 24776,
"firstname" : "Opal",
"lastname" : "Meadows",
"age" : 39,
"gender" : "M",
"address" : "963 Neptune Avenue",
"employer" : "Cedward",
"email" : "opalmeadows@cedward.com",
"city" : "Olney",
"state" : "OH"
},
"sort" : [
9
]
}
]
}
}
(1) There are only 9 pieces of data, because there are paging queries;
(2) For detailed field information, refer to: https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-search.html
The response also provides the following information about the search request:
- took – how long it took Elasticsearch to run the query, in milliseconds
- timed_out – whether or not the search request timed out
- _shards – how many shards were searched and a breakdown of how many shards succeeded, failed, or were skipped.
- max_score – the score of the most relevant document found
- hits.total.value - how many matching documents were found
- hits.sort - the document's sort position (when not sorting by relevance score)
- hits._score - the document's relevance score (not applicable when using match_all)
2)Query DSL
(1) Basic syntax format
Elasticsearch provides a Json style DSL that can execute queries. This is called Query DSL, and the query language is very comprehensive.
Typical structure of a query statement
QUERY_NAME:{
ARGUMENT:VALUE,
ARGUMENT:VALUE,...
}
For a field, its structure is as follows:
{
QUERY_NAME:{
FIELD_NAME:{
ARGUMENT:VALUE,
ARGUMENT:VALUE,...
}
}
}
GET bank/_search
{
"query": {
"match_all": {}
},
"from": 0,
"size": 5,
"sort": [
{
"account_number": {
"order": "desc"
}
}
]
}
//match_al queries all data and takes 5 data from the 0 data
Query defines how to query;
- match_all query type: represents all query types. In es, you can combine many query types in query to complete complex queries;
- In addition to the query parameter, we can also pass other parameters to change the query result, such as sort, size;
- from+size limit to complete paging function;
- Sort sort, multi field sort, will sort the subsequent fields when the previous fields are equal, otherwise the previous order will prevail;
(2) Return partial fields
GET bank/_search
{
"query": {
"match_all": {}
},
"from": 0,
"size": 5,
"sort": [
{
"account_number": {
"order": "desc"
}
}
],
"_source": ["balance","firstname"]
}
Query results:
{
"took" : 18,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1000,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "bank",
"_type" : "account",
"_id" : "999",
"_score" : null,
"_source" : {
"firstname" : "Dorothy",
"balance" : 6087
},
"sort" : [
999
]
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "998",
"_score" : null,
"_source" : {
"firstname" : "Letha",
"balance" : 16869
},
"sort" : [
998
]
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "997",
"_score" : null,
"_source" : {
"firstname" : "Combs",
"balance" : 25311
},
"sort" : [
997
]
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "996",
"_score" : null,
"_source" : {
"firstname" : "Andrews",
"balance" : 17541
},
"sort" : [
996
]
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "995",
"_score" : null,
"_source" : {
"firstname" : "Phelps",
"balance" : 21153
},
"sort" : [
995
]
}
]
}
}
(3) match query
- Basic type (non string), "account_number": 20 can be added or not. "" not added is an exact match
GET bank/_search
{
"query": {
"match": {
"account_number": "20"
}
}
}
match returns account_number=20.
Query results:
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "bank",
"_type" : "account",
"_id" : "20",
"_score" : 1.0,
"_source" : {
"account_number" : 20,
"balance" : 16418,
"firstname" : "Elinor",
"lastname" : "Ratliff",
"age" : 36,
"gender" : "M",
"address" : "282 Kings Place",
"employer" : "Scentric",
"email" : "elinorratliff@scentric.com",
"city" : "Ribera",
"state" : "WA"
}
}
]
}
}
- String, full text retrieval '' fuzzy query
GET bank/_search
{
"query": {
"match": {
"address": "kings"
}
}
}
Full text retrieval will be sorted according to the score, and the search conditions will be matched by word segmentation.
Query results:
{
"took" : 30,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : 5.990829,
"hits" : [
{
"_index" : "bank",
"_type" : "account",
"_id" : "20",
"_score" : 5.990829,
"_source" : {
"account_number" : 20,
"balance" : 16418,
"firstname" : "Elinor",
"lastname" : "Ratliff",
"age" : 36,
"gender" : "M",
"address" : "282 Kings Place",
"employer" : "Scentric",
"email" : "elinorratliff@scentric.com",
"city" : "Ribera",
"state" : "WA"
}
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "722",
"_score" : 5.990829,
"_source" : {
"account_number" : 722,
"balance" : 27256,
"firstname" : "Roberts",
"lastname" : "Beasley",
"age" : 34,
"gender" : "F",
"address" : "305 Kings Hwy",
"employer" : "Quintity",
"email" : "robertsbeasley@quintity.com",
"city" : "Hayden",
"state" : "PA"
}
}
]
}
}
(4) match_phrase [short sentence matching]
The value to be matched is retrieved as an entire word (no word segmentation)
GET bank/_search
{
"query": {
"match_phrase": {
"address": "mill road"
}
}
}
The investigation address contains mill_ All records of the road and give the correlation score
View results:
{
"took" : 32,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 8.926605,
"hits" : [
{
"_index" : "bank",
"_type" : "account",
"_id" : "970",
"_score" : 8.926605,
"_source" : {
"account_number" : 970,
"balance" : 19648,
"firstname" : "Forbes",
"lastname" : "Wallace",
"age" : 28,
"gender" : "M",
"address" : "990 Mill Road",
"employer" : "Pheast",
"email" : "forbeswallace@pheast.com",
"city" : "Lopezo",
"state" : "AK"
}
}
]
}
}
match_ For the difference between phase and match, observe the following examples:
match_phrase is phrase matching
match is word segmentation matching. For example, 990 Mill matching contains 990 or Mill results
GET bank/_search
{
"query": {
"match_phrase": {
"address": "990 Mill"
}
}
}
Query results:
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 10.806405,
"hits" : [
{
"_index" : "bank",
"_type" : "account",
"_id" : "970",
"_score" : 10.806405,
"_source" : {
"account_number" : 970,
"balance" : 19648,
"firstname" : "Forbes",
"lastname" : "Wallace",
"age" : 28,
"gender" : "M",
"address" : "990 Mill Road",
"employer" : "Pheast",
"email" : "forbeswallace@pheast.com",
"city" : "Lopezo",
"state" : "AK"
}
}
]
}
}
Use the keyword of match
GET bank/_search
{
"query": {
"match": {
"address.keyword": "990 Mill"
}
}
}
None of the query results match
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
}
}
Modify the matching criteria to "990 Mill Road"
GET bank/_search
{
"query": {
"match": {
"address.keyword": "990 Mill Road"
}
}
}
Find a piece of data
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 6.5032897,
"hits" : [
{
"_index" : "bank",
"_type" : "account",
"_id" : "970",
"_score" : 6.5032897,
"_source" : {
"account_number" : 970,
"balance" : 19648,
"firstname" : "Forbes",
"lastname" : "Wallace",
"age" : 28,
"gender" : "M",
"address" : "990 Mill Road",
"employer" : "Pheast",
"email" : "forbeswallace@pheast.com",
"city" : "Lopezo",
"state" : "AK"
}
}
]
}
}
keyword is used for text field matching. The matching condition is to display all the values of the field and to make accurate matching.
match_phrase is phrase matching. As long as the text contains matching conditions and includes this phrase, it can be matched.
(5)multi_math [multi field matching]
GET bank/_search
{
"query": {
"multi_match": {
"query": "mill",
"fields": [
"state",
"address"
]
}
}
}
state or address contains mill, and the query criteria will be segmented during the query process.
Query results:
{
"took" : 28,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 4,
"relation" : "eq"
},
"max_score" : 5.4032025,
"hits" : [
{
"_index" : "bank",
"_type" : "account",
"_id" : "970",
"_score" : 5.4032025,
"_source" : {
"account_number" : 970,
"balance" : 19648,
"firstname" : "Forbes",
"lastname" : "Wallace",
"age" : 28,
"gender" : "M",
"address" : "990 Mill Road",
"employer" : "Pheast",
"email" : "forbeswallace@pheast.com",
"city" : "Lopezo",
"state" : "AK"
}
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "136",
"_score" : 5.4032025,
"_source" : {
"account_number" : 136,
"balance" : 45801,
"firstname" : "Winnie",
"lastname" : "Holland",
"age" : 38,
"gender" : "M",
"address" : "198 Mill Lane",
"employer" : "Neteria",
"email" : "winnieholland@neteria.com",
"city" : "Urie",
"state" : "IL"
}
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "345",
"_score" : 5.4032025,
"_source" : {
"account_number" : 345,
"balance" : 9812,
"firstname" : "Parker",
"lastname" : "Hines",
"age" : 38,
"gender" : "M",
"address" : "715 Mill Avenue",
"employer" : "Baluba",
"email" : "parkerhines@baluba.com",
"city" : "Blackgum",
"state" : "KY"
}
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "472",
"_score" : 5.4032025,
"_source" : {
"account_number" : 472,
"balance" : 25571,
"firstname" : "Lee",
"lastname" : "Long",
"age" : 32,
"gender" : "F",
"address" : "288 Mill Street",
"employer" : "Comverges",
"email" : "leelong@comverges.com",
"city" : "Movico",
"state" : "MT"
}
}
]
}
}
(6) bool is used for compound queries
Compound statements can be combined with any other query statements, including matching statements. This means that between compound statements
It can be nested with each other and can express very complex logic.
Must: all conditions listed in must must be met
GET bank/_search
{
"query":{
"bool":{
"must":[
{"match":{"address":"mill"}},
{"match":{"gender":"M"}}
]
}
}
}
must_not, must not match must_ All conditions listed in not.
Should, the conditions listed in should should should be met.
Example: query the data of gender=m and address=mill
GET bank/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"gender": "M"
}
},
{
"match": {
"address": "mill"
}
}
]
}
}
}
Query results:
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : 6.0824604,
"hits" : [
{
"_index" : "bank",
"_type" : "account",
"_id" : "970",
"_score" : 6.0824604,
"_source" : {
"account_number" : 970,
"balance" : 19648,
"firstname" : "Forbes",
"lastname" : "Wallace",
"age" : 28,
"gender" : "M",
"address" : "990 Mill Road",
"employer" : "Pheast",
"email" : "forbeswallace@pheast.com",
"city" : "Lopezo",
"state" : "AK"
}
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "136",
"_score" : 6.0824604,
"_source" : {
"account_number" : 136,
"balance" : 45801,
"firstname" : "Winnie",
"lastname" : "Holland",
"age" : 38,
"gender" : "M",
"address" : "198 Mill Lane",
"employer" : "Neteria",
"email" : "winnieholland@neteria.com",
"city" : "Urie",
"state" : "IL"
}
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "345",
"_score" : 6.0824604,
"_source" : {
"account_number" : 345,
"balance" : 9812,
"firstname" : "Parker",
"lastname" : "Hines",
"age" : 38,
"gender" : "M",
"address" : "715 Mill Avenue",
"employer" : "Baluba",
"email" : "parkerhines@baluba.com",
"city" : "Blackgum",
"state" : "KY"
}
}
]
}
}
must_not: must not be specified
Example: query the data with gender=m and address=mill, but age is not equal to 38
GET bank/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"gender": "M"
}
},
{
"match": {
"address": "mill"
}
}
],
"must_not": [
{
"match": {
"age": "38"
}
}
]
}
}
Query results:
{
"took" : 4,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 6.0824604,
"hits" : [
{
"_index" : "bank",
"_type" : "account",
"_id" : "970",
"_score" : 6.0824604,
"_source" : {
"account_number" : 970,
"balance" : 19648,
"firstname" : "Forbes",
"lastname" : "Wallace",
"age" : 28,
"gender" : "M",
"address" : "990 Mill Road",
"employer" : "Pheast",
"email" : "forbeswallace@pheast.com",
"city" : "Lopezo",
"state" : "AK"
}
}
]
}
}
Should: the conditions listed in should should should be met. If reached, the score of relevant documents will be increased and the query result will not be changed. If there is only should and only one matching rule in the query, the condition of should will be used as the default matching condition. Zone 2 will change the query result.
Example: match the data that lastName should be equal to Wallace
GET bank/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"gender": "M"
}
},
{
"match": {
"address": "mill"
}
}
],
"must_not": [
{
"match": {
"age": "18"
}
}
],
"should": [
{
"match": {
"lastname": "Wallace"
}
}
]
}
}
}
Query results:
{
"took" : 5,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : 12.585751,
"hits" : [
{
"_index" : "bank",
"_type" : "account",
"_id" : "970",
"_score" : 12.585751,
"_source" : {
"account_number" : 970,
"balance" : 19648,
"firstname" : "Forbes",
"lastname" : "Wallace",
"age" : 28,
"gender" : "M",
"address" : "990 Mill Road",
"employer" : "Pheast",
"email" : "forbeswallace@pheast.com",
"city" : "Lopezo",
"state" : "AK"
}
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "136",
"_score" : 6.0824604,
"_source" : {
"account_number" : 136,
"balance" : 45801,
"firstname" : "Winnie",
"lastname" : "Holland",
"age" : 38,
"gender" : "M",
"address" : "198 Mill Lane",
"employer" : "Neteria",
"email" : "winnieholland@neteria.com",
"city" : "Urie",
"state" : "IL"
}
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "345",
"_score" : 6.0824604,
"_source" : {
"account_number" : 345,
"balance" : 9812,
"firstname" : "Parker",
"lastname" : "Hines",
"age" : 38,
"gender" : "M",
"address" : "715 Mill Avenue",
"employer" : "Baluba",
"email" : "parkerhines@baluba.com",
"city" : "Blackgum",
"state" : "KY"
}
}
]
}
}
It can be seen that the higher the correlation, the higher the score.
(7) Filter [result filtering]
Not all queries need to generate scores, especially those documents that are only used for filtering filtering. In order not to calculate the score, elastic search automatically checks the scenario and optimizes the execution of the query.
GET bank/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"address": "mill"
}
}
],
"filter": {
"range": {
"balance": {
"gte": "10000",
"lte": "20000"
}
}
}
}
}
}
First, query all documents matching address=mill, and then filter the query results according to 10000 < = balance < = 20000
Query results:
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 5.4032025,
"hits" : [
{
"_index" : "bank",
"_type" : "account",
"_id" : "970",
"_score" : 5.4032025,
"_source" : {
"account_number" : 970,
"balance" : 19648,
"firstname" : "Forbes",
"lastname" : "Wallace",
"age" : 28,
"gender" : "M",
"address" : "990 Mill Road",
"employer" : "Pheast",
"email" : "forbeswallace@pheast.com",
"city" : "Lopezo",
"state" : "AK"
}
}
]
}
}
Each must, should, and must_not element in a Boolean query is referred to as a query clause. How well a document meets the criteria in each must or should clause contributes to the document's relevance score. The higher the score, the better the document matches your search criteria. By default, Elasticsearch returns documents ranked by these relevance scores.
In the boolean query, must, should and must_not elements are called query clauses. Whether the document meets the criteria in each "must" or "should" clause determines the "relevance score" of the document. The higher the score, the better the document matches your search criteria. By default, Elasticsearch returns documents sorted by these correlation scores.
The criteria in a must_not clause is treated as a filter. It affects whether or not the document is included in the results, but does not contribute to how documents are scored. You can also explicitly specify arbitrary filters to include or exclude documents based on structured data.
Conditions in the 'must_not' clause are treated as' filters'. It affects whether the document is included in the results, but it does not affect the scoring method of the document. You can also explicitly specify any filter to include or exclude documents based on structured data.
During the use of the filter, the correlation score will not be calculated_ score:
GET bank/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"address": "mill"
}
}
],
"filter": {
"range": {
"balance": {
"gte": "10000",
"lte": "20000"
}
}
}
}
}
}
//gte:>= lte:<=
Query results:
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 213,
"relation" : "eq"
},
"max_score" : 0.0,
"hits" : [
{
"_index" : "bank",
"_type" : "account",
"_id" : "20",
"_score" : 0.0,
"_source" : {
"account_number" : 20,
"balance" : 16418,
"firstname" : "Elinor",
"lastname" : "Ratliff",
"age" : 36,
"gender" : "M",
"address" : "282 Kings Place",
"employer" : "Scentric",
"email" : "elinorratliff@scentric.com",
"city" : "Ribera",
"state" : "WA"
}
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "37",
"_score" : 0.0,
"_source" : {
"account_number" : 37,
"balance" : 18612,
"firstname" : "Mcgee",
"lastname" : "Mooney",
"age" : 39,
"gender" : "M",
"address" : "826 Fillmore Place",
"employer" : "Reversus",
"email" : "mcgeemooney@reversus.com",
"city" : "Tooleville",
"state" : "OK"
}
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "51",
"_score" : 0.0,
"_source" : {
"account_number" : 51,
"balance" : 14097,
"firstname" : "Burton",
"lastname" : "Meyers",
"age" : 31,
"gender" : "F",
"address" : "334 River Street",
"employer" : "Bezal",
"email" : "burtonmeyers@bezal.com",
"city" : "Jacksonburg",
"state" : "MO"
}
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "56",
"_score" : 0.0,
"_source" : {
"account_number" : 56,
"balance" : 14992,
"firstname" : "Josie",
"lastname" : "Nelson",
"age" : 32,
"gender" : "M",
"address" : "857 Tabor Court",
"employer" : "Emtrac",
"email" : "josienelson@emtrac.com",
"city" : "Sunnyside",
"state" : "UT"
}
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "121",
"_score" : 0.0,
"_source" : {
"account_number" : 121,
"balance" : 19594,
"firstname" : "Acevedo",
"lastname" : "Dorsey",
"age" : 32,
"gender" : "M",
"address" : "479 Nova Court",
"employer" : "Netropic",
"email" : "acevedodorsey@netropic.com",
"city" : "Islandia",
"state" : "CT"
}
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "176",
"_score" : 0.0,
"_source" : {
"account_number" : 176,
"balance" : 18607,
"firstname" : "Kemp",
"lastname" : "Walters",
"age" : 28,
"gender" : "F",
"address" : "906 Howard Avenue",
"employer" : "Eyewax",
"email" : "kempwalters@eyewax.com",
"city" : "Why",
"state" : "KY"
}
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "183",
"_score" : 0.0,
"_source" : {
"account_number" : 183,
"balance" : 14223,
"firstname" : "Hudson",
"lastname" : "English",
"age" : 26,
"gender" : "F",
"address" : "823 Herkimer Place",
"employer" : "Xinware",
"email" : "hudsonenglish@xinware.com",
"city" : "Robbins",
"state" : "ND"
}
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "222",
"_score" : 0.0,
"_source" : {
"account_number" : 222,
"balance" : 14764,
"firstname" : "Rachelle",
"lastname" : "Rice",
"age" : 36,
"gender" : "M",
"address" : "333 Narrows Avenue",
"employer" : "Enaut",
"email" : "rachellerice@enaut.com",
"city" : "Wright",
"state" : "AZ"
}
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "227",
"_score" : 0.0,
"_source" : {
"account_number" : 227,
"balance" : 19780,
"firstname" : "Coleman",
"lastname" : "Berg",
"age" : 22,
"gender" : "M",
"address" : "776 Little Street",
"employer" : "Exoteric",
"email" : "colemanberg@exoteric.com",
"city" : "Eagleville",
"state" : "WV"
}
},
{
"_index" : "bank",
"_type" : "account",
"_id" : "272",
"_score" : 0.0,
"_source" : {
"account_number" : 272,
"balance" : 19253,
"firstname" : "Lilly",
"lastname" : "Morgan",
"age" : 25,
"gender" : "F",
"address" : "689 Fleet Street",
"employer" : "Biolive",
"email" : "lillymorgan@biolive.com",
"city" : "Sunbury",
"state" : "OH"
}
}
]
}
}
You can see the "_score" of all documents: 0.0.
(8)term
Same as match. Matches the value of a property. Match is used for full-text search fields and term is used for other non text fields.
Avoid using the term query for text fields.
Avoid using "term" queries on text fields
By default, Elasticsearch changes the values of text fields as part of analysis. This can make finding exact matches for text field values difficult.
By default, Elasticsearch is used as analysis Change the value of the 'text' field as part of the. This makes it difficult to find an exact match for the "text" field value.
To search text field values, use the match.
To search for the "text" field value, use match.
https://www.elastic.co/guide/en/elasticsearch/reference/7.6/query-dsl-term-query.html
Use term to match queries
GET bank/_search
{
"query": {
"term": {
"age": "28"
}
}
}
If it is text, it cannot be found:
GET bank/_search
{
"query": {
"term": {
"gender" : "F"
}
}
}
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
}
}
None of them matched
When changing to match matching, 32 documents can be matched
[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-MtW9sEky-1624414458582)(images/image-20200502120921830.png)]
That is, match is used for full-text search fields and term is used for other non text fields.
(9) Aggregation
Aggregation provides the ability to group and extract data from data. The simplest aggregation method is roughly equal to SQL Group by and SQL aggregation function. In elastic search, the ability to perform a search to return this (hit result) and aggregate results at the same time, separating all hits in the response. This is very powerful and effective. You can execute queries and multiple aggregations, get their own (any) return results in one use, and use a concise and simplified API to avoid network roundtrip.
"size":0
size:0 do not display search data
aggs: perform aggregation. The aggregation syntax is as follows:
"aggs":{
"aggs_name The name of this aggregation can be easily displayed in the result set":{
"AGG_TYPE Type of aggregation(avg,term,terms)":{}
}
}，
Search for the age distribution and average age of all people with mill in address, but do not display the details of these people
GET bank/_search
{
"query": {
"match": {
"address": "Mill"
}
},
"aggs": {
"ageAgg": {
"terms": {
"field": "age",
"size": 10
}
},
"ageAvg": {
"avg": {
"field": "age"
}
},
"balanceAvg": {
"avg": {
"field": "balance"
}
}
},
"size": 0
}
//ageAgg: aggregation name terms: aggregation type "field": "age": aggregate by age field size:10: get the top ten ages
//avg: average aggregate type
//Don't show the details of these people, just look at the aggregation results
Query results:
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 4,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"ageAgg" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : 38,
"doc_count" : 2
},
{
"key" : 28,
"doc_count" : 1
},
{
"key" : 32,
"doc_count" : 1
}
]
},
"ageAvg" : {
"value" : 34.0
},
"balanceAvg" : {
"value" : 25208.0
}
}
}
Complex:
Aggregate by age and find the average salary of these people in these age groups
GET bank/_search
{
"query": {
"match_all": {}
},
"aggs": {
"ageAgg": {
"terms": {
"field": "age",
"size": 100
},
"aggs": {
"ageAvg": {
"avg": {
"field": "balance"
}
}
}
}
},
"size": 0
}
Output results:
{
"took" : 49,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1000,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"ageAgg" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : 31,
"doc_count" : 61,
"ageAvg" : {
"value" : 28312.918032786885
}
},
{
"key" : 39,
"doc_count" : 60,
"ageAvg" : {
"value" : 25269.583333333332
}
},
{
"key" : 26,
"doc_count" : 59,
"ageAvg" : {
"value" : 23194.813559322032
}
},
{
"key" : 32,
"doc_count" : 52,
"ageAvg" : {
"value" : 23951.346153846152
}
},
{
"key" : 35,
"doc_count" : 52,
"ageAvg" : {
"value" : 22136.69230769231
}
},
{
"key" : 36,
"doc_count" : 52,
"ageAvg" : {
"value" : 22174.71153846154
}
},
{
"key" : 22,
"doc_count" : 51,
"ageAvg" : {
"value" : 24731.07843137255
}
},
{
"key" : 28,
"doc_count" : 51,
"ageAvg" : {
"value" : 28273.882352941175
}
},
{
"key" : 33,
"doc_count" : 50,
"ageAvg" : {
"value" : 25093.94
}
},
{
"key" : 34,
"doc_count" : 49,
"ageAvg" : {
"value" : 26809.95918367347
}
},
{
"key" : 30,
"doc_count" : 47,
"ageAvg" : {
"value" : 22841.106382978724
}
},
{
"key" : 21,
"doc_count" : 46,
"ageAvg" : {
"value" : 26981.434782608696
}
},
{
"key" : 40,
"doc_count" : 45,
"ageAvg" : {
"value" : 27183.17777777778
}
},
{
"key" : 20,
"doc_count" : 44,
"ageAvg" : {
"value" : 27741.227272727272
}
},
{
"key" : 23,
"doc_count" : 42,
"ageAvg" : {
"value" : 27314.214285714286
}
},
{
"key" : 24,
"doc_count" : 42,
"ageAvg" : {
"value" : 28519.04761904762
}
},
{
"key" : 25,
"doc_count" : 42,
"ageAvg" : {
"value" : 27445.214285714286
}
},
{
"key" : 37,
"doc_count" : 42,
"ageAvg" : {
"value" : 27022.261904761905
}
},
{
"key" : 27,
"doc_count" : 39,
"ageAvg" : {
"value" : 21471.871794871793
}
},
{
"key" : 38,
"doc_count" : 39,
"ageAvg" : {
"value" : 26187.17948717949
}
},
{
"key" : 29,
"doc_count" : 35,
"ageAvg" : {
"value" : 29483.14285714286
}
}
]
}
}
}
Find out all age distributions, and the average salary of M and F in these age groups, as well as the overall average salary of this age group
GET bank/_search
{
"query": {
"match_all": {}
},
"aggs": {
"ageAgg": {
"terms": {
"field": "age",
"size": 100
},
"aggs": {
"genderAgg": {
"terms": {
"field": "gender.keyword"
},
"aggs": {
"balanceAvg": {
"avg": {
"field": "balance"
}
}
}
},
"ageBalanceAvg": {
"avg": {
"field": "balance"
}
}
}
}
},
"size": 0
}
//"field": "gender.keyword" gender is txt, which cannot be aggregated and must be added keyword exact substitution
Output results:
{
"took" : 119,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1000,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"ageAgg" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : 31,
"doc_count" : 61,
"genderAgg" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "M",
"doc_count" : 35,
"balanceAvg" : {
"value" : 29565.628571428573
}
},
{
"key" : "F",
"doc_count" : 26,
"balanceAvg" : {
"value" : 26626.576923076922
}
}
]
},
"ageBalanceAvg" : {
"value" : 28312.918032786885
}
}
]
.......//Omit others
}
}
}
3)Mapping
(1) Field type
[the external chain image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-kq6fvy0p-16244144584) (images / image-20200502161339291. PNG)]
(2) Mapping
Mapping
Mapping is used to define a document and how its contained field s are stored and indexed. For example, use Maping to define:
-
Which string attributes should be regarded as full text fields;
-
Which attributes contain numbers, dates, or geographic locations;
-
Whether all attributes in the document are indexed (all configured);
-
Format of date;
-
Custom mapping rules to dynamically add attributes;
-
Viewing mapping information
GET bank/_mapping{ "bank" : { "mappings" : { "properties" : { "account_number" : { "type" : "long" }, "address" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "age" : { "type" : "long" }, "balance" : { "type" : "long" }, "city" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "email" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "employer" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "firstname" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "gender" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "lastname" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "state" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } } } } -
Modify mapping information
[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-Whtc99bA-1624414458586)(images/image-20200502170924399.png)]
(3) New version changes
ElasticSearch7 - remove the type concept
-
Two data representations in relational databases are independent. Even if they have columns with the same name, it will not affect their use, but this is not the case in ES. Elastic search is a search engine developed based on Lucene, and the final processing method of files with the same name under different type s in ES is the same in Lucene.
- Two users under two different types_ Name, under the same index of ES, is actually considered to be the same filed. You must define the same filed mapping in two different types. Otherwise, the same field names in different types will conflict in processing, resulting in the decrease of Lucene processing efficiency.
- Removing type is to improve the efficiency of ES data processing.
-
Elasticsearch 7. The type parameter in the X URL is optional. For example, indexing a document no longer requires a document type.
-
Elasticsearch 8.x no longer supports the type parameter in URL s.
-
solve:
Migrate the index from multi type to single type, and each type of document has an independent indexMigrate all the type data under the existing index to the specified location. See data migration for details
Elasticsearch 7.x
- Specifying types in requests is deprecated. For instance, indexing a document no longer requires a document type. The new index APIs are PUT {index}/_doc/{id} in case of explicit ids and POST {index}/_doc for auto-generated ids. Note that in 7.0, _doc is a permanent part of the path, and represents the endpoint name rather than the document type.
- The include_type_name parameter in the index creation, index template, and mapping APIs will default to false. Setting the parameter at all will result in a deprecation warning.
- The _default_ mapping type is removed.
Elasticsearch 8.x
- Specifying types in requests is no longer supported.
- The include_type_name parameter is removed.
Create mapping
Create an index and specify mapping rules for attributes (equivalent to creating a new table and specifying fields and field types)
PUT /my_index
{
"mappings": {
"properties": {
"age": {
"type": "integer"
},
"email": {
"type": "keyword"
},
"name": {
"type": "text"
}
}
}
}
Output:
{
"acknowledged" : true,
"shards_acknowledged" : true,
"index" : "my_index"
}
view map
GET /my_index
Output results:
//"Index": false whether it can be retrieved by indexing. The default is true
{
"my_index" : {
"aliases" : { },
"mappings" : {
"properties" : {
"age" : {
"type" : "integer"
},
"email" : {
"type" : "keyword"
},
"employee-id" : {
"type" : "keyword",
"index" : false
},
"name" : {
"type" : "text"
}
}
},
"settings" : {
"index" : {
"creation_date" : "1588410780774",
"number_of_shards" : "1",
"number_of_replicas" : "1",
"uuid" : "ua0lXhtkQCOmn7Kh3iUu0w",
"version" : {
"created" : "7060299"
},
"provided_name" : "my_index"
}
}
}
}
Add new field mapping
PUT /my_index/_mapping
{
"properties": {
"employee-id": {
"type": "keyword",
"index": false
}
}
}
"index" here: false, indicating that the newly added field cannot be retrieved, but is only a redundant field.
Update mapping
We cannot update an existing field mapping. Update must create a new index for data migration.
data migration
Create new first_ The correct mapping of twitter. Then use the following methods for data migration.
POST reindex [Fixed writing]
{
"source":{
"index":"twitter"
},
"dest":{
"index":"new_twitters"
}
}
Migrate the data under the type of the old index
POST reindex [Fixed writing]
{
"source":{
"index":"twitter",
"twitter":"twitter"
},
"dest":{
"index":"new_twitters"
}
}
For more details, see: https://www.elastic.co/guide/en/elasticsearch/reference/7.6/docs-reindex.html
GET /bank/_search
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1000,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "bank",
"_type" : "account",//The type is account
"_id" : "1",
"_score" : 1.0,
"_source" : {
"account_number" : 1,
"balance" : 39225,
"firstname" : "Amber",
"lastname" : "Duke",
"age" : 32,
"gender" : "M",
"address" : "880 Holmes Lane",
"employer" : "Pyrami",
"email" : "amberduke@pyrami.com",
"city" : "Brogan",
"state" : "IL"
}
},
...
GET /bank/_search
[the external chain image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-IoMmyY6p-1624414458589)(images/image-20200502174825233.png)]
Want to change the age to integer
PUT /newbank
{
"mappings": {
"properties": {
"account_number": {
"type": "long"
},
"address": {
"type": "text"
},
"age": {
"type": "integer"
},
"balance": {
"type": "long"
},
"city": {
"type": "keyword"
},
"email": {
"type": "keyword"
},
"employer": {
"type": "keyword"
},
"firstname": {
"type": "text"
},
"gender": {
"type": "keyword"
},
"lastname": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"state": {
"type": "keyword"
}
}
}
}
View the mapping of "newbank":
GET /newbank/_mapping
[the external chain image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-GVKjkJHB-1624414458591)(images/image-20200502175901959.png)]
You can see that the mapping type of age is changed to integer
Migrate data from bank to newbank
POST _reindex
{
"source": {
"index": "bank",
"type": "account"
},
"dest": {
"index": "newbank"
}
}
Run output:
#! Deprecation: [types removal] Specifying types in reindex requests is deprecated.
{
"took" : 768,
"timed_out" : false,
"total" : 1000,
"updated" : 0,
"created" : 1000,
"deleted" : 0,
"batches" : 1,
"version_conflicts" : 0,
"noops" : 0,
"retries" : {
"bulk" : 0,
"search" : 0
},
"throttled_millis" : 0,
"requests_per_second" : -1.0,
"throttled_until_millis" : 0,
"failures" : [ ]
}
View data in newbank
[the external chain image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-eU4ra90c-1624414458592)(images/image-20200502181432745.png)]
4) Participle
A tokenizer receives a character stream, divides it into independent tokens (words, usually independent words), and then outputs the tokens stream.
For example: whitespace tokenizer splits text when it encounters white space characters. It will put the text "Quick brown fox!" Split into [Quick,brown,fox!].
The tokenizer is also responsible for recording the order or position position position of each term (for phrase and word proximity query), and the character offsets of the start and end of the original word (word) represented by the term (for highlighting the content of the search).
elasticsearch provides many built-in word splitters that can be used to build custom analyzers.
About word splitter: https://www.elastic.co/guide/en/elasticsearch/reference/7.6/analysis.html
POST _analyze
{
"analyzer": "standard",
"text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}
Execution results:
{
"tokens" : [
{
"token" : "the",
"start_offset" : 0,
"end_offset" : 3,
"type" : "<ALPHANUM>",
"position" : 0
},
{
"token" : "2",
"start_offset" : 4,
"end_offset" : 5,
"type" : "<NUM>",
"position" : 1
},
{
"token" : "quick",
"start_offset" : 6,
"end_offset" : 11,
"type" : "<ALPHANUM>",
"position" : 2
},
{
"token" : "brown",
"start_offset" : 12,
"end_offset" : 17,
"type" : "<ALPHANUM>",
"position" : 3
},
{
"token" : "foxes",
"start_offset" : 18,
"end_offset" : 23,
"type" : "<ALPHANUM>",
"position" : 4
},
{
"token" : "jumped",
"start_offset" : 24,
"end_offset" : 30,
"type" : "<ALPHANUM>",
"position" : 5
},
{
"token" : "over",
"start_offset" : 31,
"end_offset" : 35,
"type" : "<ALPHANUM>",
"position" : 6
},
{
"token" : "the",
"start_offset" : 36,
"end_offset" : 39,
"type" : "<ALPHANUM>",
"position" : 7
},
{
"token" : "lazy",
"start_offset" : 40,
"end_offset" : 44,
"type" : "<ALPHANUM>",
"position" : 8
},
{
"token" : "dog's",
"start_offset" : 45,
"end_offset" : 50,
"type" : "<ALPHANUM>",
"position" : 9
},
{
"token" : "bone",
"start_offset" : 51,
"end_offset" : 55,
"type" : "<ALPHANUM>",
"position" : 10
}
]
}
(1) Install ik word splitter
[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-1OJJaqcH-1624414458595)(images/image-20200502182929583.png)]
"Standard Analyzer" is used by default for all language word segmentation, but these word splitters are not friendly for Chinese word segmentation. To do this, you need to install a Chinese word splitter.
Note: you cannot use the default elasticsearch plugin to install XXX Zip for automatic installation
https://github.com/medcl/elasticsearch-analysis-ik/releases/download Corresponding es version installation
During the previous installation of elasticsearch, we have mapped the "/ usr/share/elasticsearch/plugins" directory of the elasticsearch container to the "/ mydata/elasticsearch/plugins" directory of the host machine, so it is more convenient to download the "/ elasticsearch-analysis-ik-7.6.2.zip" file and extract it into this folder. After installation, restart the elasticsearch container.
If you don't mind the trouble, you can also use the following methods.
(1) View elasticsearch version number:
[root@hadoop-104 ~]# curl http://localhost:9200
{
"name" : "0adeb7852e00",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "9gglpP0HTfyOTRAaSe2rIg",
"version" : {
"number" : "7.6.2", #The version number is 7.6.2
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
"build_date" : "2020-03-26T06:34:37.794943Z",
"build_snapshot" : false,
"lucene_version" : "8.4.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
[root@hadoop-104 ~]#
(2) Enter the plugin directory inside the es container
- docker exec -it container id /bin/bash
[root@hadoop-104 ~]# docker exec -it elasticsearch /bin/bash [root@0adeb7852e00 elasticsearch]#
- wget https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.6.2/elasticsearch-analysis-ik-7.6.2.zip
[root@0adeb7852e00 elasticsearch]# pwd /usr/share/elasticsearch #Download ik7 six point two [root@0adeb7852e00 elasticsearch]# wget https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.6.2/elasticsearch-analysis-ik-7.6.2.zip
- unzip downloaded files
[root@0adeb7852e00 elasticsearch]# unzip elasticsearch-analysis-ik-7.6.2.zip -d ink Archive: elasticsearch-analysis-ik-7.6.2.zip creating: ik/config/ inflating: ik/config/main.dic inflating: ik/config/quantifier.dic inflating: ik/config/extra_single_word_full.dic inflating: ik/config/IKAnalyzer.cfg.xml inflating: ik/config/surname.dic inflating: ik/config/suffix.dic inflating: ik/config/stopword.dic inflating: ik/config/extra_main.dic inflating: ik/config/extra_stopword.dic inflating: ik/config/preposition.dic inflating: ik/config/extra_single_word_low_freq.dic inflating: ik/config/extra_single_word.dic inflating: ik/elasticsearch-analysis-ik-7.6.2.jar inflating: ik/httpclient-4.5.2.jar inflating: ik/httpcore-4.4.4.jar inflating: ik/commons-logging-1.2.jar inflating: ik/commons-codec-1.9.jar inflating: ik/plugin-descriptor.properties inflating: ik/plugin-security.policy [root@0adeb7852e00 elasticsearch]# #Move to plugins directory [root@0adeb7852e00 elasticsearch]# mv ik plugins/
- rm -rf *.zip
[root@0adeb7852e00 elasticsearch]# rm -rf elasticsearch-analysis-ik-7.6.2.zip
Confirm that the word splitter is installed
(2) Test word splitter
Use default
GET my_index/_analyze
{
"text":"I am Chinese,"
}
Please observe the execution results:
{
"tokens" : [
{
"token" : "I",
"start_offset" : 0,
"end_offset" : 1,
"type" : "<IDEOGRAPHIC>",
"position" : 0
},
{
"token" : "yes",
"start_offset" : 1,
"end_offset" : 2,
"type" : "<IDEOGRAPHIC>",
"position" : 1
},
{
"token" : "in",
"start_offset" : 2,
"end_offset" : 3,
"type" : "<IDEOGRAPHIC>",
"position" : 2
},
{
"token" : "country",
"start_offset" : 3,
"end_offset" : 4,
"type" : "<IDEOGRAPHIC>",
"position" : 3
},
{
"token" : "people",
"start_offset" : 4,
"end_offset" : 5,
"type" : "<IDEOGRAPHIC>",
"position" : 4
}
]
}
GET my_index/_analyze
{
"analyzer": "ik_smart",
"text":"I am Chinese,"
}
Output results:
{
"tokens" : [
{
"token" : "I",
"start_offset" : 0,
"end_offset" : 1,
"type" : "CN_CHAR",
"position" : 0
},
{
"token" : "yes",
"start_offset" : 1,
"end_offset" : 2,
"type" : "CN_CHAR",
"position" : 1
},
{
"token" : "Chinese",
"start_offset" : 2,
"end_offset" : 5,
"type" : "CN_WORD",
"position" : 2
}
]
}
GET my_index/_analyze
{
"analyzer": "ik_max_word",
"text":"I am Chinese,"
}
Output results:
{
"tokens" : [
{
"token" : "I",
"start_offset" : 0,
"end_offset" : 1,
"type" : "CN_CHAR",
"position" : 0
},
{
"token" : "yes",
"start_offset" : 1,
"end_offset" : 2,
"type" : "CN_CHAR",
"position" : 1
},
{
"token" : "Chinese",
"start_offset" : 2,
"end_offset" : 5,
"type" : "CN_WORD",
"position" : 2
},
{
"token" : "China",
"start_offset" : 2,
"end_offset" : 4,
"type" : "CN_WORD",
"position" : 3
},
{
"token" : "countrymen",
"start_offset" : 3,
"end_offset" : 5,
"type" : "CN_WORD",
"position" : 4
}
]
}
(3) Custom Thesaurus
- Modify ikanalyzer.com in / usr/share/elasticsearch/plugins/ik/config cfg. xml
/usr/share/elasticsearch/plugins/ik/config
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd"> <properties> <comment>IK Analyzer Extended configuration</comment> <!--Users can configure their own extended dictionary here --> <entry key="ext_dict"></entry> <!--Users can configure their own extended stop word dictionary here--> <entry key="ext_stopwords"></entry> <!--Users can configure the remote extension dictionary here --> <entry key="remote_ext_dict">http://192.168.6.128/es/fenci.txt</entry> <!--Users can configure the remote extended stop word dictionary here--> <!-- <entry key="remote_ext_stopwords">words_location</entry> --> </properties>
Original xml
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd"> <properties> <comment>IK Analyzer Extended configuration</comment> <!--Users can configure their own extended dictionary here --> <entry key="ext_dict"></entry> <!--Users can configure their own extended stop word dictionary here--> <entry key="ext_stopwords"></entry> <!--Users can configure the remote extension dictionary here --> <!-- <entry key="remote_ext_dict">words_location</entry> --> <!--Users can configure the remote extended stop word dictionary here--> <!-- <entry key="remote_ext_stopwords">words_location</entry> --> </properties>
After the modification, restart the elasticsearch container, otherwise the modification will not take effect.
After the update is completed, es will only use the update participle for the newly added data. Historical data will not be re segmented. If you want to re segment historical data, you need to perform the following steps:
POST my_index/_update_by_query?conflicts=proceed
http://192.168.6.128/es/fenci.txt , this is the access path of the resource on nginx
Before running the following examples, you need to install nginx (see Installing nginx for the installation method), and then create a "fenci.txt" file, as follows:
echo "Cherry Sakima, bring you sweet summer" > /mydata/nginx/html/fenci.txt
Test effect:
GET my_index/_analyze
{
"analyzer": "ik_max_word",
"text":"Cherry Sakima, bring you sweet summer"
}
Output results:
{
"tokens" : [
{
"token" : "Cherry",
"start_offset" : 0,
"end_offset" : 2,
"type" : "CN_WORD",
"position" : 0
},
{
"token" : "Sakima",
"start_offset" : 2,
"end_offset" : 5,
"type" : "CN_WORD",
"position" : 1
},
{
"token" : "Take you",
"start_offset" : 6,
"end_offset" : 8,
"type" : "CN_WORD",
"position" : 2
},
{
"token" : "Sweet",
"start_offset" : 8,
"end_offset" : 10,
"type" : "CN_WORD",
"position" : 3
},
{
"token" : "summer has set in",
"start_offset" : 10,
"end_offset" : 12,
"type" : "CN_WORD",
"position" : 4
}
]
}
4,elasticsearch-Rest-Client
1)9300: TCP
- spring-data-elasticsearch:transport-api.jar;
- Different versions of springboot, ransport API Jar is different and cannot be adapted to the es version
- 7.x is no longer recommended and will be abandoned after 8
2)9200: HTTP
-
jestClient: unofficial, slow update;
-
RestTemplate: simulates HTTP requests. Many ES operations need to be encapsulated by themselves, which is troublesome;
-
HttpClient: same as above;
-
Elasticsearch rest client: the official RestClient, which encapsulates ES es ES operations. The API is hierarchical and easy to use;
Finally, elastic search rest client (elastic search rest high level client) is selected;
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high.html
5. Appendix: installation of Nginx
-
Start an nginx instance just to copy the configuration
docker run -p 8081:80 --name nginx -d nginx:1.10
-
Copy the configuration file in the container to / mydata/nginx/conf /
mkdir -p /mydata/nginx/html mkdir -p /mydata/nginx/logs mkdir -p /mydata/nginx/conf docker container cp nginx:/etc/nginx . #Copy to current folder #Since there will be an nginx folder in config after copying, its contents need to be moved to conf mv /mydata/nginx/conf/nginx/* /mydata/nginx/conf/ rm -rf /mydata/nginx/conf/nginx
-
Terminate original container:
docker stop nginx
-
Execute the command to delete the original container:
docker rm nginx
-
To create a new Nginx, execute the following command
docker run -p 8081:80 --name nginx \ -v /mydata/nginx/html:/usr/share/nginx/html \ -v /mydata/nginx/logs:/var/log/nginx \ -v /mydata/nginx/conf/:/etc/nginx \ -d nginx:1.10
-
Set startup nginx
docker update nginx --restart=always
-
Create a "/ mydata/nginx/html/index.html" file to test whether it can be accessed normally
echo '<h2>hello nginx!</h2>' >/mydata/nginx/html/index.html
visit: http://ngix IP address of host: 80 / index html
SpringBoot integrates ElasticSearch
1. Import dependency
The version here should match the ELK version.
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-high-level-client</artifactId>
<version>7.6.2</version>
</dependency>
ELK version bit 6.8.7 in spring boot dependencies
<elasticsearch.version>6.8.7</elasticsearch.version>
[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-cih65r7o-1624414458598) (images / image-2020051107437763. PNG)]
It needs to be changed to 7.6.2 in the project
<properties>
...
<elasticsearch.version>7.6.2</elasticsearch.version>
</properties>
2. Writing test classes
1) Test save data
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-document-index.html
@Test
public void indexData() throws IOException {
IndexRequest indexRequest = new IndexRequest ("users");
User user = new User();
user.setUserName("Zhang San");
user.setAge(20);
user.setGender("male");
String jsonString = JSON.toJSONString(user);
//Set what to save
indexRequest.source(jsonString, XContentType.JSON);
//Perform index creation and save data
IndexResponse index = client.index(indexRequest, GulimallElasticSearchConfig.COMMON_OPTIONS);
System.out.println(index);
}
Before test:
[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-jqtpj3tu-1624414458602) (images / image-2020051111618183. PNG)]
After test:
[the external chain image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-rvkwyhhq-1624414458605) (images / image-2020051112025327. PNG)]
2) Test acquisition data
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-search.html
@Test
public void searchData() throws IOException {
GetRequest getRequest = new GetRequest(
"users",
"_-2vAHIB0nzmLJLkxKWk");
GetResponse getResponse = client.get(getRequest, RequestOptions.DEFAULT);
System.out.println(getResponse);
String index = getResponse.getIndex();
System.out.println(index);
String id = getResponse.getId();
System.out.println(id);
if (getResponse.isExists()) {
long version = getResponse.getVersion();
System.out.println(version);
String sourceAsString = getResponse.getSourceAsString();
System.out.println(sourceAsString);
Map<String, Object> sourceAsMap = getResponse.getSourceAsMap();
System.out.println(sourceAsMap);
byte[] sourceAsBytes = getResponse.getSourceAsBytes();
} else {
}
}
Query documents with state="AK":
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 22, //22 matches
"relation": "eq"
},
"max_score": 3.7952394,
"hits": [{
"_index": "bank",
"_type": "account",
"_id": "210",
"_score": 3.7952394,
"_source": {
"account_number": 210,
"balance": 33946,
"firstname": "Cherry",
"lastname": "Carey",
"age": 24,
"gender": "M",
"address": "539 Tiffany Place",
"employer": "Martgo",
"email": "cherrycarey@martgo.com",
"city": "Fairacres",
"state": "AK"
}
},
....//Omit others
]
}
}
Search the age distribution, average age and average salary of all people with mill in address
GET bank/_search
{
"query": {
"match": {
"address": "Mill"
}
},
"aggs": {
"ageAgg": {
"terms": {
"field": "age",
"size": 10
}
},
"ageAvg": {
"avg": {
"field": "age"
}
},
"balanceAvg": {
"avg": {
"field": "balance"
}
}
}
}
java implementation
/**
* Complex search: search the bank for the age distribution, average age and average salary of all people whose address contains mill
* @throws IOException
*/
@Test
public void searchData() throws IOException {
//1. Create a retrieval request
SearchRequest searchRequest = new SearchRequest();
//1.1) specify index
searchRequest.indices("bank");
//1.2) structure search conditions
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
sourceBuilder.query(QueryBuilders.matchQuery("address","Mill"));
//1.2.1) aggregation by age distribution
TermsAggregationBuilder ageAgg=AggregationBuilders.terms("ageAgg").field("age").size(10);
sourceBuilder.aggregation(ageAgg);
//1.2.2) calculate the average age
AvgAggregationBuilder ageAvg = AggregationBuilders.avg("ageAvg").field("age");
sourceBuilder.aggregation(ageAvg);
//1.2.3) calculate average salary
AvgAggregationBuilder balanceAvg = AggregationBuilders.avg("balanceAvg").field("balance");
sourceBuilder.aggregation(balanceAvg);
System.out.println("Search criteria:"+sourceBuilder);
searchRequest.source(sourceBuilder);
//2. Perform retrieval
SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
System.out.println("Search results:"+searchResponse);
//3. Encapsulate the search results as beans
SearchHits hits = searchResponse.getHits();
SearchHit[] searchHits = hits.getHits();
for (SearchHit searchHit : searchHits) {
String sourceAsString = searchHit.getSourceAsString();
Account account = JSON.parseObject(sourceAsString, Account.class);
System.out.println(account);
}
//4. Get aggregate information
Aggregations aggregations = searchResponse.getAggregations();
Terms ageAgg1 = aggregations.get("ageAgg");
for (Terms.Bucket bucket : ageAgg1.getBuckets()) {
String keyAsString = bucket.getKeyAsString();
System.out.println("Age:"+keyAsString+" ==> "+bucket.getDocCount());
}
Avg ageAvg1 = aggregations.get("ageAvg");
System.out.println("average age:"+ageAvg1.getValue());
Avg balanceAvg1 = aggregations.get("balanceAvg");
System.out.println("Average salary:"+balanceAvg1.getValue());
}
You can try to compare the printed conditions and execution results with the previous ElasticSearch search search statements and search results;
other
1. kibana console command
ctrl+home: return to the document header;
ctril+end: return to the end of the document.