1. Introduction to elasticsearch

• Introduction to Elasticsearch
  • A distributed and Restful search engine
  • Support the retrieval of various types of data
  • Fast search speed and can provide real-time search services
  • It is easy to expand and can process PB level massive data per second
• Elasticsearch terms
  • Index, type, document, field
  • Cluster, node, sharding, replica
    

1.1 interpretation of terms

• Index: equivalent to database in MySQL database
• Type: equivalent to table in MySQL database
• Document: a row of data equivalent to a table in MySQL database. The data structure is JSON
• Field: equivalent to a column of table in MySQL database

Since Elasticsearch 6.0, the concept of type has been gradually abolished, and the meaning of index also includes type

• Cluster: distributed deployment to improve performance
• Node: each server in the cluster
• Sharding: further partition and storage of an index to improve the concurrent processing ability
• Replica: backup of shards to improve availability

Elasticsearch connection: Official website

1.2 Elasticsearch configuration

In order to be compatible with the spring boot version used, the version of elastic search is selected as 6.4.3
The file directory structure is:

Modify elasticsearch. In the config directory YML file

2. Configure environment variables

1.3 install Chinese word segmentation plug-in

1. Search on github
   
2. Unzip to the specified directory
   
   Directory structure under ik directory
   
   In the config directory, ikanalyzer cfg. New words can be configured in XML
   
   The contents are as follows:
   

1.4 installing Postman

Postman related connections: Official website
It can be installed step by step.

1.5 using the command line to operate Elasticsearch

1. Start Elasticsearch
   Double click elasticsearch. In the bin directory bat
   
2. Introduction to common commands

• View node health status
  curl -X GET "localhost:9200/_cat/health?v"
  
• View node details
  curl -X GET "localhost:9200/_cat/nodes?v"
  
• View index related information
  curl -X GET "localhost:9200/_cat/indices?v"
  
  The view status shows yellow because there is no fragmentation (backup).
• Delete index
  curl -X DELETE "localhost:9200/test" //test is the index name
  
  Query the index related information again, and there is no test
  
• Create index
  curl -X PUT "localhost:9200/test" //test is the index name
  
  Query the index related information again, and there is test again
  

1.6 accessing Elasticsearch using Postman

• Search index
  
• Delete index
  
  Query the index related information again, and there is no test
  
• New index
  
  Query the index related information again, and there is test again
  
• Submit data

//test: index_ doc: fixed format 1: id number, and then write data in the request body
PUT localhost: 9200/test/_doc/1   

• Check data
  
• Change data
  The same as adding data, the bottom layer will delete and then add
• Delete data
  
• Search function demonstration
  Add several sets of data
  

Search Demo:

When searching, you can also segment words first and then search, which is not necessarily a complete match

2. Spring integrates Elasticsearch

• Introduce dependency
  • spring-boot-starter-data-elasticsearch
• Configure Elasticsearch
  • cluster-name,cluster-nodes
• Spring Data Elasticsearch
  • ElasticsearchTemplate
  • ElasticsearchRepository
    

2.1 introducing dependencies

<!--elasticsearch-->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>

2.2 Elasticsearch related configuration

1. Spring integrates Elasticsearch in application Just configure it in the properties file

# elasticsearch
# ElasticsearchProperties
# In the installation directory config / elasticserch Found in YML
spring.data.elasticsearch.cluster-name=my-application
# 9200 HTTP access port, 9300 TCP port
spring.data.elasticsearch.cluster-nodes=127.0.0.1:9300

1. Resolving Netty conflicts
   Cause of problem: Netty is used at the bottom of Redis, and Netty is also used in Elasticsearch. An error will be reported when it is registered twice
   
   Solution: before registering Netty in Elasticsearch, it will judge whether there is a parameter. If there is, it will not be registered
   
   Specific solutions:
   
2. Annotate the DiscussPost class of the post

package com.ateam.community.entity;


import org.springframework.data.annotation.Id;
import org.springframework.data.elasticsearch.annotations.Document;
import org.springframework.data.elasticsearch.annotations.Field;
import org.springframework.data.elasticsearch.annotations.FieldType;

import java.util.Date;


// indexName: index name, type: fixed to_ doc, shards: sharding, replis: Backup
@Document(indexName = "discusspost", type = "_doc", shards = 6, replicas = 3)
public class DiscussPost {

    @Id
    private int id;
    @Field(type = FieldType.Integer)
    private int userId;
    
	//analyzer: the parser at the time of storage,
	/*
	give an example
	Deposit: Internet school recruitment -- > establish the largest index (i.e. various splits)
	In essence, the keyword is extracted according to the stored content, and then the stored content is associated with the keyword. Later, when we search, we can find the stored content according to the keyword;
	Therefore, when saving, you should split as many keywords as possible to increase the probability and scope of subsequent search
	*/
    //Search analyzer: the parser at the time of storage, which splits as few words as possible but meets your intention
    /*
    ik_max_word,ik_smart It's all the names of the participle
    */
    @Field(type = FieldType.Text, analyzer = "ik_max_word", searchAnalyzer = "ik_smart")
    private String title;
    @Field(type = FieldType.Text, analyzer = "ik_max_word", searchAnalyzer = "ik_smart")
    private String content;

    @Field(type = FieldType.Integer)
    private int discussType;
    @Field(type = FieldType.Integer)
    private int status;
    @Field(type = FieldType.Date)
    private Date createTime;
    @Field(type = FieldType.Integer)
    private int commentCount;
    @Field(type = FieldType.Double)
    private double score;


    public int getId() {
        return id;
    }

    public void setId(int id) {
        this.id = id;
    }

    public int getUserId() {
        return userId;
    }

    public void setUserId(int userId) {
        this.userId = userId;
    }

    public String getTitle() {
        return title;
    }

    public void setTitle(String title) {
        this.title = title;
    }

    public String getContent() {
        return content;
    }

    public void setContent(String content) {
        this.content = content;
    }

    public int getDiscussType() {
        return discussType;
    }

    public void setDiscussType(int discussType) {
        this.discussType = discussType;
    }

    public int getStatus() {
        return status;
    }

    public void setStatus(int status) {
        this.status = status;
    }

    public Date getCreateTime() {
        return createTime;
    }

    public void setCreateTime(Date createTime) {
        this.createTime = createTime;
    }

    public int getCommentCount() {
        return commentCount;
    }

    public void setCommentCount(int commentCount) {
        this.commentCount = commentCount;
    }

    public double getScore() {
        return score;
    }

    public void setScore(double score) {
        this.score = score;
    }

    @Override
    public String toString() {
        return "DiscussPost{" +
                "id=" + id +
                ", userId=" + userId +
                ", title='" + title + '\'' +
                ", content='" + content + '\'' +
                ", discussType=" + discussType +
                ", status=" + status +
                ", createTime=" + createTime +
                ", commentCount=" + commentCount +
                ", score=" + score +
                '}';
    }
}

2.3 data layer

Create a new elasticsearch package under dao and create the DiscussPostRepository interface

/*
ElasticsearchRepository<DiscussPost,Integer>
ElasticsearchRepository<Processed entity class, primary key type in entity class >
ElasticsearchRepository: The interface already has methods such as adding, deleting, modifying and querying access to the es service layer, which we can call directly
*/
@Repository // spring provides
public interface DiscussPostRepository extends ElasticsearchRepository<DiscussPost,Integer> {

}

2.4 testing

Create a new test package in the next test package

package com.ateam.community;


@RunWith(SpringRunner.class)
@SpringBootTest
@ContextConfiguration(classes = CommunityApplication.class)//Configuration class
public class ElasticsearchTests {

    @Resource
    private DiscussPostMapper discussPostMapper;

    @Autowired
    private DiscussPostRepository discussPostRepository;

    @Autowired
    private ElasticsearchTemplate elasticsearchTemplate;

    @Test
    public void testInert(){
        discussPostRepository.save(discussPostMapper.selectDiscussPostById(241));
        discussPostRepository.save(discussPostMapper.selectDiscussPostById(242));
        discussPostRepository.save(discussPostMapper.selectDiscussPostById(243));
    }

    @Test
    public void testInsetList(){
        discussPostRepository.saveAll(discussPostMapper.selectDiscussPosts(101,0,100,0));
        discussPostRepository.saveAll(discussPostMapper.selectDiscussPosts(102,0,100,0));
        discussPostRepository.saveAll(discussPostMapper.selectDiscussPosts(103,0,100,0));
        discussPostRepository.saveAll(discussPostMapper.selectDiscussPosts(111,0,100,0));
        discussPostRepository.saveAll(discussPostMapper.selectDiscussPosts(131,0,100,0));
        discussPostRepository.saveAll(discussPostMapper.selectDiscussPosts(132,0,100,0));
        discussPostRepository.saveAll(discussPostMapper.selectDiscussPosts(133,0,100,0));
        discussPostRepository.saveAll(discussPostMapper.selectDiscussPosts(134,0,100,0));
    }

    @Test
    public void testUpdate(){
        DiscussPost post = discussPostMapper.selectDiscussPostById(231);
        post.setContent("I'm a newcomer, pour water hard!!!!");
        discussPostRepository.save(post);
    }

    @Test
    public void testDelete(){
        discussPostRepository.deleteById(231);
    }

    @Test
    public void testSearchByRepository(){
        SearchQuery searchQuery = new NativeSearchQueryBuilder()
                .withQuery(QueryBuilders.multiMatchQuery("Internet winter","title","content"))
                .withSort(SortBuilders.fieldSort("discussType").order(SortOrder.DESC))
                .withSort(SortBuilders.fieldSort("score").order(SortOrder.DESC))
                .withSort(SortBuilders.fieldSort("createTime").order(SortOrder.DESC))
                .withPageable(PageRequest.of(0,10))
                .withHighlightFields(
                        new HighlightBuilder.Field("title").preTags("<em>").postTags("</em>"),
                        new HighlightBuilder.Field("content").preTags("<em>").postTags("</em>")
                ).build();

        //elasticsearchTemplate.queryForPage(searchQuery,class,SearchResultMapper)
        // The underlying layer gets the highlighted value, but does not return it

        Page<DiscussPost> page = discussPostRepository.search(searchQuery);
        System.out.println(page.getTotalElements()); // Total number
        System.out.println(page.getTotalPages()); // Total pages
        System.out.println(page.getNumber()); // Current page
        System.out.println(page.getSize()); // How many pieces of data per page
        for (DiscussPost post : page) {
            System.out.println(post);
        }
    }

    @Test
    public void testSearchByTemplate(){
        SearchQuery searchQuery = new NativeSearchQueryBuilder()
                .withQuery(QueryBuilders.multiMatchQuery("Internet winter","title","content"))
                .withSort(SortBuilders.fieldSort("discussType").order(SortOrder.DESC))
                .withSort(SortBuilders.fieldSort("score").order(SortOrder.DESC))
                .withSort(SortBuilders.fieldSort("createTime").order(SortOrder.DESC))
                .withPageable(PageRequest.of(0,10))
                .withHighlightFields(
                        new HighlightBuilder.Field("title").preTags("<em>").postTags("</em>"),
                        new HighlightBuilder.Field("content").preTags("<em>").postTags("</em>")
                ).build();

        Page<DiscussPost> page = elasticsearchTemplate.queryForPage(searchQuery, DiscussPost.class, new SearchResultMapper() {
            @Override
            public <T> AggregatedPage<T> mapResults(SearchResponse searchResponse, Class<T> aClass, Pageable pageable) {
                SearchHits hits = searchResponse.getHits();
                if (hits.totalHits <= 0) {
                    return null;
                }

                ArrayList<DiscussPost> list = new ArrayList<>();
                for (SearchHit hit : hits) {
                    DiscussPost post = new DiscussPost();
                    String id = hit.getSourceAsMap().get("id").toString();
                    post.setId(Integer.valueOf(id));

                    String userId = hit.getSourceAsMap().get("userId").toString();
                    post.setUserId(Integer.valueOf(userId));

                    String title = hit.getSourceAsMap().get("title").toString();
                    post.setTitle(title);

                    String content = hit.getSourceAsMap().get("content").toString();
                    post.setContent(content);

                    String discussType = hit.getSourceAsMap().get("discussType").toString();
                    post.setDiscussType(Integer.valueOf(discussType));

                    String status = hit.getSourceAsMap().get("status").toString();
                    post.setStatus(Integer.valueOf(status));

                    String createTime = hit.getSourceAsMap().get("createTime").toString();
                    post.setCreateTime(new Date(Long.valueOf(createTime)));

                    String commentCount = hit.getSourceAsMap().get("commentCount").toString();
                    post.setCommentCount(Integer.valueOf(commentCount));

                    String score = hit.getSourceAsMap().get("score").toString();
                    post.setScore(Double.valueOf(score));

                    // Process highlighted results
                    HighlightField titleField = hit.getHighlightFields().get("title");
                    if (titleField != null) {
                        // titleField.getFragments() returns an array
                        post.setTitle(titleField.getFragments()[0].toString());
                    }

                    HighlightField contentField = hit.getHighlightFields().get("content");
                    if (contentField != null) {
                        // titleField.getFragments() returns an array
                        post.setContent(contentField.getFragments()[0].toString());
                    }

                    list.add(post);


                }


                return new AggregatedPageImpl(list, pageable, hits.getTotalHits(),searchResponse.getAggregations(),
                        searchResponse.getScrollId(),hits.getMaxScore());
            }
        });

        System.out.println(page.getTotalElements()); // Total number
        System.out.println(page.getTotalPages()); // Total pages
        System.out.println(page.getNumber()); // Current page
        System.out.println(page.getSize()); // How many pieces of data per page
        for (DiscussPost post : page) {
            System.out.println(post);
        }
    }

}

3. Develop community search function

• Search service
  • Save post to Elasticsearch server
  • Delete posts from Elasticsearch server
  • Search for posts from Elasticsearch server
• Publish event
  • When publishing a post, the post is asynchronously submitted to the Elasticsearch server
  • When adding comments, posts are submitted to Elasticsearch server asynchronously
  • Add a method in the consumption component to publish events of consumption posts
• Display results
  • Process the search request in the controller and display the search results on HTML
    

3.1 search services

Under the service package, create a new class ElasticsearchService

@Service
public class ElasticsearchService {

    @Autowired
    private DiscussPostRepository discussPostRepository;

    @Autowired
    private ElasticsearchTemplate elasticsearchTemplate;

	// Add, modify
    public void saveDiscussPost(DiscussPost post) {
        discussPostRepository.save(post);
    }

	// delete
    public void deleteDiscussPost(int id) {
        discussPostRepository.deleteById(id);
    }

	// query
    public Page<DiscussPost> searchDiscussPost(String keyword, int current, int limit) {
        SearchQuery searchQuery = new NativeSearchQueryBuilder()
                .withQuery(QueryBuilders.multiMatchQuery(keyword,"title","content"))
                .withSort(SortBuilders.fieldSort("discussType").order(SortOrder.DESC))
                .withSort(SortBuilders.fieldSort("score").order(SortOrder.DESC))
                .withSort(SortBuilders.fieldSort("createTime").order(SortOrder.DESC))
                .withPageable(PageRequest.of(current,limit))
                .withHighlightFields(
                        new HighlightBuilder.Field("title").preTags("<em>").postTags("</em>"),
                        new HighlightBuilder.Field("content").preTags("<em>").postTags("</em>")
                ).build();

        return elasticsearchTemplate.queryForPage(searchQuery, DiscussPost.class, new SearchResultMapper() {
            @Override
            public <T> AggregatedPage<T> mapResults(SearchResponse searchResponse, Class<T> aClass, Pageable pageable) {
                SearchHits hits = searchResponse.getHits();
                if (hits.totalHits <= 0) {
                    return null;
                }

                ArrayList<DiscussPost> list = new ArrayList<>();
                for (SearchHit hit : hits) {
                    DiscussPost post = new DiscussPost();
                    String id = hit.getSourceAsMap().get("id").toString();
                    post.setId(Integer.valueOf(id));

                    String userId = hit.getSourceAsMap().get("userId").toString();
                    post.setUserId(Integer.valueOf(userId));

                    String title = hit.getSourceAsMap().get("title").toString();
                    post.setTitle(title);

                    String content = hit.getSourceAsMap().get("content").toString();
                    post.setContent(content);

                    String discussType = hit.getSourceAsMap().get("discussType").toString();
                    post.setDiscussType(Integer.valueOf(discussType));

                    String status = hit.getSourceAsMap().get("status").toString();
                    post.setStatus(Integer.valueOf(status));

                    String createTime = hit.getSourceAsMap().get("createTime").toString();
                    post.setCreateTime(new Date(Long.valueOf(createTime)));

                    String commentCount = hit.getSourceAsMap().get("commentCount").toString();
                    post.setCommentCount(Integer.valueOf(commentCount));

                    String score = hit.getSourceAsMap().get("score").toString();
                    post.setScore(Double.valueOf(score));

                    // Process highlighted results
                    HighlightField titleField = hit.getHighlightFields().get("title");
                    if (titleField != null) {
                        // titleField.getFragments() returns an array
                        post.setTitle(titleField.getFragments()[0].toString());
                    }

                    HighlightField contentField = hit.getHighlightFields().get("content");
                    if (contentField != null) {
                        // titleField.getFragments() returns an array
                        post.setContent(contentField.getFragments()[0].toString());
                    }

                    list.add(post);


                }

                return new AggregatedPageImpl(list, pageable, hits.getTotalHits(),searchResponse.getAggregations(),
                        searchResponse.getScrollId(),hits.getMaxScore());
            }
        });
    }

}

3.2 release event

3.2.1 when publishing a post, submit the post asynchronously to Elasticsearch server

Modify the addDiscussPost method in the DiscussPostController class

3.2.2 when adding comments, submit posts to Elasticsearch server asynchronously

Modify the addComment method in the CommentController class

3.2.3 add a method in the consumption component to publish events of consumption posts

Under the event package, add a method to consume posts and publish events in the EventConsumer class

    // Consumption posting event
    @KafkaListener(topics = {TOPIC_PUBLISH})
    public void handlePublishMessage(ConsumerRecord record) {

        if (record == null || record.value() == null) {
            logger.error("The content of the message is empty!");
            return;
        }
        // Using fastjson to convert json string into Event object
        Event event = JSONObject.parseObject(record.value().toString(), Event.class);
        if (event == null) {
            logger.error("Message format error!");
            return;
        }
		// Query this post
        DiscussPost post = discussPostService.findDiscussPostById(event.getEntityId());
        // Store this post in the es server
        elasticsearchService.saveDiscussPost(post);

    }

3.3 display results

1. Create a new searchController class under the controller package

package com.ateam.community.controller;

@Controller
public class SearchController implements CommunityConstant {

    @Autowired
    private ElasticsearchService elasticsearchService;

    @Autowired
    private UserService userService;

    @Autowired
    private LikeService likeService;

    // search?keyword=xxx
    @RequestMapping(value = "/search", method = RequestMethod.GET)
    public String search(String keyword, Page page, Model model){

        // Search posts
        org.springframework.data.domain.Page<DiscussPost> searchResult =
                elasticsearchService.searchDiscussPost(keyword,page.getCurrent() - 1, page.getLimit());
        // Aggregate data
        List<Map<String, Object>>  discussPosts = new ArrayList<>();
        if (searchResult != null) {
            for (DiscussPost post : searchResult) {
                HashMap<String, Object> map = new HashMap<>();
                // Posts
                map.put("post",post);
                // Author information
                map.put("user",userService.findUserById(post.getUserId()));
                // Number of likes
                map.put("likeCount",likeService.findEntityLikeCount(ENTITY_TYPE_POST,post.getId()));

                discussPosts.add(map);
            }
        }

        model.addAttribute("discussPosts",discussPosts);
        model.addAttribute("keyword",keyword);

        // Set paging information
        page.setPath("/search?keyword=" + keyword);
        page.setRows(searchResult == null ? 0 : (int) searchResult.getTotalElements());

        return "/site/search";
     }
}

1. Modify index Search box in HTML page
   
2. Process search HTML page