Elastic search cluster building

Posted by thomasgrant on Mon, 07 Oct 2019 04:14:21 +0200

Environment and Software

  • centos7
  • elasticsearch 7.4.0

Because ES7 already has the JDK version of the required java built-in, the java installation is not introduced here.

ES7 installation mode uses RPM installation mode.

es node list

ip Node name
192.168.1.100 es-node-master-01
192.168.1.101 es-node-master-02
192.168.1.102 es-node-master-03
192.168.1.103 es-node-data-01
192.168.1.104 es-node-data-02
192.168.1.105 es-node-data-03

Install es7

    wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.4.0-x86_64.rpm
    wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.4.0-x86_64.rpm.sha512
    shasum -a 512 -c elasticsearch-7.4.0-x86_64.rpm.sha512 
    sudo rpm --install elasticsearch-7.4.0-x86_64.rpm

The elastic search installation path installed by the above command is: / etc / elastic search

Setting up automatic startup using system D

    sudo systemctl daemon-reload
    sudo systemctl enable elasticsearch.service 

Start/stop elastic search

    sudo systemctl start elasticsearch.service
    sudo systemctl stop elasticsearch.service

Reference resources: https://www.elastic.co/guide/en/elasticsearch/reference/7.4/rpm.html

Modifying System Configuration

  • Modify the number of file descriptors

The elastic search restriction on file descriptors is at least 65536, so this configuration needs to be modified.

    echo '* soft nofile 65536' >> /etc/security/limits.conf
    echo '* hard nofile 65536' >> /etc/security/limits.conf

    #View the configuration, restart the configuration if it does not take effect
    ulimit -n
  • Modify max_map_count

Temporary amendments:

    sysctl -w vm.max_map_count=655360

Permanent settings:

    echo 'vm.max_map_count=655360' >> /etc/sysctl.conf

Basic configuration of elastic search

  • cluster.name

ES distinguishes clusters by cluster name, so the node names of all nodes in the cluster must be consistent. Such as:

    cluster.name: es-cluster
  • node.name

The name of the node in the cluster, and the name of the node in the same cluster must remain unique. Such as:

    node.name: es-master-01
  • node.master

Marker nodes can be elected master nodes. If the master node in the cluster stops, the node will participate in the election and can become the master node. Such as:

    node.master: true
  • node.data

Allow this node to store index data (default). For example:

    node.data: true
  • path.data

This path is the directory where es data is stored and can be arbitrarily specified. If the proposed path for the production environment has sufficient storage space, such as:

    path.data: /datadrive/elasticsearch/data
  • path.logs

This path stores the logs generated by es, and the production environment is recommended to be set separately from path.logs. Such as:

    path.data: /datadrive/elasticsearch/logs
  • network.host

Addresses bound to support IPv4 and IPv6. It's the monitor address of es. If a specific address is set, it can only be accessed by changing the address, or 0.0.0 can be set without restricting the access address. For example:

    network.host: 0.0.0.0
  • http.port

External access to es http port, default 9200. Such as:

    http.port: 9200
  • transport.tcp.port

The port of tcp for cluster node communication is 9300 by default. Such as:

    transport.tcp.port: 9300
  • discovery.seed_hosts

A list of hosts for cluster search. Changed by discovery.zen.ping.unicast.hosts: parameter. Such as:

   discovery.seed_hosts:["192.168.1.100:9300","192.168.1.101:9300","192.168.1.102:9300","192.168.1.103:9300","192.168.1.104:9300","192.168.1.105:9300"] 
  • cluster.initial_master_nodes

Cluster master initialization list, master election list.

    cluster.initial_master_nodes:["192.168.1.100:9300","192.168.1.101:9300","192.168.1.102:9300"]

Start elastic search

Other nodes in the elastic search cluster access the elastic search instance through the transport.tcp.port port port. External services are provided through the http.port port port. So you need to make sure that these two ports are open.

centos7 open port

    firewall-cmd --zone=public --add-port=9300/tcp --permanent
    firewall-cmd --zone=public --add-port=9200/tcp --permanent
    firewall-cmd --reload

After configuring all the nodes, start the elastic search service separately.

    systemctl start elasticsearch.service

All nodes can access any node to view the cluster status after booting, and can use the browser or curl node address. Such as:

    curl -XGET 'http://192.168.1.100:9200/_cluster/state?pretty'

Install kibana

kibana can be used to visually manage elastic search. kibana can be installed at any node. kibana can manage the entire cluster by accessing only one node in the elastic search cluster.

To be continued

Topics: Linux ElasticSearch RPM sudo firewall