Elasticsearch, Kibana version

• Elasticsearch: 7.2.0
• Kibana: 7.2.0

Cluster structure and server configuration

• There are 3 nodes in the cluster, corresponding to 3 servers
• Configuration per server: 2 cores 16G, data disk 100G solid state hard disk, system disk 40G mechanical disk

Elasticsearch installation

1. Download and Install

Create a new/data directory and mount SSD disks

mkdir /data
fdisk -u /dev/vdb
mkfs.ext4  /dev/vdb1
cp /etc/fstab /etc/fstab.bak
echo "/dev/vdb1 /data/ ext4 defaults 0 0" >> /etc/fstab
mount /dev/vdb1 /data

Create Elasticsearch data and log directories

mkdir -p /data/elasticsearch/data
mkdir -p /data/elasticsearch/logs

New/elastic directory

mkdir /elastic

Official Thunder Download, Upload to Server, Unzip ES Compressed Pack, and Copy to/elastic Directory

tar -vxzf elasticsearch-7.2.0-linux-x86_64.tar.gz
cp -r elasticsearch-7.2.0 /elastic/elasticsearch

Create an elastic user and modify/elastic, /data/elasticsearch directory and subdirectory file owners

useradd elastic
chown -R elastic:elastic /elastic
chown -R elastic:elastic /data/elasticsearch

Stability tuning

1. Tuning Linux parameters

1. Modify system resource constraints

The maximum number of files a single user can open, set to the official recommendation of 65536 or greater

echo "* - nofile 655360" >>/etc/security/limits.conf

Single User Memory Address Space

echo "* - as unlimited" >>/etc/security/limits.conf

Number of single user threads

echo "* - nproc 2056474" >>/etc/security/limits.conf

Single user file size

echo "* - fsize unlimited" >>/etc/security/limits.conf

Single User Lock Memory

echo "* - memlock unlimited" >>/etc/security/limits.conf

Maximum map memory area available to a single process

echo "vm.max_map_count = 655300" >>/etc/sysctl.conf

TCP Full Connection Queue parameter settings, which are designed to prevent the full connection queue from filling up at startup moment when a node restarts abnormally in ES clusters with a large number of nodes (such as more than 100), causing the node to hang and the entire cluster to respond slowly

echo "net.ipv4.tcp_abort_on_overflow = 1" >>/etc/sysctl.conf
echo "net.core.somaxconn = 2048" >>/etc/sysctl.conf

Reduce tcp alive time to prevent invalid links from occupying links

echo 300 >/proc/sys/net/ipv4/tcp_keepalive_time

2. Make the configuration effective

Make/etc/sysctl.conf takes effect immediately

sysctl -p

Re-login the account to/etc/security/limits.conf takes effect

II. ES Node Configuration

1. jvm.options

Set -Xms and -Xmx to the same value, recommending about half of the machine's memory and leaving the remaining half for the system cache.

• jvm memory recommendation is no less than 2G, otherwise ES may not start properly or OOM may be caused by insufficient memory
• jvm does not recommend exceeding 32G, otherwise jvm will disable memory object pointer compression technology, causing memory waste

2. elasticsearch.yml

# Cluster name
cluster.name: bl-els

# Node name, corresponding name of three nodes ["node-1","node-2","node-3"]
node.name: node-1

# ES Data Storage Path
path.data: /data/elasticsearch/data

# ES Log Storage Path
path.logs: /data/elasticsearch/logs

# memory locked
bootstrap.memory_lock: true

# HTTP Access IP, both Intranet IP and External IP can be accessed
network.host: 0.0.0.0

# HTTP Access Port
http.port: 9200

# Address list of seed nodes
discovery.seed_hosts: ["172.18.112.10", "172.18.112.11", "172.18.112.12"]

# List of names that can become primary nodes
cluster.initial_master_nodes: ["node-1", "node-2", "node-3"]

Elasticsearch Cluster, Management, Security

1. Node deployment and clustering

1. First test run after single node deployment

# Switch to elastic account
su elastic
# Jump to ES directory
cd /elastic/elasticsearch
# Foreground startup
./bin/elasticsearch

2. Multi-node deployment to form clusters

View node information

ip:9200

Common curl commands

# curl View Options Supported by ES Cluster
ip:9200/_cat

# View ES Node Information
ip:9200/_cat/nodes?v

# View the health of the ES cluster
ip:9200/_cat/health?v

3. Start-up

New/etc/init.d/elasticsearch script

vim /etc/init.d/elasticsearch

#!/bin/sh
#chkconfig: 2345 80 05
#description: elasticsearch

els_user=elastic
els_menu=/elastic/elasticsearch

case "$1" in
start)
    su $els_user<<!
    cd $els_menu
    ./bin/elasticsearch -d
!
    echo "elasticsearch startup"
    ;;
stop)
    els_pid=`ps aux|grep elasticsearch | grep -v 'grep elasticsearch' | awk '{print $2}'`
    kill $els_pid
    echo "elasticsearch stopped"
    ;;
restart)
    els_pid=`ps aux|grep elasticsearch | grep -v 'grep elasticsearch' | awk '{print $2}'`
    kill $els_pid
    echo "elasticsearch stopped"
    su $els_user<<!
    cd $els_menu
    ./bin/elasticsearch -d
!
    echo "elasticsearch startup"
    ;;
*)
    echo "start|stop|restart"
    ;;
esac

exit $?

Increase Execution Permissions for Startup Scripts

chmod +x /etc/init.d/elasticsearch

Configure Startup to Start Elasticsearch

chkconfig --add elasticsearch

4. How to close the Elasticsearch cluster correctly

• Official documents: www.elastic.co/guide/en/elasticsea...

Close Elasticsearch cluster

# Disable automatic fragmentation distribution
PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}

# Perform synchronous refresh
POST _flush/synced

# Close nodes one by one
/etc/init.d/elasticsearch stop
# Or kill manually
ps aux|grep elasticsearch
kill pid

Start the Elasticsearch cluster

# Start Nodes One by One
/etc/init.d/elasticsearch start

# Wait for all nodes to join the cluster to see if the cluster status is [yellow]
GET _cat/health
GET _cat/nodes
# Or look directly at the Elasticsearch-head tool [Cluster Health Value][yellow]

# Enable automatic fragmentation distribution
PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": null
  }
}

# Wait for the cluster to become available, monitor the availability of the cluster through its status and recovery process [green]
GET _cat/health
GET _cat/recovery
# Or look directly at the Elasticsearch-head tool [Cluster Health Value][green]

2. Cluster Management

1. elasticsearch-head management tool

github address: github.com/mobz/elasticsearch-head

Recommended Chrome Plugin Installation
chrome.google.com/webstore/detail/...

2. Kibana Installation Management

Official Thunder download, upload to server, unzip kibana package, and copy to / elastic directory

tar -vxzf kibana-7.2.0-linux-x86_64.tar.gz
cp -r kibana-7.2.0 /elastic/kibana

Modify/elastic/kibana directory and subdirectory file owners

chown -R elastic:elastic /elastic/kibana

First test run

# Switch to elastic account
su elastic
# Jump to ES directory
cd /elastic/kibana
# Foreground startup
./bin/kibana

View the Kibana website

ip:5601

Kibana Background Start and Close

# Switch to elastic account
su elastic
# Jump to ES directory
cd /elastic/kibana

# Start in the background and generate a nohup in the current directory. Out file, record kibana output log, kibana parameter [-p] means only record [error log] and [boot log]
nohup ./bin/kibana -q &

# View kibana process pid
# Method One
tail -n 10 nohup.out
# Method 2
netstat -tunlp | grep 5601

#Close kibana
kill pid

3. Communication Encryption

1. master node generation key

Generate certificates and private keys

./bin/elasticsearch-certutil ca
./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

Delete CA Certification Authority

rm -f /elastic/elasticsearch/elastic-stack-ca.p12

Create a new certs directory to store certificates

mkdir /elastic/elasticsearch/config/certs

Certificates are stored in the certs directory

mv elastic-certificates.p12 /elastic/elasticsearch/config/certs

Modify certs directory and file owner

chown -R elastic:elastic /elastic/elasticsearch/config/certs

2. Other Node Deployment Keys

Download certificate from master node

...Step omission...

Create a new certs directory to store certificates

mkdir /elastic/elasticsearch/config/certs

Upload the certificate file and store it in the certs directory

mv elastic-certificates.p12 /elastic/elasticsearch/config/certs

Modify certs directory and file owner

chown -R elastic:elastic /elastic/elasticsearch/config/certs

3. Encrypt communication between nodes in the cluster

vim config/elasticsearch.yml

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

4. Encrypt HTTP client communication [optional]

vim config/elasticsearch.yml

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12

5. Restart Elasticsearch

Close all nodes in turn

/etc/init.d/elasticsearch stop

Start all nodes in turn

/etc/init.d/elasticsearch start

4. Security Settings

1. Initialize built-in accounts

Turn on Elasticsearch security validation

vim config/elasticsearch.yml

xpack.security.enabled: true

Configure Transport Layer Security (TLS/SSL) for inter-node communication

Encrypt communication between nodes in a cluster

If Elasticsearch is not running, start all nodes

/etc/init.d/elasticsearch start

Set passwords for all built-in users

# Choose one of the following

# Automatic random password generation and output to console
./bin/elasticsearch-setup-passwords auto

# Prompt to enter password manually and fill in password step by step
./bin/elasticsearch-setup-passwords interactive

2. First login using the built-in super account elastic

elasticsearch-head logs on for the first time using account elastic

Refresh the elasticsearch-head page to automatically pop up the login box, enter the account elastic and password to login

Kibana logged in for the first time using account elastic

vim config/kibana.yml

elasticsearch.username: "elastic"
elasticsearch.password: "elasticpassword"

# Ctrl-C stops the service and restarts kibana
./bin/kibana

Refresh Kibana page automatically jumps to login page, enter account elastic and password to login

3. Use Kibana to customize role permissions, self-built accounts, and account authorization

• Log in to Kibana using super account elastic to enter security settings
• New Custom Super Role Referring to Super Account Role
• New Custom Super Account and Authorize Custom Super Role
• Exit Super Account elastic and log back in to Kibana with a custom Super Account
• Other roles and privilege assignments, created for your own team