Environmental preparation

• Click to check whether ELK is compatible with the operating system
• Click to view the list of compatibility information between ELK and JDK

Although the official website said that ES 7.12 supports JDK8, the actual test found that ES 7.12 needs at least JDK11 or above.

• Therefore, it is best for our server to use the long-term support version of JDK11.

Future versions of Elasticsearch will require Java 11; your Java version from [/usr/local/java/jdk/jdk1.8.0_161/jre] does not meet this requirement. Consider switching to a distribution of Elasticsearch with a bundled JDK. If you are already using a distribution with a bundled JDK, ensure the JAVA_HOME environment variable is not set.

ELK is the abbreviation of elasticsearch, logstash and kibana. It mainly focuses on visual log analysis and query

Elastic Stack mainly includes three parts:

• Massive data storage and retrieval ---- Elasticsearch distributed storage and query
• Visual interface and chart generation - Kibana can provide visual interface and the generation of various charts.
• Data collection ---- you can use Logstash to collect logs or FileBeat to collect logs.

ELK is just a part of the log analysis stack in Elastic Stack usage

explain:

• I wonder if you have noticed that the official website seems to have deliberately used FileBeat to replace Logstash to collect logs.
• Logstash is an open source server-side data processing pipeline launched by Elastic company. It can collect data from multiple sources at the same time, convert data, and then send the data to the specified repository. Logstash official introduction.
• Filebeat is a lightweight log collector launched by Elastic company to solve the problem of "too heavy" Logstash. When processing logs generated by a large number of servers, virtual machines and containers, the log collection method of Logstash + Filebeat can be used. Filebeat official introduction.

Installing ELK/EFK

Install Elasticsearch

• Elasticsearch, known as ES in Jianghu, is a real-time distributed storage, search and analysis engine.

Download and unzip

1. Official download address of ES: https://www.elastic.co/cn/dow...
2. Download method 2: use the command line to download

yum install curl;
curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.12.0-linux-x86_64.tar.gz;
#If the operating system does not support curl command, you can also use wget command to download
yum install wget
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.12.0-linux-x86_64.tar.gz;

The virtual machine is configured for all three servers

Three modified operating system restrictions

In the Centos7 Linux operating system, the default maximum number of files that can be opened by a single process is 1024

However, ES has some special requirements for the operating system

• The maximum number of files that can be opened by a single process of the operating system is required to be at least 65535
• Moreover, ES will use many threads, and the operating system restrictions need to be modified.
  If it is not configured, an error will be reported as follows:

bootstrap checks failed. You must address the points described in the following [3] lines before starting Elasticsearch.
bootstrap check failure [1] of [3]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
bootstrap check failure [2] of [3]: max number of threads [3687] for user [elasticsearch] is too low, increase to at least [4096]
bootstrap check failure [3] of [3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Check the maximum number of files that can be opened by a single process supported by the current operating system. If the return value is not modified, 1024 will be displayed

ulimit -n

To fix the bootstrap check failure [1] of [3] and bootstrap check failure [2] of [3] errors
Modification of configuration file of three machines

vi /etc/security/limits.conf

Add at the end

*               soft    nofile          65535
*               hard    nofile          65535
*               hard    nproc           4096
*               soft    nproc           4096

• Set the limit quantity. The first column represents users and * represents all users
• soft nproc: the maximum number of processes available to a single user (if it exceeds, it will be warned);
• hard nproc: the maximum number of processes available to a single user (an error will be reported if it exceeds);
• soft nofile: the maximum number of file descriptors that can be opened (warning will be given if it is exceeded);
• hard nofile: the maximum number of file descriptors that can be opened (an error will be reported if it exceeds);
  Original link: https://blog.csdn.net/zxljsbk...
• Disconnect the session link and log in again to take effect

To fix the bootstrap check failure [3] of [3] error, modify / etc / sysctl conf

vi /etc/sysctl.conf

#Add content
vm.max_map_count=262144

• vm. max_ map_ The count configuration ensures that the operating system has enough virtual memory
• If you are installing in the package manager mode, this operation is not required and will be configured by default.

#The refresh configuration takes effect immediately. Restart it
sysctl -p
reboot

All three need to be operated

#Create es user
useradd es_user
#Modify es user password
passwd es_user

#Create an es directory
mkdir -p /elk
cd /elk
#Upload the installation package to the elk directory. Here you can upload it uniformly. For the specific software, see the instructions above

#Enter the elk directory and unzip the file
cd /elk;
tar -zxvf elasticsearch-7.16.1-linux-x86_64.tar.gz;
tar -zxvf filebeat-7.16.1-linux-x86_64.tar.gz;
tar -zxvf logstash-7.16.2-linux-x86_64.tar.gz;
tar -zxvf kafka-3.0.0-src.tgz;
tar -zxvf kibana-7.16.1-linux-x86_64.tar.gz;

Install elasticsearch

#Enter the config configuration directory of elasticsearch
cd /elk/elasticsearch-7.16.1/config/;
#Modify profile
vim elasticsearch.yml;

Three configuration files

• If you need to set the remote access password, you need to add the following configuration

http.cors.enabled: true
http.cors.allow-origin: "*" 
http.cors.allow-headers: Authorization
xpack.security.enabled: true 
xpack.security.transport.ssl.enabled: true 

for example

Create directories separately and give them to es_user gives permission

# Assign permissions to 10.0 zero point one one
# Grant execution permission
chmod u+x  /elk/elasticsearch-7.16.1/bin
# ES folder permissions are granted to es_user user
chown -R es_user /usr/local/es/
# Create data folder
mkdir -p /var/data/es/es-node-1/
# Authorize ES data store folder permissions to es_user user
chown -R es_user /var/data/es/es-node-1/
# Create log folder
# Authorize ES log storage folder permissions to es_user user
chown -R es_user /var/log/es/es-node-1/

# Assign permissions to 10.0 zero point one two
# Grant execution permission
chmod u+x  /elk/elasticsearch-7.16.1/bin
# ES folder permissions are granted to es_user user
chown -R es_user /usr/local/es/
# Create data folder
mkdir -p /var/data/es/es-node-2/
# Authorize ES data store folder permissions to es_user user
chown -R es_user /var/data/es/es-node-2/
# Create log folder
# Authorize ES log storage folder permissions to es_user user
chown -R es_user /var/log/es/es-node-2/

# Assign permissions to 10.0 zero point one three
# Grant execution permission
chmod u+x  /elk/elasticsearch-7.16.1/bin
# ES folder permissions are granted to es_user user
chown -R es_user /usr/local/es/
# Create data folder
mkdir -p /var/data/es/es-node-3/
# Authorize ES data store folder permissions to es_user user
chown -R es_user /var/data/es/es-node-3/
# Create log folder
# Authorize ES log storage folder permissions to es_user user
chown -R es_user /var/log/es/es-node-3/

Start Elasticsearch
Development 92009300 port

firewall-cmd --zone=public --add-port=9200/tcp --permanent;
firewall-cmd --zone=public --add-port=9300/tcp --permanent;
firewall-cmd --reload;
firewall-cmd --list-all;

Switch user login

# Switch users
su es_user
# Start ES
./bin/elasticsearch

• It is recommended to start the foreground for the first time. After observing that there is no error, use the background daemon thread to start it
• After the foreground is started, if you want to exit, you can press Ctrl+C to exit
• If you need to run in the background, you can add the next startup parameter:/ bin/elasticsearch -d -p pid
  be careful:
• Where - d indicates that the background daemon is started and - p indicates that the generated process Id is stored in the file.
• ES will load $ES by default_ Elasticsearch. In the home / config / folder Configuration in YML