Centos7 installation configuration [kafka] [kafka manager] [startup Script]

Posted by webing on Wed, 15 Dec 2021 00:48:05 +0100

Background

kafka Manager (now renamed CMAK) is the most popular kafka cluster management tool. It was first opened by Yahoo. Users can perform some simple cluster management operations on the Web interface.
We install everything here from scratch.
This JDK version selects [jdk1.8.0_301]; JDK8 official website download address.
The version of Kafka is [kafka_2.12-2.4.1]; kafka official website download address.
The version of Kafka manager is [Kafka manager-1.3.3.4]; Kafka manager official website download address.
The installation package is given here. The installation package is modified and has a startup script, which can be used directly or downloaded from the official website. You need to modify and compile it yourself; [ JDK Kafka Manager installation package ][extraction code: king]
All the following commands can be directly copied and executed in the shell.

1. Install JDK8

This is determined according to the versions of Kafka and Kafka manager, and the latest cmak 3 JDK11 needs to be installed in version X. in fact, I tried the new version of cmak, and the interface is not as good as the old version. One (one) light (horse) in front of the eye (ugly) ha, I think so, so the old version is used here.

Unzip to the specified location

tar zxf jdk1.8.0_301.tgz -C /opt/

Configure environment variables

env='PATH=$PATH:$JAVA_HOME/bin'
cat << EOF >> /etc/profile

# jdk
export JAVA_HOME=/opt/jdk1.8.0_301
export $env
EOF
source /etc/profile

Verify the configuration and print out the jdk version. The configuration is OK

# View jdk version number
java -version
# View Java processes
jps

2. Install Kafka

Modify the configuration file before starting zookeeper and Kafka with a script.

Unzip to the specified location

tar zxf kafka_2.12-2.4.1.tgz -C /opt/

Add Zookeeper startup script start Zookeeper sh

#!/bin/bash

# Kafka installation directory
dir_home=/opt/kafka_2.12-2.4.1/
rm -rf $dir_home/start-zookeeper.log
nohup $dir_home/bin/zookeeper-server-start.sh $dir_home/config/zookeeper.properties >> $dir_home/start-zookeeper.log 2>&1 &

Add Kafka startup script start Kafka sh

#!/bin/bash

# Kafka installation directory
dir_home=/opt/kafka_2.12-2.4.1/
rm -rf $dir_home/start-kafka.log
nohup $dir_home/bin/kafka-server-start.sh $dir_home/config/server.properties >> $dir_home/start-kafka.log 2>&1 &

Modify Kafka's startup script

Modify Kafka's startup command, add a line after line 29, and configure jmx port, so that we can see the message writing speed.

sed -i '29 a \    export JMX_PORT=\"9999\"' /opt/kafka_2.12-2.4.1/bin/kafka-server-start.sh

Modify Kafka's configuration file

# Replace the IP with the IP of the host where Kafka is located
sed -i '31 a listeners=PLAINTEXT://192.168.110.110:9092' /opt/kafka_2.12-2.4.1/config/server.properties

Start zookeeper and Kafka using scripts

cd /opt/kafka_2.12-2.4.1
# Start Zookeeper
./bin/start-zookeeper.sh
# Check whether the log is normal
tail -f start-zookeeper.log
# Start Kafka
./bin/start-kafka.sh
# Check whether the log is normal
tail -f start-kafka.log

Normally, you can see the two processes QuorumPeerMain and Kafka

3. Install Kafka Manager

Unzip to the specified location

tar zxf kafka-manager-1.3.3.4.tgz -C /opt/

Modify profile

# Replace line 24, that is, replace the IP with the IP of the host where zookeeper is located
sed -i '24c kafka-manager.zkhosts=\"192.168.110.110:2181\"' /opt/kafka-manager-1.3.3.4/conf/application.conf

Add the startup script start kafkamanager sh

#!/bin/bash

# Installation directory of Kafka Manager
dir_home=/opt/kafka-manager-1.3.3.4
rm -rf $dir_home/{RUNNING_PID,start-kafkaManager.log}
# The specified web access port is 8888
nohup $dir_home/bin/kafka-manager -Dconfig.file=$dir_home/conf/application.conf -Dhttp.port=8888 >> $dir_home/start-kafkaManager.log 2>&1 &

Start Kafka manager using script

cd /opt/kafka-manager-1.3.3.4
# Start Zookeeper
./bin/start-kafkaManager.sh
# Check whether the log is normal
tail -f start-kafkaManager.log

The process ProdServerStart can be seen when starting normally
Then you can access the web interface with a browser: 192.168 110.110:8888 [IP is the host IP of Kafka manager]
If you need to configure the user name and password, please modify the following three lines. At that time, the browser will need to log in first