There are many Kafka cluster monitoring tools, and here we will only describe monitoring for Kafka.

1. Introduction to Kafka Offset Monitor

Clusters need to be highly available in production environments, so Kafka clusters need to be monitored.The Kafka Offset Monitor can monitor several items of the Kafka cluster:

(1) Current surviving broker set of Kafka cluster
(2) topic Collection of Current Survival of Kafka Cluster
(3) List of consumers
(4) The number of offset lag s consumed by the current consumer group in the Kafka cluster (that is, how many messages are backed up in the current topic partition and not even consumed)

Install and deploy Kafka Offset Monitor

(1) Download and Install
Download the corresponding jar package to github and download the link address

https://github.com/quantifind/KafkaOffsetMonitor/releases

After downloading, upload directly to the server where kafka is located
(2) Start Kafka Offset Monitor

java -cp KafkaOffsetMonitor-assembly-0.2.0.jar \
com.quantifind.kafka.offsetapp.OffsetGetterWeb \
--zk 192.168.12.150:2181,192.168.12.151:2181,192.168.12.152:2181 \
--port 8086 \
--refresh 10.seconds \
--retain 7.days \
1>kafka.offset.monitor.logs.stdout.log 2>kafka.offset.monitor.logs.stderr.log &

Memory parameters can also be specified

java -Xms128M -Xmx256M -Xss1024K -XX:PermSize=128m -XX:MaxPermSize=256m -cp KafkaOffsetMonitor-assembly-0.2.0.jar com.quantifind.kafka.offsetapp.OffsetGetterWeb --zk 192.168.12.150:2181,192.168.12.151:2181,192.168.12.152:2181 --port 8086 --refresh 10.seconds --retain 7.days 1>kafka.offset.monitor.logs.stdout.log 2>kafka.offset.monitor.logs.stderr.log &

Where -- zk specifies the zookeeper list, --port specifies the monitoring port to use, --refresh specifies the refresh interval of 10 seconds, --retain indicates the resident time is 7 points, the standard log is output to the kakfa.offset.monitor.logs.stdout.log file, the error is output to kafka.offset.monitor.logs.stderr.log, and finally -- indicates the backgroundFunction!

Run result:

Generate two log output files for the above configuration in the directory where jar is located

View background running processes using jps

The KafkaOffsetMonitor consumer group tested for version 0.2 is not available, and the offset is not available. It may only be compatible with pre-0.8 versions of Kafka (here I am using kafka2.11-2.2.0 versions). Port the packages to the libs directory of Kafka and download other versions of the test results day:

test
(1) You can only see the consumer group of KafkaOffsetMonitor, not the consumer group information of the consumer program you started.

java -cp KafkaOffsetMonitor-assembly-0.3.0-SNAPSHOT.jar \
com.quantifind.kafka.offsetapp.OffsetGetterWeb \
--zk 192.168.12.150:2181,192.168.12.151:2181,192.168.12.152:2181 \
--port 8086 --refresh 10.seconds \
--retain 7.days \
1>kafka.offset.monitor.logs.stdout.log 2>kafka.offset.monitor.logs.stderr.log

(2) A consumer group cannot be seen

java -cp KafkaOffsetMonitor-assembly-0.3.0-SNAPSHOT.jar \
com.quantifind.kafka.offsetapp.OffsetGetterWeb \
--offsetStorage kafka \
--zk 192.168.12.150:2181,192.168.12.151:2181,192.168.12.152:2181 \
--port 8086 --refresh 10.seconds \
--retain 7.days \
1>kafka.offset.monitor.logs.stdout.log 2>kafka.offset.monitor.logs.stderr.log &

(3) Same (1) result

java -Xms512M -Xmx512M -Xss1024K -XX:PermSize=256m -XX:MaxPermSize=512m \
     -cp KafkaOffsetMonitor-assembly-0.4.6-SNAPSHOT.jar \
     com.quantifind.kafka.offsetapp.OffsetGetterWeb \
     --offsetStorage kafka \
     --kafkaBrokers localhost:9092 \
     --kafkaSecurityProtocol PLAINTEXT \
     --zk localhost:2181 \
     --port 8787 \
     --refresh 10.seconds \
     --retain 7.days 1>stderr.log  \
     --dbName offsetapp_kafka

Note the parameters
offsetStorage: Valid options are "zookeeper", "kafka", "storm".After version 0.9, offset stores in Kafka

The above test results show that Assibar is not available for this device in the new version.

View cluster information

This tool is not very operational and cannot be used, so it is recommended to use the following tool, kafka manager, to manage multiple clusters!

Install and deploy the kafka-mamager tool

The kafka-mamager tool of yahoo is selected here. Download the binary package kafka-manager-1.3.3.7.zip first.
Install Unzip
cd /opt unzip kafka-manager-1.3.3.7.zip mv kafka-manager-1.3.3.7 kafka-manager

Edit Profile

cd /opt/kafka-manager/conf/
vim application.conf

Add Content

kafka-manager.zkhosts="kafka1:2181,kafka2:2181,kafka3:2181"

Start kafka-manager

chmod 777 /opt/kafka-manager/bin/kafka-manager
nohup /opt/kafka-manager/bin/kafka-manager &

The current directory will have a nohup.out console output log after startup

Be careful
The default port for kafka-manager is 9000, which can be accessed through
*-Dhttp.port, specified port;*
-Dconfig.file=conf/application.conf Specifies Configuration File
As shown below

nohup ./kafka-manager -Dconfig.file=conf/application.conf -Dhttp.port=8080

View startup

jps

Access Address
http://192.168.12.60:9000/
Add Family Parameter
Add Cluster

Add parameters are configured as follows

Then save it and add it as follows

Click in and you can view all the information about the cluster from the web page