kafka monitoring tool: kafka eagle installation pit

Posted by madavies on Wed, 29 Dec 2021 03:42:05 +0100

1. Background:

I use Xiaobai in kafka. When I use kafka product production data, it is inconvenient to see whether there is data in kafka topic from the background. I looked online and found that an open source tool kafka eagle works better.

2. Software preparation:

Official website link: EFAK

Just started to download an official latest version, v2 0.9, but the installation failed all the time. The jvm crashed. At first, I didn't know why. I chose a lower version v1.0 Version 4.8

3. Installation process:

(1) Get the installation package from the official website and unzip it all the time, / opt / software / kafka_eagle / Kafka Eagle bin-1.4.8 / Kafka Eagle web-1.4.8

(2) Modify conf/system-config.properties. I only need to monitor one cluster here. I mainly modified the following parameters. I commented out other parameters related to cluster2 and cluster3

kafka.eagle.zk.cluster.alias=cluster1
cluster1.zk.list=10.144.17.170:2181,10.144.17.171:2181,10.144.17.172:2181



kafka.eagle.driver=com.mysql.jdbc.Driver
kafka.eagle.url=jdbc:mysql://*****:3306/kafka_ke?createDatabaseIfNotExist=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
kafka.eagle.username=root
kafka.eagle.password=root

(3) Execute the installation sh bin/ke.sh start

During the first installation, the direct process crashes. Look at the jvm error and find the following problems:

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  java.net.Inet6AddressImpl.lookupAllHostAddr(Ljava/lang/String;)[Ljava/net/InetAddress;+0
j  java.net.InetAddress$2.lookupAllHostAddr(Ljava/lang/String;)[Ljava/net/InetAddress;+4
j  java.net.InetAddress.getAddressesFromNameService(Ljava/lang/String;Ljava/net/InetAddress;)[Ljava/net/InetAddress;+51
j  java.net.InetAddress.getAllByName0(Ljava/lang/String;Ljava/net/InetAddress;Z)[Ljava/net/InetAddress;+29
j  java.net.InetAddress.getAllByName(Ljava/lang/String;Ljava/net/InetAddress;)[Ljava/net/InetAddress;+383
j  java.net.InetAddress.getAllByName(Ljava/lang/String;)[Ljava/net/InetAddress;+2
j  java.net.InetAddress.getByName(Ljava/lang/String;)Ljava/net/InetAddress;+1
j  java.net.InetSocketAddress.<init>(Ljava/lang/String;I)V+15
j  sun.net.NetworkClient.doConnect(Ljava/lang/String;I)Ljava/net/Socket;+92
j  sun.net.www.http.HttpClient.openServer(Ljava/lang/String;I)V+4
j  sun.net.www.http.HttpClient.openServer()V+114
j  sun.net.www.http.HttpClient.<init>(Ljava/net/URL;Ljava/net/Proxy;I)V+125
j  sun.net.www.http.HttpClient.New(Ljava/net/URL;Ljava/net/Proxy;IZLsun/net/www/protocol/http/HttpURLConnection;)Lsun/net/www/http/HttpClient;+259
j  sun.net.www.http.HttpClient.New(Ljava/net/URL;Ljava/net/Proxy;ILsun/net/www/protocol/http/HttpURLConnection;)Lsun/net/www/http/HttpClient;+5
j  sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Ljava/net/URL;Ljava/net/Proxy;I)Lsun/net/www/http/HttpClient;+4
j  sun.net.www.protocol.http.HttpURLConnection.plainConnect0()V+357
j  sun.net.www.protocol.http.HttpURLConnection.plainConnect()V+71
j  sun.net.www.protocol.http.HttpURLConnection.connect()V+20
j  sun.net.www.protocol.http.HttpURLConnection.getInputStream0()Ljava/io/InputStream;+195
j  sun.net.www.protocol.http.HttpURLConnection.getInputStream()Ljava/io/InputStream;+52
j  net.sf.ehcache.util.UpdateChecker.getUpdateProperties(Ljava/net/URL;)Ljava/util/Properties;+13
j  net.sf.ehcache.util.UpdateChecker.doCheck()V+26
j  net.sf.ehcache.util.UpdateChecker.checkForUpdate()V+9
j  net.sf.ehcache.util.UpdateChecker.run()V+1
j  java.util.TimerThread.mainLoop()V+221
j  java.util.TimerThread.run()V+1
v  ~StubRoutines::call_stub

I found a lot of answers from the Internet and said it was encache There is a problem with the XML configuration. You need to close updateCheck.

1) Search the Kafka Ke installation directory for encache XML, find the configuration file Shiro - ehcache XML, add updateCheck="false", and modify as follows:

<?xml version="1.0" encoding="UTF-8"?>
<ehcache name="shirocache" updateCheck="false">

2) Edit Ke SH file, comment out the following lines, or re extract it from the war package every time

 #rm -rf $KE_HOME/kms/webapps/ke
 #rm -rf $KE_HOME/kms/work
 #mkdir -p $KE_HOME/kms/webapps/ke
 #cd $KE_HOME/kms/webapps/ke
 #${JAVA_HOME}/bin/jar -xvf $KE_HOME/kms/webapps/ke.war

(4) After modification, execute sh bin/ke.sh start again, and new errors are found. Check the log

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  java.net.Inet6AddressImpl.lookupAllHostAddr(Ljava/lang/String;)[Ljava/net/InetAddress;+0
j  java.net.InetAddress$2.lookupAllHostAddr(Ljava/lang/String;)[Ljava/net/InetAddress;+4
j  java.net.InetAddress.getAddressesFromNameService(Ljava/lang/String;Ljava/net/InetAddress;)[Ljava/net/InetAddress;+51
j  java.net.InetAddress.getAllByName0(Ljava/lang/String;Ljava/net/InetAddress;Z)[Ljava/net/InetAddress;+29
j  java.net.InetAddress.getAllByName(Ljava/lang/String;Ljava/net/InetAddress;)[Ljava/net/InetAddress;+383
j  java.net.InetAddress.getAllByName(Ljava/lang/String;)[Ljava/net/InetAddress;+2
j  java.net.InetAddress.getByName(Ljava/lang/String;)Ljava/net/InetAddress;+1
j  java.net.InetSocketAddress.<init>(Ljava/lang/String;I)V+15
j  sun.net.NetworkClient.doConnect(Ljava/lang/String;I)Ljava/net/Socket;+92
j  sun.net.www.http.HttpClient.openServer(Ljava/lang/String;I)V+4
j  sun.net.www.http.HttpClient.openServer()V+114
j  sun.net.www.http.HttpClient.<init>(Ljava/net/URL;Ljava/net/Proxy;I)V+125
j  sun.net.www.http.HttpClient.New(Ljava/net/URL;Ljava/net/Proxy;IZLsun/net/www/protocol/http/HttpURLConnection;)Lsun/net/www/http/HttpClient;+259
j  sun.net.www.http.HttpClient.New(Ljava/net/URL;Ljava/net/Proxy;ILsun/net/www/protocol/http/HttpURLConnection;)Lsun/net/www/http/HttpClient;+5
j  sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Ljava/net/URL;Ljava/net/Proxy;I)Lsun/net/www/http/HttpClient;+4
j  sun.net.www.protocol.http.HttpURLConnection.plainConnect0()V+357
j  sun.net.www.protocol.http.HttpURLConnection.plainConnect()V+71
j  sun.net.www.protocol.http.HttpURLConnection.connect()V+20
j  sun.net.www.protocol.http.HttpURLConnection.getInputStream0()Ljava/io/InputStream;+195
j  sun.net.www.protocol.http.HttpURLConnection.getInputStream()Ljava/io/InputStream;+52
j  org.quartz.utils.UpdateChecker.getUpdateProperties(Ljava/net/URL;)Ljava/util/Properties;+13
j  org.quartz.utils.UpdateChecker.doCheck()V+17
j  org.quartz.utils.UpdateChecker.checkForUpdate()V+1
j  org.quartz.utils.UpdateChecker.run()V+1
j  java.util.TimerThread.mainLoop()V+221
j  java.util.TimerThread.run()V+1
v  ~StubRoutines::call_stub

It's very similar to last time, but this is org quartz. utils. Updatechecker, we have to close this place.

Go to / opt/software/kafka_eagle/kafka-eagle-bin-1.4.8/kafka-eagle-web-1.4.8/kms/webapps/ke/WEB-INF/classes, modify alert quartz xml,mbean-quartz.xml,metrics-quartz.xml,topic-quartz.xml
The following configurations are found in these configuration files, mainly skipUpdateCheck configuration. I have modified these four files.

<bean class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
                <property name="triggers">
                        <list>
                                <!-- All scheduling lists. -->
                                <ref bean="topicRankTrigger" />
                        </list>
                </property>
                <property name="quartzProperties">
                        <props>
                           <prop key="org.quartz.scheduler.skipUpdateCheck">true</prop>
                        </props>
               </property>

        </bean>

(5) When installing again, I found that the page can finally log in, but there is no kafka information.

4. Modify kafka configuration

Enter $Kafka_ In the home / bin directory, modify Kafka run class SH file, add JMX in the first line of the file_ PORT=9988

Then restart all kafka nodes

5. Restart the kafka Eagle program again, and finally find that kafka information can be monitored.

Topics: kafka Distribution

Programmer Think

kafka monitoring tool: kafka eagle installation pit

Hot Topics