RocketMQ cluster setup

Posted by Cragsterboy on Thu, 30 Dec 2021 00:48:08 +0100

preface

Record RocketMQ

Quickly build RocketMQ cluster

Machine environment

Prepare three machines, root password root; IP address:

192.168.232.128 worker1 
192.168.232.129 worker2 
192.168.232.130 worker3

Create user

useradd oper
passwd oper (Password entry 123 qweasd)

system configuration

Switch the oper user and generate a key on worker1
ssh-kengen
Then distribute to other machines

ssh-copy-id worker1 
ssh-copy-id worker2 
ssh-copy-id worker3

In this way, you can ssh or scp directly to another machine on worker1 without entering a password.

Turn off firewall

systemctl stop firewalld.service 
firewall-cmd --state

Install java

Create / app directory for oper
Upload the tar package of jdk
Modify ~ / bash_profile to configure environment variables. source takes effect.

export JAVA_HOME=/app/jdk1.8/

Installing RocketMQ

Upload the tar package and decompress it directly. Then configure the environment variables

export ROCKETMQ_HOME=/app/rocketmq/rocketmq-all-4.7.1-bin-release 1

Before version 4.5, RocketMQ does not support automatic slave switching after master downtime. After version 4.5, the master-slave switching based on Dleger is added.

Configure RocketMQ cluster

Build a cluster with 2 master and 2 slave asynchronous disk brushing, so we will use the configuration under conf/2m-2s-async
In the actual project, in order to achieve high availability, dleger is generally used. The clusters to be designed are as follows

Therefore, the modified configuration file is the 2m-2s-async configuration file modified in the config directory of rocketmq You only need to configure the broker conf.
Various configurations suggested by rocketmq can be seen in the config directory of rocketmq:

  • 2m-2s-async: 2 master and 2 slave asynchronous disk brushing (throughput is large, but messages may be lost),
  • 2m-2s-sync:2 master and 2 slave synchronous disk brushing (throughput will decrease, but messages are safer),
  • 2m noslave: 2 master and no slave (single point of failure), and then you can directly configure the broker Conf to configure the single point environment.
  • The dleger is used to realize master-slave switching. Nodes in the cluster will randomly elect a leader based on the Raft protocol, and the others are follower s. Usually, the formal environment uses this method to build clusters.

2m-2s-async is used to build clusters

Configure the first set of brokers-a

Configure the master node of borker-a on worker2. Configure 2m-2s-async/broker-a.properties first

#Cluster name. Nodes with the same name are in the same cluster 
brokerClusterName=rocketmq-cluster 
#broker name. Nodes with the same name are a group of master-slave nodes. 
brokerName=broker-a 
#Broker ID, 0 means Master, and > 0 means Master 
Slave brokerId=0 
#nameServer address, semicolon split 
namesrvAddr=worker1:9876;
worker2:9876;worker3:9876 
#When sending messages, topic s that do not exist in the server are automatically created, and the number of queues created by default 
defaultTopicQueueNums=4 
#Whether to allow the Broker to automatically create topics. It is recommended to open offline and close online
 autoCreateTopicEnable=true 
#Whether to allow the Broker to automatically create subscription groups. It is recommended to open offline and close online 
autoCreateSubscriptionGroup=true 
#Listening port of Broker external service 
listenPort=10911 
#Delete file time point. The default is 4 a.m 
deleteWhen=04 
#File retention time, 48 hours by default 
fileReservedTime=120 
#commitLog the default size of each file is 1G 
mapedFileSizeCommitLog=1073741824 
#ConsumeQueue stores 30W files by default, which can be adjusted according to business conditions 
mapedFileSizeConsumeQueue=300000 
#destroyMapedFileIntervalForcibly=120000 
#redeleteHangedFileInterval=120000 
#Detect physical file disk space 
diskMaxUsedSpaceRatio=88 
#Storage path 
storePathRootDir=/app/rocketmq/store
 #commitLog storage path 
storePathCommitLog=/app/rocketmq/store/commitlog 
#Consumption queue storage path storage path 
storePathConsumeQueue=/app/rocketmq/store/consumequeue 
#Message index storage path 
storePathIndex=/app/rocketmq/store/index 
#checkpoint file storage path 
storeCheckpoint=/app/rocketmq/store/checkpoint 
#abort file storage path
abortFile=/app/rocketmq/store/abort 
#Limited message size 
maxMessageSize=65536 
#flushCommitLogLeastPages=4 
#flushConsumeQueueLeastPages=2 
#flushCommitLogThoroughInterval=10000 
#flushConsumeQueueThoroughInterval=60000 
#Role of Broker 
#- ASYNC_MASTER asynchronous replication master 
#- SYNC_MASTER synchronous double write master 
#- SLAVE 
brokerRole=ASYNC_MASTER 
#Disc brushing mode 
#- ASYNC_FLUSH asynchronous brush disk
#- SYNC_FLUSH synchronous brush disc 
flushDiskType=ASYNC_FLUSH 
#checkTransactionMessageEnable=false 
#Number of messaging thread pools 
#sendMessageThreadPoolNums=128 
#Number of pull message thread pools 
#pullMessageThreadPoolNums=128

The slave node corresponding to this node is on worker3. To modify 2m-2s-async/broker-a-s.properties, you only need to modify the brokerId and brokerole

#Cluster name. Nodes with the same name are in the same cluster 
brokerClusterName=rocketmq-cluster 
#broker name. Nodes with the same name are a group of master-slave nodes. 
brokerName=broker-a 
#Broker ID, 0 means Master, and > 0 means Master 
Slave brokerId=1
#nameServer address, semicolon split 
namesrvAddr=worker1:9876;
worker2:9876;worker3:9876 
#When sending messages, topic s that do not exist in the server are automatically created, and the number of queues created by default 
defaultTopicQueueNums=4 
#Whether to allow the Broker to automatically create topics. It is recommended to open offline and close online
 autoCreateTopicEnable=true 
#Whether to allow the Broker to automatically create subscription groups. It is recommended to open offline and close online 
autoCreateSubscriptionGroup=true 
#Listening port of Broker external service 
listenPort=10911 
#Delete file time point. The default is 4 a.m 
deleteWhen=04 
#File retention time, 48 hours by default 
fileReservedTime=120 
#commitLog the default size of each file is 1G 
mapedFileSizeCommitLog=1073741824 
#ConsumeQueue stores 30W files by default, which can be adjusted according to business conditions 
mapedFileSizeConsumeQueue=300000 
#destroyMapedFileIntervalForcibly=120000 
#redeleteHangedFileInterval=120000 
#Detect physical file disk space 
diskMaxUsedSpaceRatio=88 
#Storage path 
storePathRootDir=/app/rocketmq/store
 #commitLog storage path 
storePathCommitLog=/app/rocketmq/store/commitlog 
#Consumption queue storage path storage path 
storePathConsumeQueue=/app/rocketmq/store/consumequeue 
#Message index storage path 
storePathIndex=/app/rocketmq/store/index 
#checkpoint file storage path 
storeCheckpoint=/app/rocketmq/store/checkpoint 
#abort file storage path
abortFile=/app/rocketmq/store/abort 
#Limited message size 
maxMessageSize=65536 
#flushCommitLogLeastPages=4 
#flushConsumeQueueLeastPages=2 
#flushCommitLogThoroughInterval=10000 
#flushConsumeQueueThoroughInterval=60000 
#Role of Broker 
#- ASYNC_MASTER asynchronous replication master 
#- SYNC_MASTER synchronous double write master 
#- SLAVE 
brokerRole=SLAVE 
#Disc brushing mode 
#- ASYNC_FLUSH asynchronous brush disk
#- SYNC_FLUSH synchronous brush disc 
flushDiskType=ASYNC_FLUSH 
#checkTransactionMessageEnable=false 
#Number of messaging thread pools 
#sendMessageThreadPoolNums=128 
#Number of pull message thread pools 
#pullMessageThreadPoolNums=128

Configure the second set of Broker-b

The primary node of this group of brokers is on worker3, so you need to configure config / 2m-2s-async / broker B properties

#Cluster name. Nodes with the same name are in the same cluster 
brokerClusterName=rocketmq-cluster 
#broker name. Nodes with the same name are a group of master-slave nodes. 
brokerName=broker-b 
#Broker ID, 0 means Master, and > 0 means Slave 
brokerId=0 
#nameServer address, semicolon split 
namesrvAddr=worker1:9876;worker2:9876;worker3:9876 
#When sending messages, topic s that do not exist in the server are automatically created, and the number of queues created by default 
defaultTopicQueueNums=4 
#Whether to allow the Broker to automatically create topics. It is recommended to open offline and close online 
autoCreateTopicEnable=true 
#Whether to allow the Broker to automatically create subscription groups. It is recommended to open offline and close online 
autoCreateSubscriptionGroup=true 
#Listening port of Broker external service 
listenPort=10911 
#Delete file time point. The default is 4 a.m 
deleteWhen=04 
#File retention time, 48 hours by default 
fileReservedTime=120 
#commitLog the default size of each file is 1G 
mapedFileSizeCommitLog=1073741824 
#By default, there are 30W files in consumqueue. Adjust 1234567891011121314151617181920212223 according to business conditions
 Then he corresponds slave stay worker2 Up,
modify work2 Upper conf/2m-2s-async/broker-b-s.properties
mapedFileSizeConsumeQueue=300000 
#destroyMapedFileIntervalForcibly=120000 
#redeleteHangedFileInterval=120000 
#Detect physical file disk space 
diskMaxUsedSpaceRatio=88 
#Storage path 
storePathRootDir=/app/rocketmq/store 
#commitLog storage path 
storePathCommitLog=/app/rocketmq/store/commitlog 
#Consumption queue storage path storage path 
storePathConsumeQueue=/app/rocketmq/store/consumequeue 
#Message index storage path 
storePathIndex=/app/rocketmq/store/index 
#checkpoint 
File storage path 
storeCheckpoint=/app/rocketmq/store/checkpoint 
#abort file storage path 
abortFile=/app/rocketmq/store/abort 
#Limited message size 
maxMessageSize=65536 
#flushCommitLogLeastPages=4 
#flushConsumeQueueLeastPages=2 
#flushCommitLogThoroughInterval=10000 
#flushConsumeQueueThoroughInterval=60000 
#Role of Broker 
#- ASYNC_MASTER asynchronous replication master 
#- SYNC_MASTER synchronous double write master 
#- SLAVE brokerRole=ASYNC_MASTER 
#Disc brushing mode 
#- ASYNC_FLUSH asynchronous brush disk 
#- SYNC_FLUSH synchronous brush disc 
flushDiskType=ASYNC_FLUSH 
#checkTransactionMessageEnable=false
 #Number of messaging thread pools 
#sendMessageThreadPoolNums=128 
#Number of pull message thread pools 
#pullMessageThreadPoolNums=128

Then, the corresponding slave on worker2, and modify conf/2m-2s-async/broker-b-s.properties on work2

#Cluster name. Nodes with the same name are in the same cluster 
brokerClusterName=rocketmq-cluster 
#broker name. Nodes with the same name are a group of master-slave nodes. 
brokerName=broker-b 
#Broker ID, 0 means Master, and > 0 means Slave 
brokerId=1
#nameServer address, semicolon split 
namesrvAddr=worker1:9876;worker2:9876;worker3:9876 
#When sending messages, topic s that do not exist in the server are automatically created, and the number of queues created by default 
defaultTopicQueueNums=4 
#Whether to allow the Broker to automatically create topics. It is recommended to open offline and close online 
autoCreateTopicEnable=true 
#Whether to allow the Broker to automatically create subscription groups. It is recommended to open offline and close online 
autoCreateSubscriptionGroup=true 
#Listening port of Broker external service 
listenPort=10911 
#Delete file time point. The default is 4 a.m 
deleteWhen=04 
#File retention time, 48 hours by default 
fileReservedTime=120 
#commitLog the default size of each file is 1G 
mapedFileSizeCommitLog=1073741824 
#By default, there are 30W files in consumqueue. Adjust 1234567891011121314151617181920212223 according to business conditions
 Then he corresponds slave stay worker2 Up,
modify work2 Upper conf/2m-2s-async/broker-b-s.properties
mapedFileSizeConsumeQueue=300000 
#destroyMapedFileIntervalForcibly=120000 
#redeleteHangedFileInterval=120000 
#Detect physical file disk space 
diskMaxUsedSpaceRatio=88 
#Storage path 
storePathRootDir=/app/rocketmq/store 
#commitLog storage path 
storePathCommitLog=/app/rocketmq/store/commitlog 
#Consumption queue storage path storage path 
storePathConsumeQueue=/app/rocketmq/store/consumequeue 
#Message index storage path 
storePathIndex=/app/rocketmq/store/index 
#checkpoint 
File storage path 
storeCheckpoint=/app/rocketmq/store/checkpoint 
#abort file storage path 
abortFile=/app/rocketmq/store/abort 
#Limited message size 
maxMessageSize=65536 
#flushCommitLogLeastPages=4 
#flushConsumeQueueLeastPages=2 
#flushCommitLogThoroughInterval=10000 
#flushConsumeQueueThoroughInterval=60000 
#Role of Broker 
#- ASYNC_MASTER asynchronous replication master 
#- SYNC_MASTER synchronous double write master 
#- SLAVE 
brokerRole=SLAVE 
#Disc brushing mode 
#- ASYNC_FLUSH asynchronous brush disk 
#- SYNC_FLUSH synchronous brush disc 
flushDiskType=ASYNC_FLUSH 
#checkTransactionMessageEnable=false
 #Number of messaging thread pools 
#sendMessageThreadPoolNums=128 
#Number of pull message thread pools 
#pullMessageThreadPoolNums=128

This completes the broker configuration.

Configuration items needing attention:

  • The store directories of two instances on the same machine cannot be the same, otherwise the error Lock failed,MQ will be reported
    already started
  • The listenPort of two instances on the same machine cannot be the same. Otherwise, the error of port occupation will be reported. Nameserver does not need to be configured. Just start it directly. This also shows that nameserver is stateless.

Start RocketMQ

It's easier to start. Just call the script in the bin directory directly. Just pay attention to their JVM memory configuration before starting. The default configuration is relatively high.

Start nameServer first

Modify bin / runserver. On the three nodes SH, adjust the jvm memory configuration inside. Find the following line and adjust the memory

JAVA_OPT="${JAVA_OPT} -server -Xms512m -Xmx512m -Xmn256m - XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m"

Start nameServer directly on the three nodes.

nohup bin/mqadminsrv &

After the startup is completed, in nohup When you see this key log in out, it means that the startup is successful.

The Name Server boot success. serializeType=JSON

Using the jps instruction, you can see a NamesrvStartup process.
We can also see that RocketMQ is running on the runserver SH is the CMS garbage collection period used, while in runbroker The G1 garbage collection period is used in SH.

Restart the broker

The mqbroker instruction is used to start the broker. Just note that when starting the broker, you need to specify the corresponding configuration file through - c. Start the master node of broker-a and the slave node of broker-b on worker2

nohup ./mqbroker -c ../conf/2m-2s-async/broker-a.properties & 
nohup ./mqbroker -c ../conf/2m-2s-async/broker-b-s.properties &

Start the master node of broker-b and the slave node of broker-a on work3

nohup ./mqbroker -c ../conf/2m-2s-async/broker-b.properties & 
nohup ./mqbroker -c ../conf/2m-2s-async/broker-a-s.properties &

When starting a slave, if you encounter an error lock failed and MQ ready started, it is because multiple instances share the same storePath. In this case, you need to adjust the path of the store.

Startup status check

Using the jps instruction, you can see one NameSrvStartup process and two BrokerStartup processes. nohup. There is also a log of successful startup in out. Corresponding log file:

# Viewing nameServer logs 
tail -500f ~/logs/rocketmqlogs/namesrv.log 
# View broker logs 
tail -500f ~/logs/rocketmqlogs/broker.log 1234

Test the mqadmin administration tool

The source code of RocketMQ does not provide us with a console like Nacos or RabbitMQ. It only provides an mqadmin command to manage RocketMQ. The command is in the bin directory. The method of use is/ mqadmin {command} {args}

Command line quick validation

In the installation package of RocketMQ, a tool. Net is provided The SH tool can be used to quickly verify RocketMQ services on the command line. Enter the installation directory of RocketMQ on worker2:

  • Send message: 1000 messages will be sent by default
bin/tools.sh org.apache.rocketmq.example.quickstart.Producer 1
  • Receive message:
bin/tools.sh org.apache.rocketmq.example.quickstart.Consumer

Note that this is the official Demo, but in the official source code, these two classes do not specify nameserver, so there will be some problems in operation. To specify the nameserver address, you can configure an environment variable NAMESRV_ADDR, the nameserver address will be read by default. You can match it bash_ Or directly and temporarily specified in the profile.

export NAMESRV_ADDR='worker1:9876;worker2:9876;worker3:9876'

Then it can be executed normally. See the source code for how to read the NameServer address
org.apache.rocketmq.common.utils.NameServerAddressUtils

public static String getNameServerAddresses() { return System.getProperty("rocketmq.namesrv.addr", System.getenv("NAMESRV_ADDR")); }

This method is the default way to set the NameServer address in DefaultMQProducer. This rokcetmq namesrv. The addr attribute can be used in java Specified by setproperties, it can also be configured in the configuration file in SpringBoot. This tool SH encapsulates a simple environment for running RocketMQ. You can run other examples in the source code and then run it yourself
The example can also be executed in the lib directory of RocketMQ.

Set up management console

The RocketMQ source code does not provide a console, but a RocketMQ community extension project provides a console. Address: https://github.com/apache/rocketmq-externals
After downloading, enter the rocket console directory and compile it with maven

mvn clean package -Dmaven.test.skip=true

After compilation, get the jar package under target and execute it directly. But at this time, you should pay attention to the application The address of nameserver needs to be specified in properties. By default, this property is empty. You can add an application.xml file in the current directory of the jar package The properties file overrides one of the default properties in the jar package:

rocketmq.config.namesrvAddr=worker1:9876;worker2:9876;worker3:9876 1

Then execute:

java -jar rocketmq-console-ng-1.0.1.jar

After startup, you can access http://192.168.232.128:8080 See the management page
If you need to build a high availability cluster, you need to use Dleger to build a high availability cluster

Topics: Java Linux RocketMQ MQ

Hot Topics

©2024 Programmer Think