RocketMq cluster installation & spring boot integration

Posted by tommyinnn on Wed, 10 Nov 2021 20:32:06 +0100

catalogue

1, RocketMQ introduction

2, Noun interpretation

3, Installation

Installation mode

preparation

Build RocketMQ cluster in traditional way

4, Integrate Springboot

1, RocketMQ introduction

RocketMQ is a distributed messaging middleware, which was originally developed by Alibaba's messaging middleware team and applied to production systems on a large scale to meet the needs of online message accumulation. It was donated to Apache open source foundation as an incubation project at the end of 2016 and officially became Apache's top project in less than a year; In the early days, Alibaba developed a message system based on ActiveMQ. With the increase of the scale of business messages, the bottleneck gradually appeared. Later, it also considered Kafka. However, because it had no choice in terms of low latency and high reliability, it finally independently developed RocketMQ. Its performance in all aspects is better than the existing message queue. RocketMQ and Kafka are very similar in concept and principle, So it is often compared; By default, RocketMQ adopts the pull mode of long polling. A single machine supports tens of millions of levels of message accumulation, which can be well applied to mass message systems.

2, Noun interpretation

Message model: RocketMQ is mainly composed of producer, Broker and Consumer. Producer is responsible for producing messages, Consumer is responsible for consuming messages and Broker is responsible for storing messages. A Broker corresponds to a server in the actual deployment process. Each Broker can store messages of multiple topics, and messages of each Topic can also be stored in different brokers in pieces. Message Queue is used to store the physical address of messages. The message address in each Topic is stored in multiple message queues. The ConsumerGroup consists of multiple Consumer instances.

Message Producer: it is responsible for producing messages. Generally, the business system is responsible for producing messages. A message Producer will send the messages generated in the business application system to the broker server. RocketMQ provides a variety of sending methods, including synchronous sending, asynchronous sending, sequential sending and one-way sending. Both synchronous and asynchronous methods require the broker to return confirmation information, and one-way transmission is not required.

Message Consumer: it is responsible for consuming messages. Generally, the background system is responsible for asynchronous consumption. A message Consumer pulls a message from the Broker server and provides it to the application. From the perspective of user application, it provides two consumption forms: pull consumption and push consumption.

Topic: represents a collection of messages. Each topic contains several messages. Each message can only belong to one topic. It is the basic unit for RocketMQ message subscription.

Broker Server: the role of message relay, which is responsible for storing and forwarding messages. In the RocketMQ system, the proxy server is responsible for receiving and storing messages sent from producers and preparing for consumers' pull requests. The proxy server also stores metadata related to messages, including consumer groups, consumption progress offsets, and topic and queue messages.

Name Server: the name service acts as a provider for routing messages. Producers or consumers can find the corresponding Broker IP list of each topic through the name service. Multiple Namesrv instances form a cluster, but they are independent of each other without information exchange.

Pull Consumer: a type of Consumer consumption. Applications usually actively call the Consumer's pull message method to pull messages from the Broker server. The initiative is controlled by the application. Once the batch message is obtained, the application will start the consumption process.

Push Consumer: a type of Consumer consumption. In this mode, the Broker will actively push the data to the Consumer after receiving it. This consumption mode generally has high real-time performance.

Producer Group: a collection of producers of the same type. These producers send messages of the same type with the same sending logic. If a transaction message is sent and the original producer crashes after sending, the Broker server will contact other producer instances in the same Producer Group to commit or backtrack consumption.

Consumer Group: a collection of consumers of the same type, which usually consume the same type of messages and have the same consumption logic. Consumer groups make it very easy to achieve the goals of load balancing and fault tolerance in message consumption. Note that the consumer instances of the Consumer Group must subscribe to exactly the same Topic. RocketMQ supports two message modes: Clustering and Broadcasting.

Clustering: in the cluster consumption mode, each Consumer instance of the same Consumer Group allocates messages equally.

Broadcasting: in the broadcasting consumption mode, each Consumer instance of the same Consumer Group receives a full amount of messages.

Normal Ordered Message: in the normal ordered consumption mode, messages received by consumers through the same Message Queue (Topic partition, called Message Queue) are sequential, while messages received by different message queues may be non sequential.

Strictly Ordered Message: in the Strictly Ordered Message mode, all messages received by consumers are ordered.

Message: the physical carrier of information transmitted by the message system and the smallest unit of production and consumption data. Each message must belong to a subject. Each message in RocketMQ has a unique Message ID and can carry a Key with a business ID. The system provides the function of querying messages through Message ID and Key.

Tag: a flag set for messages. It is used to distinguish different types of messages under the same topic. Messages from the same business unit can set different labels under the same subject according to different business purposes. Tags can effectively maintain the clarity and consistency of the code, and optimize the query system provided by RocketMQ. Consumers can implement different consumption logic for different sub themes according to tag to achieve better scalability.

From the official website https://github.com/apache/rocketmq/blob/master/docs/cn/concept.md

3, Installation

Installation mode

Single Master mode: this mode is risky. Once the Broker is restarted or down, the whole service will be unavailable.

Multi Master mode:

A cluster has no Slave but all masters.

Advantages: the configuration is simple, and the downtime or restart maintenance of a single Master has no impact on the application. When the disk is configured as RAID10, even if the machine downtime is unrecoverable, because the RAID10 disk is very reliable, messages will not be lost (a small number of messages will be lost in asynchronous disk brushing, and none will be lost in synchronous disk brushing), and the performance is the highest;
Disadvantages: during the downtime of a single machine, messages that are not consumed on this machine cannot be subscribed before the machine is restored, and the real-time performance of messages will be affected.

Multi Master multi Slave mode - asynchronous replication:

Each Master is configured with a Slave. There are multiple pairs of Master Slave. The HA adopts asynchronous replication, and the active and standby have a short message delay (in milliseconds).

Advantages: even if the disk is damaged, very few messages are lost, and the real-time performance of messages will not be affected. At the same time, after the Master goes down, consumers can still consume from Slave. Moreover, this process is transparent to applications, does not need manual intervention, and the performance is almost the same as that of multi master mode;
Disadvantages: when the Master goes down and the disk is damaged, a small number of messages will be lost.

Multi Master multi Slave mode - synchronous double write:

Each Master is configured with a Slave. There are multiple pairs of Master Slave. The HA (dual machine cluster) adopts synchronous double write mode, that is, success is returned to the application only if the active and standby are successfully written

Advantages: there is no single point of failure in data and services. When the Master goes down, there is no message delay, and the service availability and data availability are very high;
Disadvantages: the performance is slightly lower than that of asynchronous replication mode (about 10% lower), and the RT (response time) for sending a single message will be slightly higher. In the current version, after the primary node goes down, the standby machine cannot automatically switch to the host.

This article uses multi Master multi Slave mode asynchronous replication to install

preparation

1. Because rocketmq is written in java language, you need to install the JDK environment first. This article uses the jdk17 installation reference Installation jdk17&jdk8 under linux system_ Familiar snail blog - CSDN blog_ linux Installation jdk17

2. Cluster diagram

The two servers are master-slave to each other

Build RocketMQ cluster in traditional way

1. Download package version 4.9.2

Apache Downloads

2. Upload to the server and unzip it. If unzipan is not installed, install the instructions

[root@localhost ~]# yum install unzip zip

3. Unzip file

#decompression
[root@localhost ~]# unzip rocketmq-all-4.9.2-bin-release.zip 
#Move to / usr/local and rename it rocketmq
[root@localhost ~]# mv rocketmq-4.9.2 /usr/local/rocketmq

4. Modify startup script

Note: because my jdk is 17, I need to modify the startup script. jdk8 does not need to be modified. The modified place is the garbage collector and the startup parameters (- xms256m - xmnx256m)

runbroker.sh

 #!/bin/sh
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#===========================================================================================
# Java Environment Setting
#===========================================================================================
error_exit ()
{
  echo "ERROR: $1 !!"
  exit 1
}
[ ! -e "$JAVA_HOME/bin/java" ] && JAVA_HOME=$HOME/jdk/java
[ ! -e "$JAVA_HOME/bin/java" ] && JAVA_HOME=/usr/java
[ ! -e "$JAVA_HOME/bin/java" ] && error_exit "Please set the JAVA_HOMEvariable in your environment, We need java(x64)!"
export JAVA_HOME
export JAVA="$JAVA_HOME/bin/java"
export BASE_DIR=$(dirname $0)/..



export CLASSPATH=.${JAVA_HOME}/jre/lib/ext:${BASE_DIR}/lib/*:${BASE_DIR}/conf:${CLASSPATH}
#export CLASSPATH=${BASE_DIR}/lib/rocketmq-broker-4.5.0.jar:${BASE_DIR}/lib/*:${BASE_DIR}/conf:${CLASSPATH}

#===========================================================================================
# JVM Configuration
#===========================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms256m -Xmx256m -Xmn128m"
JAVA_OPT="${JAVA_OPT} -XX:+UseG1GC -XX:G1HeapRegionSize=16m -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -XX:SoftRefLRUPolicyMSPerMB=0"
JAVA_OPT="${JAVA_OPT} -verbose:gc -Xlog:gc:/dev/shm/mq_gc_%p.log -XX:+PrintGCDetails"
JAVA_OPT="${JAVA_OPT} -XX:-OmitStackTraceInFastThrow"
JAVA_OPT="${JAVA_OPT} -XX:+AlwaysPreTouch"
JAVA_OPT="${JAVA_OPT} -XX:MaxDirectMemorySize=15g"
JAVA_OPT="${JAVA_OPT} -XX:-UseLargePages -XX:-UseBiasedLocking"
#JAVA_OPT="${JAVA_OPT} -Xdebug -Xrunjdwp:transport=dt_socket,address=9555,server=y,suspend=n"
JAVA_OPT="${JAVA_OPT} ${JAVA_OPT_EXT}"
JAVA_OPT="${JAVA_OPT} -cp ${CLASSPATH}"
numactl --interleave=all pwd > /dev/null 2>&1
if [ $? -eq 0 ]
then
   if [ -z "$RMQ_NUMA_NODE" ] ; then
     numactl --interleave=all $JAVA ${JAVA_OPT} $@
   else
     numactl --cpunodebind=$RMQ_NUMA_NODE --membind=$RMQ_NUMA_NODE $JAVA${JAVA_OPT} $@
   fi
else
     $JAVA ${JAVA_OPT} --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED $@
fi

runserver.sh

#!/bin/sh
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#===========================================================================================
# Java Environment Setting
#===========================================================================================
error_exit ()
{
  echo "ERROR: $1 !!"
  exit 1
}
[ ! -e "$JAVA_HOME/bin/java" ] && JAVA_HOME=$HOME/jdk/java
[ ! -e "$JAVA_HOME/bin/java" ] && JAVA_HOME=/usr/java
[ ! -e "$JAVA_HOME/bin/java" ] && error_exit "Please set the JAVA_HOMEvariable in your environment, We need java(x64)!"
export JAVA_HOME
export JAVA="$JAVA_HOME/bin/java"
export BASE_DIR=$(dirname $0)/..
export CLASSPATH=.:${BASE_DIR}/conf:${JAVA_HOME}/jre/lib/ext:${BASE_DIR}/lib/*
#===========================================================================================
# JVM Configuration
#===========================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms256m -Xmx256m -Xmn128m -XX:MetaspaceSize=64m -XX:MaxMetaspaceSize=160m"
# jdk17 may discard the CMS garbage collector and need to use G1 collector
JAVA_OPT="${JAVA_OPT} -XX:+UseG1GC -XX:G1HeapRegionSize=16m -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -XX:SoftRefLRUPolicyMSPerMB=0"
# JAVA_OPT="${JAVA_OPT} -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSClassUnloadingEnabled -XX:SurvivorRatio=8"
JAVA_OPT="${JAVA_OPT} -verbose:gc -Xlog:gc:/dev/shm/rmq_srv_gc.log -XX:+PrintGCDetails"
JAVA_OPT="${JAVA_OPT} -XX:-OmitStackTraceInFastThrow"
JAVA_OPT="${JAVA_OPT} -XX:-UseLargePages"
# JAVA_OPT="${JAVA_OPT} -Djava.ext.dirs=${JAVA_HOME}/jre/lib/ext:${BASE_DIR}/lib"
#JAVA_OPT="${JAVA_OPT} -Xdebug -Xrunjdwp:transport=dt_socket,address=9555,server=y,suspend=n"
JAVA_OPT="${JAVA_OPT} ${JAVA_OPT_EXT}"
JAVA_OPT="${JAVA_OPT} -cp ${CLASSPATH}"
$JAVA ${JAVA_OPT} $@

tools.sh

#!/bin/sh

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

#===========================================================================================
# Java Environment Setting
#===========================================================================================
error_exit ()
{
echo "ERROR: $1 !!"
exit 1
}

[ ! -e "$JAVA_HOME/bin/java" ] && JAVA_HOME=$HOME/jdk/java
[ ! -e "$JAVA_HOME/bin/java" ] && JAVA_HOME=/usr/java
[ ! -e "$JAVA_HOME/bin/java" ] && error_exit "Please set the JAVA_HOME variable in your environment, We need java(x64)!"

export JAVA_HOME
export JAVA="$JAVA_HOME/bin/java"
export BASE_DIR=$(dirname $0)/..
export CLASSPATH=${BASE_DIR}/lib/*:${BASE_DIR}/conf:.:${CLASSPATH}
export CLASSPATH=.${JAVA_HOME}/jre/lib/ext:${BASE_DIR}/lib/*:${BASE_DIR}/conf:${CLASSPATH}
#export CLASSPATH=.:${BASE_DIR}/conf:${CLASSPATH}
#echo "BASE_DIR:$BASE_DIR"
#echo "CLASSPATH:$CLASSPATH"

#===========================================================================================
# JVM Configuration
#===========================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms256m -Xmx256m -Xmn256m -XX:PermSize=128m-XX:MaxPermSize=128m"
JAVA_OPT="${JAVA_OPT} -cp ${CLASSPATH}"

$JAVA ${JAVA_OPT} $@

5. Modify the configuration file on server 1

broker-a.properties

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#The name of the cluster to which it belongs. The name of the same cluster is the same
brokerClusterName=rocketmq-cluster
#broker name
brokerName=broker-a
#0 means master > 0 means slave
brokerId=0
#The time when the file was deleted, 4 a.m
deleteWhen=04
#The default file retention time is 48 hours
fileReservedTime=168
#Asynchronous replication Master
brokerRole=ASYNC_MASTER
#Disc brushing mode, ASYNC_FLUSH = asynchronous disk brushing, SYNC_FLUSH = synchronous brush disc 
flushDiskType=ASYNC_FLUSH
#Listening port of Broker external service
listenPort=10911
#Nameserver address, where nameserver is a single server. If nameserver is a cluster of multiple servers, it is divided by semicolons (i.e. namesrvAddr=ip1:port1;ip2:port2;ip3:port3)
namesrvAddr=192.168.6.145:9876;192.168.6.146:9876
#The number of queues corresponding to each topic is 4 by default. In fact, the number of consumer instances should be referred to. If the value is too small, it is not conducive to consumer load balancing
defaultTopicQueueNums=8
#Whether to allow the Broker to automatically create a Topic. It is recommended to close the production
autoCreateTopicEnable=true
#Whether to allow the Broker to automatically create subscription groups. It is recommended to close production
autoCreateSubscriptionGroup=true
#Set the ip address of broker ip server 1
brokerIP1=192.168.6.145
#Storage path
storePathRootDir=/data/rocketmq/store-a
#commitLog storage path
storePathCommitLog=/data/rocketmq/store-a/commitlog
#Consumption queue storage path storage path
storePathConsumerQueue=/data/rocketmq/store-a/consumequeue
#Message index storage path
storePathIndex=/data/rocketmq/store-a/index
#checkpoint file storage path
storeCheckpoint=/data/rocketmq/store-a/checkpoint
#abort file storage path
abortFile=/data/rocketmq/store-a/abort
#commitLog the default size of each file is 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue stores 30W files by default, which can be adjusted according to business conditions
mapedFileSizeConsumeQueue=300000

broker-b-s.properties

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
brokerClusterName=rocketmq-cluster
brokerName=broker-b
#slave
brokerId=1
deleteWhen=04
fileReservedTime=168
brokerRole=SLAVE
flushDiskType=ASYNC_FLUSH
#Listening port of Broker external service
listenPort=11011
#Nameserver address, where nameserver is a single server. If nameserver is a cluster of multiple servers, it is divided by semicolons (i.e. namesrvAddr=ip1:port1;ip2:port2;ip3:port3)
namesrvAddr=192.168.6.145:9876;192.168.6.146:9876
#The number of queues corresponding to each topic is 4 by default. In fact, the number of consumer instances should be referred to. If the value is too small, it is not conducive to consumer load balancing
defaultTopicQueueNums=8
#Whether to allow the Broker to automatically create a Topic. It is recommended to close the production
autoCreateTopicEnable=true
#Whether to allow the Broker to automatically create subscription groups. It is recommended to close production
autoCreateSubscriptionGroup=true
#Set broker IP
brokerIP1=192.168.6.145
#Storage path
storePathRootDir=/data/rocketmq/store-b
#commitLog storage path
storePathCommitLog=/data/rocketmq/store-b/commitlog
#Consumption queue storage path storage path
storePathConsumerQueue=/data/rocketmq/store-b/consumequeue
#Message index storage path
storePathIndex=/data/rocketmq/store-b/index
#checkpoint file storage path
storeCheckpoint=/data/rocketmq/store-b/checkpoint
#abort file storage path
abortFile=/data/rocketmq/store-b/abort
#commitLog the default size of each file is 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue stores 30W files by default, which can be adjusted according to business conditions
mapedFileSizeConsumeQueue=300000

6. Modify the configuration file on server 2

broker-b.properties

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#Cluster name
brokerClusterName=rocketmq-cluster
#The name of the broker. Note that different configuration files here are filled in differently. For example, write broker-a in the a.properties file and broker-b in the b.properties file
brokerName=broker-b
#0 means Master, > 0 means Slave
brokerId=0
#Delete file time point. The default is 4 a.m
deleteWhen=04
#File retention time, 48 hours by default
fileReservedTime=168
#Role of Broker, ASYNC_MASTER = asynchronous replication master, SYNC_MASTER = synchronous double write master, SLAVE=slave node
brokerRole=ASYNC_MASTER
#Disc brushing mode, ASYNC_FLUSH = asynchronous disk brushing, SYNC_FLUSH = synchronous brush disc 
flushDiskType=SYNC_FLUSH
#Listening port of Broker external service
listenPort=10911
#Nameserver address, where nameserver is a single server. If nameserver is a cluster of multiple servers, it is divided by semicolons (i.e. namesrvAddr=ip1:port1;ip2:port2;ip3:port3)
namesrvAddr=192.168.6.145:9876;192.168.6.146:9876
#The number of queues corresponding to each topic is 4 by default. In fact, the number of consumer instances should be referred to. If the value is too small, it is not conducive to consumer load balancing
defaultTopicQueueNums=8
#Whether to allow the Broker to automatically create a Topic. It is recommended to close the production
autoCreateTopicEnable=true
#Whether to allow the Broker to automatically create subscription groups. It is recommended to close production
autoCreateSubscriptionGroup=true
#Set broker IP
brokerIP1=192.168.6.146
#Storage path
storePathRootDir=/data/rocketmq/store-b
#commitLog storage path
storePathCommitLog=/data/rocketmq/store-b/commitlog
#Consumption queue storage path storage path
storePathConsumerQueue=/data/rocketmq/store-b/consumequeue
#Message index storage path
storePathIndex=/data/rocketmq/store-b/index
#checkpoint file storage path
storeCheckpoint=/data/rocketmq/store-b/checkpoint
#abort file storage path
abortFile=/data/rocketmq/store-b/abort
#commitLog the default size of each file is 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue stores 30W files by default, which can be adjusted according to business conditions
mapedFileSizeConsumeQueue=300000

broker-a-s.properties

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#Cluster name
brokerClusterName=rocketmq-cluster
#The name of the broker. Note that different configuration files here are filled in differently. For example, write broker-a in the a.properties file and broker-b in the b.properties file
brokerName=broker-a
#0 means Master, > 0 means Slave
brokerId=1
#Delete file time point. The default is 4 a.m
deleteWhen=04
#File retention time, 48 hours by default
fileReservedTime=168
#Role of Broker, ASYNC_MASTER = asynchronous replication master, SYNC_MASTER = synchronous double write master, SLAVE=slave node
brokerRole=SLAVE
#Disc brushing mode, ASYNC_FLUSH = asynchronous disk brushing, SYNC_FLUSH = synchronous brush disc 
flushDiskType=SYNC_FLUSH
#Listening port of Broker external service
listenPort=11011
#Nameserver address, where nameserver is a single server. If nameserver is a cluster of multiple servers, it is divided by semicolons (i.e. namesrvAddr=ip1:port1;ip2:port2;ip3:port3)
namesrvAddr=192.168.6.145:9876;192.168.6.146:9876
#The number of queues corresponding to each topic is 4 by default. In fact, the number of consumer instances should be referred to. If the value is too small, it is not conducive to consumer load balancing
defaultTopicQueueNums=8
#Whether to allow the Broker to automatically create a Topic. It is recommended to close the production
autoCreateTopicEnable=true
#Whether to allow the Broker to automatically create subscription groups. It is recommended to close production
autoCreateSubscriptionGroup=true
#Set broker IP
brokerIP1=192.168.6.146
#Storage path
storePathRootDir=/data/rocketmq/store-a
#commitLog storage path
storePathCommitLog=/data/rocketmq/store-a/commitlog
#Consumption queue storage path storage path
storePathConsumerQueue=/data/rocketmq/store-a/consumequeue
#Message index storage path
storePathIndex=/data/rocketmq/store-a/index
#checkpoint file storage path
storeCheckpoint=/data/rocketmq/store-a/checkpoint
#abort file storage path
abortFile=/data/rocketmq/store-a/abort
#commitLog the default size of each file is 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue stores 30W files by default, which can be adjusted according to business conditions
mapedFileSizeConsumeQueue=300000

7. Start nameserver

Enter[ root@localhost ~]#CD / usr / local / rocketmq / bin / directory

Start nohup sh mqnamesrv&

Verify that the installation was successful

tail -f ~/logs/rocketmqlogs/namesrv.log

If an error is reported at this time

/usr/local/rocketmq/bin/runserver.sh: line 19: syntax error near unexpected token `$'\r''
'usr/local/rocketmq/bin/runserver.sh: line 19: `error_exit ()
Solution: the copied file format is disordered. Use the following command to see more ^ M behind the file
[root@bogon bin]# vim -b runserver.sh

Remove ^ M and restart
[root@bogon bin]# sed -i 's/\r//g' runserver.sh

8. Start broker

Start master on server 1

[root@bogon rocketmq]# nohup sh bin/mqbroker -c conf/2m-2s-async/broker-a.properties &

Start the master on server 2

[root@localhost rocketmq]# nohup sh bin/mqbroker -c conf/2m-2s-async/broker-b.properties &

Start the slave on server 1

[root@bogon rocketmq]# nohup sh bin/mqbroker -c conf/2m-2s-async/broker-b-s.properties &

Start the slave on server 2

[root@localhost rocketmq]# nohup sh bin/mqbroker -c conf/2m-2s-async/broker-a-s.properties &

9. Build visual management interface

maven package installation

mvn clean package -Dmaven.test.skip=true

java -jar target/rocketmq-console-ng-1.0.1.jar

Download installation package Rocketmq visualizer - Web server document class resources - CSDN Download

4, Integrate Springboot

maven dependency

<dependency>
    <groupId>org.apache.rocketmq</groupId>
    <artifactId>rocketmq-spring-boot-starter</artifactId>
    <version>2.2.1</version>
</dependency>

application.yml

server:
  port: 8090
rocketmq:
  name-server: 192.168.6.145:9876;192.168.6.146:9876
  producer:
    group: rocketmq-producer

producer

package com.xiaojie.rocket.rocket.producer;

import lombok.extern.slf4j.Slf4j;
import org.apache.rocketmq.client.producer.SendCallback;
import org.apache.rocketmq.client.producer.SendResult;
import org.apache.rocketmq.spring.core.RocketMQTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

/**
 * @Description: Send messages: synchronous messages, asynchronous messages, and one-way messages. The first two messages are reliable because there will be a response whether the transmission is successful or not
 * @author: yan
 * @date: 2021.11.08
 */
@Service
@Slf4j
public class MqProducer {
    @Autowired
    private RocketMQTemplate rocketMQTemplate;

    /**
     * @description: This method is mainly used in scenarios that do not particularly care about sending results, such as log sending.
     * @param:
     * @return: void
     * @author xiaojie
     * @date: 2021/11/9 23:39
     */
    public void sendMq() {
        for (int i = 0; i < 10; i++) {
            rocketMQTemplate.convertAndSend("xiaojie-test", "Test send message" + i);
        }
    }

    /**
     * @description: This reliable synchronous sending method is widely used, such as important message notification and short message notification.
     * @param:
     * @return: void
     * @author xiaojie
     * @date: 2021/11/10 22:25
     */
    public void sync() {
        SendResult sendResult = rocketMQTemplate.syncSend("xiaojie-test", "sync Send a message..........");
        log.info("Send results{}", sendResult);
    }

    /**
     * @description: Asynchronous messages are usually used in business scenarios that are sensitive to response time, that is, the sender cannot tolerate waiting for a Broker's response for a long time.
     * @param:
     * @return: void
     * @author xiaojie
     * @date: 2021/11/10 22:29
     */
    public void async() {
        String msg = "Send messages asynchronously..........";
        log.info(">msg:<<" + msg);
        rocketMQTemplate.asyncSend("xiaojie-test", msg, new SendCallback() {
            @Override
            public void onSuccess(SendResult var1) {

                log.info("Asynchronous sending succeeded{}", var1);
            }
            @Override
            public void onException(Throwable var1) {
                //Sending failed. You can retry
                log.info("Asynchronous sending failed{}", var1);
            }
        });
    }
}

consumer

package com.xiaojie.rocket.rocket.consumer;

import lombok.extern.slf4j.Slf4j;
import org.apache.rocketmq.spring.annotation.RocketMQMessageListener;
import org.apache.rocketmq.spring.core.RocketMQListener;
import org.springframework.stereotype.Component;

/**
 * @Description:
 * @author: yan
 * @date: 2021.11.08
 */
@RocketMQMessageListener(consumerGroup = "test-group", topic = "xiaojie-test")
@Slf4j
@Component
public class MqConsumer implements RocketMQListener<String> {

    @Override
    public void onMessage(String message) {
        log.info("The data received is:{}", message);
    }
}

reference resources: https://blog.csdn.net/javahongxi/article/details/84931747

Full code reference: Spring boot: spring boot integrates redis, message oriented middleware and other related codes

Topics: Java Spring Boot message queue RocketMQ

Programmer Think

RocketMq cluster installation & spring boot integration

1, RocketMQ introduction

2, Noun interpretation

3, Installation

Installation mode

preparation

Build RocketMQ cluster in traditional way

4, Integrate Springboot

Hot Topics