Sping boot integrated rocketmq multi consumer

Posted by jocknerd on Wed, 24 Jun 2020 04:34:39 +0200

1, rocketmq installation

Installation media Download
Download address: http://rocketmq.apache.org/release_notes/release-notes-4.7.0/
Download binary installation package
Upload to the server for decompression

Start command:

cd /usr/local/rocketmq/

# 1. Start mqnamesrv first
#Start command
nohup sh bin/mqnamesrv &
# Or add the output log
nohup sh bin/mqnamesrv  >  ~/logs/rocketmqlogs/namesrv.log  &
# Specify startup ip plus output log
nohup sh bin/mqnamesrv  -n "172.18.2.97:9876"  >  ./logs/namesrv.log  &

# Stop command
nohup sh bin/mqshutdown namesrv


# 2. Start broker
#startBroker.sh
#Note that the following ip address and port please fill in the ip address and port of the rocketmq machine you installed.
nohup sh bin/mqbroker -c conf/broker.conf  -n xxx.xx.xx.xx:9876  >  ~/logs/rocketmqlogs/broker.log &
# view log
tail  -f  ~/logs/rocketmqlogs/broker.log
#stopBroker.sh
nohup sh bin/mqshutdown broker

ps: rocketmq obtains the wrong ip in the multi network card, virtual network and docker. You can add the specified ip in the startup mode, and then view the acquired ip in the console
Specify broker memory size: modify profile broker.conf

vi rocketmq-all-4.7.0-bin-release/bin/runbroker.sh

3. Port Description:
rocke has 9876
Non vip channel port: 10911
vip channel port: 10909
10909 is the corresponding port of VIP channel. You can close the VIP channel in the consumer object or producer object of JAVA without opening the 10909 port
If it is a broker cluster, 10912 should also be opened, otherwise the message of the master will not be copied to the slave node

2, Rocketmq console installation (web console)

     RocketMQ console is an extension of RocketMQ project. It is a graphical management console. It provides common functions such as Broker cluster status view, Topic management, Producer, Consumer status display, message query, etc. this function needs to be installed and run separately after RocketMQ is installed.

Download address: https://github.com/apache/rocketmq-externals
1. Enter project, modify configuration file application.properties Medium mq address and port
2. Compile package

$ mvn clean package -Dmaven.test.skip=true

start-up

$ java -jar target/rocketmq-console-ng-1.0.0.jar
#If the profile does not fill in the Name Server, you can specify namesrvAddr when you start the project
$ java -jar target/rocketmq-console-ng-1.0.0.jar --rocketmq.config.namesrvAddr='localhost:9876'

#Because namesrvAddr is configured during packaging, execute the following command
$ java -jar target/rocketmq-console-ng-1.0.0.jar

visit http://127.0.0.1:8080 / port can be modified in the above configuration file

At this point, rocketmq and console are installed.

FAQ summary

The startup script is configured with 8g memory by default. When the machine is less than memory, there will be errors when starting broker. Modify the startup configuration parameters according to the actual situation.
Multiple network cards get the wrong ip, start the nameserver and broker successfully, fail to connect, or check the cluster ip information in the consle. When the ip is abnormal, specify the start ip through the command.

1. Message sending retry mechanism

3. Message consumption mode

3, springboot producer integration rocketmq

1. Message retry mechanism

The message was not sent successfully. By default, it is retried 3 times.
Define the number of retries by setting retryTimesWhenSendFailed and the timeout by setting sendMsgTimeout

Producers write messages to message queues. Different business scenarios require producers to adopt different writing strategies. For example, synchronous sending, asynchronous sending, delayed sending, sending transaction messages, etc.

There are two ways to send messages: synchronous and asynchronous. The code above uses asynchronous mode. There are four return states of message sending: FLUSH_DISK_TIMEOUT , FLUSH_SLAVE_TIMEOUT ,SLAVE_NOT_AVAILABLE ,SEND_OK, different states have different meanings under different configurations of disk swiping strategy and synchronization strategy.

FLUSH_DISK_TIMEOUT: indicates that the disk brushing is not completed within the specified time (it is required that the Broker's disk brushing policy is set to SYNC_FLUSH will report this error).
FLUSH_SLAVE_TIMEOUT: indicates that the Broker is set to sync in the active / standby mode_ Master mode, master-slave synchronization is not completed within the set time.
SLAVE_NOT_AVAILABLE: the scene and flush generated by this state_ Slave_ Timeout is similar, which means that in the active standby mode, Broker is set to SYNC_MASTER, but no Broker configured as Slave was found.
SEN_ OK: indicates that the transmission is successful. For example, whether the message has been stored in the fusion disk? Whether the message has been synchronized to the Slave? Whether the message has been written to the disk in the Slave? It needs to be determined in combination with the configured flash disk policy and master-Slave policy. This state can also be simply understood as that the three problem States listed above have not occurred are SEND OK.

2. Synchronous and asynchronous mechanism of message sending

Three ways for producers to send
Synchronization mode
It refers to that after the message is sent to mq, it waits to receive the mq response result, and then it will send the next packet after receiving the response.
Application scenario
This method has a wide range of application scenarios, such as important notification email, enrollment SMS notification, marketing SMS system, etc.
Call the send method of DefaultMQProducer

try {
                    SendResult sendResult = producer.send(msg);
                    // Send message synchronously, as long as no exception is thrown, it is successful
                    if (sendResult != null) {
                        System.out.println(new Date() + " Send mq message success. Topic is:" + msg.getTopic() + " msgId is: " + sendResult.getMessageId());
                    }
                }
                catch (Exception e) {
                    // Failed to send message, need to retry processing, can resend this message or persist this data for compensation processing
                    System.out.println(new Date() + " Send mq message failed. Topic is:" + msg.getTopic());
                    e.printStackTrace();
                }

Asynchronous mode
Asynchronous sending refers to the communication mode in which the sender sends data and then sends the next packet without waiting for the receiver's mq response. After the message is sent, you do not need to wait for the server response to return and send the next message. The sender receives the server response through the callback interface and processes the response results.
Calling the send method of DefaultMQProducer requires the user to implement the asynchronous send callback interface (SendCallback)

 // Send the message asynchronously, and send the result to the client through callback.
            producer.sendAsync(msg, new SendCallback() {
                @Override
                public void onSuccess(final SendResult sendResult) {
                    // Message sent successfully
                    System.out.println("send message success. topic=" + sendResult.getTopic() + ", msgId=" + sendResult.getMessageId());
                }

                @Override
                public void onException(OnExceptionContext context) {
                    // Failed to send message, need to retry processing, can resend this message or persist this data for compensation processing
                    System.out.println("send message failed. topic=" + context.getTopic() + ", msgId=" + context.getMessageId());
                }
            });

            // Get msgId before callback returns
            System.out.println("send message async. topic=" + msg.getTopic() + ", msgId=" + msg.getMsgID());

One way delivery
The sender is only responsible for sending the message, does not wait for the response from the server and does not trigger the callback function, that is, only sends the request and does not wait for the response. The process of sending messages in this way takes a very short time, usually at the microsecond level.
Call sendOneway (msg) method

Application scenario
It is suitable for some scenarios with very short time consumption but low reliability requirements, such as log collection.

// Because there is no request response processing when sending messages in oneway mode, once the message sending fails, data will be lost because there is no retry. If the data cannot be lost, it is recommended to choose reliable synchronous or reliable asynchronous transmission mode
                producer.sendOneway(msg);

reference resources: https://help.aliyun.com/document_detail/29547.html

Integration start

pom file import dependency

<dependency>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-web</artifactId>
		</dependency>
<dependency>
		<groupId>org.apache.rocketmq</groupId>
		<artifactId>rocketmq-client</artifactId>
		<version>4.7.0</version>
</dependency>
<dependency>
		<groupId>org.apache.rocketmq</groupId>
		<artifactId>rocketmq-common</artifactId>
		<version>4.7.0</version>
</dependency>

Create configuration class

public class JmsConfig {
	/**
     * Name Server Address. Because it is a cluster deployment, multiple addresses are separated by semicolons
     */
    public static final String NAME_SERVER = "172.18.2.97:9876";
    /**
     * The topic name topic is generally set by the server and cannot create a new topic in the code (if it is not created, the producer will send a message to the topic and report that the topic error cannot be found)
     */
    public static final String TOPIC = "topic_family";
}

Create Producer

import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.client.producer.DefaultMQProducer;
import org.springframework.stereotype.Component;

@Component
public class Producer {
    private String producerGroup = "test_producer";
    private DefaultMQProducer producer;
    
    public Producer(){
        //Sample producer
        producer = new DefaultMQProducer(producerGroup);
        //If you don't open the vip channel, the port will be reduced by 2
        producer.setVipChannelEnabled(false);
        //Bind name server
        producer.setNamesrvAddr(JmsConfig.NAME_SERVER);
        // Set instance name
        producer.setInstanceName("quick_start_producer");
        // Set retry times, default 2
        producer.setRetryTimesWhenSendFailed(3);
        //Set the transmission timeout, which is 3000 by default
        producer.setSendMsgTimeout(6000);
        // Open producer
        start();
    }
    /**
     * Object must be called once before use, it can only be initialized once
     */
    public void start(){
        try {
            this.producer.start();
        } catch (MQClientException e) {
            e.printStackTrace();
        }
    }
  
    public DefaultMQProducer getProducer(){
        return this.producer;
    }
    /**
     * Generally in the application context, use the context listener to close
     */
    public void shutdown(){
        this.producer.shutdown();
    }
}

Create Controller class, and disclose http access creation message

import java.util.ArrayList;
import java.util.List;

import org.apache.rocketmq.client.producer.SendResult;
import org.apache.rocketmq.common.message.Message;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class SendMsgController {
	
	 	@Autowired
	    private Producer producer;

	    private List<String> mesList;

	    /**
	     * Initialization message
	     */
	    public SendMsgController() {
	        mesList = new ArrayList<String>();
	        mesList.add("11111");
	        mesList.add("22222");
	        mesList.add("33333");
	        mesList.add("44444");
	        mesList.add("55555");

	    }

	    @RequestMapping("/text/rocketmq")
	    public Object callback() throws Exception {
	        //Five messages in total
	        for (String s : mesList) {
	            //Create production information
	            Message message = new Message(JmsConfig.TOPIC, "testtag", ("get meg:" + s).getBytes());
	            //send out
	            SendResult sendResult = producer.getProducer().send(message);
	            System.out.println("Output producer information={"+sendResult+"}");
	        }
	        return "success";
	    }
}

Create the startup class RocketMqSendApp

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class RocketMqSendApp {
	public static void main( String[] args )
    {
        System.out.println( "Hello World!" );
        SpringApplication.run(RocketMqSendApp.class, args);
    }
}

Start class and access
http://127.0.0.1:8080/text/rocketmq

Output producer information = {SendResult [sendStatus=SEND_OK, msgId=0A81CCE06D0073D16E937884A5FB0000, offsetMsgId=AC12026100002A9F00000000009486AA, messageQueue=MessageQueue [topic=topic_family, brokerName=broker-a, queueId=0], queueOffset=11]}
Output producer information = {SendResult [sendStatus=SEND_OK, msgId=0A81CCE06D0073D16E937884A6240001, offsetMsgId=AC12026100002A9F000000000094875F, messageQueue=MessageQueue [topic=topic_family, brokerName=broker-a, queueId=1], queueOffset=10]}
Output producer information = {SendResult [sendStatus=SEND_OK, msgId=0A81CCE06D0073D16E937884A62D0002, offsetMsgId=AC12026100002A9F0000000000948814, messageQueue=MessageQueue [topic=topic_family, brokerName=broker-a, queueId=2], queueOffset=13]}
Output producer information = {SendResult [sendStatus=SEND_OK, msgId=0A81CCE06D0073D16E937884A6370003, offsetMsgId=AC12026100002A9F00000000009488C9, messageQueue=MessageQueue [topic=topic_family, brokerName=broker-a, queueId=3], queueOffset=11]}
Output producer information = {SendResult [sendStatus=SEND_OK, msgId=0A81CCE06D0073D16E937884A6400004, offsetMsgId=AC12026100002A9F000000000094897E, messageQueue=MessageQueue [topic=topic_family, brokerName=broker-a, queueId=0], queueOffset=12]}

Sending succeeded. The sending is synchronous.
Summary:
Send synchronously: sendresult sendresult= producer.send (msg);
Asynchronous send: producer.sendAsync(msg, new SendCallback() {});
One way delivery: producer.sendOneway(msg);

4, springboot Consumer Foundation rocketmq

*RocketMQ has two consumption modes: broadcast mode and CLUSTERING mode. The default is cluster consumption mode. If you need to switch the consumption mode, you need to set the following settings on the consumer side.

DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(groupName);
		consumer.setNamesrvAddr(namesrvAddr);
		consumer.setMessageModel(MessageModel.BROADCASTING);

Integration start

Create Consumer consumption class

import java.io.UnsupportedEncodingException;

import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
import org.apache.rocketmq.client.consumer.listener.MessageListenerConcurrently;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.common.consumer.ConsumeFromWhere;
import org.apache.rocketmq.common.message.Message;
import org.springframework.stereotype.Component;

@Component
public class Consumer {

    /**
     * Consumer entity object
     */
    private DefaultMQPushConsumer consumer;
    /**
     * Consumer group
     */
    public static final String CONSUMER_GROUP = "test_consumer";
    /**
     * Instantiate an object through a constructor
     */
    public Consumer() throws MQClientException {

        consumer = new DefaultMQPushConsumer(CONSUMER_GROUP);
        consumer.setNamesrvAddr(JmsConfig.NAME_SERVER);
        //Consumption mode: a new subscription group starts to consume at the last position of the queue for the first time, and then starts to consume at the last consumption progress
        consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
        //Subscription topics and tags (* for all tags)Next information
        consumer.subscribe(JmsConfig.TOPIC, "*");
        
        // Mass consumption, take 10 pieces each time
        consumer.setConsumeMessageBatchMaxSize(10);
        
        // //Register consumption monitoring and consumption information in this monitoring, and return consumption status information
        consumer.registerMessageListener((MessageListenerConcurrently) (msgs, context) -> {
            // Only message s with the same topic, the same tag and the same key are collected in msgs
            // Different messages will be placed in different queues
            try {
                for (Message msg : msgs) {
                    //The consumer obtains the message, but only outputs it without the subsequent logic processing
                    String body = new String(msg.getBody(), "utf-8");
                    System.out.println("Consumer-Get message-theme topic by={"+msg.getTopic()+"}, Consumption message is={"+body+"}");
                }
            } catch (UnsupportedEncodingException e) {
                e.printStackTrace();
                return ConsumeConcurrentlyStatus.RECONSUME_LATER;
            }
            return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
        });
        consumer.start();
        System.out.println("Consumer launched successfully=======");
    }
}

Create JmsConfig configuration class

import java.io.UnsupportedEncodingException;

import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
import org.apache.rocketmq.client.consumer.listener.MessageListenerConcurrently;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.common.consumer.ConsumeFromWhere;
import org.apache.rocketmq.common.message.Message;
import org.springframework.stereotype.Component;

@Component
public class Consumer {

    /**
     * Consumer entity object
     */
    private DefaultMQPushConsumer consumer;
    /**
     * Consumer group
     */
    public static final String CONSUMER_GROUP = "test_consumer";
    /**
     * Instantiate an object through a constructor
     */
    public Consumer() throws MQClientException {

        consumer = new DefaultMQPushConsumer(CONSUMER_GROUP);
        consumer.setNamesrvAddr(JmsConfig.NAME_SERVER);
        //Consumption mode: a new subscription group starts to consume at the last position of the queue for the first time, and then starts to consume at the last consumption progress
        consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
        //Subscription topics and tags (* for all tags)Next information
        consumer.subscribe(JmsConfig.TOPIC, "*");
        
        // Mass consumption, take 10 pieces each time
        consumer.setConsumeMessageBatchMaxSize(10);
        
        // //Register consumption monitoring and consumption information in this monitoring, and return consumption status information
        consumer.registerMessageListener((MessageListenerConcurrently) (msgs, context) -> {
            // Only message s with the same topic, the same tag and the same key are collected in msgs
            // Different messages will be placed in different queues
            try {
                for (Message msg : msgs) {
                    //The consumer obtains the message, but only outputs it without the subsequent logic processing
                    String body = new String(msg.getBody(), "utf-8");
                    System.out.println("Consumer-Get message-theme topic by={"+msg.getTopic()+"}, Consumption message is={"+body+"}");
                }
            } catch (UnsupportedEncodingException e) {
                e.printStackTrace();
                return ConsumeConcurrentlyStatus.RECONSUME_LATER;
            }
            return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
        });
        consumer.start();
        System.out.println("Consumer launched successfully=======");
    }
}

Create startup class

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class RocketMqReceiveApp {
	
	
	public static void main(String[] args) {
		System.out.println( "Hello World!" );
	    SpringApplication.run(RocketMqReceiveApp.class, args);
	}
}

establish application.yml Modify start port

server:
  port: ${PORT:8081}

Start program
Log output

Consumer launched successfully=======
Consumer get message topic = {topic_family}, consumption message = {get meg:11111 }
Consumer get message topic = {topic_family}, consumption message = {get meg:55555 }
Consumer get message topic = {topic_family}, consumption message = {get meg:44444 }
Consumer get message topic = {topic_family}, consumption message = {get meg:22222 }
Consumer get message topic = {topic_family}, consumption message = {get meg:33333 }

summary

At this point, the integration of rocket is completed. Now review the main points:

1, Producer send mode
1. Synchronous mode
2. Asynchronous mode
3. One way mode
4. Message sending failed retry configuration
5. Message sending timeout configuration
6. Message package size configuration
2, Consumer acceptance disappears
1. Cluster mode
2. Broadcast mode

Topics: Apache Java network SpringBoot

Programmer Think

Sping boot integrated rocketmq multi consumer

1, rocketmq installation

2, Rocketmq console installation (web console)

FAQ summary

3, springboot producer integration rocketmq

1. Message retry mechanism

2. Synchronous and asynchronous mechanism of message sending

Integration start

4, springboot Consumer Foundation rocketmq

Integration start

summary

Hot Topics