Introduction to RocketMQ

For detailed explanation, you can view the following documents: rocketmq Basics

RocketMQ It is an open source distributed messaging system, based on high availability distributed cluster technology, which provides low latency and highly reliable message publishing and subscription services. At the same time, it is widely used in many fields, including asynchronous communication decoupling, enterprise solutions, financial payment, telecommunications, e-commerce, express logistics, advertising marketing, social networking, instant messaging, mobile applications, mobile games, video, Internet of things, Internet of vehicles, etc.

It has the following characteristics:

• It can ensure strict message order
• Provide rich message pull mode
• Efficient subscriber horizontal scalability
• Real time message subscription mechanism
• 100 million message accumulation capacity

How to select a RocketMQ implementation

Spring Cloud Stream

Spring Cloud Stream Is a framework for building message based microservice applications Spring Integration Connect with Broker.

Generally speaking, Message Queuing Middleware has a Broker Server (proxy server), a message relay role, which is responsible for storing and forwarding messages

For example, in RocketMQ, the Broker is responsible for receiving and storing messages sent from the producer and preparing for the pull request of the consumer. In addition, the Broker also stores metadata related to messages, including consumer groups, consumption progress offsets, topics and queue messages

Spring Cloud Stream provides a unified abstraction of message oriented middleware and introduces the unified concepts of publish subscribe, consumer groups and partition.

There are two internal concepts: cloud Binding and spring Binding

• Binder, a component integrated with message oriented middleware, is used to create a corresponding Binding. Each message oriented middleware has its own binder implementation

  • Kafka implements KafkaMessageChannelBinder
  • RabbitMQ implements RabbitMessageChannelBinder
  • RocketMQ implements RocketMQMessageChannelBinder

• Binding, including Input Binding and Output Binding. Binding provides a bridge between the message oriented middleware and the Provider and Consumer provided by the application, which enables the developer to only use the Provider or Consumer of the application to produce or consume data, and shields the contact between the developer and the underlying message oriented middleware

  

RocketMQ basic usage

This project demonstrates how to use RocketMQ Binder to subscribe and publish Spring Cloud application messages.

Source address: https://github.com/langyastudio/langya-tech/tree/master/spring-cloud

Dependent environment

Introduce Spring Cloud Alibaba RocketMQ related dependencies

<dependency>
    <groupId>com.alibaba.cloud</groupId>
    <artifactId>spring-cloud-starter-stream-rocketmq</artifactId>
</dependency>

Producer producer

configuration file

server:
  port: 28081

spring:
  application:
    name: rocketmq-produce

  cloud:
    stream:
      # Corresponding BindingProperties Map
      bindings:
        output-common:
          #RocketMQ Topic
          destination: topic-common-01
          content-type: application/json
        output-tx:
          destination: topic-tx-01
          content-type: application/json

      # Spring Cloud Stream RocketMQ configuration item
      rocketmq:
        # Corresponding RocketMQBinderConfigurationProperties class
        binder:
          name-server: 192.168.123.22:9876

        # RocketMQ custom Binding configuration item, corresponding to RocketMQBindingProperties Map
        bindings:
          output-common:
            # RocketMQ Producer configuration item, corresponding to RocketMQProducerProperties class
            producer:
              #Producer grouping
              group: group-common
              #Send messages synchronously
              sync: true
              #Maximum bytes
              maxMessageSize: 8249344
              #Timeout
              sendMessageTimeout: 3000
          tx-output:
            producer:
              #group name
              group: group-tx
              #Send transaction message
              transactional: true

logging:
  level:
    com:
      alibaba:
        cloud:
          stream:
            binder:
              rocketmq: DEBUG

• spring.cloud.stream.bindings is the Binding configuration item

  Binding can be divided into two types: Input and Output, but it cannot be reflected in the configuration item. Instead, it will be distinguished with @ Input or @Output annotation later

• spring.cloud.stream.rocketmq.binder is the configuration item of RocketMQ Binder.

  Name server: RocketMQ Namesrv address. The name service acts as a provider for routing messages. Producers or consumers can find the corresponding Broker IP list of each topic through the name service. Multiple Namesrv instances form a cluster, but they are independent of each other without information exchange.

@Output send message

public interface OutputSource
{
    @Output("output-common")
    MessageChannel sendCommon();

    @Output("output-tx")
    MessageChannel sendTx();
}

Through the @ Output annotation, an Output Binding named Output common is declared. Note that the name should be the same as spring. Com in the configuration file cloud. stream. The bindings configuration item corresponds to the

At the same time, the return result of the @ Output annotation method is of MessageChannel type, which can be used to send messages.

SenderService message sending service

The logic of Message sending is realized by assembling the Message body and calling the OutputSource interface

@Service
public class SenderService
{
    @Autowired
    private OutputSource source;

    /**
     * send message
     *
     * @param msg Message content
     * @param tag label
     * @param delay Set the consumption delay level to x seconds
     * @return
     * @throws Exception
     */
    public <T> boolean sendObject(T msg, String tag, Integer delay) throws Exception
    {
        Message<T> message = MessageBuilder.withPayload(msg)
                .setHeader(MessageConst.PROPERTY_TAGS, tag)
                .setHeader(MessageConst.PROPERTY_DELAY_TIME_LEVEL, delay)
                .setHeader(MessageHeaders.CONTENT_TYPE, MimeTypeUtils.APPLICATION_JSON)
                .build();

        return source.sendCommon().send(message);
    }

    public <T> boolean sendTransactionalMsg(T msg, int num) throws Exception
    {
        Message<T> message  = MessageBuilder.withPayload(msg)
                .setHeader("tx-state", String.valueOf(num))
                .setHeader(MessageConst.PROPERTY_TAGS, "binder")
                .setHeader(MessageHeaders.CONTENT_TYPE, MimeTypeUtils.APPLICATION_JSON)
                .build();

        return source.sendTx().send(message);
    }
}

Controller interface definition

Provides an HTTP interface for sending messages. The code is as follows:

@RestController
@RequestMapping("/produce")
public class ProduceController
{
    @Autowired
    private SenderService senderService;

    @GetMapping("/send1")
    public boolean output1(@RequestParam("msg") String msg) throws Exception
    {
        int msgId = new SecureRandom().nextInt();
        return senderService.sendObject(new FooMsg(msgId, msg), "tagObj", 0);
    }

    @GetMapping("/send3")
    public boolean output3() throws Exception
    {
        // unknown message
        senderService.sendTransactionalMsg("transactional-msg1", 1);
        // rollback message
        senderService.sendTransactionalMsg("transactional-msg2", 2);
        // commit message
        senderService.sendTransactionalMsg("transactional-msg3", 3);

        return true;
    }
}

Application entry

Start the application. The code is as follows:

@SpringBootApplication
@EnableBinding({OutputSource.class})
public class RocketMQProduceApplication
{
    public static void main(String[] args)
    {
        SpringApplication.run(RocketMQProduceApplication.class, args);
    }
}

Use the @ EnableBinding annotation to declare that the specified interface enables the Binding function and scans its @ Input and @ Output annotations.

consumer

configuration file

server:
  # Random port, easy to start multiple consumers
  port: ${random.int[10000,19999]}

spring:
  application:
    name: rocketmq-consume

  cloud:
    # Spring Cloud Stream configuration item, corresponding to BindingServiceProperties class
    stream:
      # Binding configuration item, corresponding to BindingProperties Map
      bindings:
        input-common-1:
          # destination. RocketMQ Topic is used here
          destination: topic-common-01
          content-type: application/json
          ## Consumer grouping, naming rules: group+topic name + xx
          group: group-common-1
        input-common-2:
          destination: topic-common-01
          content-type: application/json
          consumer:
            concurrency: 20
            maxAttempts: 1
          group: group-common-2
        input-common-3:
          destination: topic-common-01
          content-type: application/json
          consumer:
            concurrency: 20
          group: group-common-3
        input-tx-1:
          destination: topic-tx-01
          content-type: text/json
          consumer:
            concurrency: 5
          group: group-tx-1

      # Spring Cloud Stream RocketMQ configuration item
      rocketmq:
        binder:
          name-server: 192.168.123.22:9876

        bindings:
          input-common-1:
            # RocketMQ Consumer configuration item
            consumer:
              # Whether to enable consumption. The default value is true
              enabled: true
              # Whether to use broadcast consumption. The default value is false. Use cluster consumption. If you want to use broadcast consumption, set the value to true
              broadcasting: false
              orderly: true
          input-common-2:
            consumer:
              orderly: false
          input-common-3:
            consumer:
              tags: tagObj

spring.cloud.stream.rocketmq.bindings

• Enabled: whether to enable consumption. The default value is true. During daily development, if you do not want to consume in the local environment, you can turn it off by setting enabled to false

• broadcasting: whether to use broadcast consumption. The default value is false. Cluster consumption is used.

  Clustering: in the cluster consumption mode, each Consumer instance of the same Consumer Group allocates messages equally

  Broadcast consumption: in broadcast consumption mode, each Consumer instance of the same Consumer Group receives full messages

@Input receive message

public interface InputBinding
{
    @Input("input-common-1")
    SubscribableChannel inputCommon1();

    @Input("input-common-2")
    SubscribableChannel inputCommon2();

    @Input("input-common-3")
    SubscribableChannel inputCommon3();

    @Input("input-tx-1")
    SubscribableChannel inputTx1();
}

Through the @ Input annotation, an Input binding named Input-common-1 is declared. Note that the name should be the same as spring. Com in the configuration file cloud. stream. The bindings configuration item corresponds to the.

At the same time, the return result of the @ Input annotation method is of SubscribableChannel type, which can be used to subscribe to messages for consumption.

public interface SubscribableChannel extends MessageChannel {

 boolean subscribe(MessageHandler handler); // subscribe
 boolean unsubscribe(MessageHandler handler); // Unsubscribe

}

ReceiveService receive message service

The logic of message receiving is realized by assembling StreamListener

@Service
public class ReceiveService
{
    @StreamListener("input-common-1")
    public void receiveInput1(@Payload FooMsg receiveMsg)
    {
        System.out.println("input1 receive: " + receiveMsg);
    }

    @StreamListener("input-common-2")
    public void receiveInput2(@Payload FooMsg receiveMsg)
    {
        System.out.println("input2 receive: " + receiveMsg);
    }

    @StreamListener("input-common-3")
    public void receiveInput3(@Payload FooMsg receiveMsg)
    {
        System.out.println("input3 receive: " + receiveMsg);
    }

    @StreamListener("input-tx-1")
    public void receiveTransactionalMsg(String transactionMsg)
    {
        System.out.println("input tx receive transaction msg: " + transactionMsg);
    }

}

On the method, add @ StreamListener annotation to declare the corresponding Input Binding.

Because the consumed message is of POJO type, the @ Payload annotation needs to be added and the declaration needs to be deserialized into POJO objects.

Application entry

Start the application. The code is as follows:

@SpringBootApplication
@EnableBinding({InputBinding.class})
public class RocketMQConsumerApplication
{
    public static void main(String[] args)
    {
        SpringApplication.run(RocketMQConsumerApplication.class, args);
    }
}

Use the @ EnableBinding annotation to declare that the specified interface enables the Binding function and scans its @ Input and @ Output annotations

Test the scenario of single cluster and multiple instances

• Execute the ProducerApplication and start the instance of the producer
• Execute RocketMQConsumerApplication and start the instance of the consumer

After that, request http://localhost:28081/produce/send1?msg= Send sms-0248 Interface three times and send three messages. At this time, the consumer print log can be seen on the IDEA console as follows:

[onMessage][input-common-2 Thread number:99 Message content: FooMsg(id=-1996543838, bar=Send SMS-0248)] 
[onMessage][input-common-3 Thread number:98 Message content: FooMsg(id=-1996543838, bar=Send SMS-0248)] 
[onMessage][input-common-1 Thread number:97 Message content: FooMsg(id=-1996543838, bar=Send SMS-0248)] 

Meet expectations. It can be seen from the log that each message is consumed only once by the same group

Timing message

Timing message refers to a message that cannot be consumed by the Consumer immediately after it is sent to the Broker. It can only be consumed after a specific time

RocketMQ does not support arbitrary time precision delay, but has solidified 18 delay levels. The following table:

Delay level	time	Delay level	time	Delay level	time
1	1s	7	3m	13	9m
2	5s	8	4m	14	10m
3	10s	9	5m	15	20m
4	30s	10	6m	16	30m
5	1m	11	7m	17	1h
6	2m	12	8m	18	2h

Consumption retry

However, it should be noted that only in the cluster consumption mode can messages be retried

RocketMQ provides a mechanism for consumption retry. When the message consumption fails, RocketMQ will re deliver the message to the Consumer through the consumption retry mechanism, so that the Consumer has the opportunity to re consume the message and achieve successful consumption.

Of course, RocketMQ does not resend messages to consumers for consumption indefinitely. By default, when the number of retries reaches 16 and the Consumer still fails to consume, the message will enter the dead letter queue.

Max attempts times

Because the retry interval provided by Spring Cloud Stream is realized through sleep, which will occupy the current thread and affect the consumption speed of consumers, it is not recommended here. Therefore, set the max attempts configuration item to 1 to disable the retry function provided by Spring Cloud Stream and use the retry function provided by RocketMQ.

Delay level when next consumption policy

• -1: Do not repeat, directly into the dead letter queue
• 0: RocketMQ Broker control retry policy
• >0: RocketMQ Consumer control retry policy

The failure retry of each message has a certain interval. The first retry consumption starts with a delay level of 3. Therefore, the default is 16 retry consumption, which is also very understandable. After all, the highest delay level is 18.

Consumption exception handling mechanism

If the exception handling method succeeds and the exception is not thrown again, it will be deemed that the message has been consumed successfully, so the consumption retry will not be carried out.

Spring Cloud Stream provides a general consumption exception handling mechanism, which can intercept the exceptions that occur when consumers consume messages and carry out customized processing logic.

There are two ways to implement exception handling:

• Local exception handling: specify the error channel through subscription - < destination >< group>. errors
• Global exception handling: subscribe to global error Channel - errorChannel

When an exception occurs in the consumption message, an error message ErrorMessage will be sent to the corresponding error Channel. At the same time, all error channels are bridged to the global error channels defined by Spring Integration.

    //<destination>.<group>.errors
    //local error
    @ServiceActivator(inputChannel = "topic-common-01.group-common-3.errors")
    public void handleError(ErrorMessage errorMessage)
    {
        System.out.printf("[handleError][payload: %s] %n", ExceptionUtils.getRootCauseMessage(errorMessage.getPayload()));
        System.out.printf("[handleError][originalMessage: %s] %n", errorMessage.getOriginalMessage());
        System.out.printf("[handleError][headers: %s] %n", errorMessage.getHeaders());
    }
    // errorChannel
    // Global error
    @StreamListener(IntegrationContextUtils.ERROR_CHANNEL_BEAN_NAME)
    public void globalHandleError(ErrorMessage errorMessage)
    {
        System.out.printf("[globalHandleError][payload: %s] %n",
                          ExceptionUtils.getRootCauseMessage(errorMessage.getPayload()));
        System.out.printf("[globalHandleError][originalMessage: %s] %n", errorMessage.getOriginalMessage());
        System.out.printf("[globalHandleError][headers: %s] %n", errorMessage.getHeaders());
    }

Broadcast message

In broadcast consumption mode, each Consumer instance of the same Consumer Group receives a full amount of messages.

For example, we implement IM chat based on WebSocket. When we actively send messages to users, because we don't know which application provides WebSocket users are connected to, we can broadcast consumption through RocketMQ. Each application can judge whether the current user is connected to the WebSocket service provided by itself. If so, push messages to users.

Set the broadcasting configuration item to true

Sequential message

RocketMQ provides two sequential levels:

• Normal sequential messages: the Producer sends the associated messages to the same message queue
• Complete strict order: Based on the normal order message, the Consumer consumes in strict order

Message order refers to that when a kind of message is consumed, it can be consumed according to the sending order. For example, an order generates three messages: order creation, order payment and order completion. It is only meaningful to consume in this order, but at the same time, orders can be consumed in parallel. RocketMQ can strictly guarantee the order of messages.

The partition order is a normal order message, and the global order is a completely strict order

• Global Order: for a specified Topic, all messages are published and consumed in strict FIFO order. Applicable scenarios: scenarios where performance requirements are not high and all messages are released and consumed in strict accordance with FIFO principles
• Partition order: for a specified Topic, all messages are partitioned according to the Sharding key. Messages in the same partition are published and consumed in strict FIFO order. Sharding key is a Key field used to distinguish different partitions in sequential messages. It is a completely different concept from the Key of ordinary messages. Applicable scenario: the scenario with high performance requirements, using Sharding key as the partition field, and strictly following the FIFO principle for message publishing and consumption in the same block

Partition order:

• producer

  Add the partition key expression configuration item and set the Sharding key of the sequence message sent by the Producer

  sync: true whether to send messages synchronously. The default is false asynchronous

  server:
    port: 28081
  
  spring:
    application:
      name: rocketmq-produce
  
    cloud:
      stream:
        # Spring Cloud Stream configuration item, corresponding to BindingProperties Map
        bindings:
          output-common:
            #RocketMQ Topic
            destination: topic-common-01
            content-type: application/json
  
            # Producer configuration item, corresponding to ProducerProperties class
            producer:
              # Partition key expression. This expression is based on Spring EL and obtains the partition key from the message
              partition-key-expression: payload['id']
  
        # Spring Cloud Stream RocketMQ configuration item
        rocketmq:
          # RocketMQ Binder configuration item, corresponding to RocketMQBinderConfigurationProperties class
          binder:
            name-server: 192.168.123.22:9876
  
          # RocketMQ custom Binding configuration item, corresponding to RocketMQBindingProperties Map
          bindings:
            output-common:
              # RocketMQ Producer configuration item, corresponding to RocketMQProducerProperties class
              producer:
                #Producer grouping
                group: group-common
                #Send messages synchronously
                sync: true
  

• consumer

  Add the order configuration item and set the order consumption message of the Consumer

  server:
    # Random port, easy to start multiple consumers
    port: ${random.int[10000,19999]}
  
  spring:
    application:
      name: rocketmq-consume
  
    cloud:
      # Spring Cloud Stream configuration item, corresponding to BindingServiceProperties class
      stream:
        # Binding configuration item, corresponding to BindingProperties Map
        bindings:
          input-common-1:
            #retry count
            max-attempts: 1
            # destination. RocketMQ Topic is used here
            destination: topic-common-01
            content-type: application/json
            ## Consumer grouping, naming rules: group+topic name + xx
            group: group-common-1
  
        # Spring Cloud Stream RocketMQ configuration item
        rocketmq:
          binder:
            name-server: 192.168.123.22:9876
  
          bindings:
            input-common-1:
              # RocketMQ Consumer configuration item
              consumer:
                # Receive messages sequentially
                orderly: true
  

Message filtering

RocketMQ provides two ways to filter messages for consumers:

• Tag based filtering

  Tag: a flag set for messages, which is used to distinguish different types of messages under the same topic. Messages from the same business unit can set different labels under the same subject according to different business purposes. Tags can effectively maintain the clarity and consistency of the code and optimize the query system provided by RocketMQ. Consumers can realize different consumption logic for different sub themes according to tag to achieve better scalability

• be based on SQL92 Filtering, example: https://www.jianshu.com/p/5b13868f4451

Tag filtering is common and needs to be set as follows:

• producer

  Send a message with tag, such as:

  public <T> boolean sendObject(T msg, String tag) throws Exception
  {
      Message<T> message = MessageBuilder.withPayload(msg)
              .setHeader(MessageConst.PROPERTY_TAGS, tag)
              .setHeader(MessageHeaders.CONTENT_TYPE, MimeTypeUtils.APPLICATION_JSON)
              .build();
  
      return source.sendCommon().send(message);
  }
  

• consumer

  Subscribe to tag filtered messages, such as:

  server:
    # Random port, easy to start multiple consumers
    port: ${random.int[10000,19999]}
  
  spring:
    application:
      name: rocketmq-consume
  
    cloud:
      # Spring Cloud Stream configuration item, corresponding to BindingServiceProperties class
      stream:
        # Binding configuration item, corresponding to BindingProperties Map
        bindings:
          input-common-3:
            destination: topic-common-01
            content-type: application/json
            consumer:
              concurrency: 20
            group: group-common-3
  
        # Spring Cloud Stream RocketMQ configuration item
        rocketmq:
          binder:
            name-server: 192.168.123.22:9876
  
          bindings:
            input-common-3:
              consumer:
                # Based on Tag subscription, multiple tags are separated by 𞓜 and empty by default
                tags: tagObj
  

Transaction message

In the distributed message queue, RocketMQ is the only one that provides complete transaction messages. On this point, we can still advocate.

Consider an extreme case. When the local database transaction has been committed, if the transaction message is not committed due to network reasons or crashes, which eventually leads to the loss of this transaction message and problems in distributed transactions.

In contrast, RocketMQ provides a transaction check back mechanism. If the application fails to commit or rollback this transaction message for a certain period of time, RocketMQ will actively check the application and ask whether the transaction message is commit or rollback, so as to realize that the state of the transaction message can be committed or rollback finally, so as to achieve the consistency of the final transaction.

configuration option

RocketMQ Binder Properties

• spring.cloud.stream.rocketmq.binder.name-server

  RocketMQ NameServer address (namesrv addr configuration item is used in the old version). Default: 127.0.0.1:9876

• spring.cloud.stream.rocketmq.binder.access-key

  Alicloud account AccessKey. Default: null

• spring.cloud.stream.rocketmq.binder.secret-key

  Alibaba cloud account secret key. Default: null

• spring.cloud.stream.rocketmq.binder.enable-msg-trace

  Whether to enable message trace function for Producer and Consumer Default: true

• spring.cloud.stream.rocketmq.binder.customized-trace-topic

  The name of the topic stored after the message track is turned on. Default: RMQ_SYS_TRACE_TOPIC

RocketMQ Provider Properties

The following configurations are based on spring cloud. stream. rocketmq. bindings.< channelName>. producer. Configuration related to RocketMQ Producer with prefix.

• enable

  Whether to enable Producer. Default value: true

• group

  Producer group name. Default: empty

• maxMessageSize

  The maximum number of bytes sent by the message. Default: 8249344

• transactional

  Whether to send a transaction message. Default: false

• sync

  Whether to send messages in synchronization. Default: false

• vipChannelEnabled

  Whether to send messages on Vip Channel. Default value: true

• sendMessageTimeout

  The timeout (in milliseconds) for sending a message. Default: 3000

• compressMessageBodyThreshold

  Message body compression threshold (when the message body exceeds 4k, it will be compressed). Default: 4096

• retryTimesWhenSendFailed

  In the mode of sending messages synchronously, the number of retries of message sending failure. Default: 2

• retryTimesWhenSendAsyncFailed

  In the mode of sending messages asynchronously, the number of retries that failed to send messages. Default: 2

• retryNextServer

  Whether to retry other broker s when message sending fails. Default: false

RocketMQ Consumer Properties

The following configurations are based on spring cloud. stream. rocketmq. bindings.< channelName>. consumer. Configuration related to RocketMQ Consumer with prefix.

• enable

  Whether to enable Consumer. Default value: true

• tags

  The Consumer subscribes based on TAGS, and multiple TAGS are divided by 𞓜. Default: empty

• sql

  Consumer is based on SQL subscription. Default: empty

• broadcasting

  Whether Consumer is a broadcast consumption mode. If you want all subscribers to receive messages, you can use broadcast mode. Default: false

• orderly

  Whether the Consumer synchronizes the consumption message mode. Default: false

• delayLevelWhenNextConsume

  Retry strategy for consumption failure in asynchronous consumption message mode: - 1, no repetition, directly put into dead letter queue 0,broker control retry strategy > 0, client control retry strategy default value: 0

• suspendCurrentQueueTimeMillis

  In the synchronous consumption message mode, the time interval of re consumption after consumption failure. Default: 1000

reference resources

Rocketmq docker installation

Spring Cloud Alibaba RocketMQ

RocketMQ Example

Integration of RocketMQ and Spring Cloud Stream