As shown in the title, this paper focuses on sequential messages, delayed messages and message filtering.

1, Sequential message

RocketMQ can only ensure the order of messages at the queue level. If you want to realize the sequential execution of a certain type of messages, you must send such messages to the same queue. You can use MessageQueueSelector when sending messages, and then send the same type of messages to the same queue by specifying the sharing key. In this way, the order of messages in the CommitLog file is consistent with that when sending messages. For the queue selected by the broker, please refer to the previous article: RocketMQ learning 5 - select queue and other features.

Let's talk about the processing of the consumer side.

The event listener for sequential message consumption is MessageListenerOrderly. We know that PullMessageService will pull a batch of messages according to the offset, store them in ProcessQueue, and then use thread pool for processing. To ensure that the consumer processes the messages in a single queue in sequence, it is necessary to lock the message consumption queue in a multi-threaded scenario. The concurrency of sequential consumption at the consumer end does not depend on the size of the process pool at the consumer end, but on the number of queues allocated to consumers. Therefore, if a Topic is used in the sequential consumption scenario, it is recommended that the number of queues of consumers be set to increase, which can be 2 ~ 3 times that of non sequential consumption, which is conducive to improving the concurrency of the consumer end and facilitating horizontal capacity expansion.

The horizontal expansion of the consumer side or the change of the number of queues at the Broker side will trigger the reload of the message consumption queue. During concurrent consumption, a consumption queue may be consumed by multiple consumers at the same time, but this will not happen during sequential consumption, because sequential messages will not only lock the message consumption queue when consuming messages, but also when assigned to the message queue, To pull messages from the queue, you also need to apply for the lock of the consumption queue on the Broker side, that is, only one consumer can pull messages from the queue at the same time, so as to ensure the semantics of sequential consumption.

technological process:

1. PullMessageService gets messages from the Broker in a single thread
2. PullMessageService adds the message to ProcessQueue (ProcessMessage is a message cache), and then submits a consumption task to ConsumeMessageOrderService
3. ConsumeMessageOrderService is executed in multiple threads. Each thread needs to get the lock of MessageQueue when consuming messages
4. After getting the lock, get the message from ProcessQueue

What if the consumption fails in the process of sequential consumption? In the concurrent consumption mode, there is a retry mechanism when consumption fails. The default retry is 16 times. When retrying, the message is first sent to the Broker and then pulled to the message again. This mechanism will lose the order of consumption. In addition, if a message cannot be consumed successfully, its message consumption progress will not be able to move forward, which will lead to message backlog. Therefore, we must catch exceptions during sequential consumption.

2, Delay message

In the open source version of RocketMQ, the delay message does not support any time delay. At present, the default setting is: 1s 5S 10s 30s 1m 2m 4m 6m 7m 8m 9m 10m 20m 30m 1H 2H. From 1s to 2h, it corresponds to levels 1 to 18 respectively. The paid version in Alibaba cloud can support any time within 40 days (millisecond level).

Flow chart of delay message:

1. Producer sets the delay level on the message sent by itself (for example, set the delay level of level 3: message.setDelayTimeLevel(3)).
2. The Broker finds that this message is a delay message (the delayLevel of the message is greater than 0), and replaces the Topic with a delay Topic(SCHEDULE_TOPIC_XXXX). Each delay level will be used as a separate queue(delayLevel-1), and its own Topic will be stored as additional information (in the CommitLog#putMessage method).
3. Building ConsumerQueue
4. The scheduled task scans the ConsumerQueue of each delay level every 1s.
5. Get the Offset of CommitLog in ConsumerQueue, get the message, and judge whether the execution time has been reached
6. If yes, the Topic of the message will be restored and re delivered. If it is not reached, the task is delayed for the period of time that it is not reached.

3, Message filtering

RocketMQ supports SQL filtering and TAG filtering.

• SQL filtering: it is carried out on the broker side, which can reduce the network transmission of useless data, but the broker will have high pressure and low performance. It supports the use of complex filtering logic of SQL statements.
• TAG filtering: it is carried out at the broker and consumer side to increase the network transmission of useless data, but the broker has low pressure and high performance, and only supports simple filtering.

SQL filtering will not be analyzed first. Please refer to the article: RocketMQ source code analysis: how is message filtering implemented?
The process of tag filtering is that the broker obtains the hashcode(tag) in the corresponding ConsuemrQueue and compares it according to the tag passed in by the consumer. If it does not match, this message will be skipped; If you match the consumer, you need to compare the tags again, because there may be hash conflicts.

Filtering on the broker side:

    //Query message entry
    public GetMessageResult getMessage(final String group, final String topic, final int queueId, final long offset,
        final int maxMsgNums,
        final MessageFilter messageFilter) {
 
        //tag filtering, in consumerQueue
       if (messageFilter != null
            && !messageFilter.isMatchedByConsumeQueue(isTagsCodeLegal ? tagsCode : null, extRet ? cqExtUnit : null)) {
             if (getResult.getBufferTotalSize() == 0) {
                  status = GetMessageStatus.NO_MATCHED_MESSAGE;
             }
                continue;
        }
    //tag filtering, in the commitlog
    if (messageFilter != null
        && !messageFilter.isMatchedByCommitLog(selectResult.getByteBuffer().slice(), null)) {
        if (getResult.getBufferTotalSize() == 0) {
             status = GetMessageStatus.NO_MATCHED_MESSAGE;
         }
         // release...
         selectResult.release();
         continue;
    }
}

consumer filtering:

public PullResult processPullResult(final MessageQueue mq, final PullResult pullResult,
        final SubscriptionData subscriptionData) {
        PullResultExt pullResultExt = (PullResultExt) pullResult;

        this.updatePullFromWhichNode(mq, pullResultExt.getSuggestWhichBrokerId());
        if (PullStatus.FOUND == pullResult.getPullStatus()) {
            ByteBuffer byteBuffer = ByteBuffer.wrap(pullResultExt.getMessageBinary());
            List<MessageExt> msgList = MessageDecoder.decodes(byteBuffer);

            List<MessageExt> msgListFilterAgain = msgList;
            if (!subscriptionData.getTagsSet().isEmpty() && !subscriptionData.isClassFilterMode()) {
                msgListFilterAgain = new ArrayList<MessageExt>(msgList.size());
                for (MessageExt msg : msgList) {
                    if (msg.getTags() != null) {
                        if (subscriptionData.getTagsSet().contains(msg.getTags())) {
                            msgListFilterAgain.add(msg);
                        }
                    }
                }
            }

            if (this.hasHook()) {
                FilterMessageContext filterMessageContext = new FilterMessageContext();
                filterMessageContext.setUnitMode(unitMode);
                filterMessageContext.setMsgList(msgListFilterAgain);
                this.executeHook(filterMessageContext);
            }

            ......

        }

        pullResultExt.setMessageBinary(null);

        return pullResult;
    }

Message filtering can also be realized through Topic. We can choose whether to use Topic or Tag filtering according to specific business scenarios. Generally speaking, there is no necessary connection between messages of different topics, while Tag is used to distinguish messages related to each other under the same Topic.

Reference articles
Sequence message reference: 13. Combined with the actual scenario, sequential consumption and message filtering practice
Talk about sequential messages (implementation mechanism of RocketMQ sequential messages)
Delay message reference: Rocketmq one topic multiple group s_ What you need to know about rocketmq
Message filtering reference: rocketMQ message Tag filtering principle