This article mainly involves the following contents when sending a message:

• Queue selection mechanism for sequential messages
• RocketMQ key
• RocketMQ tag
• RocketMQ msgId

Queue selection mechanism for sequential messages

In many business scenarios, it is necessary to ensure the sequential processing of messages. For example, when orders flow to different states, they will send messages to the same topic, but consumers want to process them according to the changing order of orders. If not controlled, messages will be sent to different queues in the topic, so consumers can't consume in sequence. This paper only analyzes how the Producer sends sequential messages, and the processing of the Consumer will be analyzed later.
We know that RocketMQ supports queue level sequential messages, so when sending messages, we just need to send the messages that need to be consumed in order to a queue. RocketMQ provides a user-defined queue load mechanism when sending messages. The default queue load mechanism for message sending is polling. How to select a queue? RocketMQ provides the following APIs (only an example of one API is given here):

SendResult send(final Message msg, final MessageQueueSelector selector, final Object arg)
        throws MQClientException, RemotingException, MQBrokerException, InterruptedException;

Use example:

public static void main(String[] args) throws UnsupportedEncodingException {
        try {
            MQProducer producer = new DefaultMQProducer("please_rename_unique_group_name");
            producer.start();

            String[] tags = new String[] {"TagA", "TagB", "TagC", "TagD", "TagE"};
            for (int i = 0; i < 100; i++) {
                int orderId = i % 10;
                Message msg =
                    new Message("TopicTestjjj", tags[i % tags.length], "KEY" + i,
                        ("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET));
                SendResult sendResult = producer.send(msg, new MessageQueueSelector() {
                    @Override
                    public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg) {
                        Integer id = (Integer) arg;
                        int index = id % mqs.size();
                        return mqs.get(index);
                    }
                }, orderId);

                System.out.printf("%s%n", sendResult);
            }

            producer.shutdown();
        } catch (MQClientException | RemotingException | MQBrokerException | InterruptedException e) {
            e.printStackTrace();
        }
    }

When sending messages, we specify the implementation of the selected queue, pass in the reference arg (which can be orderId or userId, etc.), and then take the module of the whole queue, so that the same arg can be sent to the same queue.

With regard to the sequence message, the following points need special attention:

1. If we use MessageQueueSelector, the retry mechanism of message sending will fail, that is, the RocketMQ client will not retry. The high availability of message sending needs to be guaranteed by the business party. One way is to store the message in the database after sending failure, then adjust the timing, and finally send the message to MQ.
2. When the asynchronous sending method is used and the retry is performed (RocketMQ itself will not retry when it is asynchronous, but the business party may retry). For example, there are three messages 1, 2 and 3, but if message 2 fails to resend, it may resend successfully after message 3. In this case, the order of messages cannot be achieved. Therefore, when using sequential messages, we should not use asynchronous sending, but use synchronous sending.
3. When the Broker goes down and restarts, the partition will be rebalanced. At this time, the partition obtained by the production end according to the key hash module will change. At this time, the sequence of short messages will be inconsistent. To solve this problem, if the business side cannot tolerate short-term sequence consistency, either the cluster will not be available immediately after the cluster fails, or the theme will be made into a single partition, but this will greatly sacrifice the high availability of the cluster, and the single partition will also greatly reduce the performance of the cluster

Use of RocketMQ key

RocketMQ provides rich message query mechanisms, such as using message offset, message globally unique msgId and message Key.

When sending a message, RocketMQ can set an index for a message. For example, in the above example, we specify "Key" + serial number as the Key of the message, so that we can query the message through the index Key.

If you need to specify a Key for a Message, you only need to pass in the Key parameter when building the Message, such as the following API:

public Message(String topic, String tags, String keys, byte[] body)

Use of RocketMQ tag

RocketMQ can set a Tag for the Topic, so that the consumer can filter the messages in the Topic based on the Tag, that is, selectively process the messages in the Topic.

For example, the whole life process of an order: creating an order, waiting for payment, payment completion, merchant approval, merchant shipment and buyer shipment. Each change of order status will be sent to the same subject order_topic sends messages, but different downstream systems only focus on messages in certain stages of the order flow, and do not need to process all messages. We can specify different tags for each of the above states, and the consumer also specifies the corresponding tag when subscribing to the message, so that consumers can only consume the messages with their own specified tag. The API is the same as the specified message key:

public Message(String topic, String tags, String keys, byte[] body)

Specify the tag API when the consumer subscribes:

void subscribe(final String topic, final String subExpression) throws MQClientException;

On the console, we can see that the consumption status of tags that do not meet the subscription is displayed as CONSUMED_BUT_FILTERED (consumed but filtered).

RocketMQ msgId

Let's first look at the code that generates msgId:

    public static String createUniqID() {
        StringBuilder sb = new StringBuilder(LEN * 2);
        sb.append(FIX_STRING);
        sb.append(UtilAll.bytes2string(createUniqIDBuffer()));
        return sb.toString();
    }

    public static String createUniqID() {
        StringBuilder sb = new StringBuilder(LEN * 2);
        sb.append(FIX_STRING);
        sb.append(UtilAll.bytes2string(createUniqIDBuffer()));
        return sb.toString();
    }

    private static byte[] createUniqIDBuffer() {
        ByteBuffer buffer = ByteBuffer.allocate(4 + 2);
        long current = System.currentTimeMillis();
        if (current >= nextStartTime) {
            setStartTime(current);
        }
        buffer.position(0);
        buffer.putInt((int) (System.currentTimeMillis() - startTime));
        buffer.putShort((short) COUNTER.getAndIncrement());
        return buffer.array();
    }

FIX_ What is string? In the static code block of MessageClientIDSetter class:

    static {
        LEN = 4 + 2 + 4 + 4 + 2;
        ByteBuffer tempBuffer = ByteBuffer.allocate(10);
        tempBuffer.position(2);
        tempBuffer.putInt(UtilAll.getPid());
        tempBuffer.position(0);
        try {
            tempBuffer.put(UtilAll.getIP());
        } catch (Exception e) {
            tempBuffer.put(createFakeIP());
        }
        tempBuffer.position(6);
        tempBuffer.putInt(MessageClientIDSetter.class.getClassLoader().hashCode());
        FIX_STRING = UtilAll.bytes2string(tempBuffer.array());
        setStartTime(System.currentTimeMillis());
        COUNTER = new AtomicInteger(0);
    }

The components are:

• The client sends IP and supports IPV4 and IPV6
• Process PID (2 bytes)
• hashcode of class loader (4 bytes)
• Difference between current system timestamp and startup timestamp (4 bytes)
• Auto increment sequence (2 bytes)

For each producer instance, the ip is unique, so the msgId generated by different producers will not be repeated. The distinguishing factor for a single instance of producer is: time + counter. msgId is unique when the application is not restarted. As long as the system clock remains unchanged after the application is restarted, msgId is also unique. Therefore, as long as the system clock does not dial back, we can ensure the global uniqueness of msgId.

The timestamp difference in the above components is the difference between the current timestamp and the timestamp of the previous month. If the application is restarted after running for a month, msgId will repeat. From the generation algorithm, yes! However, the message of MQ is time effective. The validity period is 72 hours, that is, 3 days. At 4 a.m. every day, rocketMQ will clear the expired messages. Therefore, msgId is also globally unique.

Finally, let's mention offsetMsgId
offsetMsgId refers to the physical offset of the Broker where the message is located, that is, the offset in the commitlog file. It consists of the following two parts:

• IP and port number of the Broker
• Physical offset in commitlog

We can locate the specific message according to the offsetMsgId without knowing the Topic and other information of the message.