Summary of JAVA back-end development knowledge (continuous update...)

RocketMQ concept and usage

1, Overview

   MetaQ is a distributed message middleware with Queue model. Based on publish and subscribe mode, it supports two modes: Topic and Queue. There are two consumption modes: Push and Pull. It supports strict message sequence, 100 million level message stacking capacity, message backtracking and multi-dimensional message query.

   difference between MetaQ and RocketMQ: they are equivalent. They are called MetaQ 3.0 internally in Alibaba and RocketMQ 3.0 externally. Based on RocketMQ kernel, MetaQ adopts pull model, which mainly solves the problems of sequential messages and massive accumulation.

1.1 cluster architecture

• NameServer:
  • the lightweight domain name server specially designed for RocketMQ has the characteristics of simplicity, horizontal expansion of clusters, statelessness and no communication between nodes.
  • due to the heavy function of ZooKeeper, RocketMQ (i.e. MetaQ 3.x) removes the dependence on ZooKeeper and adopts its own NameServer.
  • only the final consistency needs to be maintained, which is the AP mode.
  • Broker cluster, Producer cluster and Consumer cluster all need to communicate with NameServer cluster.
• Broker:
  • message relay role, responsible for receiving messages, storing messages and forwarding messages.
  • a Broker cluster consists of multiple groups of Master/Slave. A Broker ID of 0 indicates a Master and a non-0 indicates a Slave.
  • when starting up, each Broker node will traverse and poll the NameServer list, establish a long connection with each NameServer, register its own information, and then report regularly.
• Consumer:
  • get the routing information of the Topic through the NameServer cluster, connect to the corresponding Broker, which is divided into Push Consumer / Pull Consumer, and regularly send heartbeat to the Master and Slave.
  • the former registers a Listener interface with the Consumer object, calls back the Listener interface method after receiving the message, and uses long polling to realize push.
  • the latter is actively pulled by the Consumer, the same as Kafka.
• Producer:
  • the message producer obtains the routing information of the Topic through the NameServer cluster, including which queues are under the Topic and which brokers these queues are distributed on.
  • Producer only sends messages to the Master node, so it only needs to establish a long connection with the Master node and send heartbeat to the Master regularly.

1.2 message domain model

• Message: company message;
• Topic: soft partition. When corresponding to the same topic, the partition ID of the producer corresponding to the consumer;
• Tag: secondary classification of messages based on topic;
• Message queue: hard partition, which physically distinguishes topics. One topic corresponds to multiple message queues;
• Group: Consumer Group is the collection name of a class of consumers. Such consumers usually consume a class of messages with the same consumption logic; Producer Group is the collection name of a class of producers. Such producers usually send a class of messages with the same sending logic;
• Offset: absolute offset value. There are two types of offsets (commitOffset and offset) in the message queue. The former is stored in the OffsetStore to represent the consumption location, and the latter is the pull message location in PullRequest;
• Broadcast consumption: Producer sends messages to some queues in turn. The queue set is called Topic. Each consumer instance consumes messages in all queues corresponding to this Topic;
• Cluster consumption: the average consumption of messages in the queue set corresponding to the topic Rebalance by multiple Consumer instances. Analysis of Kafka Rebalance mechanism.

1.3 precautions

1.3.1 publish and subscribe

• For important messages, the business party needs a retransmission compensation mechanism;
• For large messages, configure compression and decompression. It is recommended to split them:
  • MetaQ adopts RPC mode, which may lead to Buffer exceptions in the network layer;
  • the server storage is an LRU CACHE system, and too large messages will occupy more Cache;
• Set the Message Key attribute for unique id entification, such as various IDS;
• When sending a message, you can freely set the Tag to complete the filtering requirements of the subscriber;
• When subscribing to a Topic, you can filter messages (message tags) in the form of expressions;
• For non sequential message consumption, the time-consuming is not limited, and for sequential message consumption, the time-consuming is limited.

1.3.2 message repetition

• MetaQ cannot guarantee that messages are not repeated: timeout in distributed environment, short-term inconsistency in Rebalance, and unexpected downtime of subscribers;
• By de duplication such as DB or active pull, it can ensure that the pull message will not be repeated.

1.3.3 message retry

• For non sequential message consumption failure retry, you can specify the time to arrive at the Consumer next time. The number of consumption failure retries is limited. If the number of retries exceeds, the message enters the dead letter queue.
• The consumption of sequential messages fails. Try again. If the consumption of messages being consumed by a queue fails, the current queue will be suspended.

1.4 cluster

Dual master and dual Slave:

Workflow:

1. Start the NameServer. After the NameServer gets up, listen to the port and wait for the Broker, Producer and Consumer to connect.
2. The Broker starts, maintains a long connection with all nameservers, and sends heartbeat packets regularly. The heartbeat packet contains the current Broker information (IP + port, etc.) and all Topic information. After successful registration, there will be a mapping relationship between Topic and Broker in the NameServer cluster.
3. Create a Topic before sending and receiving messages. When you create a Topic, you need to specify which brokers the Topic will be stored on. You can also create a Topic automatically when sending messages.
4. Producer sends a message. When starting, it first establishes a long connection with one of the NameServer clusters, obtains the brokers of the currently sent topics from the NameServer, polls, selects a queue from the queue list, and then establishes a long connection with the Broker where the queue is located, so as to send a message to the Broker.
5. The Consumer establishes a long connection with one of the nameservers, obtains which brokers the currently subscribed Topic exists on, and then directly establishes a connection channel with the Broker to start consuming messages.

Cluster building steps

1.5 mqadmin cluster management tool

Slightly...

2, Basic use

• RocketMQ client dependency import

<dependency>
    <groupId>org.apache.rocketmq</groupId>
    <artifactId>rocketmq-client</artifactId>
    <version>4.4.0</version>
</dependency>

2.1 ordinary message sending

General steps:

1. Create a message Producer and formulate the Producer group name;
2. Specify Nameserver address;
3. Start Producer;
4. Create a message object and specify the Topic, Tag and message body;
5. Send message;
6. Close Producer.

be careful:

1. An application creates a Producer, and the Producer groupname needs to be unique by the application (global object or singleton).
2. By default, it will be automatically created when topic does not exist, and four messagequeues will be generated for topic.
3. The Producer object must be initialized once by calling start() before use (note that it is not every time a message is sent).
4. A Producer object can send(message)) messages of multiple topic s and tag s synchronously.
5. When the application exits, shut down.

• Send synchronization message

public class SyncProducer {

    public static void main(String[] args) throws Exception {
        // 1. Create a message Producer and make the Producer group name
        DefaultMQProducer producer = new DefaultMQProducer("group1");
        // 2. Specify Nameserver address: cluster configuration (use; isolation)
        producer.setNamesrvAddr("Server 1 public network IP:9876;Server 2 public network IP:9876");
        // 3. Start Producer
        producer.start();

        for (int i = 0; i < 10; i++) {
            // 4. Create a message object and specify the Topic, Tag and message body
            /**
             * Parameter 1: Message Topic
             * Parameter 2: message Tag
             * Parameter 3: message content: byte array
             */
            Message msg = new Message("base", "Tag1", ("Hello World" + i).getBytes());
            // 5. Send synchronization message
            //  Blocking waiting for return
            SendResult result = producer.send(msg);
            // Get send status
            SendStatus status = result.getSendStatus();
            // Message ID and receive queue ID
			String msgId = result.getMsgId();
			int queueId = result.getMessageQueue().getQueueId();
            System.out.println("Send results:" + result + msgId + queueId);

            // Thread sleep for 1 second
            TimeUnit.SECONDS.sleep(1);
        }
        // 6. Close the Producer
        producer.shutdown();
    }
}

• Send asynchronous message

  implement SendCallback() callback interface in send() method.

   asynchronous messages are usually used in business scenarios that are sensitive to response time, that is, the sender cannot tolerate waiting for a Broker's response for a long time.

public class AsyncProducer {

    public static void main(String[] args) throws Exception {
        // 1. Create a message Producer and make the Producer group name
        DefaultMQProducer producer = new DefaultMQProducer("group1");
        // 2. Specify Nameserver address
        producer.setNamesrvAddr("Server 1 public network IP:9876;Server 2 public network IP:9876");
        // 3. Start Producer
        producer.start();

        for (int i = 0; i < 10; i++) {
            // 4. Create a message object and specify the Topic, Tag and message body
            Message msg = new Message("base", "Tag2", ("Hello World" + i).getBytes());
            // 5. Send asynchronous message
            producer.send(msg, new SendCallback() {
                /**
                 * Send successful callback function
                 */
                public void onSuccess(SendResult sendResult) {
                    System.out.println("Send results:" + sendResult);
                }

                /**
                 * Send failed callback function
                 */
                public void onException(Throwable e) {
                    System.out.println("Send exception:" + e);
                }
            });

            // Thread sleep for 1 second
            TimeUnit.SECONDS.sleep(1);
        }

        // 6. Close the Producer
        producer.shutdown();
}

• Send one-way message

• This method is mainly used in scenarios that do not particularly care about sending results, such as log sending.
• Use sendOneway(msg) to send messages without returning results.

/**
 * Send one-way message
 */
public class OneWayProducer {

    public static void main(String[] args) throws Exception, MQBrokerException {
        // 1. Create a message Producer and make the Producer group name
        DefaultMQProducer producer = new DefaultMQProducer("group1");
        // 2. Specify Nameserver address
        producer.setNamesrvAddr("Server 1 public network IP:9876;Server 2 public network IP:9876");
        // 3. Start Producer
        producer.start();

        for (int i = 0; i < 3; i++) {
            // 4. Create a message object and specify the Topic, Tag and message body
            Message msg = new Message("base", "Tag3", ("Hello World，One way message" + i).getBytes());
            // 5. Send one-way message
            producer.sendOneway(msg);

            // Thread sleep for 1 second
            TimeUnit.SECONDS.sleep(5);
        }

        // 6. Close the Producer
        producer.shutdown();
    }
}

2.2 general message consumption

General steps:

1. Create a Consumer and make a Consumer group name;
2. Specify Nameserver address;
3. Subscribe to Topic and Tag;
4. Set callback function to process messages;
5. Start Consumer.

be careful:

1. An application creates a Consumer, and the Consumer groupname needs to be unique by the application (global object or singleton).
2. setConsumeFromWhere sets where to start consumption, and subscribe to Tags under the specified topic (wildcards are allowed).
3. setConsumeMessageBatchMaxSize() sets batch consumption.
4. If it is mass consumption, it will either succeed or fail.
5. Register the listener, customize the callback method, and execute it when the message arrives. This method is called back by RabbitMQ client multithreading, and needs to be applied to deal with concurrency safety problems.
6. If the message consumption is unsuccessful, there is a retry mechanism. When the threshold is reached, it will be discarded.
7. Call start() to initialize.

• Default load balancing mode - cluster consumption

• Consumers use load balancing to consume messages. Multiple consumers consume messages in a queue, and each consumer processes different messages.
• The current example is PushConsumer, which is pushed from the server to the client. In fact, the Consumer uses the long polling Pull method to Pull messages from the RabbitMQ server, and then calls back the Listener method.

public class Consumer {

    public static void main(String[] args) throws Exception {
        // 1. Create a Consumer and make a Consumer group name
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("group1");
        // 2. Specify Nameserver address
        consumer.setNamesrvAddr("39.96.86.4:9876;59.110.158.93:9876");
        // 3. Subscribe to Topic and Tag
        consumer.subscribe("base", "Tag1")
        
        consumer.registerMessageListener(new MessageListenerConcurrently() {
            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> list, ConsumeConcurrentlyContext consumeConcurrentlyContext) {
                System.out.println(list);
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        //5. Start consumer
        consumer.start();
    }
}

• Broadcast mode consumption message

Specify the mode through setMessageModel().

public class Consumer {

    public static void main(String[] args) throws Exception {
        // 1. Create a Consumer and make a Consumer group name
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("group1");
        // 2. Specify Nameserver address
        consumer.setNamesrvAddr("39.96.86.4:9876;59.110.158.93:9876");
        //  3. Subscribe to Topic and Tag
        consumer.subscribe("base", "Tag2");
		// Specify the consumption mode as broadcast mode
        consumer.setMessageModel(MessageModel.BROADCASTING);

        consumer.registerMessageListener(new MessageListenerConcurrently() {
            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> list, ConsumeConcurrentlyContext consumeConcurrentlyContext) {
                System.out.println(list);
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        // 5. Start Consumer
        consumer.start();
    }
}

2.3 queue messages

Message sequence problem:

1. RocketMQ can strictly guarantee the order of messages, which can be divided into queue order and global order (in the case of only one queue).
2. The premise of Broker storing messages in order is that the messages sent by Producer are in order:
   • a group of messages need to be sent by the same thread, but messages cannot be sent asynchronously. When sending synchronously, it can ensure that the Broker receives in order.
   • synchronization ensures that the previous message is sent successfully and the response is returned before sending the next one.
   • the same MessageQueue is selected for each transmission.
3. If the Consumer consumption is also sequential, it is necessary to ensure that a queue is consumed only in one thread.
4. Example: the sequential process of an order is to create, pay, push and complete. Messages with the same order number will be sent to the same queue successively. When consuming, the same OrderId must get the same queue.

2.3.1 sequential transmission

• Implement a MessageQueueSelector in send() and override the select() method of selecting MessageQueue.
• If you select MessageQueue by user, you can control sending a certain type of message to the corresponding MessageQueue.
• The last parameter of send() will be passed as a parameter to arg in the select method.

public class Producer {

    public static void main(String[] args) throws Exception {
        // 1. Create a message Producer and make the Producer group name
        DefaultMQProducer producer = new DefaultMQProducer("group1");
        // 2. Specify Nameserver address
        producer.setNamesrvAddr("Server 1 public network IP:9876;Server 2 public network IP:9876");
        // 3. Start Producer
        producer.start();
        // Build message (order) collection
        List<OrderStep> orderSteps = OrderStep.buildOrders();
        // send message
        for (int i = 0; i < orderSteps.size(); i++) {
            String body = orderSteps.get(i) + "";
            Message message = new Message("OrderTopic", "Order", "i" + i, body.getBytes());
            /**
             * Parameter 1: message object
             * Parameter 2: selector of message queue
             * Parameter 3: select the business ID (order ID) of the queue
             */
            SendResult sendResult = producer.send(message, new MessageQueueSelector() {
                /**
                 *
                 * @param mqs: Queue collection
                 * @param msg: Message object
                 * @param arg: Parameters of business identification
                 */
                @Override
                public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg) {
                    long orderId = (long) arg;
                    long index = orderId % mqs.size();
                    return mqs.get((int) index);
                }
            }, orderSteps.get(i).getOrderId());

            System.out.println("Send result:" + sendResult);
        }
        producer.shutdown();
    }
}

• For the official Demo, it can be seen that the orders with orderId 6 are put into the queue with queueId 6 in order, and the queueOffset increases in order at the same time.

2.3.2 sequential consumption

• The registered listener needs to implement the MessageListenerOrderly interface.
• In the cluster mode, distributed locks are used to ensure that a queue will only be locked and consumed by one consumer.
• Locking ensures that only one thread can consume, the messages obtained from ProcessQueue are orderly, and the Consumer ensures the order of consumption.

public class Consumer {
    public static void main(String[] args) throws MQClientException {
        // 1. Create a Consumer and make a Consumer group name
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("group1");
        // 2. Specify Nameserver address
        consumer.setNamesrvAddr("Server 1 public network IP:9876;Server 2 public network IP:9876");
        // 3. Subscribe to Topic and Tag
        consumer.subscribe("OrderTopic", "*");

        // 4. Register message listener
        consumer.registerMessageListener(new MessageListenerOrderly() {

            @Override
            public ConsumeOrderlyStatus consumeMessage(List<MessageExt> msgs, ConsumeOrderlyContext context) {
                for (MessageExt msg : msgs) {
                    System.out.println("Thread Name:[" + Thread.currentThread().getName() + "]:" + new String(msg.getBody()));
                }
                return ConsumeOrderlyStatus.SUCCESS;
            }
        });

        // 5. Start consumers
        consumer.start();
        System.out.println("Consumer startup");
    }
}

Sequential consumption results:

2.4 delayed messages

• For example, in e-commerce, you can send a delay message after submitting an order, and check the status of the order after 1h. If it is still unpaid, cancel the order and release the inventory.
• This logic can be compared with the delay queue constructed by dead letter queue in RabbitMQ to realize distributed transactions.

• RocketMq does not support any time delay. It is necessary to set several fixed delay levels, from 1s to 2h, corresponding to levels 1 to 18 respectively.

// MessageStoreConfig.java
private String messageDelayLevel = "1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h";

• Sender: directly set the delay time in the Message - setDelayTimeLevel(2)

public class Producer {

    public static void main(String[] args) throws InterruptedException, RemotingException, MQClientException, MQBrokerException {
        // 1. Create a message Producer and make the Producer group name
        DefaultMQProducer producer = new DefaultMQProducer("group1");
        // 2. Specify Nameserver address
        producer.setNamesrvAddr("Server 1 public network IP:9876;Server 2 public network IP:9876");
        // 3. Start Producer
        producer.start();

        for (int i = 0; i < 10; i++) {
            // 4. Create a message object and specify the Topic, Tag and message body
            Message msg = new Message("DelayTopic", "Tag1", ("Hello World" + i).getBytes());
            // Set delay time
            msg.setDelayTimeLevel(2);
            // 5. Send message
            SendResult result = producer.send(msg);
            SendStatus status = result.getSendStatus();
            System.out.println("Send results:" + result);

            TimeUnit.SECONDS.sleep(1);
        }

        // 6. Close the Producer
        producer.shutdown();
    }
}

• Consumer side

public class Consumer {

    public static void main(String[] args) throws Exception {
        // 1. Create a Consumer and make a Consumer group name
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("group1");
        // 2. Specify Nameserver address
        consumer.setNamesrvAddr("Server 1 public network IP:9876;Server 2 public network IP:9876");
        // 3. Subscribe to Topic and Tag
        consumer.subscribe("DelayTopic", "*");

        // 4. Set callback function to process messages
        consumer.registerMessageListener(new MessageListenerConcurrently() {

            //Accept message content
            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
                for (MessageExt msg : msgs) {
                    System.out.println("news ID: [" + msg.getMsgId() + "],Delay time:" + (System.currentTimeMillis() - msg.getStoreTimestamp()));
                }
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        // 5. Start Consumer
        consumer.start();
        System.out.println("Consumer startup");
    }
}

2.5 batch message sending

   sending messages in batch can significantly improve the performance of delivering small messages. Batch messages should have the same Topic and waitStoreMsgOK, and should not be delayed messages. The single sending of batch messages should not exceed 4MB. When it is greater than 4MB, it is best to divide the messages.

• Batch sending and message segmentation

public class ListSplitter implements Iterator<List<Message>> {
   private final int SIZE_LIMIT = 1024 * 1024 * 4;
   private final List<Message> messages;
   private int currIndex;
   public ListSplitter(List<Message> messages) {
           this.messages = messages;
   }
    @Override 
    public boolean hasNext() {
       return currIndex < messages.size();
   }
   	@Override 
    public List<Message> next() {
       int nextIndex = currIndex;
       int totalSize = 0;
       for (; nextIndex < messages.size(); nextIndex++) {
           Message message = messages.get(nextIndex);
           int tmpSize = message.getTopic().length() + message.getBody().length;
           Map<String, String> properties = message.getProperties();
           for (Map.Entry<String, String> entry : properties.entrySet()) {
               tmpSize += entry.getKey().length() + entry.getValue().length();
           }
           tmpSize = tmpSize + 20; // Increase log overhead by 20 bytes
           if (tmpSize > SIZE_LIMIT) {
               // A single message exceeds the maximum limit
               // Ignore, otherwise it will block the split process
               if (nextIndex - currIndex == 0) {
                  // If the next sub list has no elements, add this sub list and exit the loop. Otherwise, just exit the loop
                  nextIndex++;
               }
               break;
           }
           if (tmpSize + totalSize > SIZE_LIMIT) {
               break;
           } else {
               totalSize += tmpSize;
           }

       }
       List<Message> subList = messages.subList(currIndex, nextIndex);
       currIndex = nextIndex;
       return subList;
   }
}
// Split a big message into several small messages
ListSplitter splitter = new ListSplitter(messages);
while (splitter.hasNext()) {
  try {
      List<Message>  listItem = splitter.next();
      producer.send(listItem);
  } catch (Exception e) {
      e.printStackTrace();
      //Processing error
  }
}

2.6 filtering messages

2.6.1 Tag filtering

  consumers will receive messages containing TAGA, TAGB or TAGC, but the limit is that a message can only have one Tag tag, which may not work in complex scenarios.

DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("CID_EXAMPLE");
consumer.subscribe("TOPIC", "TAGA || TAGB || TAGC");

2.6.2 SQL filtering

   you can use SQL expressions to filter messages. The SQL feature can be calculated by the attributes when sending messages. Under the syntax defined by RocketMQ, you can implement some simple logic:

• Numerical comparison, such as: >, > =, <, < =, BETWEEN, =;
• Character comparison, such as: =, < >, IN;
• IS NULL or IS NOT NULL;
• Logical symbols AND, OR, NOT;

Support type:

• Value: 123, 3.1415;
• The character 'abc' must be in single quotation marks
• NULL, special constant
• Boolean: TRUE or FALSE

• Produce

Message calls putUserProperty() to import some parameters for SQL.

public class Producer {

    public static void main(String[] args) throws InterruptedException, RemotingException, MQClientException, MQBrokerException {
        // 1. Create a message Producer and make the Producer group name
        DefaultMQProducer producer = new DefaultMQProducer("group1");
        // 2. Specify Nameserver address
        producer.setNamesrvAddr("Server 1 public network IP:9876;Server 2 public network IP:9876");
        // 3. Start Producer
        producer.start();

        for (int i = 0; i < 10; i++) {
            // 4. Create a message object and specify the Topic, Tag and message body
            Message msg = new Message("FilterSQLTopic", "Tag1", ("Hello World" + i).getBytes());
            // Set some properties
			msg.putUserProperty("i", String.valueOf(i));
            // 5. Send message
            SendResult result = producer.send(msg);
            SendStatus status = result.getSendStatus();
            System.out.println("Send results:" + result);

            TimeUnit.SECONDS.sleep(1);
        }

        // 6. Close the Producer
        producer.shutdown();
    }
}

• Consumer

subscribe() passes in the selector MessageSelector and calls bySql() to write SQL filtering statements.

DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("please_rename_unique_group_name_4");

// Only subscribed messages have attributes i: i > = 0 and i < = 3
consumer.subscribe("FilterSQLTopic", MessageSelector.bySql("a between 0 and 3");
consumer.registerMessageListener(new MessageListenerConcurrently() {
   @Override
   public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
       return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
   }
});
consumer.start();

2.7 transaction messages - solving distributed transaction problems

2.7.1 basic process

Transaction messages have three states: commit state, rollback state and intermediate state:

• TransactionStatus.CommitTransaction: commit a transaction that allows the consumer to consume this message.
• TransactionStatus.RollbackTransaction: rollback transaction, which means that the message will be deleted and cannot be consumed.
• TransactionStatus.Unknown: intermediate status, which represents the need to check the message queue to determine the status.

Transaction flow:

• Sending and submitting transaction messages

1. Send a message (half message);
2. The writing result of the server response message;
3. Execute the local transaction according to the sending result. If the write fails, the half message is invisible to the business and the local logic will not execute;
4. Execute Commit or Rollback according to the local transaction status. The Commit operation generates a message index, and the message is visible to the consumer.

• Transaction compensation

1. For transaction messages without Commit/Rollback (messages in pending status), initiate a backcheck from the server;
2. The Producer receives the backcheck message and checks the status of the local transaction corresponding to the backcheck message;
3. Recommit or Rollback according to the local transaction status;
4. The compensation phase is used to solve the timeout or failure of the message Commit or Rollback.

2.7.2 basic use

• Producer

   use TransactionMQProducer class to create a producer and specify a unique producer group. You can set a custom thread pool to process these check requests. After executing local transactions, you need to reply to the message queue according to the execution results.

• setTransactionListener() adds a transaction listener as an entry to execute local transactions.

  when sending the HALF message successfully, use the executelocetransaction () method to execute the local transaction, which returns one of the three transaction states. The checklocaltransmission () method is used to check the local transaction status and respond to the check request of the message queue.

public class Producer {

    public static void main(String[] args) throws Exception {
        // 1. Create a message Producer and make the Producer group name
        TransactionMQProducer producer = new TransactionMQProducer("group5");
        // 2. Specify Nameserver address
        producer.setNamesrvAddr("39.96.86.4:9876;59.110.158.93:9876");

        // Add transaction listener
        producer.setTransactionListener(new TransactionListener() {
            /**
             * Local transactions are performed in this method
             */
            @Override
            public LocalTransactionState executeLocalTransaction(Message msg, Object arg) {
                if (StringUtils.equals("TAGA", msg.getTags())) {
                    return LocalTransactionState.COMMIT_MESSAGE;
                } else if (StringUtils.equals("TAGB", msg.getTags())) {
                    return LocalTransactionState.ROLLBACK_MESSAGE;
                } else if (StringUtils.equals("TAGC", msg.getTags())) {
                    return LocalTransactionState.UNKNOW;
                }
                return LocalTransactionState.UNKNOW;
            }
            /**
             * This method is the logic for MQ to check the status of message transactions
             */
            @Override
            public LocalTransactionState checkLocalTransaction(MessageExt msg) {
                System.out.println("Informative Tag:" + msg.getTags());
                return LocalTransactionState.COMMIT_MESSAGE;
            }
        });

        // 3. Start Producer
        producer.start();
        String[] tags = {"TAGA", "TAGB", "TAGC"};
        for (int i = 0; i < 3; i++) {
            // 4. Create a message object and specify the Topic, Tag and message body
            Message msg = new Message("TransactionTopic", tags[i], ("Hello World" + i).getBytes());
            // 5. Send message
            // Pass in null for transaction control of all messages
            SendResult result = producer.sendMessageInTransaction(msg, null);
            // Send status
            SendStatus status = result.getSendStatus();
            System.out.println("Send results:" + result);
            TimeUnit.SECONDS.sleep(2);
        }
        // 6. Close the Producer
        //producer.shutdown();
    }
}

• summary

1. Transaction messages do not support delayed messages and batch messages.
2. In order to avoid the accumulation of semi queue messages caused by multiple checks of a single message, the number of checks of a single message is limited to 15 by default. This limit can be modified through the transactionCheckMax parameter of the Broker configuration file.
3. If a message has been checked more than N times, the Broker will discard the message and print the error log at the same time by default. This behavior can be modified by rewriting the AbstractTransactionCheckListener class.
4. The transaction message will be checked after a specific time configured by the parameter transactionMsgTimeout in the Broker configuration file. When sending a transaction message, you can set CHECK_IMMUNITY_TIME_IN_SECONDS to change this limit. This parameter takes precedence over the transactionMsgTimeout parameter.
5. Transactional messages may be checked or consumed more than once.
6. The target subject message submitted to the user may fail. Its high availability is guaranteed by RocketMQ's own high availability mechanism. If you want to ensure that the transaction message is not lost and the transaction integrity is guaranteed, you can use the synchronous dual write mechanism.
7. The producer ID of the transaction message cannot be shared with the producer ID of other types of messages. The transaction message allows reverse query, and the MQ server can query the consumer through the producer ID.

2.8 consumers actively Pull

RocketMq: two ways to consume messages pull and push

• Examples

public class PullConsumer {
    private static final Map<MessageQueue, Long> offseTable = new HashMap<MessageQueue, Long>();


    public static void main(String[] args) throws MQClientException {
        DefaultPullConsumer consumer = new DefaultPullConsumer("please_rename_unique_group_name_5");

        consumer.start();

        Set<MessageQueue> mqs = consumer.fetchSubscribeMessageQueues("TopicTest");
        for (MessageQueue mq : mqs) {
            System.out.println("Consume from the queue: " + mq);
            PullResult pullResult;
            try {
                pullResult = consumer.pullBlockIfNotFound(mq, null, getMessageQueueOffset(mq), 32);
                System.out.println(pullResult);
                putMessageQueueOffset(mq, pullResult.getNextBeginOffset());
                switch (pullResult.getPullStatus()) {
                    case FOUND:
                        // TODO
                        break;
                    case NO_MATCHED_MSG:
                        break;
                    case NO_NEW_MSG:
                        break;
                    case OFFSET_ILLEGAL:
                        break;
                    default:
                        break;
                }
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
        consumer.shutdown();
    }

    private static long getMessageQueueOffset(MessageQueue mq) {
        Long offset = offseTable.get(mq);
        if (offset != null)
            return offset;

        return 0;
    }

    private static void putMessageQueueOffset(MessageQueue mq, long offset) {
        offseTable.put(mq, offset);
    }
}

3, RocketMQ supplement

3.1 message storage + High Availability Mechanism + load balancing

RocketMQ: message storage, high availability mechanism, load balancing

• ACK mechanism ensures persistent storage
• Messages are stored as sequential writes
• mmap zero copy mechanism improves the speed of message storage and network transmission

3.2 message retry

• For sequential message consumption, RocketMQ will automatically retry messages continuously. At this time, message consumption is blocked, so the time-consuming is limited.
• Non sequential message consumption, with no time limit, is only effective for cluster consumption.
• If the maximum number of retries is exceeded, it will be put into the dead letter queue.
• Number of retries:

• Retry configuration mode

• Retry count configuration

3.3 dead letter queue

characteristic:

Reference documents

Deep understanding of NameServer

Interpretation of RocketMQ message sending process

RocketMQ source code catalog

RocketMQ learning