catalogue
I have learned the basic knowledge of Rocket and set up MQ stand-alone version and cluster environment. Now I will start the actual development and explain the basic use of RocketMQ according to the RocketMQ source code downloaded earlier:
Producer Product
In the example subproject of RocketMq's source code, it is related to the use of rocketMQ API:
rocketMQ dependency needs to be introduced for Maven development:
<dependency>
<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-client</artifactId>
<version>4.7.1</version>
</dependency>There are three main forms of sending messages:
- Send messages synchronously
- Send message asynchronously
- One way send message
- Send messages synchronously
DefaultMQProducer producer = new DefaultMQProducer("poc-mq-product");
//Cluster environment
// producer.setNamesrvAddr("192.168.36.132:9876;192.168.36.131:9876");
producer.setNamesrvAddr("192.168.10.21:9876");
//Start producer
producer.start();
for (int i = 0; i < 2; i++) {
try {
User user = new User();
user.setAddress("Henan"+i);
user.setId(i);
user.setName("millet"+i);
//Create a message instance and specify the subject, label and message body.
Message msg = new Message("pocTopic" ,
"TagA" ,"keys"+i,
user.toString().getBytes(RemotingHelper.DEFAULT_CHARSET)
);
//messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h
//Time delay level
msg.setDelayTimeLevel(2);
//Call to send a message to deliver the message to one of the brokers. Synchronous transmission
SendResult sendResult = producer.send(msg);
System.out.printf("%s%n", sendResult);
} catch (Exception e) {
e.printStackTrace();
Thread.sleep(1000);
}
}
//Once the producer instance is no longer used, close it
producer.shutdown();- Send message asynchronously
DefaultMQProducer producer = new DefaultMQProducer("Jodie_Daily_test");
producer.start();
producer.setRetryTimesWhenSendAsyncFailed(0);
int messageCount = 100;
//Because it is sent asynchronously, a countDownLatch is introduced here to ensure that all the callback methods sent by the Producer are executed before stopping the Producer service.
final CountDownLatch countDownLatch = new CountDownLatch(messageCount);
for (int i = 0; i < messageCount; i++) {
try {
final int index = i;
Message msg = new Message("Jodie_topic_1023",
"TagA",
"OrderID188",
"Hello world".getBytes(RemotingHelper.DEFAULT_CHARSET));
//Callback SendCallback
producer.send(msg, new SendCallback() {
@Override
public void onSuccess(SendResult sendResult) {
countDownLatch.countDown();
System.out.printf("%-10d OK %s %n", index, sendResult.getMsgId());
}
@Override
public void onException(Throwable e) {
countDownLatch.countDown();
System.out.printf("%-10d Exception %s %n", index, e);
e.printStackTrace();
}
});
} catch (Exception e) {
e.printStackTrace();
}
}
countDownLatch.await(5, TimeUnit.SECONDS);
producer.shutdown();- One way send message
DefaultMQProducer producer = new DefaultMQProducer("please_rename_unique_group_name");
producer.setNamesrvAddr("localhost:9876");
producer.start();
for (int i = 0; i < 100; i++) {
Message msg = new Message("TopicTest" /* Topic */,
"TagA" /* Tag */,
("Hello RocketMQ " +
i).getBytes(RemotingHelper.DEFAULT_CHARSET) /* Message body */
);
//Call send message to deliver message to one of brokers.
producer.sendOneway(msg);
}
//Wait for sending to complete
Thread.sleep(5000);
producer.shutdown();
Consumer
There are two modes of consumer consumption news:
One is the pull mode in which consumers take the initiative to pull messages from the Broker, and the other is the push mode in which consumers wait for the Broker to push messages.
- push mode
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("poc-mq-consumer");
//consumer.setNamesrvAddr("192.168.36.132:9876;192.168.36.131:9876");
consumer.setNamesrvAddr("192.168.10.21:9876");
//If the specified consumer group is a new consumer group, please specify where to start.
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
consumer.subscribe("pocTopic", "*");
//Registered listener cluster consumption
consumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
consumer.start();- pull mode
Generally, the push mode is relatively simple. In fact, the push mode of RocketMQ is also encapsulated by the pull mode. In version 4.7.1, the consumer class DefaultMQPullConsumerImpl has been marked as expired, but it can still be used. The replaced class is DefaultLitePullConsumerImpl.
1: DefaultLitePullConsumerImpl:
DefaultLitePullConsumer litePullConsumer = new DefaultLitePullConsumer("lite_pull_consumer_test");
litePullConsumer.setNamesrvAddr("localhost:9876");
litePullConsumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
litePullConsumer.subscribe("TopicTest", "*");
litePullConsumer.start();
try {
while (running) {
List<MessageExt> messageExts = litePullConsumer.poll();
System.out.printf("%s%n", messageExts);
}
} finally {
litePullConsumer.shutdown();
}
DefaultLitePullConsumer litePullConsumer = new DefaultLitePullConsumer("please_rename_unique_group_name");
litePullConsumer.setAutoCommit(false);
litePullConsumer.start();
Collection<MessageQueue> mqSet = litePullConsumer.fetchMessageQueues("TopicTest");
List<MessageQueue> list = new ArrayList<>(mqSet);
List<MessageQueue> assignList = new ArrayList<>();
for (int i = 0; i < list.size() / 2; i++) {
assignList.add(list.get(i));
}
litePullConsumer.assign(assignList);
litePullConsumer.seek(assignList.get(0), 10);
try {
while (running) {
List<MessageExt> messageExts = litePullConsumer.poll();
System.out.printf("%s %n", messageExts);
litePullConsumer.commitSync();
}
} finally {
litePullConsumer.shutdown();
}
2: DefaultMQPullConsumer:
private static final Map<MessageQueue, Long> OFFSE_TABLE = new HashMap<MessageQueue, Long>();
public static void main(String[] args) throws MQClientException {
DefaultMQPullConsumer consumer = new DefaultMQPullConsumer("please_rename_unique_group_name_5");
consumer.setNamesrvAddr("127.0.0.1:9876");
consumer.start();
Set<MessageQueue> mqs = consumer.fetchSubscribeMessageQueues("broker-a");
for (MessageQueue mq : mqs) {
System.out.printf("Consume from the queue: %s%n", mq);
SINGLE_MQ:
while (true) {
try {
PullResult pullResult = consumer.pullBlockIfNotFound(mq, null, getMessageQueueOffset(mq), 32);
System.out.printf("%s%n", pullResult);
putMessageQueueOffset(mq, pullResult.getNextBeginOffset());
switch (pullResult.getPullStatus()) {
case FOUND:
break;
case NO_MATCHED_MSG:
break;
case NO_NEW_MSG:
break SINGLE_MQ;
case OFFSET_ILLEGAL:
break;
default:
break;
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
consumer.shutdown();
}
private static long getMessageQueueOffset(MessageQueue mq) {
Long offset = OFFSE_TABLE.get(mq);
if (offset != null){
return offset;
}
return 0;
}
private static void putMessageQueueOffset(MessageQueue mq, long offset) {
OFFSE_TABLE.put(mq, offset);
}
}In the actual development, we will encounter the situations of sending messages in sequence and consuming messages in sequence
- Sequential production message
MQProducer producer = new DefaultMQProducer("orderProduce");
producer.start();
String[] tags = new String[] {"TagA", "TagB", "TagC", "TagD", "TagE"};
for (int i = 0; i < 100; i++) {
int orderId = i % 10;
Message msg =
new Message("TopicTestjjj", tags[i % tags.length], "KEY" + i,
("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET));
SendResult sendResult = producer.send(msg, new MessageQueueSelector() {
@Override
public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg) {
Integer id = (Integer) arg;
int index = id % mqs.size();
return mqs.get(index);
}
}, orderId);
System.out.printf("%s%n", sendResult);
}
producer.shutdown();
} catch (MQClientException | RemotingException | MQBrokerException | InterruptedException e) {
e.printStackTrace();
}- Sequential consumption message
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("please_rename_unique_group_name_3");
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
consumer.subscribe("TopicTest", "TagA || TagC || TagD");
consumer.registerMessageListener(new MessageListenerOrderly() {
AtomicLong consumeTimes = new AtomicLong(0);
@Override
public ConsumeOrderlyStatus consumeMessage(List<MessageExt> msgs, ConsumeOrderlyContext context) {
context.setAutoCommit(true);
System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);
this.consumeTimes.incrementAndGet();
if ((this.consumeTimes.get() % 2) == 0) {
return ConsumeOrderlyStatus.SUCCESS;
} else if ((this.consumeTimes.get() % 3) == 0) {
return ConsumeOrderlyStatus.ROLLBACK;
} else if ((this.consumeTimes.get() % 4) == 0) {
return ConsumeOrderlyStatus.COMMIT;
} else if ((this.consumeTimes.get() % 5) == 0) {
context.setSuspendCurrentQueueTimeMillis(3000);
return ConsumeOrderlyStatus.SUSPEND_CURRENT_QUEUE_A_MOMENT;
}
return ConsumeOrderlyStatus.SUCCESS;
}
});
consumer.start();
System.out.printf("Consumer Started.%n");During verification, you can start multiple Consumer instances to observe the message allocation of each order and the consumption order of multiple steps under each order. No matter how the order is allocated before multiple Consumer instances, the order of multiple messages under each order is fixed from 0 to 5. RocketMQ guarantees the local order of messages, not the global order.
First, see what List mqs is from the console. Looking back at our example, in fact, RocketMQ only ensures that all messages of each OrderID are in order (sent to the same queue), but not all messages are in order. Therefore, this involves the principle of RocketMQ message ordering. In order to ensure that the final consumed messages are orderly, we need to ensure that the messages are orderly in three steps: Producer, Broker and Consumer.
First, on the sender side: by default, the message sender will send messages to different messagequeues (partition queues) by Round Robin polling, and consumers will also pull messages from multiple messagequeues when consuming. In this case, the order of messages cannot be guaranteed. Only when a group of ordered messages are sent to the same MessageQueue can the first in first out feature of MessageQueue be used to ensure the order of this group of messages.
Messages in a queue in a Broker can be ordered. Then on the consumer side: consumers will get messages from multiple Message queues. At this time, although the messages on each Message queue are orderly, the messages between multiple queues are still out of order. To ensure the order of messages, consumers need to get messages one by one, that is, after getting the messages of one queue, they can get the messages of the next queue. For the MessageListenerOrderly object injected into the consumer, RocketMQ will ensure that the messages are retrieved one queue by one by locking the queue. Messagelistenercurrently, the Message listener will not lock the queue. Each time, it takes a batch of data from multiple messages (no more than 32 by default). Therefore, the order of messages cannot be guaranteed.
- Broadcast consumption
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("please_rename_unique_group_name_1");
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
consumer.setMessageModel(MessageModel.BROADCASTING);
consumer.subscribe("TopicTest", "TagA || TagC || TagD");
consumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
ConsumeConcurrentlyContext context) {
System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
consumer.start();There is no specific example of broadcast message consumers, because it involves the cluster consumption mode of consumers. In the cluster state (MessageModel.CLUSTERING), each message will only be consumed by one instance in the same consumer group (this is the same as the cluster mode of kafka and rabbitMQ). The broadcast mode sends the message to all consumers who have subscribed to the corresponding topic, regardless of whether the consumers are the same consumer group or not.
- Delayed message
// Instantiate a producer to send scheduled messages
DefaultMQProducer producer = new DefaultMQProducer("ExampleProducerGroup");
// Launch producer
producer.start();
int totalMessagesToSend = 100;
for (int i = 0; i < totalMessagesToSend; i++) {
Message message = new Message("TestTopic", ("Hello scheduled message " + i).getBytes());
// This message will be delivered to consumer 10 seconds later.
message.setDelayTimeLevel(3);
// Send the message
producer.send(message);
}
// Shutdown producer after use.
producer.shutdown();The effect of delaying Message implementation is to call producer After the send method, the Message will not be sent immediately, but will be sent after a period of time. This is a unique feature of RocketMQ. How long will it be delayed? Setting the delay time is to set a delay level Message on the Message object setDelayTimeLevel(3); In the open source version of RocketMQ, the delay setting at any time is not supported for delay messages (supported in the commercial version), but only 18 fixed delay levels are supported. 1 to 18 correspond to messagedelaylevel = 1s 5S 10s 30s 1m 2m 4m 7m 8m 9m 10m 20m 30m 1H 2H respectively. Where does this come from? In fact, it can be seen from the RocketMQ console. These 18 delay levels also support self-definition, but generally it is best not to customize and modify them. How is such an easy-to-use delay Message implemented? In addition to using these 18 delay levels in delay messages, where else are they used? Don't worry, we'll explain it in detail later.
- Batch message
Batch message refers to combining multiple messages into one batch message and sending it at one time. This has the advantage of reducing network IO and improving throughput. For example of message producers of batch messages, see: org apache. rocketmq. example. batch. Simplebackproducer and org apache. rocketmq. example. batch. SplitBatchProducer
DefaultMQProducer producer = new DefaultMQProducer("BatchProducerGroupName");
producer.start();
//If you just send messages of no more than 1MiB at a time, it is easy to use batch
//Messages of the same batch should have: same topic, same waitStoreMsgOK and no schedule support
String topic = "BatchTest";
List<Message> messages = new ArrayList<>();
messages.add(new Message(topic, "Tag", "OrderID001", "Hello world 0".getBytes()));
messages.add(new Message(topic, "Tag", "OrderID002", "Hello world 1".getBytes()));
messages.add(new Message(topic, "Tag", "OrderID003", "Hello world 2".getBytes()));
producer.send(messages);org.apache.rocketmq.example.batch.SplitBatchProducer;
public class SplitBatchProducer {
public static void main(String[] args) throws Exception {
DefaultMQProducer producer = new DefaultMQProducer("BatchProducerGroupName");
producer.start();
//large batch
String topic = "BatchTest";
List<Message> messages = new ArrayList<>(100 * 1000);
for (int i = 0; i < 100 * 1000; i++) {
messages.add(new Message(topic, "Tag", "OrderID" + i, ("Hello world " + i).getBytes()));
}
//split the large batch into small ones:
ListSplitter splitter = new ListSplitter(messages);
while (splitter.hasNext()) {
List<Message> listItem = splitter.next();
producer.send(listItem);
}
}
}
class ListSplitter implements Iterator<List<Message>> {
private int sizeLimit = 1000 * 1000;
private final List<Message> messages;
private int currIndex;
public ListSplitter(List<Message> messages) {
this.messages = messages;
}
@Override
public boolean hasNext() {
return currIndex < messages.size();
}
@Override
public List<Message> next() {
int nextIndex = currIndex;
int totalSize = 0;
for (; nextIndex < messages.size(); nextIndex++) {
Message message = messages.get(nextIndex);
int tmpSize = message.getTopic().length() + message.getBody().length;
Map<String, String> properties = message.getProperties();
for (Map.Entry<String, String> entry : properties.entrySet()) {
tmpSize += entry.getKey().length() + entry.getValue().length();
}
tmpSize = tmpSize + 20; //for log overhead
if (tmpSize > sizeLimit) {
//it is unexpected that single message exceeds the sizeLimit
//here just let it go, otherwise it will block the splitting process
if (nextIndex - currIndex == 0) {
//if the next sublist has no element, add this one and then break, otherwise just break
nextIndex++;
}
break;
}
if (tmpSize + totalSize > sizeLimit) {
break;
} else {
totalSize += tmpSize;
}
}
List<Message> subList = messages.subList(currIndex, nextIndex);
currIndex = nextIndex;
return subList;
}
@Override
public void remove() {
throw new UnsupportedOperationException("Not allowed to remove");
}
}I believe you have seen the key comments on the official website and the test code: if the batch message is greater than 1MB, do not send it in one batch, but split it into multiple batches. In other words, the size of a batch message should not exceed 1MB. In actual use, the 1MB limit can be slightly expanded. The actual maximum limit is 4194304 bytes, about 4MB. However, when using batch messages, the message length is indeed a problem that must be considered. Moreover, there are certain restrictions on the use of batch messages. These messages should have the same Topic and the same waitStoreMsgOK. And it cannot be delay message, transaction message, etc.
- Filter messages
In most cases, you can use the Tag attribute of Message to filter information simply and quickly.
For a message producer case that uses Tag to filter messages, see: org apache. rocketmq. example. filter. TagFilterProducer
For the message consumer case of using Tag to filter messages, see: org apache. rocketmq. example. filter. TagFilterConsumer
public class TagFilterProducer {
public static void main(String[] args) throws Exception {
DefaultMQProducer producer = new DefaultMQProducer("please_rename_unique_group_name");
producer.start();
String[] tags = new String[] {"TagA", "TagB", "TagC"};
for (int i = 0; i < 60; i++) {
Message msg = new Message("TagFilterTest", tags[i % tags.length],
"Hello world".getBytes(RemotingHelper.DEFAULT_CHARSET));
SendResult sendResult = producer.send(msg);
System.out.printf("%s%n", sendResult);
}
producer.shutdown();
}
}public class TagFilterConsumer {
public static void main(String[] args) throws InterruptedException, MQClientException, IOException {
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("please_rename_unique_group_name");
consumer.subscribe("TagFilterTest", "TagA || TagC");
consumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,ConsumeConcurrentlyContext context) {
System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
consumer.start();
System.out.printf("Consumer Started.%n");
}
}However, this method has a big limitation, that is, a message can only have one TAG, which is a little insufficient in some complex scenarios. At this time, you can use SQL expressions to filter messages.
For the message producer case of SQL filtering, see: org apache. rocketmq. example. filter. SqlFilterProducer
For the message consumer cases of SQL filtering, see: org apache. rocketmq. example. filter. SqlFilterConsumer
The key to this model is to use MessageSelector on the consumer side A MessageSelector returned by bysql (string SQL). The SQL statements are executed according to the SQL92 standard. The parameters that can be used IN SQL include the default TAGS AND an a attribute added to the producer. SQL92 syntax: RocketMQ only defines some basic syntax to support this feature. You can also easily expand it. Numerical comparison, such as: >, > =, character comparison, such as: =, < >, IN; IS NULL OR IS NOT NULL; Logical symbols: AND, OR, NOT; The supported types of constants are: numeric values, such as 123, 3.1415; Characters, such as: 'abc', must be wrapped IN single quotation marks; NULL, special constant Boolean, TRUE OR FALSE. Note: only push mode consumers can use SQL filtering. Pull mode is useless.
Let's think about it. Is this message filtering done on the Broker side or on the Consumer side?
public class SqlFilterProducer {
public static void main(String[] args) throws Exception {
DefaultMQProducer producer = new DefaultMQProducer("please_rename_unique_group_name");
producer.start();
String[] tags = new String[] {"TagA", "TagB", "TagC"};
for (int i = 0; i < 10; i++) {
Message msg = new Message("SqlFilterTest",
tags[i % tags.length],
("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET)
);
msg.putUserProperty("a", String.valueOf(i));
SendResult sendResult = producer.send(msg);
System.out.printf("%s%n", sendResult);
}
producer.shutdown();
}
}public class SqlFilterConsumer {
public static void main(String[] args) throws Exception {
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("please_rename_unique_group_name");
// Don't forget to set enablePropertyFilter=true in broker
consumer.subscribe("SqlFilterTest", MessageSelector.bySql("(TAGS is not null and TAGS in ('TagA', 'TagB'))" +
"and (a is not null and a between 0 and 3)"));
consumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
ConsumeConcurrentlyContext context) {
System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
consumer.start();
System.out.printf("Consumer Started.%n");
}
}
- Transaction message
This transaction message is a very distinctive function provided by RocketMQ, which needs to be understood. First, let's understand what transaction messages are. The introduction of the official website is: transaction message is a two-stage message implementation that ensures the final consistency in the distributed system. It can ensure the atomicity of local transaction execution and message sending, that is, the two operations succeed or fail together.
Secondly, let's understand the programming model of transaction messages. Transaction messages only guarantee the atomicity of the two operations of the message sender's local transaction and message sending. Therefore, the example of transaction messages only involves the message sender, which is nothing special for the message consumer. For the case of transaction message producers, see: org apache. rocketmq. example. transaction. TransactionProducer
The key of transaction message is to specify a TransactionListener transaction listener in TransactionMQProducer, which is the key controller of transaction message. The case in the source code is a little complex. I have prepared a clearer example of transaction listener here.
public class TransactionListenerImpl implements TransactionListener {
//Execute after submitting the transaction message.
//Return to commit_ Messages with message status will be consumed by consumers immediately.
//Return to rollback_ Messages with message status will be discarded.
//For the message that returns the UNKNOWN status, the Broker will check the status of the transaction again after a period of time.
@Override
public LocalTransactionState executeLocalTransaction(Message msg, Object arg) {
String tags = msg.getTags();
//TagA messages will be consumed by consumers immediately
if(StringUtils.contains(tags,"TagA")){
return LocalTransactionState.COMMIT_MESSAGE;
//TagB messages are discarded
}else if(StringUtils.contains(tags,"TagB")){
return LocalTransactionState.ROLLBACK_MESSAGE;
//Other messages will wait for the Broker to check the transaction status.
}else{
return LocalTransactionState.UNKNOW;
}
}
//It is executed when the status of the UNKNOWN message is checked back. The results returned are the same.
@Override
public LocalTransactionState checkLocalTransaction(MessageExt msg) {
String tags = msg.getTags();
//TagC messages will be consumed by consumers over time
if(StringUtils.contains(tags,"TagC")){
return LocalTransactionState.COMMIT_MESSAGE;
//TagD messages will also be discarded during status check
}else if(StringUtils.contains(tags,"TagD")){
return LocalTransactionState.ROLLBACK_MESSAGE;
//The remaining tag messages will be discarded after multiple status checks
}else{
return LocalTransactionState.UNKNOW;
}
}
}
public class TransactionProducer {
public static void main(String[] args) throws MQClientException, InterruptedException {
TransactionListener transactionListener = new TransactionListenerImpl();
TransactionMQProducer producer = new TransactionMQProducer("please_rename_unique_group_name");
producer.setNamesrvAddr("127.0.0.1:9876");
//A thread pool is defined
ExecutorService executorService = new ThreadPoolExecutor(2, 5, 100, TimeUnit.SECONDS, new ArrayBlockingQueue<Runnable>(2000), new ThreadFactory() {
@Override
public Thread newThread(Runnable r) {
Thread thread = new Thread(r);
thread.setName("client-transaction-msg-check-thread");
return thread;
}
});
producer.setExecutorService(executorService);
producer.setTransactionListener(transactionListener);
producer.start();
String[] tags = new String[] {"TagA", "TagB", "TagC", "TagD", "TagE"};
for (int i = 0; i < 10; i++) {
try {
Message msg =
new Message("TopicTest", tags[i % tags.length], "KEY" + i,
("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET));
SendResult sendResult = producer.sendMessageInTransaction(msg, null);
System.out.printf("%s%n", sendResult);
Thread.sleep(10);
} catch (MQClientException | UnsupportedEncodingException e) {
e.printStackTrace();
}
}
for (int i = 0; i < 100000; i++) {
Thread.sleep(1000);
}
producer.shutdown();
}
}
Then, we need to understand the usage restrictions of the following transaction messages:
1. Transaction messages do not support delayed messages and batch messages.
2. In order to avoid the accumulation of semi queue messages caused by too many times of single message inspection, we limit the number of times of single message inspection to 15 by default, but users can modify this limit through the {transactionCheckMax parameter of the Broker configuration file. If a message has been checked more than N times (N = transactionCheckMax), the Broker will discard the message and print the error log at the same time by default. Users can modify this behavior by overriding the AbstractTransactionCheckListener class. The number of backchecks is determined by brokerconfig transactionCheckMax is configured with this parameter. It is 15 times by default and can be set in the Broker Conf. Then, the actual number of checks will save a user attribute messageconst. In message PROPERTY_ TRANSACTION_ CHECK_ TIMES. If the value of this attribute is greater than transactionCheckMax, it will be discarded. The value of this user attribute will be incremented according to the number of lookbacks. You can also override this attribute in Producer.
3. The transaction message will be checked after a specific length of time such as the parameter transactionMsgTimeout in the Broker configuration file. When sending a transaction message, the user can also set the user's CHECK_IMMUNITY_TIME_IN_SECONDS to change this limit. This parameter takes precedence over the "transactionMsgTimeout" parameter.
By brokerconfig Transactiontimeout is configured with this parameter. The default is 6 seconds, which can be displayed in the broker Conf. In addition, you can also configure a messageconst PROPERTY_ CHECK_ IMMUNITY_ TIME_ IN_ The seconds attribute is used to specify a specific message check back time for the message. msg.putUserProperty(MessageConst.PROPERTY_CHECK_IMMUNITY_TIME_IN_SECONDS, "10000"); That's 10 seconds.
4. Transactional messages may be checked or consumed more than once.
5. The target topic message submitted to the user may fail. At present, this depends on the log records. Its high availability is guaranteed by RocketMQ's own high availability mechanism. If you want to ensure that transaction messages are not lost and transaction integrity is guaranteed, it is recommended to use synchronous dual write mechanism.
6. The producer ID of transaction messages cannot be shared with the producer ID of other types of messages. Unlike other types of messages, transaction messages allow reverse queries, and MQ servers can query consumers through their producer ID.
The schematic diagram of transaction message implementation mechanism is as follows:
The key of the transaction message mechanism is that when sending a message, it will be converted into a half message and stored in an RMQ inside RocketMQ_ SYS_ TRANS_ HALF_ Topic is a topic that is invisible to consumers. After passing a series of transaction checks, the message is transferred to the target topic, which is visible to consumers.
Transaction message scheme:
- Synchronous sending + multiple retries is the most common scheme
- Transaction message mechanism provided by RocketMQ
Finally, we need to think about the role of transaction messages.
Let's think about the relationship between this transaction message and distributed transactions? Why does it involve two-phase commit related to distributed transactions? Transaction message only guarantees the atomicity of the sender's local transaction and sending message, but does not guarantee the atomicity of the consumer's local transaction. Therefore, transaction message only guarantees half of the distributed transaction. But even so, for complex distributed transactions, the transaction message provided by RocketMQ is also the best degradation scheme in the industry.
- ACL permission control
Permission control (ACL) mainly provides Topic resource level user access control for RocketMQ. When using RocketMQ permission control, users can inject AccessKey and SecretKey signatures into the Client through RPCHook; At the same time, set the corresponding permission control attributes (including Topic access permission, IP white list, AccessKey and secret key signature, etc.) in $RocketMQ_ HOME/conf/plain_ acl. In the configuration file of YML. The Broker side verifies the permissions of the AccessKey. If the permissions are verified, an exception is thrown; ACL Client can refer to: org apache. RocketMQ. example. The AclClient code under the simple package.
Note that if you want to use the ACL function of RocketMQ in your own client, you also need to introduce a separate dependency package
<dependency> <groupId>org.apache.rocketmq</groupId> <artifactId>rocketmq-acl</artifactId> <version>4.7.1</version> </dependency>
For the specific configuration information of the Broker side, please refer to docs / CN / ACL / user under the source package_ guide. md. Mainly in Broker Flag to open ACL in conf: aclEnable=true. Then you can use plain_acl.yml for permission configuration. And the configuration file is hot loaded, that is, when you want to modify the configuration, you only need to modify the configuration file without restarting the Broker service. Let's briefly analyze the plan in the source code_ acl. YML configuration:
#Global whitelist, not controlled by ACL #It is usually necessary to add all nodes in the master-slave architecture globalWhiteRemoteAddresses: - 10.10.103.* - 192.168.0.* accounts: #First account - accessKey: RocketMQ secretKey: 12345678 whiteRemoteAddress: admin: false defaultTopicPerm: DENY #The default Topic access policy is deny defaultGroupPerm: SUB #The default Group access policy is to allow subscriptions only topicPerms: - topicA=DENY #topicA reject - topicB=PUB|SUB #topicB allows you to publish and subscribe to messages - topicC=SUB #topicC only allows subscriptions groupPerms: # the group should convert to retry topic - groupA=DENY - groupB=PUB|SUB - groupC=SUB #The second account, as long as it is from 192.168.1* You can access all resources through IP - accessKey: rocketmq2 secretKey: 12345678 whiteRemoteAddress: 192.168.1.* # if it is admin, it could access all resources admin: true
The above mainly explains the main methods used in RocketMQ API development, as well as the explanation and configuration of Acl permission control.