Release confirmation

Release confirmation principle

The producer sets the channel to confirm mode. Once the channel enters the confirm mode, all messages published on the channel will be assigned a unique ID (starting from 1). Once the message is delivered to all matching queues, the broker will send a confirmation to the producer (including the unique ID of the message), This enables the producer to know that the message has correctly arrived at the destination queue. If the message and queue are persistent, the confirmation message will be sent after writing the message to the disk. The delivery tag field of the confirmation message returned by the broker to the producer contains the sequence number of the confirmation message. In addition, the broker can also set basic The multiple field of ACK indicates that all messages before this serial number have been processed.
The biggest advantage of confirm mode is that it is asynchronous. Once a message is published, the producer application can continue to send the next message while waiting for the channel to return the confirmation. After the message is finally confirmed, the producer application can process the confirmation message through the callback method. If RabbitMQ loses the message due to its own internal error, A nack message is sent, and the producer application can also process the nack message in the callback method.

Policy for publishing confirmation

Method for opening release confirmation

Publishing confirmation is not enabled by default. If you want to enable it, you need to call the method confirmSelect. Whenever you want to use publishing confirmation, you need to call this method in channel

Single confirmation release

This is a simple confirmation method. It is a synchronous confirmation publishing method, that is, after publishing a message, only it is confirmed to be published, and subsequent messages can continue to be published. waitForConfirmsOrDie(long) returns only when the message is confirmed. If the message is not confirmed within the specified time range, it will throw an exception.
The biggest disadvantage of this confirmation method is that the publishing speed is particularly slow, because if the published messages are not confirmed, the publishing of all subsequent messages will be blocked. This method provides a throughput of no more than hundreds of published messages per second. Of course, this may be sufficient for some applications.

public static void publishMessageIndividually() throws Exception {
 try (Channel channel = RabbitMqUtils.getChannel()) {
 String queueName = UUID.randomUUID().toString();
 channel.queueDeclare(queueName, false, false, false, null);
 //Open release confirmation
 channel.confirmSelect();
 long begin = System.currentTimeMillis();
 for (int i = 0; i < MESSAGE_COUNT; i++) {
 String message = i + "";
 channel.basicPublish("", queueName, null, message.getBytes());
 //If the server returns false or does not return within the timeout, the producer can resend the message
 boolean flag = channel.waitForConfirms();
 if(flag){
 System.out.println("Message sent successfully");
 }
 }
 long end = System.currentTimeMillis();
 System.out.println("release" + MESSAGE_COUNT + "Separate confirmation messages,time consuming" + (end - begin) + 
"ms");
 }
}

Batch confirmation release

The above method is very slow. Compared with a single message waiting for confirmation, publishing a batch of messages first and then confirming together can greatly improve the throughput. Of course, the disadvantage of this method is that when a failure causes a problem in publishing, we don't know which message has a problem. We must save the whole batch in memory, To record important information and then republish the message. Of course, this scheme is still synchronous and blocks the release of messages

public static void publishMessageBatch() throws Exception {
 try (Channel channel = RabbitMqUtils.getChannel()) {
 String queueName = UUID.randomUUID().toString();
 channel.queueDeclare(queueName, false, false, false, null);
 //Open release confirmation
 channel.confirmSelect();
 //Bulk confirmation message size
 int batchSize = 100;
 //Number of unacknowledged messages
 int outstandingMessageCount = 0;
 long begin = System.currentTimeMillis();
 for (int i = 0; i < MESSAGE_COUNT; i++) {
 String message = i + "";
 channel.basicPublish("", queueName, null, message.getBytes());
 outstandingMessageCount++;
 if (outstandingMessageCount == batchSize) {
 channel.waitForConfirms();
 outstandingMessageCount = 0;
 }
 }
 //To ensure that there are still unconfirmed messages left, confirm again
 if (outstandingMessageCount > 0) {
 channel.waitForConfirms();
 }
 long end = System.currentTimeMillis();
 System.out.println("release" + MESSAGE_COUNT + "Batch confirmation messages,time consuming" + (end - begin) + 
"ms");
 }
}

Asynchronous confirmation Publishing

Although the programming logic of asynchronous confirmation is more complex than the above two, it has the highest cost performance. It can not be said whether it is reliable or efficient. It uses callback function to achieve reliable message delivery. This middleware also ensures whether it is delivered successfully through function callback. Let's explain in detail how asynchronous confirmation is realized.

public static void publishMessageAsync() throws Exception {
 try (Channel channel = RabbitMqUtils.getChannel()) {
 String queueName = UUID.randomUUID().toString();
 channel.queueDeclare(queueName, false, false, false, null);
 //Open release confirmation
 channel.confirmSelect();
 /**
 * A thread safe and orderly hash table, which is suitable for high concurrency
 * 1.Easily associate sequence numbers with messages
 * 2.Easily delete entries in batches as long as the serial number is given
 * 3.Support concurrent access
 */
 ConcurrentSkipListMap<Long, String> outstandingConfirms = new 
ConcurrentSkipListMap<>();
 /**
 * A callback to acknowledge receipt of a message
 * 1.Message sequence number
 * 2.true Messages less than or equal to the current serial number can be confirmed
 * false Confirm current serial number message
 */
 ConfirmCallback ackCallback = (sequenceNumber, multiple) -> {
 if (multiple) {
 //The returned unconfirmed message is less than or equal to the current serial number. It is a map
 ConcurrentNavigableMap<Long, String> confirmed = 
outstandingConfirms.headMap(sequenceNumber, true);
 //Clear this part of the unacknowledged message
 confirmed.clear();
 }else{
 //Only messages with the current serial number are cleared
 outstandingConfirms.remove(sequenceNumber);
 }
 };
 ConfirmCallback nackCallback = (sequenceNumber, multiple) -> {
 String message = outstandingConfirms.get(sequenceNumber);
 System.out.println("Published messages"+message+"Unconfirmed, serial number"+sequenceNumber);
 };
 /**
 * Add a listener for asynchronous acknowledgement
 * 1.Callback to acknowledge receipt of message
 * 2.Callback for message not received
 */
 channel.addConfirmListener(ackCallback, null);
 long begin = System.currentTimeMillis();
 for (int i = 0; i < MESSAGE_COUNT; i++) {
 String message = "news" + i;
 /**
 * channel.getNextPublishSeqNo()Gets the sequence number of the next message
 * An association is made with the message body through the serial number
 * All are unacknowledged message bodies
 */
 outstandingConfirms.put(channel.getNextPublishSeqNo(), message);
 channel.basicPublish("", queueName, null, message.getBytes());
 }
 long end = System.currentTimeMillis();
 System.out.println("release" + MESSAGE_COUNT + "Asynchronous acknowledgement message,time consuming" + (end - begin) + 
"ms");
 }
}

How to handle asynchronous unacknowledged messages

The best solution is to put unconfirmed messages into a memory based queue that can be accessed by the publishing thread. For example, use the ConcurrentLinkedQueue queue to transfer messages between confirm callbacks and the publishing thread.

More than 3 kinds of release speed comparison

• Publish message separately
  Synchronization waiting for confirmation is simple, but the throughput is very limited.
• Batch publish message
  Batch synchronization waits for confirmation. It is simple and reasonable throughput. Once there is a problem, it is difficult to infer that the message has a problem.
• Asynchronous processing:
  Optimal performance and resource usage can be well controlled in case of errors, but it is slightly difficult to implement

public static void main(String[] args) throws Exception {
 //Set the number of messages to 1000 or it will take too long
 publishMessagesIndividually();
 publishMessagesInBatch();
 handlePublishConfirmsAsynchronously();
}
//Operation results
 Release 1,000 50 individual acknowledgement messages,278 ms
 Release 1,000 635 batch confirmation messages ms
 Release 1,000 Asynchronous acknowledgement messages took 92 minutes ms