Publisher confirmation

Posted by shazly on Tue, 04 Jan 2022 04:17:30 +0100

Publisher confirmation

Publisher confirms(using the java client)

Publisher validation is an extension of rabbitMQ for reliable publishing. When publisher acknowledgement is enabled on the channel, messages published by the client will be asynchronously acknowledged by the agent, which means that they will be processed by the server.

overview

In this chapter, we will use publisher confirmation to ensure that the published message has arrived at the agent safely. It will also introduce several strategies for using publisher validation, identify and explain their advantages and disadvantages.

Enable publisher acknowledgement on channel

Publisher confirmed AMQP 0.9 1 protocol, so it is not enabled by default. It can be enabled on the channel through the method confirmSelect:

Channel channel = connection.createChannel();
channel.confirmSelect();

This method must be called on each channel that you want to use publisher acknowledgement, and acknowledgement should be enabled only once, not for each published message.

Strategy 1: publish messages separately

Let's start with the simplest way to publish with confirmation, that is, publish a message and wait for its confirmation synchronously:

while (thereAreMessagesToPublish()) {
	byte[] body = ...;
    BasicProperties properties = ...;
    channel.basicPublish(exchange, queue, properties, body);
    // uses a 5 second timeout
    channel.waitForConfirmsOrDie(5_000);
}

In the above example, we publish the message as usual and use the Channel#waitForConfirmsOrDie(long) method to wait for the message confirmation. Once the message is acknowledged, the method returns. This method throws an exception if the message is not acknowledged within the timeout, or if the agent cannot process it (NACK ED) for some reason. Exception handling usually includes logging error messages and / or retrying sending messages.
It should be noted that different client libraries have different methods to synchronize publisher confirmation, so you must carefully read the client documentation you are using.
This technique is very simple, but it also has one main disadvantage: it reduces the speed of publishing, because the confirmation of the message will block the publishing of all subsequent messages. This approach does not provide throughput of more than hundreds of published messages per second. However, for some applications, this is good enough.

Does the publisher confirm that it is not asynchronous?

We mentioned at the beginning that the agent confirms the published message asynchronously, but in the first example, the code waits synchronously until the message is confirmed. However, the client actually receives the acknowledgement asynchronously and unblocks the waitForConfirmsOrDie call accordingly. Think of waitForConfirmsOrDie as a synchronous helper that relies on asynchronous notifications at the bottom.

Strategy 2: publish messages in batches

To improve the previous example, we can publish a batch of messages and wait for the entire batch of messages to be confirmed. The following example uses 100 batches:

int batchSize = 100;
int outstandingMessageCount = 0;
while (thereAreMessagesToPublish()) {
	byte[] body = ...;
    BasicProperties properties = ...;
    channel.basicPublish(exchange, queue, properties, body);
    outstandingMessageCount++;
    
    if (outstandingMessageCount == batchSize) {
        ch.waitForConfirmsOrDie(5_000);
        outstandingMessageCount = 0;
    }
}

if (outstandingMessageCount > 0) {
    ch.waitForConfirmsOrDie(5_000);
}

Compared with waiting for the confirmation of a single message, waiting for the confirmation of a batch message can greatly improve the throughput (up to 20-30 times for remote rabbitMQ nodes). One disadvantage is that in the event of a failure, you cannot know exactly where the error occurred, so you may have to save the entire batch in memory to record meaningful content or republish messages. This solution is still synchronous, so it blocks the publication of messages.

Strategy 3: asynchronous processing of publisher confirmation

The agent asynchronously confirms the published messages. It only needs to register a callback at the client to get the notification of these confirmation messages:

Channel channel = connection.createChannel();
channel.confirmSelect();
channel.addConfirmListener((deliveryTag, multiple) -> {
    // code when message is confirmed
}, (deliveryTag, multiple) -> {
    // code when message is nack-ed
});

There are two callbacks: one for acknowledged messages and one for messages that may be considered missing by the agent (NACK ED). Each callback has two parameters:

deliveryTag(long): identifies the serial number of the acknowledged or unacknowledged message. We'll soon see how to relate it to published messages.
multiple(boolean): This is a Boolean value. If false, only confirm whether a message is confirmed or not; If true, all messages with lower or equal serial numbers are acknowledged.

The serial number can be obtained by calling the Channel#getNextPublishSeqNo() method before publishing:

int deliveryTag = channel.getNextPublishSeqNo());
ch.basicPublish(exchange, queue, properties, body);

A simple way to associate a message with a sequence number is to use a mapping. Suppose you want to publish strings because they can easily be converted to an array of bytes for publishing. The following is a code example that uses a mapping to associate the publication serial number with the string body of the message:

ConcurrentNavigableMap<Long, String> outstandingConfirms = new ConcurrentSkipListMap<>();
// ... code for confirm callbacks will come later
String body = "...";
outstandingConfirms.put(channel.getNextPublishSeqNo(), body);
channel.basicPublish(exchange, queue, properties, body.getBytes());

The release code now uses the mapping to trace out the message. When the acknowledgement arrives, we need to clean up the mapping and do something extra, such as recording a warning message when the message is not answered.

ConcurrentNavigableMap<Long, String> outstandingConfirms = new ConcurrentSkipListMap<>();
ConfirmCallback cleanOutstandingConfirms = (deliveryTag, multiple) -> {
	if (multiple) {
		ConcurrentNavigableMap<Long, String> confirmed = outstandingConfirms.headMap(
			deliveryTag, true
        );
        confirmed.clear();
    } else {
        outstandingConfirms.remove(deliveryTag);
    }
};

channel.addConfirmListener(cleanOutstandingConfirms, (deliveryTag, multiple) -> {
	String body = outstandingConfirms.get(deliveryTag);
    System.err.format(
		"Message with body %s has been nack-ed. Sequence number: %d, multiple: %b%n",
		body, deliveryTag, multiple
    );
	cleanOutstandingConfirms.handle(deliveryTag, multiple);
});
// ... publishing code

The above example contains a callback that cleans up the map when the acknowledgement arrives. Note that this callback function handles both single and multiple acknowledgments. This callback is called when the acknowledgement arrives (as the first parameter of the Channel#addConfirmListener). The callback function without a reply message will retrieve the message body and issue a warning. Then, it reuses the previous callback to clean up the mapping that did not perform confirmation (whether the message is confirmed or no response, their corresponding entries in the mapping must be deleted).

How to track outstanding confirmations

Our example uses concurrent navigablemap to track outstanding confirmations. This data structure is convenient for several reasons. It allows you to easily associate a sequence number with a message (regardless of the message data) and easily clear entries before a given sequence id (to handle multiple acknowledgements / no replies). Finally, it supports concurrent access because the acknowledgement callback is invoked in the thread owned by the client library, which should be different from the publishing thread.
In addition to using a complex mapping implementation, there are other methods to track outstanding confirmations, such as using a simple concurrent hash map and a variable to track the lower bound of the publication sequence, but these methods are usually more complex.

In summary, asynchronous processing of publisher validation typically requires the following steps:

Provides a method of associating a publication serial number with a message.
Register an acknowledgement listener on the channel to receive a notification when the publisher's acknowledgement / no reply arrives to perform appropriate operations, such as recording or republishing a no reply message. In this step, the mechanism by which the serial number is associated with the message may also require some appropriate cleanup.
Track the publication serial number before publishing a message.

Republish unresponsive messages?

It is very appealing to re release non response messages from the corresponding callbacks, but this should be avoided because the acknowledgement callbacks are scheduled in the I/O thread that the channel should not perform the operation. A better solution is to queue messages into a memory queue that is polled by the publishing thread. Classes like ConcurrentLinkedQueue are good candidates for transmitting messages between the acknowledgement callback and the publishing thread.

summary

In some applications, it is critical to ensure that published messages are sent to the agent. Publisher validation is a feature of rabbitMQ that can help meet this requirement. Publisher confirmations are asynchronous in nature, but they can also be processed synchronously. There is no clear way to implement publisher validation, which usually comes down to the constraints of the application and the whole system. Typical technologies include:

Publish messages separately, synchronize and wait for confirmation: very simple, but the throughput is very limited.
Batch publish messages and synchronously wait for batch confirmation: simple and reasonable throughput, but it is difficult to judge when an error occurs.
Asynchronous processing: best performance and resource use, good control in case of errors, but may need to be implemented correctly.

Integration code

public class PublisherConfirms {

    static final int MESSAGE_COUNT = 50_000;

    static Connection createConnection() throws Exception {
        ConnectionFactory cf = new ConnectionFactory();
        cf.setHost("192.168.1.254");
        cf.setUsername("admin");
        cf.setPassword("admin123");

        return cf.newConnection();
    }

    public static void main(String[] args) throws Exception {
        publishMessagesIndividually();
        publishMessagesInBatch();
        handlePublishConfirmsAsynchronously();
    }

    static void publishMessagesIndividually() throws Exception {
        try (Connection connection = createConnection()) {
            Channel ch = connection.createChannel();
            String queue = UUID.randomUUID().toString();
            ch.queueDeclare(queue, false, false, true, null);
            ch.confirmSelect();
            long start = System.nanoTime();

            for (int i = 0; i < MESSAGE_COUNT; i++) {
                String body = String.valueOf(i);
                ch.basicPublish("", queue, null, body.getBytes());
                ch.waitForConfirmsOrDie(5_000);
            }

            long end = System.nanoTime();

            System.out.format("Published %,d messages individually in %,d ms%n",
                    MESSAGE_COUNT, Duration.ofNanos(end - start).toMillis());
        }
    }

    static void publishMessagesInBatch() throws Exception {
        try (Connection connection = createConnection()) {
            Channel ch = connection.createChannel();
            String queue = UUID.randomUUID().toString();
            ch.queueDeclare(queue, false, false, true, null);
            ch.confirmSelect();
            int batchSize = 100;
            int outstandingMessageCount = 0;
            long start = System.nanoTime();

            for (int i = 0; i < MESSAGE_COUNT; i++) {
                String body = String.valueOf(i);
                ch.basicPublish("", queue, null, body.getBytes());
                outstandingMessageCount++;

                if (outstandingMessageCount == batchSize) {
                    ch.waitForConfirmsOrDie(5_000);
                    outstandingMessageCount = 0;
                }
            }

            if (outstandingMessageCount > 0) {
                ch.waitForConfirmsOrDie(5_000);
            }

            long end = System.nanoTime();

            System.out.format("Published %,d messages in batch in %,d ms%n",
                    MESSAGE_COUNT, Duration.ofNanos(end - start).toMillis());
        }
    }

    static void handlePublishConfirmsAsynchronously() throws Exception {
        try (Connection connection = createConnection()) {
            Channel ch = connection.createChannel();
            String queue = UUID.randomUUID().toString();
            ch.queueDeclare(queue, false, false, true, null);
            ch.confirmSelect();
            ConcurrentNavigableMap<Long, String> outstandingConfirms = new ConcurrentSkipListMap<>();
            ConfirmCallback cleanOutstandingConfirms = (sequenceNumber, multiple) -> {
                if (multiple) {
                    ConcurrentNavigableMap<Long, String> confirmed = outstandingConfirms.headMap(
                            sequenceNumber, true
                    );
                    confirmed.clear();
                } else {
                    outstandingConfirms.remove(sequenceNumber);
                }
            };

            ch.addConfirmListener(cleanOutstandingConfirms, (sequenceNumber, multiple) -> {
                String body = outstandingConfirms.get(sequenceNumber);
                System.err.format(
                        "Message with body %s has been nack-ed. Sequence number: %d, multiple: %b%n",
                        body, sequenceNumber, multiple
                );
                cleanOutstandingConfirms.handle(sequenceNumber, multiple);
            });

            long start = System.nanoTime();

            for (int i = 0; i < MESSAGE_COUNT; i++) {
                String body = String.valueOf(i);
                outstandingConfirms.put(ch.getNextPublishSeqNo(), body);
                ch.basicPublish("", queue, null, body.getBytes());
            }

            if (!waitUntil(Duration.ofSeconds(60), () -> outstandingConfirms.isEmpty())) {
                throw new IllegalStateException("All messages could not be confirmed in 60 seconds");
            }

            long end = System.nanoTime();
            System.out.format("Published %,d messages and handled confirms asynchronously in %,d ms%n", 
                    MESSAGE_COUNT, Duration.ofNanos(end - start).toMillis());
        }
    }

    static boolean waitUntil(Duration timeout, BooleanSupplier condition) throws InterruptedException {
        int waited = 0;
        while (!condition.getAsBoolean() && waited < timeout.toMillis()) {
            Thread.sleep(100L);
            waited = +100;
        }
        return condition.getAsBoolean();
    }
}

If the client and server are on the same machine, the output on the computer should be similar. The performance of publishing messages alone is not as good as expected, but the results of asynchronous processing are a little disappointing compared with batch publishing.
Publisher validation is very network dependent, so we'd better try to use remote nodes, which is more realistic, because in production, the client and server are usually not on the same machine. Java can easily be changed to use non local nodes (just modify the corresponding information in the ConnectionFactory).

Topics: Java RabbitMQ Distribution

Programmer Think

Publisher confirmation