3, Work Queues mode

The main idea of work queue (also known as task queue) is to avoid executing resource intensive tasks immediately and having to wait for them to complete. Instead, we arranged for the task to be carried out later. We encapsulate the task as a message and send it to the queue. The worker process running in the background will pop up the task and finally execute the job. When there are multiple worker threads, these worker threads will handle these tasks together.

3.1. Polling distribution messages

In the case of multithreading, tasks are distributed by polling by default

In this case, we will start two worker threads, a message sending thread. Let's see how their two worker threads work.

3.1.1 extraction tools

public class RabbitMqUtils {
    //Get a connected channel
    public static Channel getChannel() throws Exception{
        //Create a connection factory
        ConnectionFactory factory = new ConnectionFactory();
        factory.setHost("192.168.181.128");
        factory.setUsername("admin");
        factory.setPassword("admin");
        Connection connection = factory.newConnection();
        Channel channel = connection.createChannel();
        return channel;
    }
}

3.1.2. Start two worker threads

public class Worker01 {
    private static final String QUEUE_NAME = "hello";

    public static void main(String[] args) throws Exception {
        Channel channel = RabbitMqUtils.getChannel();

        //How to perform interface callback for consumption of pushed messages
        DeliverCallback deliverCallback = (consumerTag, delivery) -> {
            String message = new String(delivery.getBody());
            System.out.println("Received message" + message);
        };

        //A callback interface for canceling consumption, such as when the queue is deleted during consumption
        CancelCallback cancelCallback = consumerTag -> {
            System.out.println(consumerTag + "Consumer cancels consumer interface callback logic");
        };

        System.out.println("C1 Consumers start waiting for consumption......");

        /**
         * Consumer News
         * 1.Which queue to consume
         * 2.Whether to answer automatically after consumption is successful. true means to answer automatically. false means to answer manually
         * 3.Callback logic of consumer consumption message
         * 4.Consumer cancels the callback of consumption
         */
        channel.basicConsume(QUEUE_NAME,true,deliverCallback,cancelCallback);
    }
}

Because the code is the same, use ideal to directly start the code twice without writing the code of another worker thread

Start worker thread

3.1.3. Start a send thread

public class Task01 {
    private static final String QUEUE_NAME = "hello";

    public static void main(String[] args) throws Exception {
        Channel channel = RabbitMqUtils.getChannel();

        /**
         * Generate a queue
         * 1.Queue name
         * 2.Is the message in the queue persistent? The default message is stored in memory
         * 3.Whether the queue is only used by one consumer or shared. true can be used by multiple consumers
         * 4.Whether to automatically delete the queue after the last consumer is disconnected. true: automatically delete
         * 5.Other parameters
         */
        channel.queueDeclare(QUEUE_NAME,false,false,false,null);

        //Receive information from the console
        Scanner scanner = new Scanner(System.in);
        while (scanner.hasNext()){
            String message = scanner.next();
            /**
             * Send a message
             * 1.To which switch
             * 2.Which key is the routing key
             * 3.Other parameter information
             * 4.The body of the message sent
             */
            channel.basicPublish("",QUEUE_NAME,null,message.getBytes());
            System.out.println("Send message complete:"+message);
        }
    }
}

3.1.4. Result display

Through program execution, it is found that the producer sends a total of four messages, and consumer 1 and consumer 2 receive two messages respectively, and receive one message in order

3.2 message response

3.2.1 concept

It may take some time for a consumer to complete a task. What happens if one of the consumers processes a long task and only completes part of it, and suddenly it hangs up. Once RabbitMQ delivers a message to the consumer, it immediately marks the message for deletion. In this case, a consumer suddenly hangs up. We will lose the message being processed and the subsequent message sent to the consumer because it cannot be received.

In order to ensure that the message is not lost during sending, rabbitmq introduces a message response mechanism. The message response is: after receiving and processing the message, the consumer tells rabbitmq that it has been processed, and rabbitmq can delete the message.

3.2.2 automatic response

The message is considered to have been successfully transmitted immediately after it is sent. This mode requires a trade-off between high throughput and data transmission security, because in this mode, if the connection or channel is closed on the consumer's side before the message is received, the message will be lost. On the other hand, of course, this mode can deliver overloaded messages on the consumer's side, There is no limit on the number of messages delivered. Of course, this may cause consumers to receive too many messages that are too late to process, resulting in the backlog of these messages, eventually running out of memory, and finally these consumer threads are killed by the operating system, Therefore, this model is only applicable when consumers can process these messages efficiently and at a certain rate.

3.2.3 message response method

A.Channel. Basicack (for positive confirmation)

• RabbitMQ knows the message and processes it successfully. It can be discarded

B.Channel. Basicnack (for negative confirmation)

C.Channel. Basicreject (for negative confirmation)

• And channel One less parameter than basicnack (Multiple)
• If the message is rejected without processing, it can be discarded

3.2.4 interpretation of multiple

The advantage of manual response is that it can respond in batches and reduce network congestion

true and false of multiple mean different things

• true means to batch respond to messages that are not answered on the channel
  • For example, if there are messages 5, 6, 7, and 8 on the channel that transmit the tag, and the current tag is 8, then the unacknowledged messages of 5-8 will be confirmed to receive the message response
• false compared with the above
  • Only messages 5, 6 and 7 with tag=8 will be answered, and the three messages will not be confirmed

3.2.5 automatic message rejoining

If the consumer loses the connection for some reason (its channel has been closed, the connection has been closed or the TCP connection has been lost), resulting in the message not sending ACK confirmation, RabbitMQ will understand that the message has not been fully processed and will queue it again. If other consumers can handle it at this time, it will soon redistribute it to another consumer. In this way, even if a consumer dies occasionally, it can be ensured that no message will be lost.

3.2.6 message manual response code

The default message adopts automatic response, so if we want to realize that the message is not lost in the process of consumption, we need to change the automatic response to manual response. The consumer adds the code drawn in red below on the basis of the above code.

producer

public class Task2 {
    private static final String ACK_QUEUE_NAME="ack_queue";

    public static void main(String[] args) throws Exception {
        Channel channel = RabbitMqUtils.getChannel();

        channel.queueDeclare(ACK_QUEUE_NAME,false,false,false,null);

        //Receive information from the console
        Scanner scanner = new Scanner(System.in);
        System.out.println("Please enter a message");
        while (scanner.hasNext()){
            String message = scanner.next();
            channel.basicPublish("",ACK_QUEUE_NAME,null,message.getBytes());
            System.out.println("Producer sends message:"+message);
        }
    }
}

Consumer 01

public class Work03 {
    private static final String ACK_QUEUE_NAME="ack_queue";

    public static void main(String[] args) throws Exception {
        Channel channel = RabbitMqUtils.getChannel();
        System.out.println("C1 Short waiting time for receiving messages");

        //How to process messages when consuming messages
        DeliverCallback deliverCallback=(consumerTag, delivery)->{
            String message= new String(delivery.getBody());
            try {
                Thread.sleep(1 * 1000);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
            System.out.println("Message received:"+message);
            /**
             * 1.Message tag
             * 2.Whether to answer unanswered messages in batch
             */
            channel.basicAck(delivery.getEnvelope().getDeliveryTag(),false);
        };
        //Manual response is adopted
        boolean autoAck=false;
        channel.basicConsume(ACK_QUEUE_NAME,autoAck,deliverCallback,(consumerTag)->{
            System.out.println(consumerTag+"Consumer cancels consumer interface callback logic");
        });

    }
}

Consumer 02

public class Work04 {
    private static final String ACK_QUEUE_NAME="ack_queue";

    public static void main(String[] args) throws Exception {
        Channel channel = RabbitMqUtils.getChannel();
        System.out.println("C1 Long waiting time for receiving messages");

        //How to process messages when consuming messages
        DeliverCallback deliverCallback=(consumerTag, delivery)->{
            String message= new String(delivery.getBody());
            try {
                Thread.sleep(30 * 1000);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
            System.out.println("Message received:"+message);
            /**
             * 1.Message tag
             * 2.Whether to answer unanswered messages in batch
             */
            channel.basicAck(delivery.getEnvelope().getDeliveryTag(),false);
        };
        //Manual response is adopted
        boolean autoAck=false;
        channel.basicConsume(ACK_QUEUE_NAME,autoAck,deliverCallback,(consumerTag)->{
            System.out.println(consumerTag+"Consumer cancels consumer interface callback logic");
        });

    }
}

3.2.7 demonstration of manual response effect

Under normal circumstances, the message sender sends two messages C1 and C2, receives and processes the messages respectively

After the sender sends the message dd, the C2 consumer is stopped. Normally, the C2 processes the message. However, because it takes a long time to process it, C2 is stopped before it is finished, that is, C2 has not executed the ack code. At this time, it will be seen that the message is received by C1, indicating that the message dd is re queued, It is then allocated to C1 that can process messages

3.3 RabbitMQ persistence

3.3.1 concept

We have just seen how to handle the situation that tasks are not lost, but how to ensure that the messages sent by the message producer are not lost after the RabbitMQ service is stopped. By default, when RabbitMQ exits or crashes for some reason, it ignores queues and messages unless told not to do so. Two things need to be done to ensure that messages are not lost: we need to mark both queues and messages as persistent.

3.3.2 how to implement queue persistence

The queues we created before are all non persistent. rabbitmq if restarted, the queue will be deleted. If you want the queue to be persistent, you need to set the durable parameter to persistent when the (producer) declares the queue

However, it should be noted that if the previously declared queue is not persistent, you need to delete the original queue or re create a persistent queue, otherwise an error will occur

The following is the UI display area of persistent and non persistent queues in the console. At this time, even if rabbitmq queue is restarted, it still exists

3.3.3 message persistence

To make the message persistent, you need to modify the code in the message producer, messageproperties PERSISTENT_ TEXT_ Plan adds this attribute.

Marking messages as persistent does not completely guarantee that messages will not be lost. Although it tells RabbitMQ to save the message to disk, there is still an interval point where the message is still cached when it is just ready to be stored on disk. There is no real write to disk at this time. The persistence guarantee is not strong, but it is more than enough for our simple task queue. If you need a stronger persistence strategy, refer to the following courseware release confirmation chapter

3.4 unfair distribution

At the beginning, we learned that RabbitMQ distributes messages in rotation, but this strategy is not very good in a certain scenario. For example, there are two consumers processing tasks, one consumer 1 processes tasks very fast, while the other consumer 2 processes tasks very slowly, At this time, if we still use rotation training distribution, the fast processing consumer will be idle for a large part of the time, while the slow processing consumer has been working. This distribution method is not very good in this case, but RabbitMQ doesn't know this situation, and it still distributes fairly.

To avoid this situation, we can set the channel parameter channel basicQos(1);

This means that if I haven't finished processing this task or I haven't answered you, don't assign it to me. I can only process one task at present, and then rabbitmq will assign the task to the idle consumer who is not so busy. Of course, if all consumers haven't finished their tasks, the queue is still adding new tasks, The queue may be full. At this time, you can only add a new worker or change the strategy of other storage tasks.

3.5 pre value

The message itself is sent asynchronously, so there must be more than one message on the channel at any time. In addition, the manual confirmation from the consumer is also asynchronous in nature. Therefore, there is an unacknowledged message buffer here. Therefore, we hope that developers can limit the size of this buffer to avoid the problem of unlimited unacknowledged messages in the buffer. At this time, you can use basic The QoS method sets the "prefetch count" value. This value defines the maximum number of unacknowledged messages allowed on the channel. Once the number reaches the configured number, RabbitMQ will stop delivering more messages on the channel unless at least one unprocessed message is confirmed. For example, if there are unconfirmed messages 5, 6, 7 and 8 on the channel and the prefetch count of the channel is set to 4, RabbitMQ will not deliver any messages on the channel, Unless at least one unanswered message is acked. For example, the message tag=6 has just been acknowledged, and RabbitMQ will sense the situation and send another message. Message response and QoS pre value have a significant impact on user throughput. Generally, increasing prefetching will improve the speed of message delivery to consumers. Although the automatic response transmission message rate is the best, in this case, the number of messages delivered but not processed will also increase, thus increasing the RAM consumption of consumers (random access memory). Care should be taken to use the automatic confirmation mode or manual confirmation mode with infinite preprocessing. Consumers consume a large number of messages if there is no confirmation, It will cause the memory consumption of consumers' connection nodes to become larger, so finding the appropriate pre value is a process of repeated experiments. The value is also different for different loads. The value in the range of 100 to 300 can usually provide the best throughput and will not bring too much risk to consumers. The pre value of 1 is the most conservative. Of course, this will make the throughput very low, especially when the consumer connection delay is very serious, especially in the environment where the consumer connection waiting time is long. For most applications, a slightly higher value will be optimal.

3.6 summary distribution method

channel.basicQos(prefetchCount);

• prefetchCount is not set or set to 0 to distribute data in polling mode
• prefetchCount is set to 1 to distribute data unfairly. The prefetchCount is 1
• prefetchCount is set to be greater than or equal to 1 to distribute data for prefetch value

The so-called pre value can be regarded as the size of the unconfirmed message buffer in the consumer channel and the maximum number of people that can be accommodated in a lounge. When it is full, it can't go in again unless someone leaves.

4, Release confirmation

4.1 release confirmation principle

The producer sets the channel to confirm mode. Once the channel enters the confirm mode, all messages published on the channel will be assigned a unique ID (starting from 1). Once the message is delivered to all matching queues, the broker will send a confirmation to the producer (including the unique ID of the message), This enables the producer to know that the message has correctly arrived at the destination queue. If the message and queue are persistent, the confirmation message will be sent after writing the message to the disk. The delivery tag field of the confirmation message returned by the broker to the producer contains the sequence number of the confirmation message. In addition, the broker can also set basic The multiple field of ACK indicates that all messages before this serial number have been processed.

The biggest advantage of confirm mode is that it is asynchronous. Once a message is published, the producer application can continue to send the next message while waiting for the channel to return the confirmation. After the message is finally confirmed, the producer application can process the confirmation message through the callback method. If RabbitMQ loses the message due to its own internal error, A nack message is sent, and the producer application can also process the nack message in the callback method.

4.2 release confirmation strategy

4.2.1. Method for opening release confirmation

Publishing confirmation is not enabled by default. If you want to enable it, you need to call the method confirmSelect. Whenever you want to use publishing confirmation, you need to call this method in channel

        Channel channel = connection.createChannel();
        channel.confirmSelect();

4.2.2 single confirmation release

This is a simple confirmation method. It is a synchronous confirmation publishing method, that is, after publishing a message, only it is confirmed to be published, and subsequent messages can continue to be published. waitForConfirmsOrDie(long) returns only when the message is confirmed. If the message is not confirmed within the specified time range, it will throw an exception.

The biggest disadvantage of this confirmation method is that the publishing speed is particularly slow, because if the published messages are not confirmed, the publishing of all subsequent messages will be blocked. This method provides a throughput of no more than hundreds of published messages per second. Of course, this may be sufficient for some applications.

//Single confirmation
    public static void publishMessageIndividually() throws Exception{
        Channel channel = RabbitMqUtils.getChannel();
        String queueName = UUID.randomUUID().toString();
        channel.queueDeclare(queueName, false, false, false, null);
        //Open release confirmation
        channel.confirmSelect();
        long begin = System.currentTimeMillis();

        for (int i = 0;i < MESSAGE_COUNT;i++){
            String message = i + "";
            channel.basicPublish("",queueName,null,message.getBytes());
            //If the server returns false or does not return within the timeout, the producer can resend the message
            boolean flag = channel.waitForConfirms();
            if(flag){
                System.out.println("Message sent successfully");
            }

        }

        long end = System.currentTimeMillis();
        System.out.println("release" + MESSAGE_COUNT + "Separate confirmation messages,time consuming" + (end - begin) +
                "ms");
    }

4.2.3. Batch confirmation release

The above method is very slow. Compared with a single message waiting for confirmation, publishing a batch of messages first and then confirming together can greatly improve the throughput. Of course, the disadvantage of this method is that when a failure causes a problem in publishing, we don't know which message has a problem. We must save the whole batch in memory, To record important information and then republish the message. Of course, this scheme is still synchronous and blocks the release of messages.

//Batch confirmation
    public static void publishMessageBatch() throws Exception{
        Channel channel = RabbitMqUtils.getChannel();
        String queueName = UUID.randomUUID().toString();
        channel.queueDeclare(queueName, false, false, false, null);
        //Open release confirmation
        channel.confirmSelect();
        //Bulk confirmation message size
        int batchSize = 100;
        //Number of unacknowledged messages
        int outstandingMessageCount = 0;

        long begin = System.currentTimeMillis();

        for (int i = 0;i < MESSAGE_COUNT;i++){
            String message = i + "";
            channel.basicPublish("",queueName,null,message.getBytes());
            outstandingMessageCount++;
            if(outstandingMessageCount == batchSize){
                channel.waitForConfirms();
                outstandingMessageCount = 0;
            }
        }

        //To ensure that there are still unconfirmed messages left, confirm again
        if (outstandingMessageCount > 0) {
            channel.waitForConfirms();
        }

        long end = System.currentTimeMillis();
        System.out.println("release" + MESSAGE_COUNT + "Batch confirmation messages,time consuming" + (end - begin) +
                "ms");
    }

4.2.4. Asynchronous confirmation Publishing

Although the programming logic of asynchronous confirmation is more complex than the above two, it has the highest cost performance. It can not be said whether it is reliable or efficient. It uses callback function to achieve reliable message delivery. This middleware also ensures whether it is delivered successfully through function callback. Let's explain in detail how asynchronous confirmation is realized.

4.2.5 how to handle asynchronous unacknowledged messages

The best solution is to put unconfirmed messages into a memory based queue that can be accessed by the publishing thread. For example, use the ConcurrentLinkedQueue queue to transfer messages between confirm callbacks and the publishing thread.

    //Asynchronous batch confirmation
    public static void publishMessageAsync() throws Exception{
        Channel channel = RabbitMqUtils.getChannel();
        String queueName = UUID.randomUUID().toString();
        channel.queueDeclare(queueName, false, false, false, null);
        //Open release confirmation
        channel.confirmSelect();
        /**
         * A thread safe and orderly hash table, which is suitable for high concurrency
         * 1.Easily associate sequence numbers with messages
         * 2.Easily delete entries in batches as long as the serial number is given
         * 3.Support concurrent access
         */
        ConcurrentSkipListMap<Long,String> outstandingConfirms = new ConcurrentSkipListMap<>();

        /**
         * A callback to acknowledge receipt of a message
         * 1.Message sequence number
         * 2.true Messages less than or equal to the current serial number can be confirmed
         * false Confirm current serial number message
         */
        ConfirmCallback ackCallback = (deliveryTag, multiple) -> {
            if (multiple) {
                //The returned unconfirmed message is less than or equal to the current serial number. It is a map
                ConcurrentNavigableMap<Long, String> confirmed =
                        outstandingConfirms.headMap(deliveryTag, true);
                //Clear this part of the unacknowledged message
                confirmed.clear();
            }else{
                //Only messages with the current serial number are cleared
                outstandingConfirms.remove(deliveryTag);
            }
        };

        ConfirmCallback nackCallback = (deliveryTag, multiple) -> {
            String message = outstandingConfirms.get(deliveryTag);
            System.out.println("Published messages"+message+"Unconfirmed, serial number"+deliveryTag);
        };


        /**
         * Add a listener for asynchronous acknowledgement
         * 1.Callback to acknowledge receipt of message
         * 2.Callback for message not received
         */
        channel.addConfirmListener(ackCallback,nackCallback);

        long begin = System.currentTimeMillis();
        for (int i = 0;i < MESSAGE_COUNT;i++){
            String message = i + "";
            /**
             * channel.getNextPublishSeqNo()Gets the sequence number of the next message
             * An association is made with the message body through the serial number
             * All are unacknowledged message bodies
             */
            outstandingConfirms.put(channel.getNextPublishSeqNo(),message);
            channel.basicPublish("",queueName,null,message.getBytes());

        }

        long end = System.currentTimeMillis();
        System.out.println("release" + MESSAGE_COUNT + "Asynchronous acknowledgement message,time consuming" + (end - begin) +
                "ms");
    }

4.2.6. Comparison of the above three release confirmation speeds

• Publish message separately
  • Synchronization waiting for confirmation is simple, but the throughput is very limited.
• Batch publish message
  • Batch synchronization waits for confirmation. It is simple and reasonable throughput. Once there is a problem, it is difficult to infer that the message has a problem.
• Asynchronous processing:
  • Optimal performance and resource usage can be well controlled in case of errors, but it is slightly difficult to implement

    public static void main(String[] args) throws Exception {
//        1. Single confirmation
//        ConfirmMessage.publishMessageIndividually(); // Issue 1000 individual confirmation messages, taking 630ms
//        2. Batch confirmation
//        ConfirmMessage.publishMessageBatch(); // It takes 126ms to publish 1000 batch confirmation messages
//        3. Asynchronous batch confirmation
        ConfirmMessage.publishMessageAsync();  //Publish 1000 asynchronous confirmation messages, taking 40ms
    }

4.4 persistence summary

Must do

• Queue persistence
• Message persistence
• Release confirmation

Reference video: RabbitMQ tutorial in Silicon Valley - quickly master MQ message middleware