Idempotency

concept

The results of one request or multiple requests initiated by the user for the same operation are consistent, and there will be no side effects due to multiple clicks.

The simplest example is payment. The user pays after purchasing goods, and the payment deduction is successful. However, when the result is returned, the network is abnormal. At this time, the money has been deducted. The user clicks the button again, and the second deduction will be made. The result is returned successfully. The user queries the balance and finds that more money has been deducted, and the daily record has become two.

In the previous single application system, we only need to put the data operation into the transaction and roll back immediately when an error occurs, but there may be network interruption or exceptions when responding to the client

Solution ideas

• The idempotency of MQ consumers is generally solved by using the global ID or writing a unique ID, such as timestamp or UUID or order. Consumers can also use the ID of MQ to judge whether they consume messages in MQ, or they can generate a global unique ID according to their own rules. Each time they consume a message, they use the ID to judge whether the message has been consumed.
• During the business peak of massive order generation, the manufacturer may send messages repeatedly. At this time, the consumer must realize idempotency, which means that our messages will never be consumed many times, even if we receive the same message. The mainstream idempotency in the industry has two operations: a. unique ID + fingerprint code mechanism, which uses the database primary key to remove duplication, and b. It is realized by using the atomicity of redis

Unique ID + fingerprint code mechanism:

Fingerprint code: the unique information code given by some of our rules or timestamps plus other services. It is not necessarily generated by our system. It is basically spliced from our business rules, but we must ensure the uniqueness. Then we use the query statement to judge whether this id exists in the database. The advantage is to realize a simple splicing, Then query and judge whether it is repeated; The disadvantage is that in the case of high concurrency, if it is a single database, there will be a write performance bottleneck. Of course, we can also use database and table to improve the performance, but it is not our most recommended way.

Redis atomicity:

Using redis to execute setnx command naturally has idempotency, so as to realize no repeated consumption

Priority queue

Usage scenario

In our system, there is a scene of urging payment of orders. Taobao will push the orders placed by our customers on tmall to us in time. If they fail to pay within the time set by the user, they will push a SMS reminder to the user. It's a simple function, right? However, tmall businesses must be divided into large customers and small customers for us, right, For example, big businesses such as apple and Xiaomi can at least make us a lot of profits a year, so it should be. Of course, their orders must be given priority. In the past, our back-end system used redis. To store timed polling. We all know that redis can only use List as a simple message queue, but can not achieve a priority scenario, Therefore, after the order quantity is large, RabbitMQ is used for transformation and optimization. If it is found that the order of a large customer is given a relatively high priority, otherwise it is the default priority.

How to add priority to a queue

a. Console page add

b. Add priority to code in queue

To enable the queue to achieve priority, the following things need to be done: the queue needs to be set as the priority queue, the message needs to be set as the priority of the message, and the consumer needs to wait until the message has been sent to the queue, because this is the opportunity to sort the messages

producer

public class Task01 {
    // Queue name
    public static final String QUEUE_NAME = "hello";

    // Send a large number of messages
    public static void main(String[] args) throws Exception {
        Channel channel = RabbitMqUtils.getInstance().getChannel();
            // Declaration of queue
            channel.queueDeclare(QUEUE_NAME, false, false, false, null);
         
        Map<String, Object> arguments = new HashMap<>();
        //The official allowable range is 0-255. Set 10 here and the allowable optimization level range is 0-10. Do not set it too large to waste CPU and memory
        //Set queue priority
        arguments.put("x-max-priority",10);
        channel.queueDeclare(QUEUE_NAME,true,false,false,arguments);
        // Send a message
        for (int i = 0; i < 10; i++) {
            String message = "info" + i;
            if(i == 5){
                //Set the priority of the current message --- it cannot exceed the maximum priority of the message set by the queue
                AMQP.BasicProperties properties = new AMQP.BasicProperties().builder().priority(5).build();
                channel.basicPublish("",QUEUE_NAME,properties,message.getBytes(StandardCharsets.UTF_8));
            }else {
                channel.basicPublish("",QUEUE_NAME,null,message.getBytes(StandardCharsets.UTF_8));

            }
        }
        System.out.println("Message sent!");
    }
}

consumer

public class Worker01 {

    // Queue name
    public static final String QUEUE_NAME = "hello";

    // Accept message
    public static void main(String[] args) throws IOException, TimeoutException {
        Channel channel = RabbitMqUtils.getInstance().getChannel();

        // Declare accept message
        DeliverCallback deliverCallback = (consumerTag,message) -> {
            System.out.println(new String(message.getBody()));
        };
        // Declare cancellation message
        CancelCallback cancelCallback = consumer -> {
            System.out.println("Message consumption interrupted");
        };
        System.out.println("C2 Waiting to receive message.......");
        channel.basicConsume(QUEUE_NAME,true,deliverCallback,cancelCallback);

    }
}

Inert queue

Usage scenario

RabbitMQ introduced the concept of lazy queues from version 3.6.0. Inert queue will store messages to disk as much as possible, and they will be loaded into memory when consumers consume the corresponding messages. One of its important design goals is to support longer queues, that is, to support more message storage. When consumers cannot consume messages for a long time due to various reasons (such as offline, downtime, or shutdown due to maintenance, etc.), it is necessary to have an inert queue.

By default, when the producer sends messages to RabbitMQ, the messages in the queue will be stored in memory as much as possible, so that messages can be sent to consumers more quickly. Even for persistent messages, a backup will reside in memory while being written to disk. When RabbitMQ needs to free memory, it will page the messages in memory to disk. This operation will take a long time, block the queue operation, and then fail to receive new messages. Although RabbitMQ developers have been upgrading relevant algorithms, the effect is not ideal, especially when the message volume is very large.

Two modes

Queues have two modes: default and lazy. The default mode is the default mode, and there is no need to make any change in versions before 3.6.0. Lazy mode is the mode of inert queue, which can be set in the parameter when calling the channel.queueDeclare method or by Policy. If a queue is set by both methods, the Policy method has higher priority. If you want to change the mode of an existing queue by declaration, you can only delete the queue first and then declare a new queue again.

When the queue is declared, you can set the mode of the queue through the "x-queue-mode" parameter, with the values of "default" and "lazy". The following example demonstrates the declaration details of an inert queue:

Map<String, Object> args = new HashMap<String, Object>();
args.put("x-queue-mode","lazy");
channel.queueDeclare( "myqueue", false, false, false,args);

Memory comparison

When sending 1 million service messages, each message occupies about 1KB, the memory occupied by the ordinary queue is 1.2GB, while the inert queue only occupies 1.5MB