RabbitMQ delay queue

Posted by mikejs on Sat, 08 Jan 2022 09:38:41 +0100

1 delay queue concept

The delay queue is orderly. The most important feature is reflected in its delay attribute. The elements in the delay queue want to be taken out and processed after or before the specified time. In short, the delay queue is a queue used to store the elements that need to be processed at the specified time.

2 delay queue usage scenario

  • If the order is not paid within ten minutes, it will be automatically cancelled.
  • If the newly created store has not uploaded products within ten days, it will automatically send a message reminder.
  • After successful registration, if the user does not log in within three days, a short message reminder will be sent.
  • The user initiates a refund, and if it is not handled within three days, notify the relevant operation personnel.
  • After the scheduled meeting, all participants shall be notified to attend the meeting ten minutes before the scheduled time point.

   these scenarios have a feature that a task needs to be completed at a specified time point after or before an event occurs. For example, when an order generation event occurs, check the payment status of the order ten minutes later, and then close the unpaid order; It seems that using a scheduled task, polling the data all the time, checking once a second, taking out the data to be processed, and then processing is finished? If the amount of data is small, this can be done. For example, for the demand of "automatic settlement if the bill is not paid within one week", if the time is not strictly limited, but a week in a loose sense, running a regular task every night to check all unpaid bills is indeed a feasible scheme. However, for scenes with large amount of data and strong timeliness, For example: "if the order is not paid within ten minutes, it will be closed ", there may be a lot of unpaid order data in the short term, even reaching the level of millions or even tens of millions during the event. It is obviously undesirable to still use polling for such a huge amount of data. It is likely that the inspection of all orders can not be completed in one second. At the same time, it will put great pressure on the database, fail to meet business requirements and have low performance.

3 TTL in rabbitmq

  what is TTL? TTL is the attribute of a message or queue in RabbitMQ, indicating the maximum lifetime of a message or all messages in the queue, in milliseconds. In other words, if a message has the TTL attribute set or enters the queue with the TTL attribute set, the message will become a "dead letter" if it is not consumed within the time set by the TTL. If both the TTL of the queue and the TTL of the message are configured, the smaller value will be used. There are two ways to set the TTL.

3.1 message setting TTL

One way is to set TTL for each message

3.2 queue setting TTL

Another way is to set the "x-message-ttl" attribute of the queue when creating the queue

3.3 difference between the two

   if the TTL attribute of the queue is set, once the message expires, it will be discarded by the queue (if the dead letter queue is configured, it will be thrown into the dead letter queue). In the second way, even if the message expires, it will not necessarily be discarded immediately, because whether the message expires is determined before it is delivered to the consumer. If the current queue has a serious message backlog, Then expired messages may survive for a long time; In addition, it should be noted that if TTL is not set, the message will never expire. If TTL is set to 0, the message will be discarded unless it can be delivered directly to the consumer at this time.
    in the previous article, we introduced the dead letter queue and just introduced TTL. So far, the two elements of using RabbitMQ to realize the delay queue have been collected. Next, we just need to integrate them and add a little seasoning to make the delay queue fresh. Think about it. The delay queue is just how long you want messages to be processed. TTL can just make messages become dead letters after how long they are delayed. On the other hand, messages that become dead letters will be delivered to the dead letter queue. In this way, consumers only need to consume the messages in the dead letter queue all the time, because the messages inside are messages that they want to be processed immediately.

4. Integrate SpringBoot

4.1 adding dependencies

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-amqp</artifactId>
</dependency>

4.2 yml configuration

spring:
  rabbitmq:
    host: 127.0.0.1
    username: guest
    password: guest
    port: 5672

5 queue TTL

5.1 code architecture diagram

    create two queues QA and QB, and set their TTL S to 10S and 40S respectively. Then create a switch X and a dead letter switch Y, both of which are direct. Create a dead letter queue QD, and their binding relationship is as follows:

5.2 configuration file class code

@Configuration
public class TtlQueueConfig {
    /**
     * Name of common switch
     */
    private static final String X_EXCHANGE = "X";
    /**
     * Name of dead letter switch
     */
    private static final String Y_DEAD_LETTER_EXCHANGE = "Y";
    /**
     * Name of the normal queue
     */
    private static final String QUEUE_A = "QA";
    /**
     * Name of normal queue QB
     */
    private static final String QUEUE_B = "QB";
    /**
     * Name of the dead letter queue
     */
    private static final String DEAD_LETTER_QUEUE = "QD";

    /**
     * Declare common switch X_EXCHANGE
     */
    @Bean
    public DirectExchange xExchange() {
        return new DirectExchange(X_EXCHANGE);
    }

    /**
     * Declare dead letter switch Y_DEAD_LETTER_EXCHANGE
     */
    @Bean
    public DirectExchange yExchange() {
        return new DirectExchange(Y_DEAD_LETTER_EXCHANGE);
    }

    /**
     * Declare that the general queue QA TTL is 10 s
     */
    @Bean
    public Queue queueA() {
        Map<String, Object> arguments = new HashMap<>(3);
        //Set up dead letter switch
        arguments.put("x-dead-letter-exchange", Y_DEAD_LETTER_EXCHANGE);
        //Set dead letter RoutingKey
        arguments.put("x-dead-letter-routing-key", "YD");
        //Set expiration time
        arguments.put("x-message-ttl", 10000);
        return QueueBuilder.durable(QUEUE_A).withArguments(arguments).build();
    }

    /**
     * Declare that the QB TTL of the normal queue is 40 s
     */
    @Bean
    public Queue queueB() {
        Map<String, Object> arguments = new HashMap<>(3);
        //Set up dead letter switch
        arguments.put("x-dead-letter-exchange", Y_DEAD_LETTER_EXCHANGE);
        //Set dead letter RoutingKey
        arguments.put("x-dead-letter-routing-key", "YD");
        //Set expiration time
        arguments.put("x-message-ttl", 40000);
        return QueueBuilder.durable(QUEUE_B).withArguments(arguments).build();
    }

    /**
     * Declare dead letter queue QD
     */
    @Bean
    public Queue queueD() {
        return QueueBuilder.durable(DEAD_LETTER_QUEUE).build();
    }


    /**
     * queueA  Bind common switch X_EXCHANGE
     */
    @Bean
    public Binding queueABindingX(Queue queueA, DirectExchange xExchange) {
        return BindingBuilder.bind(queueA).to(xExchange).with("XA");
    }

    /**
     * queueB  Bind common switch X_EXCHANGE
     */
    @Bean
    public Binding queueBBindingX(Queue queueB, DirectExchange xExchange) {
        return BindingBuilder.bind(queueB).to(xExchange).with("XB");
    }

    /**
     * queueD Bind dead letter switch DEAD_LETTER_QUEUE
     */
    @Bean
    public Binding queueDBindingY(Queue queueD, DirectExchange yExchange) {
        return BindingBuilder.bind(queueD).to(yExchange).with("YD");
    }
}

5.3 message producer code

@RestController
@RequestMapping("ttl")
@Slf4j
public class SendMsgController {

    @Autowired
    private RabbitTemplate rabbitTemplate;

    @GetMapping("/sendMsg/{message}")
    public void sendMsg(@PathVariable("message") String message) {
        log.info("Current time:{}，Send a message to two TTL Queue:{}", new Date(), message);
        rabbitTemplate.convertAndSend("X", "XA", "Message from TTL For 10 s Queue for:" + message);
        rabbitTemplate.convertAndSend("X", "XB", "Message from TTL For 40 s Queue for:" + message);
    }
}

5.4 message consumer code

@Slf4j
@Component
public class DeadLetterQueueConsumer {
    /**
     * receive messages
     */
    @RabbitListener(queues = "QD")
    public void receiveD(Message message, Channel channel) {
        String msg = new String(message.getBody(), StandardCharsets.UTF_8);
        log.info("current time {}，Messages received from dead letter queue:{}", new Date(), msg);
    }
}

Initiate request: http://localhost:8080/ttl/sendMsg/ Delayed message

    the first message becomes a dead letter message after 10S and is consumed by the consumer. The second message becomes a dead letter message after 40S and is consumed. Such a delay queue is completed.
   however, if it is used in this way, it is not necessary to add a queue every time a new time demand is added. There are only two time options: 10S and 40S. If it needs to be processed after one hour, it is necessary to add a queue with TTL of one hour. If it is a preset meeting room and notify such a scenario in advance, Isn't it necessary to add countless queues to meet the demand?

6 delay queue optimization

6.1 code architecture diagram

A new queue QC is added here. The binding relationship is as follows. The queue does not set TTL time

6.2 add the following code to the configuration file class

/**
 * Name of normal queue
 */
private static final String QUEUE_C = "QC";

/**
 * Declare that normal queue QC does not set TTL (optimization)
 */
@Bean
public Queue queueC() {
    Map<String, Object> arguments = new HashMap<>(3);
    //Set up dead letter switch
    arguments.put("x-dead-letter-exchange", Y_DEAD_LETTER_EXCHANGE);
    //Set dead letter RoutingKey
    arguments.put("x-dead-letter-routing-key", "YD");
    return QueueBuilder.durable(QUEUE_C).withArguments(arguments).build();
}

/**
 * queueC  Bind common switch X_EXCHANGE
 */
@Bean
public Binding queueCBindingX(Queue queueC, DirectExchange xExchange) {
    return BindingBuilder.bind(queueC).to(xExchange).with("XC");
}

6.3 message producer code

/**
 * Start sending message TTL
 * Delay queue optimization
 */
@ApiOperation("Message interface with expiration time")
@GetMapping("/sendExpirationMsg/{message}/{ttlTime}")
public void sendMessage(@PathVariable("message") String message, @PathVariable("ttlTime") String ttlTime) {
    log.info("Current time:{}，Send a message with a duration of{}Milliseconds of information to the queue QC: {}", new Date(), ttlTime, message);
    rabbitTemplate.convertAndSend("X", "XC", message, msg -> {
        //Set the delay time for sending messages
        msg.getMessageProperties().setExpiration(ttlTime);
        return msg;
    });
}

Initiate request

  • http://localhost:8080/ttl/sendExpirationMsg/40 Messages / 40000 seconds
  • http://localhost:8080/ttl/sendExpirationMsg/10 Messages per 10000 seconds


    it seems that there is no problem, but at the beginning, I introduced how to set TTL on message properties, Messages may not "die" on time, because RabbitMQ will only check whether the first message expires. If it expires, it will be thrown into the dead letter queue. If the delay time of the first message is very long and the delay time of the second message is very short, the second message will not be executed first.

7 RabbitMQ plug-in implements delay queue

   the problem mentioned above is indeed a problem. If the TTL on message granularity cannot be realized and it dies in time at the set TTL time, it cannot be designed into a general delay queue. Then how to solve it? Next, let's solve the problem.

7.1 installing the delay plug-in

Download on the official website https://www.rabbitmq.com/community-plugins.html , Download rabbitmq_delayed_message_exchange plug-in, and then unzip and place it in the plug-in directory of RabbitMQ.

I won't say much about this. Baidu has a lot of installation methods.

  • Before installation:

  • After installation: multiple switch types

7.2 code architecture diagram

A new queue, delayed, is added here Queue, a custom switch delayed For exchange, the binding relationship is as follows:

7.3 configuration file class code

   in our customized switch, this is a new switching type. This type of message supports the delayed delivery mechanism. After the message is delivered, it will not be delivered to the target queue immediately, but will be stored in the mnesia (a distributed data system) table. When the delivery time is reached, it will be delivered to the target queue.

@Configuration
public class DelayedQueueConfig {
    /**
     * Delay switch name
     */
    public static final String DELAYED_EXCHANGE_NAME = "delayed.exchange";
    /**
     * Delay queue name
     */
    public static final String DELAYED_QUEUE_NAME = "delayed.queue";
    /**
     * routingKey
     */
    public static final String DELAYED_ROUTING_KEY = "delayed.routingKey";

    @Bean
    public Queue delayedQueue() {
        return QueueBuilder.durable(DELAYED_QUEUE_NAME).build();
    }

    /**
     * Custom delay switch
     */
    @Bean
    public CustomExchange delayedExchange() {
        Map<String, Object> arguments = new HashMap<>();
        //Delay type
        arguments.put("x-delayed-type", "direct");
        return new CustomExchange(DELAYED_EXCHANGE_NAME, "x-delayed-message", true, false, arguments);
    }

    @Bean
    public Binding delayedQueueBindingDelayedExchange(Queue delayedQueue, CustomExchange delayedExchange) {
        return BindingBuilder.bind(delayedQueue).to(delayedExchange).with(DELAYED_ROUTING_KEY).noargs();
    }
}

7.4 message producer code

/**
 * Plug in based delayed sending time
 */
@GetMapping("/sendDepayMsg/{message}/{delayTime}")
public void sendDepayMsg(@PathVariable("message") String message, @PathVariable("delayTime") Integer delayTime) {
    log.info("Current time:{}，Send a message with a duration of{}Millisecond message to delayTime queue delayed.queue: {}", new Date(), delayTime, message);
    rabbitTemplate.convertAndSend("delayed.exchange", "delayed.routingKey", message, msg -> {
        //Set the delay time for sending messages
        msg.getMessageProperties().setDelay(delayTime);
        return msg;
    });
}

7.5 message consumer code

@Component
@Slf4j
public class DelayQueueConsumer {
    @RabbitListener(queues = DelayedQueueConfig.DELAYED_QUEUE_NAME)
    public void receiveDelayedQueue(Message message, Channel channel) {
        String msg = new String(message.getBody(), StandardCharsets.UTF_8);
        log.info("Current time:{}，Message received from delay queue:{}", new Date(), msg);
    }
}

Initiate request

  • http://localhost:9045/ttl/sendDepayMsg/40 Messages / 40000 seconds
  • http://localhost:9045/ttl/sendDepayMsg/40 Messages per 10000 seconds


10 seconds were consumed first, which is in line with our expectations

8 summary

   delay queue is very useful in situations where delay processing is required. Using RabbitMQ to implement delay queue can make good use of RabbitMQ's characteristics, such as reliable message sending, reliable message delivery and dead letter queue to ensure that messages are consumed at least once and messages that have not been correctly processed will not be discarded. In addition, through the characteristics of RabbitMQ cluster, the single point of failure problem can be well solved, and the delay queue will not be unavailable or messages will not be lost because a single node hangs up.
   of course, there are many other options for delay queue, such as using Java's DelayQueue, Redis's zset, Quartz or kafka's time wheel. These methods have their own characteristics, depending on the applicable scenarios.

Topics: Java RabbitMQ

Hot Topics

©2024 Programmer Think