RabbitMQ learning -- learning of delay queue

Posted by sickness01 on Fri, 14 Jan 2022 08:12:01 +0100

Concept of delay queue

The delay queue is orderly. Its most important feature is reflected in its delay attribute. The elements in the delay queue * * want to be taken out and processed after or before the specified time. In short, the delay queue is a queue used to store the elements that need to be processed at the specified time.

Delay queue in RabbitMQ

What is TTL? TTL is the attribute of a message or queue in RabbitMQ, indicating the maximum lifetime of a message or all messages in the queue, in milliseconds. In other words, if a message has the TTL attribute set or enters the queue with the TTL attribute set, the message will become a "dead letter" if it is not consumed within the time set by the TTL. If the TTL of the queue and the TTL of the message are configured at the same time, the smaller value will be used. There are two ways to set the TTL;

Queue settings TTL

When creating a queue, set the "x-message-ttl" attribute of the queue

Message setting TTL

Set TTL for each message:

The difference between the two

If the TTL attribute of the queue is set, once the message expires, it will be discarded by the queue (if the dead letter queue is configured, it will be thrown into the dead letter queue). In the second way, even if the message expires, it may not be discarded immediately, because * * whether the message expires is determined before it will be delivered to the consumer * * if the current queue has a serious message backlog, Then expired messages may survive for a long time; In addition, it should be noted that if TTL is not set, the message will never expire. If TTL is set to 0, the message will be discarded unless it can be delivered directly to the consumer at this time.
In the previous section, we introduced the dead letter queue and just introduced TTL. So far, the two elements of using RabbitMQ to realize the delay queue have been collected. Next, we just need to integrate them and add a little seasoning to make the delay queue fresh. Think about it. The delay queue is just how long you want messages to be processed. TTL can just make messages become dead letters after how long they are delayed. On the other hand, messages that become dead letters will be delivered to the dead letter queue. In this way, consumers only need to consume the messages in the dead letter queue all the time, because the messages inside are messages that they want to be processed immediately

Usage scenario of delay queue

1. If the order is not paid within ten minutes, it will be automatically cancelled;
2. If the newly created store has not uploaded goods within ten days, it will automatically send a message reminder;
3. After successful registration, if the user does not log in within three days, a short message reminder will be sent;
4. The user initiates a refund, and if it is not handled within three days, notify the relevant operators;
5. After the scheduled meeting, all participants shall be notified to attend the meeting ten minutes before the scheduled time point;

These scenarios have a feature that a task needs to be completed at a specified time point after or before an event occurs. For example, when an order generation event occurs, check the payment status of the order ten minutes later, and then close the unpaid order; It seems that using a scheduled task, polling the data all the time, checking once a second, taking out the data to be processed, and then processing is finished? If the amount of data is small, this can be done. For example, for the demand of "automatic settlement if the bill is not paid within one week", if the time is not strictly limited, but a week in a loose sense, running a regular task every night to check all unpaid bills is indeed a feasible scheme. However, for scenes with large amount of data and strong timeliness, For example: "if the order is not paid within ten minutes, it will be closed ", there may be a lot of unpaid order data in the short term, even reaching the level of millions or even tens of millions during the event. It is obviously undesirable to still use polling for such a huge amount of data. It is likely that the inspection of all orders can not be completed in one second. At the same time, it will put great pressure on the database, fail to meet business requirements and have low performance.

SpringBoot integrates RabbitMQ

Create a Springboot project and introduce related dependencies:

<dependencies>

        <!--RabbitMQ rely on-->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-amqp</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.47</version>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
        </dependency>
        <!--swagger-->
        <dependency>
            <groupId>io.springfox</groupId>
            <artifactId>springfox-swagger2</artifactId>
            <version>2.9.2</version>
        </dependency>
        <dependency>
            <groupId>io.springfox</groupId>
            <artifactId>springfox-swagger-ui</artifactId>
            <version>2.9.2</version>
        </dependency>
        <!--RabbitMQ Test dependency-->
        <dependency>
            <groupId>org.springframework.amqp</groupId>
            <artifactId>spring-rabbit-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

configuration file

spring.rabbitmq.host=192.168.37.139
spring.rabbitmq.port=5672
spring.rabbitmq.username=admin
spring.rabbitmq.password=123456

Swagger configuration

/**
 * @author vleus
 * @date 2021 16:34, July 25
 */
@Configuration
@EnableSwagger2
public class SwaggerConfig {

    @Bean
    public Docket webApiConfig(){
        return new Docket(DocumentationType.SWAGGER_2)
                .groupName("webApi")
                .apiInfo(webApiInfo())
                .select()
                .build();
    }
    private ApiInfo webApiInfo(){
        return new ApiInfoBuilder()
                .title("rabbitmq Interface documentation")
                .description("This document describes rabbitmq Microservice interface definition")
                .version("1.0")
                .contact(new Contact("enjoy6288", "http://baidu.com",
                        "1071309217@qq.com"))
                .build();
    }
}

Queue TTL

Code architecture diagram

Create two queues QA and QB, and set the TTL of the two queues to 10S and 40S respectively. Then create a common switch X and a dead letter switch Y, both of which are direct. Create a dead letter queue QD, and their binding relationship is as follows;

Configuration file class code

Declare queues, switches, and bindings between them

package com.vleus.rabbitmq.springbootrabbitmq.config;

import org.springframework.amqp.core.*;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.util.HashMap;
import java.util.Map;

/**
 * @author vleus
 * @date 2021 17:14, July 25
 * TTL Queue: profile class code
 */
@Configuration
public class TLLQueueConfig {

    //Common switch name
    public static final String NORMAL_EXCHANGE = "X";
    //Dead letter switch name
    public static final String DEAD_LETTER_EXCHANGE = "Y";
    //Common queue name 1
    public static final String NORMAL_QUEUE_A = "QA";
    //Common queue name 2
    public static final String NORMAL_QUEUE_B = "QB";
    //Dead letter queue
    public static final String DEAD_LETTER_QUEUE = "QD";

    //Declare a common switch and start a switch alias
    @Bean("xEXCHANGE")
    public DirectExchange normalExchange() {
        return new DirectExchange(NORMAL_EXCHANGE);
    }

    @Bean("yEXCHANGE")
    public DirectExchange deadLetterExchange() {
        return new DirectExchange(DEAD_LETTER_EXCHANGE);
    }

    //Declaration queue
    @Bean("queueA")
    public Queue queueA() {
        Map<String, Object> argumentMap = new HashMap<>(3);
        //Set up dead letter switch
        argumentMap.put("x-dead-letter-exchange", DEAD_LETTER_EXCHANGE);
        //Set dead letter RoutingKey
        argumentMap.put("x-dead-letter-routing-key", "YD");
        //Set TTL
        argumentMap.put("x-message-ttl", 10000);
        return QueueBuilder.durable(NORMAL_QUEUE_A).withArguments(argumentMap).build();
    }

    @Bean("queueB")
    public Queue queueB() {
        Map<String, Object> argumentMap = new HashMap<>(3);
        //Set up dead letter switch
        argumentMap.put("x-dead-letter-exchange", DEAD_LETTER_EXCHANGE);
        //Set dead letter RoutingKey
        argumentMap.put("x-dead-letter-routing-key", "YD");
        //Sets the TTL unit to ms
        argumentMap.put("x-message-ttl", 40000);
        return QueueBuilder.durable(NORMAL_QUEUE_B).withArguments(argumentMap).build();
    }

    //Dead letter queue
    @Bean("deadLetterQueue")
    public Queue deadLetterQueue() {
        return QueueBuilder.durable(DEAD_LETTER_QUEUE).build();
    }

    //Queue and switch binding
    @Bean
    public Binding QueueABindingX(@Qualifier("queueA")Queue queueA,
                                  @Qualifier("xEXCHANGE")DirectExchange exchangeA) {
      return BindingBuilder.bind(queueA).to(exchangeA).with("XA");
    }

    @Bean
    public Binding QueueABindingY(@Qualifier("queueB")Queue queueA,
                                  @Qualifier("xEXCHANGE")DirectExchange exchangeA) {
        return BindingBuilder.bind(queueA).to(exchangeA).with("XB");
    }

    @Bean
    public Binding QueueABindingD(@Qualifier("deadLetterQueue")Queue queueA,
                                  @Qualifier("yEXCHANGE")DirectExchange exchangeA) {
        return BindingBuilder.bind(queueA).to(exchangeA).with("YD");
    }
}

Producer code

package com.vleus.rabbitmq.springbootrabbitmq.controller;

import lombok.extern.slf4j.Slf4j;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.Date;

/**
 * @author vleus
 * @date 2021 17:35, July 25
 *  Act as producer: send delayed messages
 */
@Slf4j
@RestController
@RequestMapping(value ="/ttl")
public class SendMessageController {

    @Autowired
    private RabbitTemplate rabbitTemplate;

    @GetMapping("/sendMsg/{message}")
    public void sendMsg(@PathVariable("message") String message) {
        log.info("Current time is: {},Send a message to two ttl queue: {}", new Date().toString(), message);
        rabbitTemplate.convertAndSend("X","XA","Message from ttl For 10 s: " + message);
        rabbitTemplate.convertAndSend("X","XB","Message from ttl For 40 s: " + message);
    }
}

Consumer code

package com.vleus.rabbitmq.springbootrabbitmq.consumer;

import com.rabbitmq.client.Channel;
import lombok.extern.slf4j.Slf4j;
import org.springframework.amqp.core.Message;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.stereotype.Component;

import java.util.Date;

/**
 * @author vleus
 * @date 2021 17:40, July 25
 *
 * Queue ttl consumers
 */
@Slf4j
@Component
public class DealLetterQueueConsumer {

    //receive messages
    @RabbitListener(queues = "QD")
    public void receiveD(Message message, Channel channel) throws Exception {
        String msg = new String(message.getBody(), "UTF-8");
        log.info("current time : {},The message received from the dead letter queue is: {}", new Date().toString(), msg);
    }
}

Request a record:

http://localhost:8080/ttl/sendMsg/ are you there

The result is:

Delay queue optimization

However, if it is used in this way, it is necessary to add a queue every time a new time demand is added. There are only two time options: 10S and 40S. If it needs to be processed after one hour, it is necessary to add a queue with TTL of one hour. If it is to book a meeting room and notify it in advance, Isn't it necessary to add countless queues to meet the demand?

Code architecture diagram

Add queue without expiration time

Configuration file class code

package com.vleus.rabbitmq.springbootrabbitmq.config;

import org.springframework.amqp.core.*;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.util.HashMap;
import java.util.Map;

/**
 * @author vleus
 * @date 2021 17:14, July 25
 * TTL Queue: profile class code
 */
@Configuration
public class TLLQueueConfig {

    //Common switch name
    public static final String NORMAL_EXCHANGE = "X";
    //Dead letter switch name
    public static final String DEAD_LETTER_EXCHANGE = "Y";
    //Common queue name 1
    public static final String NORMAL_QUEUE_A = "QA";
    //Common queue name 2
    public static final String NORMAL_QUEUE_B = "QB";

    //Common queue name 3, no expiration time is set
    public static final String NORMAL_QUEUE_C = "QC";

    //Dead letter queue
    public static final String DEAD_LETTER_QUEUE = "QD";

    //Declare a common switch and start a switch alias
    @Bean("xEXCHANGE")
    public DirectExchange normalExchange() {
        return new DirectExchange(NORMAL_EXCHANGE);
    }

    @Bean("yEXCHANGE")
    public DirectExchange deadLetterExchange() {
        return new DirectExchange(DEAD_LETTER_EXCHANGE);
    }

    //Declaration queue
    @Bean("queueA")
    public Queue queueA() {
        Map<String, Object> argumentMap = new HashMap<>(3);
        //Set up dead letter switch
        argumentMap.put("x-dead-letter-exchange", DEAD_LETTER_EXCHANGE);
        //Set dead letter RoutingKey
        argumentMap.put("x-dead-letter-routing-key", "YD");
        //Set TTL
        argumentMap.put("x-message-ttl", 10000);
        return QueueBuilder.durable(NORMAL_QUEUE_A).withArguments(argumentMap).build();
    }

    @Bean("queueB")
    public Queue queueB() {
        Map<String, Object> argumentMap = new HashMap<>(3);
        //Set up dead letter switch
        argumentMap.put("x-dead-letter-exchange", DEAD_LETTER_EXCHANGE);
        //Set dead letter RoutingKey
        argumentMap.put("x-dead-letter-routing-key", "YD");
        //Sets the TTL unit to ms
        argumentMap.put("x-message-ttl", 40000);
        return QueueBuilder.durable(NORMAL_QUEUE_B).withArguments(argumentMap).build();
    }

    @Bean("queueC")
    public Queue queueC() {
        Map<String, Object> argumentMap = new HashMap<>(3);
        //Set up dead letter switch
        argumentMap.put("x-dead-letter-exchange", DEAD_LETTER_EXCHANGE);
        //Set dead letter RoutingKey
        argumentMap.put("x-dead-letter-routing-key", "YD");
        return QueueBuilder.durable(NORMAL_QUEUE_C).withArguments(argumentMap).build();
    }

    //Dead letter queue
    @Bean("deadLetterQueue")
    public Queue deadLetterQueue() {
        return QueueBuilder.durable(DEAD_LETTER_QUEUE).build();
    }

    //Queue and switch binding
    @Bean
    public Binding QueueABindingX(@Qualifier("queueA")Queue queueA,
                                  @Qualifier("xEXCHANGE")DirectExchange exchangeA) {
      return BindingBuilder.bind(queueA).to(exchangeA).with("XA");
    }

    @Bean
    public Binding QueueABindingY(@Qualifier("queueB")Queue queueA,
                                  @Qualifier("xEXCHANGE")DirectExchange exchangeA) {
        return BindingBuilder.bind(queueA).to(exchangeA).with("XB");
    }

    @Bean
    public Binding QueueABindingC(@Qualifier("queueC")Queue queueC,
                                  @Qualifier("xEXCHANGE")DirectExchange exchangeA) {
        return BindingBuilder.bind(queueC).to(exchangeA).with("XC");
    }

    @Bean
    public Binding QueueABindingD(@Qualifier("deadLetterQueue")Queue queueA,
                                  @Qualifier("yEXCHANGE")DirectExchange exchangeA) {
        return BindingBuilder.bind(queueA).to(exchangeA).with("YD");
    }
}

Producer code

package com.vleus.rabbitmq.springbootrabbitmq.controller;

import lombok.extern.slf4j.Slf4j;
import org.springframework.amqp.core.MessagePostProcessor;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.Date;

/**
 * @author vleus
 * @date 2021 17:35, July 25
 *  Act as producer: send delayed messages
 */
@Slf4j
@RestController
@RequestMapping(value ="/ttl")
public class SendMessageController {

    @Autowired
    private RabbitTemplate rabbitTemplate;

    @GetMapping("/sendMsg/{message}")
    public void sendMsg(@PathVariable("message") String message) {
        log.info("Current time is: {},Send a message to two ttl queue: {}", new Date().toString(), message);
        rabbitTemplate.convertAndSend("X","XA","Message from ttl For 10 s: " + message);
        rabbitTemplate.convertAndSend("X","XB","Message from ttl For 40 s: " + message);
    }

    //Start sending messages and set ttl
    @GetMapping("/sendExpireMsg/{msg}/{ttlTime}")
    public void sendMsg(@PathVariable String msg, @PathVariable String ttlTime) {
        log.info("Current time is: {},Send a message with a duration of{}Milliseconds of information to a normal queue C: {}",
                new Date(),
                ttlTime,
                msg);

        MessagePostProcessor messagePostProcessor = message -> {
            //Set the delay length of the message
            message.getMessageProperties().setExpiration(ttlTime);
            return message;
        };
        rabbitTemplate.convertSendAndReceive("X", "XC", msg, messagePostProcessor);

    }
}

It seems that there is no problem, but at the beginning, I introduced how to set TTL on message properties, Messages may not "die" on time, because RabbitMQ will only check whether the first message expires. If it expires, it will be thrown into the dead letter queue. If the delay time of the first message is very long and the delay time of the second message is very short, the second message will not be executed first

The Rabbitmq plug-in implements delay queues

The problem mentioned above is indeed a problem. If the TTL on message granularity cannot be realized and it dies in time at the set TTL time, it cannot be designed into a general delay queue. Then how to solve it? Next, let's solve the problem;

Download plug-ins

rabbitmq_delayed_message_exchange-3.8.0.ez
Transfer it to: / usr/lib/rabbitmq/lib/rabbitmq_server-3.8.8/plugins/；
Install plug-ins

rabbitmq-plugins enable rabbitmq_delayed_message_exchange-3.8.0.ez

Restart RabbitMq

The delay time is controlled by the switch;

Code architecture diagram

A new queue, delayed, is added here Queue, a custom switch delayed For exchange, the binding relationship is as follows:

Configuration file class code

package com.vleus.rabbitmq.springbootrabbitmq.config;

import com.rabbitmq.client.AMQP;
import org.springframework.amqp.core.Binding;
import org.springframework.amqp.core.BindingBuilder;
import org.springframework.amqp.core.CustomExchange;
import org.springframework.amqp.core.Queue;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.util.HashMap;
import java.util.Map;

/**
 * @author vleus
 * @date 2021 18:38, July 25
 */
@Configuration
public class DelayQueueConfig {

    //Prepare switch
    public static final String DELAY_EXCHANGE_NAME = "delayed.exchange";
    //Prepare queue
    public static final String DELAY_QUEUE_NAME = "delayed.queue";
    //routingKey
    public static final String DELAY_ROUTING_KEY = "delayed.routing.key";

    //Claim switch
    @Bean
    public CustomExchange delayedExchange() {

        /**
         * 1,Switch name
         * 2,Switch Type 
         * 3,Need persistence
         * 4,Do you want to delete it automatically
         * 5,Other parameters
         */
        Map<String, Object> argumentMap = new HashMap<>();
        argumentMap.put("x-delayed-type", "direct");
        return new CustomExchange(DELAY_EXCHANGE_NAME,"x-delayed-message",true,false,
                argumentMap);
    }

    @Bean("delayQueue")
    public Queue delayQueue() {
        return new Queue(DELAY_QUEUE_NAME);
    }

    //binding
    @Bean
    public Binding delayQueueBindingDelayExchangge(@Qualifier("delayQueue")Queue delayQueue,
                                                   @Qualifier("delayedExchange")CustomExchange customExchange){
        return BindingBuilder.bind(delayQueue).to(customExchange).with(DELAY_ROUTING_KEY).noargs();
    }
}

Producer code

package com.vleus.rabbitmq.springbootrabbitmq.controller;

import com.vleus.rabbitmq.springbootrabbitmq.config.DelayQueueConfig;
import lombok.extern.slf4j.Slf4j;
import org.springframework.amqp.core.MessagePostProcessor;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.Date;

/**
 * @author vleus
 * @date 2021 17:35, July 25
 * Act as producer: send delayed messages
 */
@Slf4j
@RestController
@RequestMapping(value = "/ttl")
public class SendMessageController {

    @Autowired
    private RabbitTemplate rabbitTemplate;

    @GetMapping("/sendMsg/{message}")
    public void sendMsg(@PathVariable("message") String message) {
        log.info("Current time is: {},Send a message to two ttl queue: {}", new Date().toString(), message);
        rabbitTemplate.convertAndSend("X", "XA", "Message from ttl For 10 s: " + message);
        rabbitTemplate.convertAndSend("X", "XB", "Message from ttl For 40 s: " + message);
    }

    //Start sending messages and set ttl
    @GetMapping("/sendExpireMsg/{msg}/{ttlTime}")
    public void sendMsg(@PathVariable String msg, @PathVariable String ttlTime) {
        log.info("Current time is: {},Send a message with a duration of{}Milliseconds of information to a normal queue C: {}",
                new Date(),
                ttlTime,
                msg);

        MessagePostProcessor messagePostProcessor = message -> {
            //Set the delay length of the message
            message.getMessageProperties().setExpiration(ttlTime);
            return message;
        };
        rabbitTemplate.convertSendAndReceive("X", "XC", msg, messagePostProcessor);

    }

    //Send message: plug-in based message and delay time
    @GetMapping("/sendDelayMsg/{msg}/{delayTime}")
    public void sendDelayMsg(@PathVariable String msg, @PathVariable Integer delayTime) {
        log.info("Current time is: {},Send a message with a duration of{}Milliseconds of information to the delay queue delay.queue: {}",
                new Date(),
                delayTime,
                msg);

        rabbitTemplate.convertAndSend(DelayQueueConfig.DELAY_EXCHANGE_NAME,
            DelayQueueConfig.DELAY_ROUTING_KEY, msg, message -> {
                //When sending a message, delay duration: unit: ms
                message.getMessageProperties().setDelay(delayTime);
                return message;
        });
    }
}

Consumer code

package com.vleus.rabbitmq.springbootrabbitmq.consumer;

import com.rabbitmq.client.Channel;
import lombok.extern.slf4j.Slf4j;
import org.springframework.amqp.core.Message;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.stereotype.Component;

import java.util.Date;

/**
 * @author vleus
 * @date 2021 17:40, July 25
 *
 * Queue ttl consumers
 */
@Slf4j
@Component
public class DealLetterQueueConsumer {

    //receive messages
    @RabbitListener(queues = "QD")
    public void receiveD(Message message, Channel channel) throws Exception {
        String msg = new String(message.getBody(), "UTF-8");
        log.info("current time : {},The message received from the dead letter queue is: {}", new Date().toString(), msg);
    }
}

http://localhost:8080/ttl/sendDelayMsg/come on baby1/20000
http://localhost:8080/ttl/sendDelayMsg/come on baby2/2000

The implementation effect is as follows:

Topics: Java RabbitMQ MQ

Programmer Think

RabbitMQ learning -- learning of delay queue

Concept of delay queue

Delay queue in RabbitMQ

Queue settings TTL

Message setting TTL

The difference between the two

Usage scenario of delay queue

SpringBoot integrates RabbitMQ

Queue TTL

Code architecture diagram

Configuration file class code

Producer code

Consumer code

Delay queue optimization

Code architecture diagram

Configuration file class code

Producer code

The Rabbitmq plug-in implements delay queues

Download plug-ins

Code architecture diagram

Configuration file class code

Producer code

Consumer code

Hot Topics