Following the previous implementation of delay queue with rabbitMQ, Redis can also implement delay operation due to its own Zset data structure

In essence, Zset adds a sorting function to the Set structure. In addition to adding data value, it also provides another attribute score. This attribute can be specified when adding and modifying elements. After each assignment, Zset will automatically adjust the order according to the new value. It can be understood as a data table with two columns of fields, one for value and the other for sequence number. In the operation, the key is understood as the name of Zset, so what is the use for the delay queue? Imagine that if score represents the timestamp of the execution time, and it is inserted into the Zset Set at a certain time, it will be sorted according to the timestamp size, that is, before and after the execution time. In this way, an endless loop thread will be started to continuously take the first key value, If the current timestamp is greater than or equal to the key value, the socre will take it out for consumption and deletion, so as to delay the execution. Note that it is not necessary to traverse the whole Zset Set to avoid performance waste.

The arrangement effect of Zset is shown in the following figure:

The java code is implemented as follows:

package cn.chinotan.service.delayQueueRedis;

import org.apache.commons.lang3.StringUtils;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import redis.clients.jedis.Tuple;

import java.text.SimpleDateFormat;
import java.util.Calendar;
import java.util.Date;
import java.util.Set;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;

/**
 * @program: test
 * @description: redis Implement delay queue
 * @author: xingcheng
 * @create: 2018-08-19
 **/
public class AppTest {

    private static final String ADDR = "127.0.0.1";
    private static final int PORT = 6379;
    private static JedisPool jedisPool = new JedisPool(ADDR, PORT);
    private static CountDownLatch cdl = new CountDownLatch(10);

    public static Jedis getJedis() {
        return jedisPool.getResource();
    }

    /**
     * Producer, generate 5 orders
     */
    public void productionDelayMessage() {
        for (int i = 0; i < 5; i++) {
            Calendar instance = Calendar.getInstance();
            // Execute in 3 seconds
            instance.add(Calendar.SECOND, 3 + i);
            AppTest.getJedis().zadd("orderId", (instance.getTimeInMillis()) / 1000, StringUtils.join("000000000", i + 1));
            System.out.println("Production order: " + StringUtils.join("000000000", i + 1) + " Current time:" + new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(new Date()));
            System.out.println((3 + i) + "Execute in seconds");
        }
    }

    //Consumer, take order
    public static void consumerDelayMessage() {
        Jedis jedis = AppTest.getJedis();
        while (true) {
            Set<Tuple> order = jedis.zrangeWithScores("orderId", 0, 0);
            if (order == null || order.isEmpty()) {
                System.out.println("There are currently no tasks waiting");
                try {
                    TimeUnit.MICROSECONDS.sleep(500);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
                continue;
            }
            Tuple tuple = (Tuple) order.toArray()[0];
            double score = tuple.getScore();
            Calendar instance = Calendar.getInstance();
            long nowTime = instance.getTimeInMillis() / 1000;
            if (nowTime >= score) {
                String element = tuple.getElement();
                Long orderId = jedis.zrem("orderId", element);
                if (orderId > 0) {
                    System.out.println(new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(new Date()) + ":redis Consume a task: consume orders OrderId by" + element);
                }
            }
        }
    }

    static class DelayMessage implements Runnable{
        @Override
        public void run() {
            try {
                cdl.await();
                consumerDelayMessage();
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }
    }
    
    public static void main(String[] args) {
        AppTest appTest = new AppTest();
        appTest.productionDelayMessage();
        for (int i = 0; i < 10; i++) {
            new Thread(new DelayMessage()).start();
            cdl.countDown();
        }
    }
}

The results are as follows:

Notes for use in production environment:

Because this implementation method is simple, but in the production environment, it is mostly multi instance deployment, so there is a concurrency problem, that is, the search and deletion of the cache are not atomic (zrangeWithScores and zrem operations are not a command and are not atomic), which will lead to the problem of multiple message sending. The methods to avoid this problem are as follows:

1. A single instance can be used to deploy the solution (it does not have high availability, so it is easy to send messages in time after a single machine fails)

2. The lua script of redis is used for atomic operation, that is, atomic operation search and deletion (difficult to implement)

Therefore, the implementation of delay queue is best implemented by rabbitMQ. rabbitMQ naturally has distributed characteristics and can be well used in multi service and multi instance environment. For specific implementation, please refer to my first blog https://my.oschina.net/u/3266761/blog/1926588