[redis] redis's "message queue": redis stream (analysis)

Posted by thunderbox on Sun, 02 Jan 2022 22:40:09 +0100

Article catalog

About redis stream

Previously, I only knew that redis had a publish / subscribe similar to message queue. I really didn't know that it had a "message queue" quietly.

redis stream implements most of the functions of message queues, such as:

news ID Serialization generation of
 Message traversal
 Blocking and non blocking reads of messages
 Packet consumption of messages
ACK Confirmation mechanism

Publish / subscribe mode can not be regarded as a real message queue. It has certain real-time performance and does not persist. However, message queues such as redis stream and Kafka cannot be compared. After all, they are in memory and small.

redis stream usage example

Official website command document reference

Add message

The XADD command can send a message to the specified Stream message flow (created if it does not exist).

Use example:

XADD mystream 1526919030474-55 Key value

XADD Stream Name message ID message Message content

As for what is mentioned in official documents or other blogs *, I won't say anything.

If the specified ID parameter is a * character, the XADD command automatically generates a unique ID. When the ID is automatically generated, the first part is the Unix time (in milliseconds) of the Redis instance that generated the ID. the second part is only the serial number, which is used to distinguish the ID generated in the same millisecond.

The ID is guaranteed to be always incremented, so the entries are completely ordered in the flow. To ensure this property, if the time of the current top ID in the stream is greater than the current local time of the instance, the top entry time will be used, and the sequence part of the ID will be increased. For example, this may happen when the local clock jumps back, or when the new host has a different absolute time after failover.

When the user specifies an explicit ID for XADD, the minimum valid ID is 0-1, and the user must specify an ID greater than any other ID currently in the stream, otherwise the command will fail and return an error.

This tells us that if there are no special requirements, use the ID automatically generated by the system.

Version change:

>=6.2: Increased NOMKSTREAM Options MINID Fine tuning strategy and LIMIT Options.
>=7.0: Added pair<ms>-*Explicit ID Form support.

Use example

redis>  XADD mystream * name Sara surname OConnor

"1640412909358-0"

redis>  XADD mystream * field1 value1 field2 value2 field3 value3

"1640412909358-1"

redis>  XLEN mystream

(integer) 2

redis>  XRANGE mystream - +

1) 1) "1640412909358-0"
   2) 1) "name"
      2) "Sara"
      3) "surname"
      4) "OConnor"
2) 1) "1640412909358-1"
   2) 1) "field1"
      2) "value1"
      3) "field2"
      4) "value2"
      5) "field3"
      6) "value3"

redis>

Time complexity

O(1) when adding a new entry, O(N) when trimming where N being the number of entries evicted.

Read message

XREAD

XREAD can be used to read data from the message flow.

The format should be obvious.

The last parameter is the message ID. redis will return a message larger than this ID. "0-0" is a special ID that represents the smallest message ID. it can be used to require redis to read messages from scratch.

XREAD can also block clients waiting for new messages to be received in the message flow. To block, the block option is used and the number of milliseconds to block before the timeout.

Usually, the Redis blocking command times out in seconds, but this command requires a timeout of milliseconds, even though the timeout resolution of the server is usually close to 0.1 seconds.

It is usually better to use this command as follows:

XREAD BLOCK 1000 STREAMS mystream $

$is also a special ID, indicating the current maximum message ID. Use it to ask redis to read the latest messages.

Time complexity

Time complexity: For each stream mentioned: O(N) with N being the number of elements being returned, it means that XREAD-ing with a fixed COUNT is O(1). Note that when the BLOCK option is used, XADD will pay O(M) time in order to serve the M clients blocked on the stream getting new data.

XRANGE

Range read, literally.

There are also special characters here to see: +- The - and + special IDs mean respectively the minimum ID possible and the maximum ID possible inside a stream.

You can use count to limit the quantity:

removal message

XDEL

There's nothing to say about this. Just give an example.

> XADD mystream * a 1
1538561698944-0
> XADD mystream * b 2
1538561700640-0
> XADD mystream * c 3
1538561701744-0
> XDEL mystream 1538561700640-0
(integer) 1
127.0.0.1:6379> XRANGE mystream - +
1) 1) 1538561698944-0
   2) 1) "a"
      2) "1"
2) 1) 1538561701744-0
   2) 1) "c"
      2) "3"

XTRIM

Use XTRIM convection to trim and limit the length.

127.0.0.1:6379> XADD mystream * field1 A field2 B field3 C field4 D
"1601372434568-0"
127.0.0.1:6379> XTRIM mystream MAXLEN 2
(integer) 0
127.0.0.1:6379> XRANGE mystream - +
1) 1) "1601372434568-0"
   2) 1) "field1"
      2) "A"
      3) "field2"
      4) "B"
      5) "field3"
      6) "C"
      7) "field4"
      8) "D"
127.0.0.1:6379>
 
redis>

I don't want to write too much now. After all, I haven't used it yet. The official document is at the beginning and can be read by yourself. I'll write it after I use it next month.