Analysis of quicklist source code of Redis

1, Introduction to quicklist

Redis list is a simple list of strings, sorted by insertion order. You can add an element to the head (left) or tail (right) of the list.

A list can contain up to 232 - 1 elements (4294967295, more than 4 billion elements per list).

The internal data structure that its underlying implementation relies on is quicklist, which has the following main features:

1. List is a two-way linked list.
2. It is very convenient to add and delete data at both ends of the list, and the time complexity is O(1).
3. list also supports access operations at any intermediate location with O(N) time complexity.

Before looking at the source code (version 3.2.2), let's take a look at several main data structures in quicklist:

A quicklist consists of multiple quicklistnodes. Each quicklistNode points to a ziplist. A ziplist contains multiple entry elements. Each entry element is a list element. The schematic diagram is as follows:

Figure 1: quicklist

II. Source code of quicklist data structure

Look at the source code of quicklist and quicklistNode respectively (the code file is Quicklist.h, which will be analyzed later in ziplist):

quicklist:

/*
The quicklist structure takes 32 bytes (64 bit system), in which the fields:
head: points to the first quicklistNode.
tail: point to the last quicklistNode.
count: the total number of entries in all ziplist s.
len: number of quicklistnodes.
fill: ziplist size limit, given by server.list_max_ziplist_size.
Compress: node compression depth setting, given by server.list-compress-depth, 0 means compression is turned off.
*/
typedef struct quicklist {

quicklistNode *head;
quicklistNode *tail;
unsigned long count;        /* total count of all entries in all ziplists */
unsigned int len;           /* number of quicklistNodes */
int fill : 16;              /* fill factor for individual nodes */
unsigned int compress : 16; /* depth of end nodes not to compress;0=off */

} quicklist;

quicklistNode:

/*
prev: points to the previous quicklistNode.
Next: points to the next quicklistNode.
zl: ziplist pointing to the current node.
sz: the number of bytes occupied by ziplist.
Count: number of elements in ziplist.
Encoding: encoding type, RAW==1 or LZF==2.
Container: container type, NONE==1 or ZIPLIST==2
recompress: bool type. true indicates that the data of this node has been temporarily decompressed.
attempted_compress: bool type, used for the test phase.
extra: fill in the dictionary, which may be used in the future.
*/
typedef struct quicklistNode {

struct quicklistNode *prev;
struct quicklistNode *next;
unsigned char *zl;
unsigned int sz;             /* ziplist size in bytes */
unsigned int count : 16;     /* count of items in ziplist */
unsigned int encoding : 2;   /* RAW==1 or LZF==2 */
unsigned int container : 2;  /* NONE==1 or ZIPLIST==2 */
unsigned int recompress : 1; /* was this node previous compressed? */
unsigned int attempted_compress : 1; /* node can't compress; too small */
unsigned int extra : 10; /* more bits to steal for future usage */

} quicklistNode;

3, The addition, deletion, modification and search of quicklist

1. Create quicklist

When executing the push command (for example, lpush), if no such key is found, a quicklist will be created and initialized, as follows:

void pushGenericCommand(client *c, int where) {

int j, waiting = 0, pushed = 0;
robj *lobj = lookupKeyWrite(c->db,c->argv[1]);

if (lobj && lobj->type != OBJ_LIST) {
    addReply(c,shared.wrongtypeerr);
    return;
}

for (j = 2; j < c->argc; j++) {
    c->argv[j] = tryObjectEncoding(c->argv[j]);
    if (!lobj) {  // If the key does not exist, first create the key object and add it to the db
        lobj = createQuicklistObject(); // Initializing the quicklist object
        quicklistSetOptions(lobj->ptr, server.list_max_ziplist_size,
                            server.list_compress_depth); // Use the configuration item of redis server for initialization
        dbAdd(c->db,c->argv[1],lobj); // Add quicklist to redis db
    }
    // Add new elements to the list
    listTypePush(lobj,c->argv[j],where);
    pushed++;
}
addReplyLongLong(c, waiting + (lobj ? listTypeLength(lobj) : 0));
if (pushed) {
    char *event = (where == LIST_HEAD) ? "lpush" : "rpush";

    signalModifiedKey(c->db,c->argv[1]);
    notifyKeyspaceEvent(NOTIFY_LIST,event,c->argv[1],c->db->id);
}
server.dirty += pushed;

}

Take a look at createQuicklistObject:

/* Create a new quicklist.

• Free with quicklistRelease(). */

1. *quicklistCreate(void) {
   struct quicklist *quicklist;

   quicklist = zmalloc(sizeof(*quicklist));
   quicklist->head = quicklist->tail = NULL;
   quicklist->len = 0;
   quicklist->count = 0;
   quicklist->compress = 0;
   quicklist->fill = -2;
   return quicklist;
   }

1. Additive elements

Continue to see the listTypePush method in the source code above:

void listTypePush(robj subject, robj value, int where) {

if (subject->encoding == OBJ_ENCODING_QUICKLIST) {
    int pos = (where == LIST_HEAD) ? QUICKLIST_HEAD : QUICKLIST_TAIL;
    value = getDecodedObject(value);
    size_t len = sdslen(value->ptr);// Calculate new element length
    quicklistPush(subject->ptr, value->ptr, len, pos); // Join quicklist
    decrRefCount(value); 
} else {
    serverPanic("Unknown list encoding");
}

}

Continue to quicklistPush:

/ Wrapper to allow argument-based switching between HEAD/TAIL pop /
void quicklistPush(quicklist quicklist, void value, const size_t sz,

               int where) {
if (where == QUICKLIST_HEAD) {  // Add to list header
    quicklistPushHead(quicklist, value, sz);
} else if (where == QUICKLIST_TAIL) {  // Add to end of list
    quicklistPushTail(quicklist, value, sz);
}

}

/* Add new entry to head node of quicklist.
*

• Returns 0 if used existing head.
• Returns 1 if new head created.
  Add a new element to the quicklist header node:

If the new element is added in the head, return 0, otherwise return 1
*/
int quicklistPushHead(quicklist quicklist, void value, size_t sz) {

quicklistNode *orig_head = quicklist->head;
// If the head is not empty and the space size meets the storage requirements of the new element, the new element is added to the head, otherwise a quicklistNode is added
if (likely(
        _quicklistNodeAllowInsert(quicklist->head, quicklist->fill, sz))) {
    quicklist->head->zl =
        ziplistPush(quicklist->head->zl, value, sz, ZIPLIST_HEAD);
    quicklistNodeUpdateSz(quicklist->head);
} else {
    // Create a new quicklistNode
    quicklistNode *node = quicklistCreateNode();
    // Add new elements to the new ziplist
    node->zl = ziplistPush(ziplistNew(), value, sz, ZIPLIST_HEAD);
    // Update the length of ziplist to sz field of quicklist node, and then add the new node to quicklist, that is, before the original head
    quicklistNodeUpdateSz(node);
    _quicklistInsertNodeBefore(quicklist, quicklist->head, node);
}
quicklist->count++;
quicklist->head->count++;
return (orig_head != quicklist->head);

}

ziplistpush will add new elements to ziplist, which will be analyzed later.

1. Get elements

The method to get the element is quicklistPop method (quicklist.c), as follows:

/* Default pop function
*

• Returns malloc'd value from quicklist */

1. quicklistPop(quicklist quicklist, int where, unsigned char *data,

               unsigned int *sz, long long *slong) {

   unsigned char *vstr;
   unsigned int vlen;
   long long vlong;
   if (quicklist->count == 0)

      return 0;

   //pop an element
   int ret = quicklistPopCustom(quicklist, where, &vstr, &vlen, &vlong,

                               _quicklistSaver);

   if (data)

      *data = vstr;

   if (slong)

      *slong = vlong;

   if (sz)

      *sz = vlen;

   return ret;
   }

The specific implementation is in quicklistPopCustom:

/* pop from quicklist and return result in 'data' ptr. Value of 'data'

• is the return value of 'saver' function pointer if the data is NOT a number.
  *
• If the quicklist element is a long long, then the return value is returned in
• 'sval'.
  *
• Return value of 0 means no elements available.
• Return value of 1 means check 'data' and 'sval' for values.
• If 'data' is set, use 'data' and 'sz'. Otherwise, use 'sval'.
  If there is no element in the quicklist, return 0, otherwise return 1

When returning 1, you need to check the data and sval fields:

1. If it is of string type, the result address is saved in the data pointer, and the length is saved in sz.
2. If it is of type long long, the value is saved in the sval field.
   */

int quicklistPopCustom(quicklist quicklist, int where, unsigned char *data,

                   unsigned int *sz, long long *sval,
                   void *(*saver)(unsigned char *data, unsigned int sz)) {
unsigned char *p;
unsigned char *vstr;
unsigned int vlen;
long long vlong;
int pos = (where == QUICKLIST_HEAD) ? 0 : -1;

if (quicklist->count == 0)
    return 0;

if (data)
    *data = NULL;
if (sz)
    *sz = 0;
if (sval)
    *sval = -123456789;

quicklistNode *node;
if (where == QUICKLIST_HEAD && quicklist->head) {
    node = quicklist->head;
} else if (where == QUICKLIST_TAIL && quicklist->tail) {
    node = quicklist->tail;
} else {
    return 0;
}
// p: 0 takes the first element of ziplist; - 1 takes the last element of ziplist;
p = ziplistIndex(node->zl, pos);
if (ziplistGet(p, &vstr, &vlen, &vlong)) {
    if (vstr) {
        if (data)
            *data = saver(vstr, vlen);
        if (sz)
            *sz = vlen;
    } else {
        if (data)
            *data = NULL;
        if (sval)
            *sval = vlong;
    }
    // Remove the element from the quicklist
    quicklistDelIndex(quicklist, node, &p);
    return 1;
}
return 0;

}

Take a look at quicklistDelIndex:

/* Delete one entry from list given the node for the entry and a pointer

• to the entry in the node.
  *
• Note: quicklistDelIndex() requires uncompressed nodes because you
• already had to get *p from an uncompressed node somewhere.
  *
• Returns 1 if the entire node was deleted, 0 if node still exists.
• Also updates in/out param 'p' with the next offset in the ziplist.
  Remove an entry from quicklistNode:

1. Remove entry from ziplist.
2. The number of entries in quicklistNode minus 1:
   If the number of entries in quicklistNode is 0, the current quicklistNode is removed from the quicklist.
   Otherwise, update the sz field in quicklistNode.
   */

REDIS_STATIC int quicklistDelIndex(quicklist quicklist, quicklistNode node,

                               unsigned char **p) {
int gone = 0;

node->zl = ziplistDelete(node->zl, p);
node->count--;
if (node->count == 0) {
    gone = 1;
    __quicklistDelNode(quicklist, node);
} else {
    quicklistNodeUpdateSz(node);
}
quicklist->count--;
/* If we deleted the node, the original node is no longer valid */
return gone ? 1 : 0;

}

At this point, the main structure code of quicklist, and the code of the main two methods push and pop are analyzed. The next chapter analyzes the source code of ziplist stored in the underlying quicklist.

https://github.com/tomliugen
Original address https://www.cnblogs.com/xinghebuluo/p/12722103.html