Turn:

Top down redis4 0 (4) time events and expire

redis4. Time events of 0 and expire

brief introduction

Time events and file events have similar interfaces. They are called in aeProcessEvents. The difference is that the bottom layer of file events is delegated to multiplex interfaces such as select and epoll. The time event checks whether the trigger time of the time event has expired through each tick. redis4. Only one time event serverCron is registered in version 0. It is registered in initServer and called at the end of each aeProcessEvents function. As mentioned above, the aeMain function is the main event loop of redis, which will continuously call aeProcessEvents.

The expire instruction inserts the key value and expiration time of the internal data type of sds into the server - > expires dictionary dict, and triggers the key space event. In the databaseCron function of serverCron, activeExpireCycle(ACTIVE_EXPIRE_CYCLE_SLOW) is called to randomly extract the key value in expires, if it expires, the corresponding key value is deleted in server->dict.

text

Time event registration

First, let's look at the structure of time events. Although there are many members in the structure, it can be said that when is actually used_ sec ，when_ms, timeProc3 members, and the return value of timeProc. We can observe that aeTimeProc will return a value of int type. If it is not - 1, it will be used as the interval of the next call.

/* Time event structure */
typedef struct aeTimeEvent {
    long long id; /* time event identifier. */
    long when_sec; /* seconds */
    long when_ms; /* milliseconds */
    aeTimeProc *timeProc;
    aeEventFinalizerProc *finalizerProc;
    void *clientData;
    struct aeTimeEvent *prev;
    struct aeTimeEvent *next;
} aeTimeEvent;

typedef int aeTimeProc(struct aeEventLoop *eventLoop, long long id, void *clientData);

The registered function is located in initServer. aeCreateTimeEvent function will generate an aeTimeEvent object and assign it to EventLoop - > timeeventhead.

    if (aeCreateTimeEvent(server.el, 1, serverCron, NULL, NULL) == AE_ERR) {
        serverPanic("Can't create event loop timers.");
        exit(1);
    }

Time event trigger

The function that really handles time events is processTimeEvents, but let's go back to aeProcessEvents and learn a trick in redis. aeSearchNearestTimer will find the closest time event. If yes (there must be a serverCron function under normal circumstances), the interval event from the next time event will be written into the tvp parameter, and the tvp will be passed in the aeApiPoll parameter. If no file event has been triggered, the aeApiPoll function will wait for the appropriate time to return, and the function can handle the time event just after it returns.

int aeProcessEvents(aeEventLoop *eventLoop, int flags)
{
    int processed = 0, numevents;
    int j;
    aeTimeEvent *shortest = NULL;
    struct timeval tv, *tvp;

    shortest = aeSearchNearestTimer(eventLoop);
    if (shortest) {
        long now_sec, now_ms;
        aeGetTime(&now_sec, &now_ms);
        tvp = &tv;

        /* How many milliseconds we need to wait for the next
             * time event to fire? */
        long long ms =
            (shortest->when_sec - now_sec)*1000 +
            shortest->when_ms - now_ms;

        if (ms > 0) {
            tvp->tv_sec = ms/1000;
            tvp->tv_usec = (ms % 1000)*1000;
        } else {
            tvp->tv_sec = 0;
            tvp->tv_usec = 0;
        }
    }

    numevents = aeApiPoll(eventLoop, tvp);

    for (j = 0; j < numevents; j++) {
        //process file events
    }
    
    /* Check time events */
    if (flags & AE_TIME_EVENTS)
        processed += processTimeEvents(eventLoop);

    return processed; /* return the number of processed file/time events */
}

In the processTimeEvents function, the previously registered functions will be traversed. If the time conditions are met, the corresponding functions will be called. If the value returned by the function is not - 1, it means that the function will use the return value as the interval for the next call to the function. The frequency of servercron is defined in server Hz, which means calling the servercron function several times a second. The default is 10 times / second.

expire command

Before understanding the expire command, let's review the previous content. In the process of file event processing, redis will convert the content in querybuf into client - > argc and client - > argv. The way is to convert the createStringObject into the corresponding string object. Therefore, the encoding type of redisObject in argv can only be embstr or raw.

#define OBJ_ENCODING_EMBSTR_SIZE_LIMIT 44
robj *createStringObject(const char *ptr, size_t len) {
    if (len <= OBJ_ENCODING_EMBSTR_SIZE_LIMIT)
        return createEmbeddedStringObject(ptr,len);
    else
        return createRawStringObject(ptr,len);
}

If the passed time to live parameter is a negative number, the exit instruction will be converted into a del instruction and the corresponding key value will be deleted directly.

Otherwise, add the corresponding expiration time in server - > expire internal data type dict.

void expireGenericCommand(client *c, long long basetime, int unit) {
    robj *key = c->argv[1], *param = c->argv[2];
    long long when; /* unix time in milliseconds when the key will expire. */

    if (getLongLongFromObjectOrReply(c, param, &when, NULL) != C_OK)
        return;

    if (unit == UNIT_SECONDS) when *= 1000;
    when += basetime;

    /* No key, return zero. */
    if (lookupKeyWrite(c->db,key) == NULL) {
        addReply(c,shared.czero);
        return;
    }

    if (when <= mstime() && !server.loading && !server.masterhost) {
        robj *aux;

        int deleted = dbSyncDelete(c->db,key);
        serverAssertWithInfo(c,key,deleted);
        server.dirty++;

        aux =  shared.del;
        rewriteClientCommandVector(c,2,aux,key);
        signalModifiedKey(c->db,key);
        notifyKeyspaceEvent(NOTIFY_GENERIC,"del",key,c->db->id);
        addReply(c, shared.cone);
        return;
    } else {
        setExpire(c,c->db,key,when);
        addReply(c,shared.cone);
        signalModifiedKey(c->db,key);
        notifyKeyspaceEvent(NOTIFY_GENERIC,"expire",key,c->db->id);
        server.dirty++;
        return;
    }
}

Delete expired key values

There are three ways to delete expired key values: regular deletion, regular deletion and passive deletion. redis combines periodic deletion and passive deletion.

Passive deletion

When the client sends get, expire and other requests to the server, expireifneed (C - > dB, C - > argv [J]) will be called; Function to delete expired key values. Interestingly, the del request will also call expireifneed, that is, it is possible to call the dbSyncDelete function twice.

Active deletion / periodic deletion

In the time event serverCron function mentioned earlier, if it is not from the library and active is enabled_ expire_ Enabled (enabled by default), the activeExpireCycle function will be called to actively clean up expired key values.

By default, Cron_ DBS_ PER_ The value of call is 16, which is also the value of dbnum, which means that activeExpireCycle will process 16 databases at a time. Moreover, if the last call times out, it will be processed according to the database that processes dbnum once.

At least one round of processing will be conducted for each database. 20 samples will be taken in one round of processing. If the samples expire, the key will be deleted. Moreover, if the expired keys in the sample exceed 25% and there is no timeout, the iteration will continue and another round of processing will be carried out.

The unit of timelimit is microseconds. If the processing of the current db times out, the processed db will only be processed once.

Therefore, regular deletion will not delete all expired keys. Under the normal operation of the server, the expired keys will be maintained within 25%.

void activeExpireCycle(int type) {
	//static global
    static unsigned int current_db = 0; /* Last DB tested. */
    static int timelimit_exit = 0;      /* Time limit hit in previous call? */

    int j, iteration = 0;
    int dbs_per_call = CRON_DBS_PER_CALL;
    long long start = ustime(), timelimit;
	
    //How many db are processed at a time
    if (dbs_per_call > server.dbnum || timelimit_exit)
        dbs_per_call = server.dbnum;

	//Time limit: if the total time exceeds the limit, only one round of current db will be processed
    timelimit = 1000000*ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC/server.hz/100;
    timelimit_exit = 0;
    if (timelimit <= 0) timelimit = 1;

    if (type == ACTIVE_EXPIRE_CYCLE_FAST)
        timelimit = ACTIVE_EXPIRE_CYCLE_FAST_DURATION; /* in microseconds. */

    for (j = 0; j < dbs_per_call && timelimit_exit == 0; j++) {
        int expired;
        redisDb *db = server.db+(current_db % server.dbnum);
        current_db++;

        do {
            unsigned long num, slots;
            long long now, ttl_sum;
            int ttl_samples;
            iteration++;

            if ((num = dictSize(db->expires)) == 0) {
                db->avg_ttl = 0;
                break;
            }
            slots = dictSlots(db->expires);
            now = mstime();

            expired = 0;

            if (num > ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP)
                num = ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP;

            while (num--) {
                dictEntry *de;

                if ((de = dictGetRandomKey(db->expires)) == NULL) break;
                if (activeExpireCycleTryExpire(db,de,now)) expired++;
            }
            total_ += expired;

            if ((iteration & 0xf) == 0) { /* check once every 16 iterations. */
                elapsed = ustime()-start;
                if (elapsed > timelimit) {
                    timelimit_exit = 1;
                    server.stat_expired_time_cap_reached_count++;
                    break;
                }
            }

        } while (expired > ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP/4);
    }
}

reference

redis document

Redis design and Implementation

Turn:

Top down redis4 0 (4) time events and expire

--Posted from Rpc