brief introduction

redis-full-check is an open source tool used by the Redis&MongoDB team in Aliyun to verify the consistency of the two redis data and is often used to verify the correctness of the redis data migration (redis-shake).
Support: Homogeneous or heterogeneous comparisons between single-node, master-slave, cluster, proxy cloud cluster (Ali cloud), version 2.x-5.x.
redis-full-check validates data by fully comparing the data in redis at the source and destination ends in multiple rounds: each time the data at the source and destination ends is captured for differential comparison, and inconsistent data is recorded for the next round of comparison (recorded in sqlite3 db). Then the data consistency between source and destination databases due to incremental data synchronization is reduced by continuous convergence through multiple comparisons. Finally, the data present in sqlite is the final difference.

Comparison principle

There are three options for comparemode:

KeyOutline: Only compare if the key values are equal.
ValueOutline: Only compare if the length of the value is equal.
FullValue: Contrast whether key, value length, and value are equal.

In the first round, all key s on the source library are found, and then the comparison is made between the source library and the target library.
The second iteration begins with a comparison of key s and field s that are still inconsistent after the last iteration ends.

For key inconsistencies, including lack_source, lack_target and type, retrieve key and value from source library and target library for comparison.
Value inconsistent string, re-compare key: take key from source and destination, value comparison.
hash, set, and zset with inconsistent value s, only inconsistent field s are compared again, and previously compared and identical files are no longer compared. This is to prevent frequent updates to large key s from causing the check to never pass.
Value inconsistent list, re-compare key: take key from source and destination, value comparison.
Interval stops between rounds.
_For hash, set, zset, list large key processing follows:

Len <= 5192, directly take the full field, value to compare, using the following commands: hgetall, smembers, zrange 0-1 withscores, lrange 0-1.
Len > 5192, use hscan, sscan, zscan, lrange to batch field and value.

Parameter Description

The main parameters in redis-full-check are as follows:

 -s, --source=SOURCE               source redis Library address ( ip:port)，If it's a clustered version, you need to use a semicolon (;)Divide different db，Only one of the master or slave needs to be configured. For example:10.1.1.1:1000;10.2.2.2:2000;10.3.3.3:3000. 
  -p, --sourcepassword=Password     source redis Library Password
      --sourceauthtype=AUTH-TYPE    Source Library Management Rights, Open Source reids This parameter is useless below.
      --sourcedbtype=               Source library category, 0: db(standalone Single-node, master-slave)，1: cluster(Cluster edition),2: Ali Yun
      --sourcedbfilterlist=         Source Library Requires Capture Logic db Whitelist, semicolon (;)Split, for example:0;5;15 Express db0,db5 and db15 Will be captured
  -t, --target=TARGET               objective redis Library address ( ip:port)
  -a, --targetpassword=Password     objective redis Library Password
      --targetauthtype=AUTH-TYPE    Destination Library Management Rights, Open Source reids This parameter is useless below.
      --targetdbtype=               Reference resources sourcedbtype
      --targetdbfilterlist=         Reference resources sourcedbfilterlist
  -d, --db=Sqlite3-DB-FILE          For Difference key Stored sqlite3 db Location, default result.db
      --comparetimes=COUNT          Compare rounds
  -m, --comparemode=                Comparison mode, 1 for full comparison, 2 for comparison only value Length, 3 Comparisons key Existence or not, 4 Ignore large in case of full comparison key Comparison
      --id=                         For playing metric
      --jobid=                      For playing metric
      --taskid=                     For playing metric
  -q, --qps=                        qps Speed Limit Threshold
      --interval=Second             Time interval between rounds
      --batchcount=COUNT            Number of batch aggregations
      --parallel=COUNT              Number of concurrent coprocesses compared, default 5
      --log=FILE                    log file
      --result=FILE                 Inconsistent results are recorded to result File, format:'db    diff-type    key    field'
      --metric=FILE                 metric file
      --bigkeythreshold=COUNT       large key Split threshold, used for comparemode=4
  -f, --filterlist=FILTER           Needs comparison key List, with semicolons (;)Division. For example:"abc*|efg|m*"Show contrast'abc', 'abc1', 'efg', 'm', 'mxyz'，No contrast'efgh', 'p'. 
  -v, --version

Download Address

wget https://github.com/alibaba/RedisFullCheck/releases/download/release-v1.4.8-20200212/redis-full-check-1.4.8.tar.gz

Comparison results

Command reference:

./redis-full-check -s "192.168.1.204:6460;192.168.1.205:6446;192.168.1.205:6447" -t "192.168.1.204:6458;192.168.1.205:6444;192.168.1.205:6445" --comparemode=1 --comparetimes=1 --qps=100   --sourcedbtype=1 --targetdbtype=1 

The result is saved in sqlite3 db file, which, if not specified, is the result of the current directory. DB file
_sqlite3 result.db login
. quit logon
_sqlite> select * from key; View exception key