1: Overview of Redis persistence
Persistence function: Redis is an in memory database, and all data are stored in memory. In order to avoid permanent loss of data caused by process exit, it is necessary to regularly save the data in Redis from memory to hard disk in some form (data or command); When Redis restarts next time, use persistent files to realize data recovery. In addition, for disaster backup, persistent files can be copied to a remote location
1: RDB persistence mode
RDB (Redis DataBase) persistence is to save the snapshot generated by the data in the current process to the hard disk (so it is also called snapshot persistence) within a set time point, and the saved file suffix is RDB; When Redis restarts, it will automatically read the snapshot file recovery data; Manual persistence can also be achieved through the save and bgsave commands
Note: RDB mode is to save the stored data} to the hard disk
2: AOF persistence mode
Aof (append only file) persistence: record each addition, deletion and modification command in the form of independent log, and re execute the command in AOF file when restarting, so as to recover data. The main function of AOF is to solve the real-time of data persistence. At present, AOF has become the mainstream way of Redis persistence. When we restart Redis service next time, we will first execute AOF file to recover data
Note: RDB mode is to save the # addition, deletion and modification commands stored by us to the hard disk
2: RDB persistence
1: RDB persistence trigger mode
Manual trigger:
save Command:
Executing this command will block the current Redis The server will not be used in the production environment until the persistence is completed (because it will block the access operations of other clients).
bgsave Command:
This trigger method will fork A child process, which is responsible for the persistence process, so blocking will only occur inforkChild process.
Auto trigger:
①: According to what we configured in the configuration file save m n Automatically triggered by the rules of (described later)
②: Execute exitshutdownThe command will be triggered automatically (provided that the persistence mechanism we configured is RDB)
Execute exitshutdown nosaveCommand is not triggered automatically
③: implementflushAllPersistence will be triggered automatically (but there is no data, only an empty one).rdb (file)
implementflushdbPersistence is not triggered
④: implementdebug reloadPersistence is automatically triggered
①: implement bgsave Orders, Redis The parent process determines whether there are currently executing child processes, such as RDB/AOF Subprocess, if any bgsave The command returns directly.
②: Parent process execution fork Operation to create a child process, fork During the operation, the parent process will block through info stats Command view latest_fork_usec Options,
You can get the latest one fork The time taken by the operation, in microseconds.
③: Parent process fork When it's done, bgsave Command return“ Background saving started"The message no longer blocks the parent process and can continue to respond to other commands.
④: Child process creation RDB File, generate a temporary snapshot file according to the memory of the parent process, and atomic replace the original file after completion. implement lastsave Command can get the last
One time generation RDB Corresponding time info Statistical rdb_last_save_time Options.
⑤: The process sends a signal to the parent process to indicate completion, and the parent process updates the statistics.
2: RDB configuration file
Redis Common configuration: # By default Redis Data will be saved automatically: # 3600 After seconds (1 hour), if at least one key is changed # 300 After seconds (5 minutes), if at least 100 keys are changed # 60 After seconds (1 minute), if at least 10000 keys are changed # You can set these explicitly (customizable) by uncommenting the following three lines save (second key) save 3600 1 save 300 100 save 60 10000 # When bgsave When an error occurs, Redis Whether to stop executing the write command; # yes: When there is a problem with the hard disk, it can be found in time to avoid a large amount of data loss; # no: be Redis ignore bgsave The error continues to execute the write command when Redis Server system(Especially hard disk)When monitoring is used, this option is considered to be set to no stop-writes-on-bgsave-error yes # Open or not RDB File compression rdbcompression yes # Open or not RDB File verification plays a role in both writing and reading files; close checksum It can bring about 10 when writing files and startup files%Improved performance, # However, it cannot be found when the data is damaged rdbchecksum yes # After storage RDB File name default dump.rdb dbfilename dump.rdb # Delete the created during master-slave replication RDB File (in instances where persistence is not enabled, it is best to delete it) rdb-del-sync-files no # The directory where persistent files are stored is used above“ dbfilename"Configuration instructions, additional files will also be created in this directory dir ./
3: Basic operation demonstration
# implement save client: 127.0.0.1:6379> save OK Server: 910:M 12 Dec 2021 14:59:31.373 * DB saved on disk # implement bgsave client: 127.0.0.1:6379> bgsave Background saving started Server: 910:M 12 Dec 2021 15:00:44.755 * Background saving started by pid 913 913:C 12 Dec 2021 15:00:44.777 * DB saved on disk 910:M 12 Dec 2021 15:00:44.825 * Background saving terminated with success # implement flushall client: 127.0.0.1:6379> FLUSHALL OK Server: 910:M 12 Dec 2021 15:01:55.217 * DB saved on disk # Automatic storage( save 30 2)When the command is executed twice within 30 seconds, automatic persistence will be executed after 30 seconds RDB client: 127.0.0.1:6379> set name zhangsan OK 127.0.0.1:6379> set age 22 OK Server: 245:M 12 Dec 2021 15:06:09.045 * 2 changes in 30 seconds. Saving... 245:M 12 Dec 2021 15:06:09.092 * Background saving started by pid 247 247:C 12 Dec 2021 15:06:09.118 * DB saved on disk 245:M 12 Dec 2021 15:06:09.200 * Background saving terminated with success # In there RDB Open with file Redis service Server: 1240:M 12 Dec 2021 15:06:35.975 * Loading RDB produced by version 6.2.5 1240:M 12 Dec 2021 15:06:35.975 * RDB age 26 seconds 1240:M 12 Dec 2021 15:06:35.975 * RDB memory usage when created 0.48 Mb 1240:M 12 Dec 2021 15:06:35.976 * DB loaded from disk: 0.001 seconds 1240:M 12 Dec 2021 15:06:35.976 * Ready to accept connections
4: save m n automatic persistence principle
Redis of save m n，Yes serverCron Functions dirty Counters, and lastsave Time stamp.
serverCron:
yes Redis The periodic operation function of the server. By default, every 100 ms Execute once; This function maintains the state of the server, one of which is
inspect save m n Whether the configured conditions are met. If so, execute bgsave.
dirty:
The counter is Redis A state maintained by the server that records the last execution bgsave/save How many times did the server state change after the command(Including addition, deletion and modification)；
And when save/bgsave After execution, the dirty Reset to 0.
For example: if Redis Yes set name jack，be dirty Value meeting+1；If implemented sadd myset v1 v2 v3，be dirty Value meeting+3；
be careful dirty It records the number of modifications made by the server, not the number of commands executed by the client to modify the data.
lastsave So is the timestamp Redis A state maintained by the server that records the last successful execution save/bgsave Time of day.
save m n The principle of is as follows:
Every 100 ms，implement serverCron Function; stay serverCron Function, traversal save m n As long as one of the storage conditions of the configuration is met,
Proceed bgsave. For each save m n Conditions are met only when the following two conditions are met at the same time:
lastsave > m
dirty >= n
5: RDB file processing and repair
preservation: RDB File saved in dir Under the directory specified by the configuration, the file name is passed dbfilename Configuration assignment.
config set dir {newDir} Temporary settings rdb File storage directory (Note: log and AOF The file storage directory also refers to this configuration)
config set dbfilename {newFileName} Temporary settings rdb File name of the file store (enabled for the next automatic persistent refresh after setting)
Compression: Redis Default use LZF Algorithm pair generated RDB The file is compressed. The compressed file is far smaller than the memory size,
It is enabled by default and can be accessed through parameters config set rdbcompression {yes|no} Dynamic modification.
Note: if the above modified configuration wants to be permanently modified, change the configuration file!!
Verification: if Redis Load corrupted RDB Refuse to start the file and print the following log:
# Short read or OOM loading DB. Unrecoverable error, aborting now.
You can use Redis Provided redis-check-dump Tool detection RDB File and get the corresponding error report
3: AOF persistence
1: AOF persistence execution mode
RDB persistence mode is automatically triggered by default. If you want AOF persistence mode, we must first enable this function in the configuration file; appendonly yes is not enabled by default; The file name of AOF persistence needs to be configured through appendfilename ， and the default AOF file name is appendonly aof； The save path is consistent with the RDB persistence mode and is specified through dir configuration.
The execution method of AOF is as follows: as long as the add, delete and modify command is executed, an append operation will be executed to appendonly Add, delete and modify commands in AOF file
technological process:
①: All addition, deletion and modification commands will be appended to aof_buf(AOF Buffer).
②: According to different synchronization policies aof_buf Synchronize the contents of to the hard disk.
③: along with AOF Files are getting larger and larger, and need to be updated regularly AOF The file is rewritten to achieve the purpose of compression.
④: When Redis When the server restarts, it can be loaded AOF File for data recovery.
AOF Command append mode( append):
Redis First append the write command to the buffer instead of directly writing to the file, mainly to avoid directly writing to the hard disk every time there is a write command IO become Redis Load bottleneck
The format of command append is Redis The protocol format of command request, which is a plain text format, has the advantages of good compatibility, strong readability, easy processing, simple operation and avoiding secondary overhead
The specific format is omitted. stay AOF File, except for the select Command (if any) select 0 Is selected by Redis Added,
Others are write commands sent by the client.
2: AOF profile
# Open or not AOF Logging, default redis Using rdb Mode persistence, which is enough in many applications, but redis If it goes down halfway, # This may result in data loss for several minutes(Depending on dump Data interval)，according to save To persist the policy, # Append Only File It is another persistence method, which can provide better persistence features, Redis The data written each time will be written after receiving # appendonly.aof File, every time you start Redis Will first read the data of this file into memory and ignore it first RDB Documents. This feature is not enabled by default appendonly yes # text file AOF The file name of the file is stored in the directory dir Configuration in configuration appendfilename "appendonly.aof" # aof Configuration of persistence policy( appendfsync Attribute description) # always: # Command write aof_buf Call the system immediately after fsync Synchronize operations to AOF Documents, fsync When finished, the thread returns. In this case, every addition, deletion and modification command should be synchronized to AOF Documents, # This hard disk IO Will become a performance bottleneck, Redis It can only support about a few hundred TPS Write, severely reduced Redis Performance of; Even with solid state drives( SSD)，Every second can only # Handle tens of thousands of commands, and it will be greatly reduced SSD Life span. # Summary: each addition, deletion and modification shall be implemented fsync，To ensure data synchronization to disk,High safety,But the performance is poor # no: # Command write aof_buf Post call system write Operation, No AOF File do fsync Synchronization; Synchronization is the responsibility of the operating system, and usually the synchronization cycle is 30 seconds. In this case, # The time of file synchronization is uncontrollable, and there will be a lot of data accumulated in the buffer, so the data security cannot be guaranteed. # Summary: the operating system ensures data synchronization to disk,Linux Default fsync The policy is 30 seconds and a maximum of 30 seconds will be lost s Data # everysec: # Command write aof_buf Post call system write Operation, write After completion, the thread returns; fsync The synchronous file operation is called by a dedicated thread once per second. # everysec It is a compromise between the above two strategies and a balance between performance and data security Redis The default configuration is also our recommended configuration. # appendfsync always appendfsync everysec # appendfsync no # stay AOF rewrite period,Right aof Newly recorded append Suspend file synchronization policy,Disk is mainly considered IO Expense and request blocking time # no: # express"No suspension",new aof Records will still be synchronized to disk immediately, which is the safest way. There will be no data loss, but we have to endure the problem of blocking # yes: # Equivalent to appendfsync Set to no，This indicates that the disk operation is not performed, but the buffer is written, so this is not necessary # It will cause blocking (because there is no competing disk), but if at this time redis Hang up and lose data. How much data is lost? Linux # Default fsync The policy is 30 seconds and a maximum of 30 seconds will be lost s Data,But because yes It has good performance and can avoid blocking, so it is recommended rewrite # That is right aof Organize documents,Reclaim free space,This reduces data recovery time no-appendfsync-on-rewrite no
# aof_current_size and aof_base_size can execute info persistence or info directly # implement AOF When overridden, the current AOF size(Namely aof_current_size)And last rewrite AOF size(aof_base_size)Ratio of; One of the trigger conditions for file rewriting auto-aof-rewrite-percentage 100 # implement AOF When rewriting, the minimum volume of the file. The default value is 64 MB. One of the trigger conditions for file rewriting auto-aof-rewrite-min-size 64mb # Whether to load the end exception caused by some reason AOF file(The main process is kill/Power failure, etc)，proposal yes aof-load-truncated yes # redis4.0 newly added RDB-AOF Mixed persistence format. After this function is enabled, AOF The file generated by rewriting will also contain RDB Content and format AOF The content of the format, # among RDB The content of the format is used to record the existing data, and AOF The content of the format is used to record the data that has changed recently Redis You can have both RDB hold # Jiuhuahe AOF Advantages of persistence (it can not only generate rewritten files quickly, but also load data quickly in case of problems),Default to no,This feature is not enabled aof-use-rdb-preamble yes
3: AOF file rewrite
Over time, Redis The server executes more and more write commands, AOF Documents will become larger and larger; Oversized AOF Files will not only affect the normal operation of the server,
It will also cause data recovery to take too long.
File rewriting refers to periodic rewriting AOF Files, reducing AOF The volume of the file. It should be noted that, AOF Rewriting is Redis The data in the process is converted into write commands,
Sync to new AOF Documents; Not for the old AOF Any read or write operation of the file!
Another thing to note about file rewriting is: AOF For persistence, although file rewriting is strongly recommended, it is not necessary; Even without file rewriting,
Data can also be persisted and stored in Redis Import at startup; Therefore, in some implementations, automatic file rewriting is turned off and then executed at a certain time of the day through a scheduled task.
The reason why file rewriting can be compressed AOF File because:
①: Expired data is no longer written to the file
②: Invalid commands are no longer written to the file: for example, some data are set repeatedly(set name jack, set name tom),
Some data has been deleted(sadd myset v1, del myset)wait
③: Multiple commands can be combined into one: for example: sadd myset v1, sadd myset v2, sadd myset v3 Can be merged into sadd myset v1 v2 v3
However, in order to prevent client buffer overflow caused by too large a single command, for list,set,hash,zset Type key，It is not necessary to use only one command;
Instead, the command is split into multiple commands bounded by a constant.
This constant is in redis.h/REDIS_AOF_REWRITE_ITEMS_PER_CMD Defined in, cannot be changed, 3.0 The median version is 64.
As can be seen from the above, due to the rewriting AOF The number of commands executed is reduced. File rewriting can not only reduce the space occupied by files, but also speed up recovery
4: Trigger of file rewriting (manual trigger)
Directly call the bgrewriteaof command. The execution of this command is somewhat similar to that of bgsave. It is a fork subprocess that performs specific work and is blocked only when forking
Perform manual trigger AOF file overwrite client: 127.0.0.1:6379> bgrewriteaof Background append only file rewriting started Server: 1751:M 12 Dec 2021 20:33:45.401 * Background append only file rewriting started by pid 1753 1751:M 12 Dec 2021 20:33:45.736 * AOF rewrite child asks to stop sending diffs. 1753:C 12 Dec 2021 20:33:45.752 * Parent agreed to stop sending diffs. Finalizing AOF... 1753:C 12 Dec 2021 20:33:45.754 * Concatenating 0.00 MB of AOF diff received from parent. 1753:C 12 Dec 2021 20:33:45.769 * SYNC append only file rewrite performed 1751:M 12 Dec 2021 20:33:45.845 * Background AOF rewrite terminated with success 1751:M 12 Dec 2021 20:33:45.848 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB) 1751:M 12 Dec 2021 20:33:45.863 * Background AOF rewrite finished successfully
5: Trigger of file rewriting (automatic trigger)
Auto trigger is based on auto AOF rewrite min size and auto AOF rewrite percentage configuration parameters, as well as aof_current_size and aof_base_size status determines the trigger timing
auto-aof-rewrite-min-size:
implement AOF When rewriting, the minimum volume of the file. The default value is 64 MB.
auto-aof-rewrite-percentage:
implement AOF When overridden, the current AOF size(Namely aof_current_size)And last rewrite AOF size(aof_base_size)Ratio of
Can pass config get Command view:
127.0.0.1:6379[12]> config get auto-aof-rewrite-percentage
1) "auto-aof-rewrite-percentage"
2) "100" -- Percentage 100%
127.0.0.1:6379[12]> config get auto-aof-rewrite-min-size
1) "auto-aof-rewrite-min-size"
2) "67108864" -- Size 64 M
Status can be passed info persistence see:(Other states are not introduced here)
aof_base_size: After last rewrite AOF File space
aof_current_size: current AOF File space
Only whenauto-aof-rewrite-min-sizeandauto-aof-rewrite-percentageWhen the two parameters are met at the same time,
Will trigger automatically AOF Rewrite, i.e bgrewriteaof operation
Automatic trigger condition:
aof_current_size > auto-aof-rewrite-min-size &&
(aof_current_size - aof_base_size) / aof_base_size >= auto-aof-rewrite-percentage
To see the execution effect, I modify the parameters: auto-aof-rewrite-percentage 1 auto-aof-rewrite-min-size 1mb When AOF When the size peak is reached, the rewriting will be performed automatically; But it will be rewritten aof The file will be compressed very small Redis Server Trigger Auto rewrite journal: 1611:M 13 Dec 2021 13:51:45.941 * Starting automatic rewriting of AOF on 105343400% growth 1611:M 13 Dec 2021 13:51:45.986 * Background append only file rewriting started by pid 1621 1611:M 13 Dec 2021 13:51:46.594 * AOF rewrite child asks to stop sending diffs. 1621:C 13 Dec 2021 13:51:46.610 * Parent agreed to stop sending diffs. Finalizing AOF... 1621:C 13 Dec 2021 13:51:46.610 * Concatenating 0.02 MB of AOF diff received from parent. 1621:C 13 Dec 2021 13:51:46.616 * SYNC append only file rewrite performed 1611:M 13 Dec 2021 13:51:46.671 * Background AOF rewrite terminated with success 1611:M 13 Dec 2021 13:51:46.674 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB) 1611:M 13 Dec 2021 13:51:46.681 * Background AOF rewrite finished successfully
6: File rewriting process
For the process of file rewriting, please note:
①: Override by parent process fork Create a child process for persistence
②: Rewrite period Redis The executed write command needs to be appended to a new one AOF File, for this purpose Redis Introduced aof_rewrite_buf cache
Compared with the above figure, the process of automatic file rewriting is as follows:
1: Redis The parent process first determines whether there is currently executing bgsave/bgrewriteaofChild processes, if any bgrewriteaof The command returns directly,
If it exists bgsave Command, etc bgsave Execute after execution. As mentioned earlier, this is mainly based on performance considerations.
If the current process is executing AOF(bgrewriteaof)Rewrite, the request is not executed and the following response is returned:
ERR Background append only file rewriting already in progress
If the current process is executing bgsave Operation, override command delayed to bgsave Execute after completion and return the following response:
Background append only file rewriting scheduled
2: Parent process execution fork Operation creates a child process, in which the parent process is blocked (the cost is equivalent to bgsave (process)
3.1: Parent process fork After, bgrewriteaof Command returnBackground append only file rewrite started"Information;
At the same time, the parent process is no longer blocked, and can continue to respond to other commands;All addition, deletion and modification commands are still written AOF Buffer and according to appendfsync Policy synchronization to hard disk,
Guarantee the original AOF Correctness of mechanism.
3.2: because fork The operation uses write time replication technology, and child processes can only be shared fork Memory data during operation.Because the parent process is still responding to commands, the Redis use AOF rewrite
buffer(In the figure aof_rewrite_buf)Save this part of data to prevent new AOF This data was lost during file generation. in other words, bgrewriteaof Execution period
Redis The write command of is appended to aof_buf and aof_rewirte_buf Two buffers.
4: The subprocess writes to the new memory according to the memory snapshot and the command merge rules AOF Documents. The amount of data written to the hard disk in batches each time is configured by the aof-rewrite-incremental-fsync control
The default is 32 MB，Prevent the hard disk from blocking due to too much data in a single disk brushing.
5.1: Subprocess writes new AOF After the file, send a signal to the parent process, and the parent process updates the statistical information. For details, you can info persistence see.
5.2: Parent process AOF The data of the rewriting buffer is written to the new buffer AOF Documents, which ensures the new AOF The database state saved by the file is consistent with the current state of the server.
5.3: Use new AOF File replace old file, done AOF rewrite
8: File verification and AOF file repair
File verification start:
127.0.0.1:6379> config get aof-load-truncated -- skip AOF Incomplete end of file
1) "aof-load-truncated"
2) "yes"
AOF and RDB File loading is similar, Redis load AOF The file will be AOF Check the file. If the file is found to be damaged (i.e. modified), the log will be printed
Wrong, Redis The startup failed;
if AOF The file is because the end of the file is incomplete( Redis Sudden server downtime (power failure), and AOF Configured aof-load-truncated by yes(default)When,
The incomplete end of the file will be ignored and a warning will be output in the log; (Note: it can only be the end+The line is damaged and cannot be modified or deleted)
450:M 14 Dec 2021 09:40:41.141 # Server initialized
450:M 14 Dec 2021 09:40:41.142 # !!! Warning: short read while loading the AOF file !!!
450:M 14 Dec 2021 09:40:41.142 # !!! Truncating the AOF at offset 90 !!!
450:M 14 Dec 2021 09:40:41.144 # AOF loaded anyway because aof-load-truncated is enabled
450:M 14 Dec 2021 09:40:41.145 * DB loaded from append only file: 0.003 seconds
450:M 14 Dec 2021 09:40:41.146 * Ready to accept connections
if aof If the file is corrupt enough to fail to start, you need to AOF File repair:
Note: AOF The file repair will locate the wrong line until the last line is deleted (operate properly after backup)
use Redis Incidental redis-check-aof Program, for the original AOF Repair the file and restart it redis
Linux The following operations: redis-check-aof --fix To be repaired AOF file
Windows The following operations:
PS C:\Users\redis-x64-6.2.5> .\redis-check-aof.exe --fix .\appendonly.aof
0x 56: Expected \r\n, got: 3233
AOF analyzed: size=113, ok_up_to=60, ok_up_to_line=19, diff=53
This will shrink the AOF from 113 bytes, with 53 bytes, to 60 bytes
Continue? [y/N]: y
Successfully truncated AOF
4: Restart loading
Both AOF and RDB files can be used for data recovery when Redis server is restarted; However, when Redis is started, AOF files will be loaded first to recover data. RDB files will be loaded to recover data only when AOF is closed
When AOF When persistence is off and there is no .RDB Persistent file
1775:M 13 Dec 2021 18:07:47.923 # Server initialized
1775:M 13 Dec 2021 18:07:47.924 * Ready to accept connections
When AOF When persistence is turned off, and .RDB Persistent file -- No files are read
80:M 13 Dec 2021 18:08:38.896 # Server initialized
80:M 13 Dec 2021 18:08:38.897 * Loading RDB produced by version 6.2.5
80:M 13 Dec 2021 18:08:38.897 * RDB age 5 seconds
80:M 13 Dec 2021 18:08:38.897 * RDB memory usage when created 0.51 Mb
80:M 13 Dec 2021 18:08:38.897 * DB loaded from disk: 0.001 seconds
80:M 13 Dec 2021 18:08:38.898 * Ready to accept connections
When AOF When persistence is on and there is no .AOF Persistent file -- No files are read
515:M 13 Dec 2021 18:09:36.702 # Server initialized
1515:M 13 Dec 2021 18:09:36.704 * Ready to accept connections
When AOF When persistence is on, and .AOF Persistent file(Note: if.AOF If the file is empty, it is empty.AOF File is not executed)
178:M 13 Dec 2021 18:10:08.803 # Server initialized
178:M 13 Dec 2021 18:10:08.804 * DB loaded from append only file: 0.001 seconds
178:M 13 Dec 2021 18:10:08.804 * Ready to accept connections
5: Advantages and disadvantages of AOF and RDB
RDB persistence:
Advantages: RDB files are compact, small in size, fast in network transmission, and suitable for full replication; Recovery is much faster than AOF. Of course, one of the most important advantages of RDB over AOF is that it has a relatively small impact on performance.
Disadvantages: the fatal disadvantage of RDB file is that its data snapshot persistence mode determines that real-time persistence is inevitable. Today, when data is becoming more and more important, a large amount of data loss is often unacceptable. Therefore, AOF persistence has become the mainstream. In addition, RDB files need to meet specific formats and have poor compatibility (for example, the old version of Redis is not compatible with the new version of RDB files).
AOF persistence:
Corresponding to RDB persistence, AOF has the advantages of supporting second level persistence and good compatibility. Its disadvantages are large files, slow recovery speed and great impact on performance.