Persistence
Redis is an in memory database. It needs to be persistent in case of power failure and power loss. RDB is used by default. Generally, we can use it without modifying RDB configuration.
RDB (Redis DataBase)
What is RDB
Redis will create a fork subprocess separately for persistence. In the subprocess, it will cycle all data and write the data to the binary file. First, it will write the data to a temporary file. When the persistence process is over, it will replace the last persistent file with this temporary file. In the whole process, the main process does not perform any IO operations to ensure high performance. If large-scale data recovery is required and the integrity of data recovery is not very sensitive, RDB mode is more efficient than AOF mode. The power off of RDB is the last persistence, and the data may be lost.
flow chart:
The default file saved by rdb is dump rdb can be configured in the snapshot of the configuration file.
RDB trigger mechanism
1. When the rule of configuration file save is satisfied, rdb will be triggered automatically to generate rdb files.
2. Executing the flush command will also trigger rdb and automatically generate an rdb file.
3. Exit redis and enter the command shutdown to shut down the redis service. At this time, rdb files will also be generated.
Recover RDB files
1. Just put the rdb file under the Redis startup folder. When Redis starts, the rdb file will be automatically loaded and recovered.
2. Check the location of the startup folder we need to store
127.0.0.1:6379> config get dir 1) "dir" 2) "/opt/redis-6.2.6" # If dump exists in this directory RDB file. Redis will automatically load recovery data when it starts
Advantages and disadvantages of RDB
advantage:
1. Suitable for large-scale data recovery
2. It can be used when the integrity of data is not high
Disadvantages:
1. Operations need to be performed at a certain interval. If redis goes down unexpectedly, the last modified data will not be available
2. The fork process takes up a certain amount of memory space
AOF(Append Only File)
What is AOF
Let's start with a flow chart:
Aof is to record all our commands, and then re execute them when recovering. Only write commands are recorded, and read commands are not recorded. Append only files. Therefore, the speed of AOF will be slower. Especially in the case of big data, it will take a long time to recover the data
The default file saved by AOF is appendonly aof
It is not enabled by default. I don't need to configure it manually. Then we can restart Redis to take effect.
AOF trigger mechanism
1. Modify the configuration file and restart Redis to trigger AOF persistence.
2. When the rules of the configuration file appendfsync are met, aof will be triggered automatically to generate an aof file
# 1. Delete appendonly aof [root@VM-0-16-centos bin]# ls appendonly.aof busybox-x86_64 dump.rdb redis-benchmark redis-check-aof redis-check-rdb redis-cli redis.conf redis-sentinel redis-server [root@VM-0-16-centos bin]# rm -rf appendonly.aof [root@VM-0-16-centos bin]# ls busybox-x86_64 dump.rdb redis-benchmark redis-check-aof redis-check-rdb redis-cli redis.conf redis-sentinel redis-server # 2. Log in to redis and restart after modifying the configuration file [root@VM-0-16-centos bin]# redis-cli -p 6379 127.0.0.1:6379> auth hecan1992 OK 127.0.0.1:6379> SHUTDOWN not connected> exit [root@VM-0-16-centos bin]# ./redis-server /opt/redis-6.2.6/redis.conf # 3. Check whether appendonly. Is generated aof [root@VM-0-16-centos bin]# ls appendonly.aof busybox-x86_64 dump.rdb redis-benchmark redis-check-aof redis-check-rdb redis-cli redis.conf redis-sentinel redis-server # 4. Log in to redis and assign a key [root@VM-0-16-centos bin]# redis-cli -p 6379 127.0.0.1:6379> auth hecan1992 OK 127.0.0.1:6379> set k1 v1 OK 127.0.0.1:6379> set k2 v2 OK 127.0.0.1:6379> set k3 v3 OK 127.0.0.1:6379> set k4 v4 OK 127.0.0.1:6379> exit # 5. View appendonly Aof file content [root@VM-0-16-centos bin]# vim appendonly.aof *2 $6 SELECT $1 0 *3 $3 set $2 k1 $2 v1 *3 $3 set $2 k2 $2 v2 *3 $3 set $2 k3 $2 v3 *3 $3 set $2 k4 $2 v4
Above, we successfully opened AOF and generated appendonly Aof file. From the contents of the file, we can see that we recorded the command records of K1, K2, K3 and K4.
AOF appendonly AOF file repair function
We do a very interesting operation, we will append only The contents of the AOF file command are randomly changed to see what happens?
# 1. View the files in the redis startup directory and delete dump rdb [root@VM-0-16-centos bin]# ls appendonly.aof busybox-x86_64 redis-benchmark redis-check-aof redis-check-rdb redis-cli redis.conf redis-sentinel redis-server [root@VM-0-16-centos bin]# rm -rf dump.rdb [root@VM-0-16-centos bin]# ls appendonly.aof busybox-x86_64 redis-benchmark redis-check-aof redis-check-rdb redis-cli redis.conf redis-sentinel redis-server # 2. Log in to the redis client and shut down the redis server [root@VM-0-16-centos bin]# redis-cli -p 6379 127.0.0.1:6379> auth hecan1992 OK 127.0.0.1:6379> ping PONG 127.0.0.1:6379> SHUTDOWN not connected> exit # 3. Modify appendonly Aof, and then randomly add some content at the end of the file [root@VM-0-16-centos bin]# vim appendonly.aof [root@VM-0-16-centos bin]# cat appendonly.aof *2 $6 SELECT $1 0 *3 $3 set $2 k1 $2 v1 *3 $3 set $2 k2 $2 v2 *3 $3 set $2 k3 $2 v3 *3 $3 set $2 k4 $2 v4 fgGgegsagaggsgdzgggdgq # Random content # 4. Restart redis. We found that the startup failed [root@VM-0-16-centos bin]# ./redis-server /opt/redis-6.2.6/redis.conf [root@VM-0-16-centos bin]# ps -ef|grep redis root 14516 13825 0 09:25 pts/0 00:00:00 grep --color=auto redis
We found that when we will appendonly After the AOF file is modified, redis cannot be started. We need to fix appendonly Aof file.
The root directory of Redis restart provides us with a repair tool Redis check AOF
# 1. Command: redis check AOF -- fix appendonly Aof repair file [root@VM-0-16-centos bin]# redis-check-aof --fix appendonly.aof 0x 8b: Expected prefix '*', got: 'f' AOF analyzed: size=163, ok_up_to=139, ok_up_to_line=34, diff=24 This will shrink the AOF from 163 bytes, with 24 bytes, to 139 bytes Continue? [y/N]: y Successfully truncated AOF # 2. View repaired files [root@VM-0-16-centos bin]# cat appendonly.aof *2 $6 SELECT $1 0 *3 $3 set $2 k1 $2 v1 *3 $3 set $2 k2 $2 v2 *3 $3 set $2 k3 $2 v3 *3 $3 set $2 k4 $2 v4 # 3. We found that the file has been successfully repaired, and the start-up and link of redis are successful again [root@VM-0-16-centos bin]# ./redis-server /opt/redis-6.2.6/redis.conf [root@VM-0-16-centos bin]# ps -ef|grep redis root 16926 1 0 09:38 ? 00:00:00 ./redis-server *:6379 root 16957 13825 0 09:38 pts/0 00:00:00 grep --color=auto redis [root@VM-0-16-centos bin]# redis-cli -p 6379 127.0.0.1:6379> auth hecan1992 OK 127.0.0.1:6379> ping PONG 127.0.0.1:6379>
Summary: when the file appendonly When there is a problem with AOF, we execute the command under the root directory: redis check AOF -- fix appendonly Aof can repair files.
Rewriting of AOF
View redis Conf profile discovery:
aof file is the default file. The file will be larger and larger if it is appended infinitely. When we open the file no appendfsync on rewrite to yes in the configuration file and set the file size, Redis will automatically detect whether the aof file is greater than 64M. If it is greater than 64M, Redis will automatically fork a new process to rewrite our aof file.
AOF advantages and disadvantages
advantage:
1. Every modification is synchronized, and the integrity of the file will be better
2. Sync once per second, and you may lose one second of data
3. Never synchronized, most efficient
appendfsync always #Each modification consumes sync performance appendfsync everysec #sync is executed once every second, and this 1s of data may be lost appendfsync no #When sync is not executed, the operating system synchronizes the data itself, which is the fastest
Disadvantages:
1. Compared with data files, AOF is much larger than RDB, and repairing data is slower than RDB.
2. The running efficiency of AOF is also slower than that of RDB, so the default configuration of Redis is RDB
extend
- RDB persistence can snapshot and store your data within a specified time interval
- Aof persistence records every write operation to the server. When the server restarts, these commands will be re executed to recover the original data. The AOF command additionally saves each write operation to the end of the file with redis protocol. Redis can also rewrite the AOF file in the background so that the volume of the AOF file is not too large
- Only cache. If you only want your data to exist when the server is running, you can completely avoid any persistence
- Enable two persistence methods at the same time
1: In this case, when redis restarts, AOF files will be preferentially loaded to recover data, because in general, the data set saved by AOF files is more complete than that saved by RDB files
2: RDB data is not real-time. When using both, the server will only find AOF files when restarting. Do you want to use AOF only? The author suggests no, because RDB is more suitable for backing up the database (AOF is changing and hard to back up), fast restart, and there will be no potential bug s in AOF. Keep it as a means in case - Performance recommendations
1: Because RDB files are only used for back-end purposes, it is recommended to persist RDB files on Slave, and only backup once every 15 minutes is enough. Only save 900 1 is retained
2: If you Enable AOF, the advantage is that in the worst case, only less than two seconds of data will be lost. The startup script is relatively simple, and you can only load your own AOF file. Firstly, it brings continuous IO, and secondly, the blocking caused by writing new data generated in the process of AOF rewriter to new files is inevitable. As long as the hard disk is allowed, the frequency of AOF rewriting should be minimized. The default value of 64M for the basic size of AOF rewriting is too small and can shoot above 5G. By default, it exceeds 100% of the original size, and the size rewriting can be changed to an appropriate value
3: If AOF is not enabled, high availability can be achieved only by master slave replay, which can save a lot of IO and reduce the system fluctuation caused by rewriting. The price is that if the master / slave is dropped at the same time, more than ten minutes of data will be lost. The startup script also needs to compare the RDB files in the master / slave and load the newer one. This is the architecture of microblog
Acknowledge
This article is the learning notes of Redis
Learning address: https://www.bilibili.com/video/BV1S54y1R7SB?p=28