Architecture diagram
- Introduction to sentry
Sentinel, Chinese is sentinel.
Sentry is a very important component in the redis cluster architecture. Its main functions are as follows:
(1) Cluster monitoring: responsible for monitoring whether the reidis master and slave processes work normally;
(2) Message notification: if a redis instance fails, the sentry is responsible for sending a message as an alarm notification to the administrator;
(3) Failover: if the master node hangs, it will be automatically transferred to the slave node;
(4) Configuration center: if failover occurs, notify the client of the new master address;
Sentinels are also distributed. They operate as a sentinel cluster and work together with each other;
(1) During failover, it needs the consent of most sentinels to judge that a master node is down, which involves distributed election;
(2) Even if some sentinel nodes hang up, the sentinel cluster can still work normally, because if a failover system itself, which is an important part of the high availability mechanism, is a single point, it will be in trouble; Sentinel 2 is currently used. Compared with sentinel 1, sentinel 2 rewrites a lot of code, mainly to make the failover mechanism and algorithm more robust and simple. - Sentinel core knowledge
(1) Sentinels need at least 3 instances to ensure their robustness;
(2) The sentry + redis master-slave deployment architecture will not guarantee data 0 loss, but only the high availability of the redis cluster;
(3) For the responsible deployment architecture of sentry + redis master-slave, try to conduct sufficient tests and drills in both the test environment and the production environment
deploy
Prepare docker environment
hostnamectl set-hostname l-gz-txy-lyy-uat-redis-master-001
sudo yum install -y yum-utils
sudo yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
sudo yum install docker-ce docker-ce-cli containerd.io -y
systemctl start docker
systemctl enable docker
Three virtual machines
10.180.3.125:7000 master node
10.180.3.181:7000 slave node
10.180.3.90:7000 slave node
The master node creates redis data and conf directories
mkdir -p /data/redis/7000 mkdir /data/redis/7000/conf mkdir /data/redis/7000/data cd /data/redis/7000/conf
Modify redis Conf main configuration
cat > /data/redis/7000/conf/redis.conf <<EOF port 7000 masterauth "YACEheyuan@123" requirepass "YACEheyuan@123" protected-mode yes tcp-backlog 511 timeout 0 tcp-keepalive 300 daemonize no supervised no pidfile /var/run/redis_6379.pid loglevel notice #logfile "/var/redis.log" databases 16 always-show-logo yes save 900 1 save 300 10 save 60 10000 stop-writes-on-bgsave-error yes rdbcompression yes rdbchecksum yes dbfilename dump.rdb dir ./ replica-serve-stale-data yes replica-read-only yes repl-diskless-sync no repl-diskless-sync-delay 5 repl-timeout 1 repl-disable-tcp-nodelay no replica-priority 100 maxmemory 1024000kb maxmemory-policy volatile-lru lazyfree-lazy-eviction no lazyfree-lazy-expire no lazyfree-lazy-server-del no replica-lazy-flush no appendonly no appendfilename "appendonly.aof" appendfsync everysec no-appendfsync-on-rewrite no auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 512mb aof-load-truncated yes aof-use-rdb-preamble yes lua-time-limit 5000 cluster-enabled no slowlog-log-slower-than 10000 slowlog-max-len 128 latency-monitor-threshold 0 notify-keyspace-events "" hash-max-ziplist-entries 512 hash-max-ziplist-value 64 list-max-ziplist-size -2 list-compress-depth 0 set-max-intset-entries 512 zset-max-ziplist-entries 128 zset-max-ziplist-value 64 hll-sparse-max-bytes 3000 stream-node-max-bytes 4096 stream-node-max-entries 100 activerehashing yes client-output-buffer-limit normal 0 0 0 client-output-buffer-limit replica 256mb 64mb 60 client-output-buffer-limit pubsub 32mb 8mb 60 hz 10 dynamic-hz yes aof-rewrite-incremental-fsync yes rdb-save-incremental-fsync yes EOF
Configure sentinel
cat >/data/redis/7000/conf/sentinel.conf <<EOF port 27000 daemonize yes sentinel monitor mymaster 10.180.3.125 7000 1 sentinel auth-pass mymaster YACEheyuan@123 EOF
Copy the entire / data/redis directory to the standby node
scp -r /data/redis root@10.180.3.181:/data/
Both the master and slave nodes start the redis container
cat >start.sh<<EOF docker run -d -ti -p 7000:7000 -p 27000:27000 \ -v /data/redis/7000/conf/redis.conf:/usr/local/etc/redis/redis.conf \ -v /data/redis/7000/conf/sentinel.conf:/usr/local/etc/redis/sentinel.conf \ -v /data/redis/7000/data:/data \ -v /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime \ --privileged=true --restart always --name redis_7000 \ uhub.service.ucloud.cn/jackhe/redis-sentinel:latest redis-server /usr/local/etc/redis/redis.conf EOF chmod +x start.sh sh start.sh
Enter the container from the node and set yourself as the slave node
docker exec -it redis_7000 bash redis-cli -h 127.0.0.1 -c -p 7000 -a YACEheyuan@123 SLAVEOF 10.180.3.125 7000
Enter the containers of the master and slave nodes respectively and start sentinel
redis-sentinel /usr/local/etc/redis/sentinel.conf
Enter info to check whether the setting is successful
After startup, close the redis container of the primary node and wait 30 seconds to see whether the standby node switches to the primary node
Start the redis container of the primary node, check whether the primary node is connected as a standby node, and start sentinel
Deployment logic
docker starts redis - declare slave node - start sentinel
Verify availability
- set key on master
# Keyspace 127.0.0.1:7000> set a 1 OK 127.0.0.1:7000> get a "1"
- get on any slave
# Keyspace 127.0.0.1:7000> get a "1"
- Script test
from redis.sentinel import Sentinel
sentinel = Sentinel([('10.180.3.198', 27000)],
password='YACEheyuan@123',socket_timeout=0.5)
master = sentinel.discover_master("mymaster")
print(master)
Pressure measurement
redis-benchmark -h 10.180.3.125 -p 27000 -a YACEheyuan@123 -c 100 -n 100000 -q