Middleware learning notes - Introduction to Redis

Posted by rakennedy75 on Sun, 27 Feb 2022 09:33:32 +0100

1. Redis (CACHE)

1.1 Nosql

1.1.1 general

Single machine mysql era:

In the 1990s, the number of visits to a basic website will not be too large, and a single database is enough!

More use of static web pages html

Website bottleneck:

  • The amount of data is too large for one machine
  • The index of data can't fit into the memory of a machine
  • Too many visits, mixed reading and writing, performance problems

Cache age:

Memcached + mysql + read write separation (vertical split)

80% of the website is reading. It's too troublesome to query the database every time. To reduce the pressure on the database, you can use Cache to ensure efficiency -- solve the problem of reading

Development process: optimize the underlying structure and index of Mysql – > file caching (IO operation) - > memcached was the most popular technology at that time!

Sub database and sub table + horizontal split + mysql Cluster:

With the development of technology and business, the technical requirements are becoming higher and higher

Essence: database (read, write)

Early years MyISAM: watch lock, affecting efficiency, high efficiency, and serious lock problems

Innodb in earlier years: row lock

Slowly use sub database and sub table to solve the pressure of writing! Not many companies use it

Mysql Cluster can well meet individual needs

Recent years:

2010-2020 rapid technological development (positioning, music, hot list)

Mysql and other relational databases are not enough. There is a lot of data and changes quickly

At present, there is a basic Internet Project:

Why use Nosql

  • Users' personal information, geographical location, social networks, data generated by users themselves, user logs and so on are growing explosively!

At this time, you can use Nosql database to deal with the above situation!

1.1.2 what is NoSQL


NoSQL=not only sql

SQL: table, row, column

Generally refers to non relational database, with web2 0 the birth of the Internet, the traditional relational database is difficult to deal with web2 0 era! Especially large-scale and highly concurrent communities!

NoSQL is developing rapidly in today's big data era, and REDIS is developing fastest

NoSQL features

  • Convenient expansion (no relationship between data, good expansion, no coupling)

  • High performance under large amount of data (redis can write 8W times and read 11w times per second. NoSQL cache level is a fine-grained cache, and the performance will be relatively high!)

  • Multiple types (no need to design the database in advance! Use as you go! If it is a table with a large amount of data, many people can't design it)

  • Traditional RDBM and NoSQL:

    Structured organization


    Data and relationships exist in separate tables

    Data manipulation language, data definition language,

    Strict consistency

    Basic transaction


Not just data

There is no fixed query language

Key value pair storage, column storage, document storage, graphic database (social relationship)

Final consistency

CAP theorem, BASE (remote multi activity) junior architect!

Basic high performance, high availability and high scalability

​ ...

Expansion: 3v and 3 high of big data

Describe the problem: massive, diverse and real-time

Program requirements: high concurrency, high availability and high performance

1.1.3 classification of NoSQL

KV key value pair

  • Sina: redis
  • Meituan: redis+Tair
  • Alibaba, baidu: redis+memcache

Document database (json format)

  • MongoDB (generally required)

    Database based on distributed file storage, written in C + +, processing a large number of documents!

    It is a product between relational database and non relational database. It is the most functional and most like relational database in non relational database!

  • ConthDB

Column storage database

  • HBase
  • distributed file system

Graphical relational database

  • Not pictures, but relationships: circle of friends, social networks, advertising recommendations!

  • Ne04j, infoGrid

Comparison of NoSQL

1.1.4 Redis

redis–Remote Dictionary Server

Redis (Remote Dictionary Server), that is, remote dictionary service, is an open source log type and key value database written in ANSI C language, supporting network, memory based and persistent, and provides API s in multiple languages. Free and open source, one of the most popular NoSql

What can I do?

  • Memory storage and persistence. Memory is lost in case of power failure, so persistence is very important (rdb/aof)

  • Efficient and can be used for caching

  • Publish subscribe system

  • Map information analysis

  • Timer, counter (Views)

  • ...


  • Diverse data types
  • Persistence
  • colony
  • affair
  • ...

Things needed in learning

  • Official website
  • Chinese network
  • Download address: through the official website (it is not recommended to use windows development officially)

1.2 Redis installation

1.2.1 windows installation

  • Download installation package https://github.com/microsoftarchive/redis/releases/tag/win-3.2.100

  • Extract to a specific folder

  • Start Redis: double click to run the service (start in seconds)

Default port: 6379

  • Connect using Redis client

Test connection:

  • ping test returns PONG
  • Set a key value and get it

It is really easy to use under windows, but it is recommended to use linux for development

1.2.2 Linux Installation download installation package

redis-6.0.6.tar.gz unzip the installation package

And enter

tar -zxvf redis-6.0.6.tar.gz

Profile: redis conf basic environment installation

  1. yum install gcc-c++

  2. gcc -v

  1. Make (it will take some time)

Error report, cause analysis: gcc version is too low, upgrade it

# Upgrade gcc version
$ yum -y install centos-release-scl
$ yum -y install devtoolset-9-gcc devtoolset-9-gcc-c++ devtoolset-9-binutils
$ scl enable devtoolset-9 bash
# View the next version after upgrading
$ gcc -v

Re execute make

  1. make install (you can also confirm the installation without executing it)

Everything has been installed confirm the installation path

Redis default installation path: / usr/local/bin copying configuration files

Copy Redis configuration file to current directory

# Create a profile folder and copy profiles
$ mkdir hconfig
$ cp /opt/redis-6.0.6/redis.conf hconfig

Then use this configuration file to start configure background startup

Redis is not started in the background by default, and the configuration file needs to be modified

# Modify the daemon to yes
$ vim redis.conf start Redis service

Start Redis service

Start the service through the specified profile

redis-server hconfig/redis.conf test connection

Test connection

Connect using Redis client

# connect
$ redis-cli -p 6379> ping
PONG> set name yunmx   # Set a key
OK> get name		# Read a key
"yunmx"> keys *			# Query all key s
1) "name"> 

9.2 viewing process

Check whether the Redis process is started

ps -ef |grep redis turn off Redis service

Turn off redis service

shutdown> ping
PONG> shutdown
not connected> exit

1.3 Redis performance test tool

1.3.1 redis-benchmark

Official performance testing tool - >


Command parameters:

Simple test:

# Test 100 concurrent connections, each with 20 concurrent requests
$ redis-benchmark -h localhost -p 6379 -c 100 -n 20

1.3.2 how to analyze

View analysis:

====== PING_INLINE ======
  100000 requests completed in 1.46 seconds   	# 100000 ping requests completed in 1.46s
  100 parallel clients							# Concurrency of 100
  3 bytes payload								# 3-byte data write
  keep alive: 1									# Ensure one server test (stand-alone performance)
  host configuration "save": 900 1 300 10 60 10000
  host configuration "appendonly": no
  multi-thread: no

0.00% <= 0.5 milliseconds						# 0 percent of requests completed in 0.5 milliseconds
1.37% <= 0.6 milliseconds
11.42% <= 0.7 milliseconds
23.22% <= 0.8 milliseconds
35.29% <= 0.9 milliseconds
47.60% <= 1.0 milliseconds
60.01% <= 1.1 milliseconds
72.40% <= 1.2 milliseconds
84.82% <= 1.3 milliseconds
94.78% <= 1.4 milliseconds
97.65% <= 1.5 milliseconds
98.50% <= 1.6 milliseconds
98.95% <= 1.7 milliseconds
99.28% <= 1.8 milliseconds
99.51% <= 1.9 milliseconds
99.67% <= 2 milliseconds
99.99% <= 3 milliseconds
100.00% <= 3 milliseconds						# 100% requests completed in 3 milliseconds (100000)
68587.11 requests per second					# Average number of requests completed per second

1.4 Redis Basics

1.4.1 default database

The default number of Redis databases is 16, Redis The conf configuration file can be viewed, and the default database is 0

1.4.2 switching database

# Use the select command to switch
[root@yunmx bin]# redis-cli -p 6379> select 1
OK[1]> select 2
OK[2]> select 16
(error) ERR DB index is out of range[2]> select 15

1.4.3 viewing database size

# DBSIZE view database size> set name hejie
(integer) 1> SET AGE NAN
(integer) 2>

1.4.4 view all key s

# Using keys*> keys *
1) "AGE"
2) "name">

1.4.5 clearing the database

# Clear the current database flushdb> keys *
1) "AGE"
2) "name"> flushdb
OK> keys *
(empty array)>
# Empty all databases flush> select 0
OK> set name hejie
OK> set key1 hejie1
OK> set key2 hejie2
OK> keys *
1) "key1"
2) "key2"
3) "name"> select 1
OK[1]> set name yunmx
OK[1]> set key1 yunmx1
OK[1]> set key2 yunmx2
OK[1]> keys *
1) "key1"
2) "key2"
3) "name"[1]> flushall
OK[1]> keys *
(empty array)[1]> select 0
OK> keys *
(empty array)>

1.4.6 determine whether the key exists

# Use exists to judge> set name hejie
OK> set age 1
OK> keys *
1) "age"
2) "name"> exists age     
(integer) 1							# existence> exists name
(integer) 1> exists key1  
(integer) 0							# non-existent>

1.4.7 move a key

# Move move a key> set key1 hejie
OK> set key2 hejie1
OK> set key3 hejie2
OK> move key1 1      	# Move key1 to database 1
(integer) 1> select 1
OK[1]> keys *
1) "key1"[1]>

1.4.8 setting the effective time of key

# EXPIRE sets the expiration time of a key. Unit time: s
# TTL view the expiration time of a key> keys *
1) "key3"
2) "key2"> get key3
"hejie2"> EXPIRE key3 10     # Set Key3 to expire in 10 seconds
(integer) 1> ttl key3
(integer) -2					   # Indicates that it has expired> ttl key3
(integer) -2> keys *
1) "key2">

1.4.9 Redis single thread

Why is single thread so fast?

Written in C language, it is officially said that the QPS per second is 100000 +. Based on memory operation, Redis's bottleneck is related to the server's memory and network bandwidth. Since it can be implemented by single thread, it can be implemented by single thread.


High performance server must be multi-threaded?

Must multithreading be more efficient than single thread?

Core: Redis puts all data in memory, so using a single thread to operate is the most efficient, because multithreading will produce CPU context switching and time-consuming operations. For the memory system, without context switching, the efficiency is the highest, and multiple reads and writes are on one CPU

1.5 Redis data type

Five data types

Redis is an open source (BSD licensed) in memory data structure storage system, which can be used as database, cache and message middleware. It supports many types of data structures, such as strings, hashes, lists, sets, sorted sets and range queries, bitmaps, hyperlogs and geospatial index radius queries. Redis has built-in replication, Lua scripting, LRU events, transactions and different levels of disk persistence, and provides high availability through redis Sentinel and Cluster.

Usage scenario: value can be not only a string, but also our number

  • Counter
  • Count the quantity of multiple units
  • Number of fans
  • Object cache storage

1.5.1 String

### Common usage> set kye1 yunmx								# Set a key
OK> get key1									# Get the value of a key
(nil)> keys *										# View all keys in the current library and output the key name
1) "kye1"> EXISTS kye1									# Determine whether the key exists
(integer) 1> EXISTS key1
(integer) 0> APPEND kye1 "helloworld"					# Add content to the key. If the key does not exist, set a key
(integer) 15> get kye1
"yunmxhelloworld"> STRLEN kye1 								# Get the length of a key
(integer) 15> 

### Self growth usage> set count 0
OK> get count
"0"> incr count									# Self increment 1
(integer) 1> get count
"1"> incr count									
(integer) 2> get count
"2"> decr count									# Self subtraction 1
(integer) 1> get count
"1"> decr count
(integer) 0> get count
"0"> incrby count 5								# Self increasing; Step 5
(integer) 5> get count
"5"> decrby count 5								# Self subtraction; Step 5
(integer) 0> get count

### Get string range> set key1 yunmxhejie							# Set a key
OK> getrange key1 0 4							# Gets a string in the range from 0 to 4
"yunmx"> getrange key1 0 -1							# Get all strings

### replace> set key1 yunmx123456						# Set a key
OK> SETRANGE key1 0 hejie						# Replace from the position with subscript 0
(integer) 11> get key1

### Set expiration time> setex key1 10 yunmx							# Set a key with an expiration time of 10 seconds
OK> ttl key1									# View the expiration time of the key
(integer) 7> ttl key1									# Returning - 2 means it has expired
(integer) -2> keys *
(empty array)> 

### No settings exist (often used in distributed locks)> setnx key2 redis							# setnx: create if there is no key value
(integer) 1> setnx key2 mysql							# setnx: creation fails if there is a key value
(integer) 0> keys *
1) "key2"> 

### Batch setting and acquisition> mset key1 1 key2 2 key3 3 key4 4			# Set multiple values at the same time
OK> keys *
1) "key1"
2) "key4"
3) "key3"
4) "key2"> mget key1 key2 key3 key4					# Get multiple values at the same time
1) "1"
2) "2"
3) "3"
4) "4"> msetnx key5 5 key4 4						# Atomic operations: either succeed or fail together
(integer) 0> get key5

### object> set user:1 {name:yunmx,age:25}				# Common setting object: key plus json string
OK> get user:1
"{name:yunmx,age:25}"> mset user:1:name yunmx user:1:age 25		# Skillfully designed objects; Such a design is entirely possible
OK> mget user:1:name user:1:age
1) "yunmx"
2) "25"

### Combination command> getset db mysql								# If not, set
(nil)> get db
"mysql"> getset db MongoDB							# If it exists, return the current value and set a new value
"mysql"> get db

1.5.2 List

Basic data types

Basic data types, lists

In redis, list can complete stack, queue and blocking queue!


  • It is actually a linked list. Before node after, left right can insert values
  • If the key does not exist, create a new linked list
  • If the key exists, add content
  • If all values are removed, the empty linked list indicates that it does not exist
  • Insert or change values on both sides for the highest efficiency; Modifying intermediate elements is relatively inefficient

Message queuing, message queuing (lpush Rpop) stack (lpush lpop)

### Basic command> LPUSH key1 1								# Add one or more values to the header
(integer) 1> LPUSH key1 2
(integer) 2> LPUSH key1 3
(integer) 3> LRANGE key1 0 -1							
1) "3"
2) "2"
3) "1"> LRANGE key1 0 1
1) "3"
2) "2"> rpush key1 5								# Add a value to the tail
(integer) 4> LRANGE key1 0 -1
1) "3"
2) "2"
3) "1"
4) "5"

### Removing Elements> LRANGE key1 0 -1
1) "5"
2) "4"
3) "3"
4) "2"
5) "1"> LPOP key1									# Remove the first element of the list
"5"> rPOP key1									# Removes the last element of the list
"1"> LRANGE key1 0 -1
1) "4"
2) "3"
3) "2"

### Get element, list length> LRANGE key1 0 -1
1) "4"
2) "3"
3) "2"> LINDEX key1 0								# Gets the first element in the list
"4"> LINDEX key1 1
"3"> LINDEX key1 2
"2"> LINDEX key1 3
(nil)> LLEN key1									# Get list length
(integer) 3

### Removes the specified value> LRANGE key1 0 -1
1) "2"
2) "4"
3) "4"
4) "3"
5) "2"> LREM key1 1 2								# Remove a 2
(integer) 1> LRANGE key1 0 -1
1) "4"
2) "4"
3) "3"
4) "2"> LREM key1 2 4								# 4 removed
(integer) 2> LRANGE key1 0 -1
1) "3"
2) "2"

### intercept> LRANGE key1 0 -1
1) "15"
2) "14"
3) "13"
4) "12"
5) "11"
6) "10"> LTRIM key1 0 3								# The list has been changed by intercepting the specified length by subscript
OK> LRANGE key1 0 -1
1) "15"
2) "14"
3) "13"
4) "12"

### Combined command: rpoppush> rpush list1 1 2 3 4 5 
(integer) 5> LRANGE list1 0 -1
1) "1"
2) "2"
3) "3"
4) "4"
5) "5"> rpoplpush list1 list2					# Removes the last element from the list and adds it to the new list
"5"> LRANGE list1 0 -1
1) "1"
2) "2"
3) "3"
4) "4"> LRANGE list2 0 -1
1) "5"> 

### Element update> EXISTS list1						# Determine whether the list exists
(integer) 0> lset list1 0 yunmx					# There is no error report
(error) ERR no such key> lpush lista1 1 2 3 4
(integer) 4> LRANGE list1 0 0
(empty array)> LRANGE list1 0 -1
(empty array)> LRANGE lista1 0 0
1) "4"> lset lista1 0 yunmx						# There is a value to modify the current subscript
OK> LRANGE lista1 0 0
1) "yunmx"

### Element insertion> LRANGE lista1 0 -1
1) "yunmx"
2) "3"
3) "2"
4) "1"> LINSERT lista1 before yunmx hejie		# Insert in front of the current element
(integer) 5> LINSERT lista1 after yunmx hejie1		# Insert after current element
(integer) 6> LRANGE lista1 0 -1
1) "hejie"
2) "yunmx"
3) "hejie1"
4) "3"
5) "2"
6) "1"

1.5.3 Set

The values in set cannot be repeated d and are out of order

### Basic usage> sadd set1 yunmx													# Save values into the set
(integer) 1> sadd set1 hejie1
(integer) 1> sadd set1 hejie01
(integer) 1> sadd set1 hejie02
(integer) 1> sadd set1 hejie02
(integer) 0> SMEMBERS set1													# Query all elements in the collection
1) "yunmx"
2) "hejie02"
3) "hejie01"
4) "hejie1"> SISMEMBER set1 hejie											# Determine whether the element exists
(integer) 0> SISMEMBER set1 hejie5
(integer) 0> SISMEMBER set1 hejie1											# Presence return 1
(integer) 1

### Gets the number of elements in the collection> SCARD set1
(integer) 4> sadd set1 hejie09
(integer) 1> SCARD set1
(integer) 5

### Removes the specified element from the set collection> SREM set1 yunmx													# Remove: yunmx from set set
(integer) 1> SCARD set1							
(integer) 4> SMEMBERS set1												
1) "hejie01"
2) "hejie09"
3) "hejie1"
4) "hejie02"

### Random usage (unordered and not repeated)> SRANDMEMBER set1												# Select an element at random
"hejie02"> SRANDMEMBER set1
"hejie09"> SRANDMEMBER set1
"hejie02"> SRANDMEMBER set1
"hejie09"> SRANDMEMBER set1
"hejie01"> SRANDMEMBER set1 2												# Randomly select the specified number of elements
1) "hejie1"
2) "hejie01"> SRANDMEMBER set1 2
1) "hejie09"
2) "hejie02"> SRANDMEMBER set1 2
1) "hejie1"
2) "hejie01"

### Randomly remove elements> SMEMBERS set1
1) "hejie01"
2) "hejie09"
3) "hejie1"
4) "hejie02"> spop set1														# Remove an element at random
"hejie02"> spop set1
"hejie1"> SMEMBERS set1
1) "hejie01"
2) "hejie09"

### Moves the specified value to another collection> SMEMBERS set1
1) "1"
2) "2"
3) "3"> SMEMBERS set2
1) "er"
2) "san"
3) "yi"> smove set1 set2 1												# Move the element in set1 to set2. The value to be moved is 1 in set1
(integer) 1> SMEMBERS set1
1) "2"
2) "3"> SMEMBERS set2
1) "1"
2) "er"
3) "san"
4) "yi"

### Common concern: station B, microblog (combined)
# Digital set: difference set, intersection set, union set> sadd set1 1 2 3 4 5 6
(integer) 6> sadd set2 4 5 6 7 8 9
(integer) 6>> SDIFF set1 set2												# Difference set: the element in set1 does not exist in set2
1) "1"
2) "2"
3) "3"> SINTER set1 set2											# intersection
1) "4"
2) "5"
3) "6"> SDIFF set2 set1
1) "7"
2) "8"
3) "9"> SUNION set1 set2										  	# Union
1) "1"
2) "2"
3) "3"
4) "4"
5) "5"
6) "6"
7) "7"
8) "8"
9) "9"

1.5.4 Hash

The essence of Map set and key Map set is not much different from that of string. It is still a simple key vlaue


  • User information and so on
  • Frequently changing information
  • It is more suitable for storing objects, and string is more suitable for storing strings
### Basic usage> hset hash1 name yunmx										# Set a key vlaue 
(integer) 1> hget hash1 name												# Get a field value
"yunmx"> hmset hash1 name hejie age 28								# Set multiple
OK> hgetall hash1												# Get all elements
1) "name"
2) "hejie"
3) "age"
4) "28"> hget hash1 age												# Get a field value

### Delete a value> HDEL hash1 name												# Delete the specified key field
(integer) 1> hgetall hash1
1) "age"
2) "28"

### Get all the lengths of the hash (key value pairs)> HLEN hash1													# Gets the number of key value pairs in the hash
(integer) 1> hset hash1 age 18
(integer) 0> hset hash1 age1 19
(integer) 1> hset hash1 name hejie
(integer) 1> HLEN hash1
(integer) 3

### Determine whether the field in the hash exists> HGETALL hash1
1) "age"
2) "18"
3) "age1"
4) "19"
5) "name"
6) "hejie"> HEXISTS hash1 age											# Determine whether the age field exists in the hash
(integer) 1> HEXISTS hash1 yunmx
(integer) 0

### Get only all fields or values> hkeys hash1
1) "age"
2) "age1"
3) "name"> HVALS hash1
1) "18"
2) "19"
3) "hejie"

### Specified self increment> HINCRBY hash1 age 1											# Self increment 1
(integer) 19> HINCRBY hash1 age -1
(integer) 18> hsetnx hash1 age 1											# If present, do not set
(integer) 0> hsetnx hash1 age2 1											# If it does not exist, add a new key value pair
(integer) 1

1.5.5 Zset (ordered set)

On the basis of set, a value is added and a counting digit is added

Scenario idea:

  • set can do everything it can. It can store the class grade sheet and payroll sorting
  • Ordinary message 1 and important message 2 are judged with weight
  • Application implementation of leaderboard: TOP N
### Basic usage> zadd myset 1 one											# Add a value
(integer) 1> zadd myset 2 two
(integer) 1> zadd myset 3 three
(integer) 1> zadd myset 4 four 5 five									# Add multiple values
(integer) 2> ZRANGE myset 0 -1											# View all values
1) "one"
2) "two"
3) "three"
4) "four"
5) "five"

### How to implement sorting
# Usage: from minimum to maximum
# ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count]> zadd salary 12000 yunmx										# Add 4 users
(integer) 1> zadd salary 11000 hejie
(integer) 1> zadd salary 10000 hejie01
(integer) 1> zadd salary 9000 yunmx01
(integer) 1> ZRANGEBYSCORE salary -inf +inf								# -inf +inf negative infinity to positive infinity from small to large
1) "yunmx01"
2) "hejie01"
3) "hejie"
4) "yunmx"> ZRANGEBYSCORE salary -inf +inf withscores					# WithCores means with some other parameter descriptions
1) "yunmx01"
2) "9000"
3) "hejie01"
4) "10000"
5) "hejie"
6) "11000"
7) "yunmx"
8) "12000"

### Removing Elements> ZRANGE salary 0 -1											# View all elements
1) "yunmx01"
2) "hejie01"
3) "hejie"
4) "yunmx"> zrem salary yunmx01											# Remove specific elements
(integer) 1> ZRANGE salary 0 -1
1) "hejie01"
2) "hejie"
3) "yunmx"

### Gets the number of in the ordered collection> ZRANGE salary 0 -1
1) "hejie01"
2) "hejie"
3) "yunmx"> ZCARD salary
(integer) 3

### Calculated by interval> ZRANGE salary 0 -1
1) "hejie01"
2) "hejie"
3) "yunmx"> zcount salary 1000 9000										# View quantities between 1000 and 9000
(integer) 0> zcount salary 1000 10000									# View the quantity value between 1000 and 1-000
(integer) 1

1.5.6 geospatial (geospatial)

City longitude and latitude query: http://www.jsons.cn/lngcode/

  • Positioning of circle of friends
  • people nearby
  • Taxi distance calculation
  • ...

The underlying principle of GEO implementation is actually Zset! You can use Zset to operate GEO!

There are only six commands

### Add location
# Geodd key longitude latitude member [longitude latitude member...] Multiple can be added
# Parameter: key value (longitude dimension name)> geoadd china:city 104.065735 30.659462 chengdu
(integer) 1> geoadd china:city 106.504962 29.533155 chongqing
(integer) 1

### Get the latitude and longitude of the current positioning> GEOPOS china:city chengdu									# Get the longitude and latitude of Chengdu
1) 1) "104.06573742628097534"
   2) "30.65946118872339099"> GEOPOS china:city chongqing									# Get the longitude and latitude of Chongqing
1) 1) "106.50495976209640503"
   2) "29.53315530684997015"

### Calculate the distance between two places
#Unit: m/km/mi/ft> geodist china:city chengdu chongqing m						# Calculate the straight-line distance from Chengdu to Chongqing, in m 
"266056.2971"> geodist china:city chengdu chongqing km						# Calculate the straight-line distance from Chengdu to Chongqing, in km

### Take the given longitude and latitude as the center, and return all position elements whose distance from the center does not exceed the given maximum distance among the position elements contained in the key.
# GEORADIUS:> GEORADIUS china:city 103 30 1 km							# Take 103 30 as the center and 1km as the radius
(empty array)> GEORADIUS china:city 103 30 2 km
(empty array)> GEORADIUS china:city 103 30 20 km
(empty array)> GEORADIUS china:city 103 30 200 km
1) "chengdu-qingyangqu"
2) "chengdu"
3) "chengdu-jinjiangqu"> GEORADIUS china:city 103 30 20000 km
1) "chengdu-qingyangqu"
2) "chongqing"
3) "chengdu"
4) "chengdu-jinjiangqu"> GEORADIUS china:city 103 30 20000 km withcoord				# Longitude and latitude are displayed	
1) 1) "chengdu-qingyangqu"
   2) 1) "104.06151026487350464"
      2) "30.67387107851423167"
2) 1) "chongqing"
   2) 1) "106.50495976209640503"
      2) "29.53315530684997015"
3) 1) "chengdu"
   2) 1) "104.06573742628097534"
      2) "30.65946118872339099"
4) 1) "chengdu-jinjiangqu"
   2) 1) "104.06573742628097534"
      2) "30.65946118872339099"

### Find the element within the specified range. The center point is determined by the given location element
# GEORADIUSBYMEMBER> GEORADIUSBYMEMBER china:city chengdu 10 km					# City coordinates of 10km
1) "chengdu"
2) "chengdu-jinjiangqu"
3) "chengdu-qingyangqu"> GEORADIUSBYMEMBER china:city chengdu 100 km
1) "chengdu"
2) "chengdu-jinjiangqu"
3) "chengdu-qingyangqu"> GEORADIUSBYMEMBER china:city chengdu 10 km
1) "chengdu"
2) "chengdu-jinjiangqu"
3) "chengdu-qingyangqu"> GEORADIUSBYMEMBER china:city chengdu 1000 km
1) "chengdu-qingyangqu"
2) "chongqing"
3) "chengdu"
4) "chengdu-jinjiangqu"> GEORADIUSBYMEMBER china:city chengdu 1000 km withcoord		# With coordinate information
1) 1) "chengdu-qingyangqu"
   2) 1) "104.06151026487350464"
      2) "30.67387107851423167"
2) 1) "chongqing"
   2) 1) "106.50495976209640503"
      2) "29.53315530684997015"
3) 1) "chengdu"
   2) 1) "104.06573742628097534"
      2) "30.65946118872339099"
4) 1) "chengdu-jinjiangqu"
   2) 1) "104.06573742628097534"
      2) "30.65946118872339099"

### Redis GEOHASH command - returns the Geohash representation of one or more location elements, which is not commonly used

1.5.7 hyperlog (cardinality Statistics)

Cardinality: the number of elements that are not repeated. The error is acceptable

A(1 2 3 45 5 78)

B(3 4 6 7 8 9 0 9 78 )

Algorithm of cardinality statistics

  • UV of web page (a person visits a website, but still counts as one)

    The traditional way: set saves the user's ID, and then you can count the number of elements in set as the standard judgment. If you save a large number of user IDs, it will be more troublesome. Our purpose is to count rather than save the ID

  • The occupied memory is fixed. 2 ^ 64 different element technologies only need to waste 12KB of memory

0.81% error rate! UV statistics can be ignored> PFADD key1 a 1 2 3 4 b c 6 7 9								# Set up a HyperLOGLOG		
(integer) 1> PFCOUNT key1												# Check the number of non duplicates			
(integer) 10> PFADD key2 a a  1 2 3 4 b c 6 7 9 10 10 101 10 10 8 6
(integer) 1> PFCOUNT key2
(integer) 13> PFMERGE key3 key1 key2										# Number of elements merging two key s
OK> PFCOUNT key3												# View the number of non repeating elements
(integer) 13

1.5.8 Bitmap

Bit store 0 and 1

This data type can be used for both statuses

  • Active, inactive
  • Number of infected and uninfected persons> SETBIT key 0 1												# Use Bitmaps to record the clock out of a week
(integer) 0> SETBIT key 1 0
(integer) 0> SETBIT key 2 0
(integer) 0> SETBIT key 3 1
(integer) 0
## Check whether there is a clock out on a certain day> getbit key 3
(integer) 1> getbit key 2
(integer) 0

### Statistical operation
# Count open days> BITCOUNT key
(integer) 2

1.6 Redis transaction

A transaction can execute multiple commands at one time with the following two important guarantees:

  • A transaction is a separate isolation operation: all commands in the transaction are serialized and executed sequentially. During the execution of the transaction, it will not be interrupted by the command request sent by other clients
  • A transaction is an atomic operation: all commands in the transaction are either executed or not executed at all

1.6.1 basic transaction operations

Redis transaction essence: a collection of commands! All commands in a transaction will be serialized and executed in sequence during transaction execution

Redis single command guarantees atomicity, but transactions do not guarantee atomicity!

  • disposable
  • Order
  • exclusiveness
  • There is no concept of isolation level, no unreal reading, dirty reading
---Join the team---

Three stages:

  1. Open transaction (MULTI)
  2. commands
  3. Execute transaction (EXEC)
### Normal execution> MULTI								# Open transaction
OK> set key1 1							# Order to join the team
QUEUED> set key2 2
QUEUED> set key3 3
QUEUED> get key1
QUEUED> get key2
QUEUED> exec								# Transaction execution, in order
1) OK
2) OK
3) OK
4) "1"
5) "2"

### Abandon transaction> MULTI								# Open transaction
OK> set key1 1
QUEUED> set key2 2
QUEUED> set key3 3
QUEUED> DISCARD								# Cancel transaction
OK> exec
(error) ERR EXEC without MULTI> get key1							# Failed to get the relevant value after abandoning the transaction

### Transaction error 1: if the command is wrong, all commands in the transaction will not be executed!> MULTI
OK> set key1 1
QUEUED> set key2 2
QUEUED> getset key3											# Wrong command, direct error reporting
(error) ERR wrong number of arguments for 'getset' command> set key3 3
QUEUED> exec
(error) EXECABORT Transaction discarded because of previous errors.	# Error in executing transaction> get key1											# As a result, all commands will not be executed

### Transaction error 2: operation exception. If there is syntax in the transaction queue, other commands can be executed normally when executing the command, and the error command throws an exception> MULTI
OK> set key1 "yunmx"
QUEUED> incr key1
QUEUED> set key2 2
QUEUED> set key3 3
QUEUED> get key3
QUEUED> exec
1) OK
2) (error) ERR value is not an integer or out of range				# The code structure is OK, and an exception is thrown
3) OK
4) OK
5) "3"> get key2

1.6.2 optimistic lock

Monitoring: watch

Pessimistic lock: very pessimistic; I think there will be problems at any time. No matter what I do, I will lock it, which will affect the performance

Optimistic lock: very optimistic. The task will not have problems at any time, so it will not be locked. When updating the data, judge whether anyone has modified the data during this period. The version field used by mysql!

### Redis test monitoring> set mymoney 1000
OK> set myout 0
OK> WATCH mymoney										# monitor
OK> DECRby mymoney 20
QUEUED> exec												# The transaction ended normally, and there was no change in the data period. It was executed normally
1) (integer) 980
2) (integer) 20>

### Use the watch lock to test the multithreading. After modifying the value, the transaction submission fails
# The first process starts the transaction>  watch mymoney
OK> DECRBY mymoney 10
QUEUED> exec												# Execute the second transaction to modify the data before committing
(nil)																# After the transaction is committed, the modification fails,>
# The second process modifies our values> get mymoney
"980"> set mymoney 20
OK> get mymoney
"20"> set mymoney 1000
OK> get mymoney

### You can use unwatch to unlock, use watch again to get the latest value, and then execute the transaction

1.7 detailed explanation of redis configuration file

1.7.1 redis.conf

When starting, it is started through the configuration file

The configuration file is case insensitive

Include other profiles


bind		# Bound ip
protected-mode yes	# Protection mode
port 6379			# port settings

GENERAL configuration

daemonize yes		# It has been run in daemon mode. The default is no. We need to start it ourselves
pidfile /var/run/redis_6379.pid	# If we run in the background mode, we need to specify a pid file
# Log (4 levels)
# Specify the server verbosity level.
# This can be one of:
# debug (a lot of information, useful for development/testing)
# verbose (many rarely useful info, but not a mess like the debug level)
# notice (moderately verbose, what you want in production probably)
# warning (only very important / critical messages are logged)
loglevel notice	#
logfile ""	#File location name of the log
databases 16	# The default number of databases is 16
always-show-logo yes	# Whether the logo is always displayed. It can not be displayed. It is displayed by default


# rule
save 900 1		# If at least one key is modified within 900 seconds, we will perform the persistence operation
save 300 10		
save 60 10000
# We can set our own rules
stop-writes-on-bgsave-error yes	# Whether to continue working when persistence fails. The default is yes
rdbcompression yes				# Whether to compress rdb files, which will consume CPU resources
rdbchecksum yes					# Check the rdb file or not. If there is an error, it will be repaired
dir ./							# Directory where rdb files are saved


Persistence: the number of operations performed within the specified time will be persisted to the file rdb .aof, if there is no persistence, the data will be powered off and lost

Master slave REPLICATION


# requirepass foobared no password by default
# You can set the password by modifying the configuration file
# For example:
requirepass 123456

# verification> ping
(error) NOAUTH Authentication required.> auth 123456
OK> ping
# You can also set the password through the command> config get requirepass
1) "requirepass"
2) "123456"> config set requirepass "110"
OK> ping
(error) NOAUTH Authentication required.> auth 110
OK> ping


# Some limitations of the client
maxclients 10000   # Set the limit on the number of clients that can connect to redis
maxmemory <bytes>	# Maximum memory capacity of redis configuration
maxmemory-policy noeviction	# Processing strategy after the memory reaches the upper limit
	# Remove some expired key s
	# report errors
	# ...
1,volatile-lru: Only for those with expiration time set key conduct LRU((default) 
2,allkeys-lru :  delete lru Algorithmic key   
3,volatile-random: Random deletion is about to expire key   
4,allkeys-random: Random deletion   
5,volatile-ttl :  Delete expiring   
6,noeviction :  Never expire, return error

APPEND ONLY MODE aof configuration

appendonly no   		# aof mode is not enabled by default. rdb persistent mode is used by default
appendfilename "appendonly.aof"			# The name of the persistent file

# appendfsync always			# The value will be written every time it is modified, which is slow
appendfsync everysec			# sync is executed once every second, and 1s of data may be lost!
# appendfsync no				# Out of sync. At this time, the operating system synchronizes data by itself. The speed is the fastest!

1.8 Redis persistence

Redis is an in memory database. If the database state in memory is not saved to the hard disk, the data state in the server will disappear once the server process exits. Therefore, redis provides the persistence function

1.8.1 RDB(Redis DataBase)


Within the specified time interval, the collective Snapshot of data in memory is written to disk, that is, Snapshot snapshot Snapshot. When it is restored, the Snapshot file is directly read into memory.


Create (fork) a sub process separately for persistence. The data will be written to a temporary file. After the persistence process is completed, this temporary file will be used to replace the last persistent file. During the whole process, the main process does not perform any IO operations. This ensures extremely high performance. If large-scale data recovery is required and the integrity of data recovery is not very sensitive, RDB method is more effective than AOF method. The disadvantage of RDB is that the data may be lost after one persistence. Our default is RDB. Generally, we don't need to modify this configuration!

The file saved by RDB is dump rdb

# Trigger mechanism
1.save When the rules are met
2.Yes FLUSHALL command
3.sign out redis Can also trigger

How to recover rdb files

  1. You only need to put the rdb file in the redis startup directory, and dump will be checked automatically during startup rdb restores the data in it
# View storage location> config get dir
1) "dir"
2) "/usr/local/bin"

Advantages and disadvantages


  1. Suitable for large-scale data recovery
  2. If you don't have high requirements for data integrity


  1. A certain time interval is required for operation! If redis goes down unexpectedly, the last modified data will be gone
  2. The fork process takes up a certain amount of memory space

1.8.2 AOF(Append Only File)

Record all our commands, history, and execute all the files when recovering


Record each operation in the form of a log, and record all instructions executed by redis (the read operation will not be recorded). Only files can be added, but files cannot be rewritten. After redis starts, the file will be read to rebuild the data. In other words, if redis restarts, the write instructions will be executed from front to back according to the contents of the daily record file to complete the data recovery

The file saved by aof is appendonly aof file

# After destroying the aof file, try to connect
[root@yunmx bin]# redis-cli
Could not connect to Redis at Connection refused
# Use the detection file to automatically repair the AOF file: redis check AOF
[root@yunmx bin]# redis-check-aof --fix appendonly.aof
0x              62: Expected \r\n, got: 6173
AOF analyzed: size=122, ok_up_to=23, diff=99
This will shrink the AOF from 122 bytes, with 99 bytes, to 23 bytes
Continue? [y/N]: y
Successfully truncated AOF

Advantages and disadvantages


  1. Each modification is synchronized to improve file integrity,
  2. Sync once per second, which will lose one second of data
  3. Never synchronized, the most efficient!


  1. Compared with data files, aof is much larger than rdb, and the repair speed is slower than rdb
  2. aof runs slower than rdb and performs IO operations


  • RBD
    RDB persistence can snapshot and store your data at a specified time interval.

  • AOF
    Aof persistence records every write operation to the server. When the server restarts, these commands will be re executed to recover the original data. AOF command saves each write operation to the end of the file with redis protocol. Redis can also rewrite AOF files in the background, so that the volume of AOF files will not be too large.

  • Cache only
    If you only want your data to exist when the server is running, you can also not use any persistence method.

  • Enable two persistence methods at the same time

    • When AOF is restarted, saving the original dataset takes precedence over saving the original dataset
    • RDB data is not real-time. When using both, the server will only find AOF files when restarting. Do you want to use AOF only? The author suggests not, because RDB is more suitable for backing up the database (AOF is changing and hard to back up), fast restart, and there will be no potential bug s in AOF, so keep it as a means in case.
  • Performance recommendations

    • Because RDB files are only used for backup purposes, it is recommended to only persist RDB files on Slave, and only backup once every 15 minutes is enough. Only save 900 1 is retained.
    • If Enalbe AOF, the advantage is that in the worst case, only less than two seconds of data will be lost. The startup script is relatively simple, and only load your own AOF file. The first cost is that it brings continuous IO. The second is that AOF rewrite s. Finally, the new data generated in the rewriting process is written to a new file, and the blocking caused by it is almost inevitable. As long as the hard disk is allowed, the frequency of AOF rewriting should be minimized. The default value of 64M is too small and can be set to more than 5G. When the default size exceeds 100% of the original size, the override can be changed to an appropriate value.
    • If AOF is not enabled, high availability can be achieved only by master slave replication. It can save a lot of IO and reduce the system fluctuation caused by rewriting. The price is that if the Master/Slave is dropped at the same time, more than ten minutes of data will be lost. The startup script also needs to compare the RDB files in the two Master/Slave and load the newer one. Sina Weibo chose this architecture.

1.9 Redis publish and subscribe

1.9.1 Redis publish and subscribe

Subscribe and publish message diagram:

Redis publish / subscribe (pub/sub) is a message communication mode: the sender (pub) sends messages and the subscriber (sub) receives messages

Redis client can subscribe to any number of channels

The first is the message sender, the second is the channel, and the third is the message subscriber

The following figure shows the relationship between channel 1 and the three clients subscribing to this channel - client2, client5 and client1:

When a new message is sent to channel 1 through the PUBLISH command, the message will be sent to the three clients subscribing to it:

Redis publish subscribe command
The following table lists the common commands for publishing and subscribing to redis:

  • PSUBSCRIBE pattern [pattern ...]
    Subscribe to one or more channels that match a given pattern.
  • PUBSUB subcommand [argument [argument ...]]
    View subscription and publishing system status.
  • PUBLISH channel message
    Send information to the specified channel.
  • PUNSUBSCRIBE [pattern [pattern ...]]
    Unsubscribe from all channels in a given mode.
  • SUBSCRIBE channel [channel ...]
    Subscribe to information for a given channel or channels.
  • UNSUBSCRIBE [channel [channel ...]]
    Unsubscribe from a given channel.
# test
# Subscribe to a channel> SUBSCRIBE yunmx										# Subscribe to a channel to yunmx
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "yunmx"
3) (integer) 1

# New window sends a message: Publisher
[root@yunmx ~]# redis-cli> PUBLISH yunmx test									# The publisher publishes a message to a channel with a specified name
(integer) 1

# Received message> SUBSCRIBE yunmx		
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "yunmx"
3) (integer) 1
1) "message"														# Received messages
2) "yunmx"															# Which channel
3) "test"															# Message content

Principle of Redis publishing and subscription

It is implemented in C by analyzing PubSub. In redis source code C file to understand the underlying implementation logic of publish and subscribe mechanism and deepen the understanding of redis. (I don't know C, and I'm not a developer. How to fix it)

Publish and subscribe functions are realized through PUBLISH/SUBSCRIBE/PSUBSCRIBE and other commands

  • SUBSCRIBE: after subscribing to channels through this command, a dictionary is maintained in redis server. The key of the dictionary is each channel, and the value of the dictionary is a linked list, which saves all clients subscribing to this channel----- Is to add the client to the subscription list of a given channel
  • PUBLISH: Send a message to subscribers. Redis server will use the given channel as the key, query and record all client connections subscribing to this channel in the channel dictionary maintained by it, traverse the whole linked list and send the message to all subscribers
  • PUB and SUB: publish and subscribe. In redis, you can set a key value for message publishing and message subscription. When a key value is published, all clients subscribing to it will receive corresponding messages. This function is obviously used as a real-time message system, such as ordinary instant chat, group chat and other functions

1.9.2 usage scenarios

  • Real time message system
  • Real time chat ---- the channel can be used as a chat room and the information can be echoed back to everyone
  • Subscription, official account subscription, etc.
  • In a slightly complex scenario, we will use message oriented middleware (MQ)

2.0 Redis master-slave replication

2.0.1 concept

Master-slave replication

It refers to copying the data of one Redis server to other Redis servers. The former is called master and the latter is called slave; Data replication is unidirectional and can only be from master node to slave node. Master mainly writes, slave mainly reads.

By default, each Redis server is the master node; And a master node can have multiple slave nodes (or no slave nodes), but a slave node can only have one master node.

Role of master-slave replication

  1. Data redundancy: master-slave replication realizes the hot backup of data, which is a way of data redundancy other than persistence.
  2. Fault recovery: when the master node has problems, the slave node can provide services to achieve rapid fault recovery; In fact, it is a kind of redundancy of services.
  3. Load balancing: on the basis of master-slave replication, combined with read-write separation, the master node can provide write services, and the slave node can provide read services (that is, the application connects to the master node when writing Redis data, and the application connects to the slave node when reading Redis data), sharing the server load; Especially in the scenario of less writing and more reading, the concurrency of Redis server can be greatly improved by sharing the reading load among multiple slave nodes.
  4. High availability cornerstone: in addition to the above functions, master-slave replication is also the basis for sentinels and clusters to implement. Therefore, master-slave replication is the basis for Redis high availability.

Generally speaking, to apply redis to production projects, it is absolutely impossible to use only one redis

  • Structurally, a single redis server will have a single point of failure, and a single machine processing all requests will lead to excessive load
  • In terms of capacity, the memory capacity of a single machine is limited. Generally speaking, the maximum memory used by a single redis should not exceed 20G

2.0.2 configure master-slave

Master-slave copy, read-write separation! 80% of the cases are read operations! Ease the pressure on the server and often use it in the architecture! One master and two slaves!

Test scenario

Environment configuration

Configure only slave libraries, not master libraries!

# Start a redis to view the information> info replication
# Replication
role:master														# role
connected_slaves:0												# Number of slaves

Copy a configuration file, modify the port number, the file name of log saving, rdb file name and pid file, and start two redis

[root@yunmx bin]# redis-server redis-conf/redis.conf1			# 6380
6038:C 12 Dec 2021 13:06:38.166 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
6038:C 12 Dec 2021 13:06:38.166 # Redis version=6.0.6, bits=64, commit=00000000, modified=0, pid=6038, just started
6038:C 12 Dec 2021 13:06:38.166 # Configuration loaded
[root@yunmx bin]# redis-server redis-conf/redis.conf2			# 6381
6045:C 12 Dec 2021 13:06:39.640 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
6045:C 12 Dec 2021 13:06:39.641 # Redis version=6.0.6, bits=64, commit=00000000, modified=0, pid=6045, just started
6045:C 12 Dec 2021 13:06:39.641 # Configuration loaded

Before configuring master-slave, all three nodes are master nodes

Recognize the boss

One master (79) and two slaves (80,81)

# 6380 configuration> SLAVEOF 6379							# redis, who recognizes the 6379 port service of this machine, is the boss
OK> info replication
# Replication
role:slave														# Becomes a slave node
master_host:											# Master node information
# View host information> info replication
# Replication
slave0:ip=,port=6380,state=online,offset=168,lag=0			# Slave information

The real master-slave configuration should be configured in the configuration file. In this case, it is permanent. The above commands are used, but only temporary

replicaof <masterip> <masterport>   # IP address and port number of the host
masterauth <master-password>		# If a password is set, configure the password

2.0.3 characteristic verification

The master can set values, but the slave cannot write. All information and data in the master will be saved by the slave

# Set a key for the host> set key1 yunmx
# There will also be in the slave. The slave cannot set the key> keys *
1) "key1"> get key1
"yunmx"> set key2 yunmx2
(error) READONLY You can't write against a read only replica.>

After the boss goes down, the slave or slave will only show that the host state is abnormal; When the host is restored, the slave can still directly obtain the information written by the host, ensuring a certain high availability

If we use the master-slave configured on the command line, if the slave goes down, the slave will be separated from the master-slave. We need to configure the slave on the command line again. After it becomes a slave, we can get the value of the key

# Stop the 6379 host service and check the status of the slave cluster> info replication
# Replication

# Restore 6379 host services and verify that data will also be synchronized
[root@yunmx ~]# redis-cli -p 6379> set key2 yunmx2													# Restore primary node, set one key
# Read the key set after host recovery from the node> get key2														# It can be read normally

# Stop the slave service, set a key on the host, and verify when the slave is restored> set key3 yunmx3
OK> get key3														# The slave cannot obtain the data of key3
(nil)> get key3

2.0.4 replication principle

After Slave is successfully started and connected to the master, it will send a sync synchronization command

After receiving the command, the master determines the background save process and collects all received commands for modifying the dataset. After the background process is executed, the master will transfer the whole data file to the slave and complete a complete synchronization

Full copy: the slave service will save and load the database file data into memory after receiving it

Incremental replication: Mster continues to transmit all new collected modification commands to slave in turn to complete synchronization

However, as long as you reconnect to the master, a full synchronization will be performed automatically! Our data can certainly be seen in the slave!

2.0.5 layer by layer link

The previous M links to the next S!

Master slave replication can be completed!

If there is no 79, can you choose a boss at this time? You need to configure it manually at this time!

Seek to usurp the throne: slaveof no one make yourself a host! If the boss comes back, it also needs to be configured manually

2.1 Redis sentinel mode

Automatic version selection mode

2.1.1 general

  • The way of master-slave switching technology is: when the master server goes down, one server needs to be manually switched to the main service, which requires manual intervention, which is time-consuming and laborious, and the service can not be used for a period of time. This is not a recommended way, but rather we consider the Sentinel mode. Redis has officially provided Sentinel architecture since 2.8 to solve this problem.

  • It can monitor whether the host fails in the background. If it fails, it will automatically convert from the library to the main library according to the number of votes

  • Sentinel mode is a special mode. Firstly, Redis provides sentinel commands. It is an independent process. As a process, it will run independently. The principle is that the sentinel monitors multiple running Redis instances by sending commands and waiting for a response from the Redis server.

2.1.2 basic structure

The role of sentinel mode:

  • By sending commands, the Redis server can return to the monitoring operation status, including master service and slave service
  • When the sentinel detects that the master is down, it will automatically switch the slave to the master, and then modify the configuration file through other slave servers through publish and subscribe mode to let them switch the host

If a sentinel process monitors the redis server, there may be problems. Therefore, we can use multiple sentinels for monitoring, and each sentinel will also monitor. In this way, a multi sentinel mode is formed

Using sentinel mode, at least 6 processes will be started

Assuming that the main service is down, sentry 1 detects this result first, and the system will not immediately carry out the failover process. Only sentry 1 subjectively thinks that the main service is unavailable. This site is called subjective offline. When the following sentry also detects that the main service is unavailable and the number reaches a certain value, the Sentry will vote once, and the voting result is initiated by a sentry for failover failover. After the handover is successful, each sentinel will switch the monitored from the service to the host through the publish and subscribe mode. This process is called objective offline

2.1.3 scenario test

The architecture we are currently testing is one master and two slave

Configure the profile for sentinel mode

# Create a new profile and edit the following
sentinel monitor myredis 6379 1# Syntax: sentinel monitor the name of the monitored host port 1 (1 represents who will take over as the host after the host goes down)

Activate the sentry

[root@yunmx bin]# redis-sentinel redis-conf/sentinel.conf				# Start a sentry
12680:X 12 Dec 2021 15:13:42.570 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
12680:X 12 Dec 2021 15:13:42.570 # Redis version=6.0.6, bits=64, commit=00000000, modified=0, pid=12680, just started
12680:X 12 Dec 2021 15:13:42.570 # Configuration loaded
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 6.0.6 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in sentinel mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 26379
 |    `-._   `._    /     _.-'    |     PID: 12680
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           http://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'

12680:X 12 Dec 2021 15:13:42.571 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
12680:X 12 Dec 2021 15:13:42.575 # Sentinel ID is ea7ccf0119a4cf2873cf3bb108da5c7af86d36bd
12680:X 12 Dec 2021 15:13:42.575 # +monitor master myredis 6379 quorum 1
12680:X 12 Dec 2021 15:13:42.575 * +slave slave 6380 @ myredis 6379
12680:X 12 Dec 2021 15:13:42.580 * +slave slave 6381 @ myredis 6379

Manual downtime test

# Turn off the host> SHUTDOWN
not connected>
# Some information monitored by the sentry
12680:X 12 Dec 2021 15:16:49.174 # +failover-state-select-slave master myredis 6379
12680:X 12 Dec 2021 15:16:49.241 # +selected-slave slave 6381 @ myredis 6379
12680:X 12 Dec 2021 15:16:49.241 * +failover-state-send-slaveof-noone slave 6381 @ myredis 6379
12680:X 12 Dec 2021 15:16:49.324 * +failover-state-wait-promotion slave 6381 @ myredis 6379
12680:X 12 Dec 2021 15:16:50.181 # +promoted-slave slave 6381 @ myredis 6379
12680:X 12 Dec 2021 15:16:50.181 # +failover-state-reconf-slaves master myredis 6379
12680:X 12 Dec 2021 15:16:50.237 * +slave-reconf-sent slave 6380 @ myredis 6379
12680:X 12 Dec 2021 15:16:51.182 * +slave-reconf-inprog slave 6380 @ myredis 6379
12680:X 12 Dec 2021 15:16:51.182 * +slave-reconf-done slave 6380 @ myredis 6379
12680:X 12 Dec 2021 15:16:51.234 # +failover-end master myredis 6379
12680:X 12 Dec 2021 15:16:51.234 # +switch-master myredis 6379 6381		# The sentry shows that the host is automatically switched to 6381
12680:X 12 Dec 2021 15:16:51.234 * +slave slave 6380 @ myredis 6381
12680:X 12 Dec 2021 15:16:51.234 * +slave slave 6379 @ myredis 6381
12680:X 12 Dec 2021 15:17:21.244 # +sdown slave 6379 @ myredis 6381
# Check the information of 6381 host> info replication
# Replication
role:master								# Become a master

If the master goes down, it will randomly select a server (with its own voting algorithm) from the slave as the host

Host recovery

# Restart the host that was previously down. Observe the reaction of the sentry
12680:X 12 Dec 2021 15:23:41.461 * +convert-to-slave slave 6379 @ myredis 6381
# View information about previous hosts> info replication
# Replication
role:slave								# Become a slave
# It can only be merged into the new host and treated as a slave. This is the rule of sentinel mode!

Advantages and disadvantages

  • Sentinel cluster, based on master-slave replication mode, has many advantages of all master-slave configurations

  • The master-slave can be switched and the fault can be transferred, so the availability of the system will be better

  • It is the upgrade of master-slave mode, from manual to automatic, which is more robust!

  • Online capacity expansion is not good. Once the cluster capacity reaches the online level, online capacity expansion will be very troublesome!

  • The configuration of sentinel mode is actually very troublesome. There are many choices!

Full configuration of sentinel mode

< follow up detailed study >

# Example sentinel.conf
# port <sentinel-port>
# The port that this sentinel instance will run on
# The port on which the sentinel instance runs
port 26379												# The port number on which the sentinel process runs
# sentinel announce-ip <ip>
# sentinel announce-port <port>
# The above two configuration directives are useful in environments where,
# because of NAT, Sentinel is reachable from outside via a non-local address.
# When announce-ip is provided, the Sentinel will claim the specified IP address
# in HELLO messages used to gossip its presence, instead of auto-detecting the
# local address as it usually does.
# Similarly when announce-port is provided and is valid and non-zero, Sentinel
# will announce the specified TCP port.
# The two options don't need to be used together, if only announce-ip is
# provided, the Sentinel will announce the specified IP and the server port
# as specified by the "port" option. If only announce-port is provided, the
# Sentinel will announce the auto-detected local IP and the specified port.
# Example:
# sentinel announce-ip
# dir <working-directory>
# Every long running process should have a well-defined working directory.
# For Redis Sentinel to chdir to /tmp at startup is the simplest thing
# for the process to don't interferer with administrative tasks such as
# unmounting filesystems.
dir /tmp
# sentinel monitor <master-name> <ip> <redis-port> <quorum>
# Master Name: the name of the master redis server
# IP: the IP address of the master redis server
# Redis port: the port number of the master redis server
# Quorum: if the primary instance is judged to be invalid, it needs the consent of at least quorum Sentinel processes. As long as the number of Sentinel processes agreed is not up to standard, the automatic failover will not be executed
# Tells Sentinel to monitor this master, and to consider it in O_DOWN
# (Objectively Down) state only if at least <quorum> sentinels agree.
# Note that whatever is the ODOWN quorum, a Sentinel will require to
# be elected by the majority of the known Sentinels in order to
# start a failover, so no failover can be performed in minority.
# Slaves are auto-discovered, so you don't need to specify slaves in
# any way. Sentinel itself will rewrite this configuration file adding
# the slaves using additional configuration options.
# Also note that the configuration file is rewritten when a
# slave is promoted to master.
# Note: master name should not include special characters or spaces.
# The valid charset is A-z 0-9 and the three characters ".-_".
sentinel monitor mymaster 6379 2
# sentinel auth-pass <master-name> <password>
# Set the password to use to authenticate with the master and slaves.
# Useful if there is a password set in the Redis instances to monitor.
# Note that the master password is also used for slaves, so it is not
# possible to set a different password in masters and slaves instances
# if you want to be able to monitor these instances with Sentinel.
# However you can have Redis instances without the authentication enabled
# mixed with Redis instances requiring the authentication (as long as the
# password set is the same for all the instances requiring the password) as
# the AUTH command will have no effect in Redis instances with authentication
# switched off.
# Example:
# sentinel auth-pass mymaster MySUPER--secret-0123passw0rd
# sentinel down-after-milliseconds <master-name> <milliseconds>
# Number of milliseconds the master (or any attached slave or sentinel) should
# be unreachable (as in, not acceptable reply to PING, continuously, for the
# specified period) in order to consider it in S_DOWN state (Subjectively
# Down).
# Option specifies the number of milliseconds required for Sentinel to think that the Redis instance has expired. When an instance does not return PING after this time, or an error is returned directly, Sentinel marks the instance as subjectively down (SDOWN for short)
# Default is 30 seconds.
sentinel down-after-milliseconds mymaster 30000
# sentinel parallel-syncs <master-name> <numslaves>
# How many slaves we can reconfigure to point to the new slave simultaneously
# during the failover. Use a low number if you use the slaves to serve query
# to avoid that all the slaves will be unreachable at about the same
# time while performing the synchronization with the master.
# This option specifies the maximum number of secondary Redis instances that can synchronize new primary instances during failover. When there are many secondary Redis instances, the smaller the number, the longer the synchronization time, and the longer the time required to complete failover.
sentinel parallel-syncs mymaster 1
# sentinel failover-timeout <master-name> <milliseconds>
# Specifies the failover timeout in milliseconds. It is used in many ways:
# - The time needed to re-start a failover after a previous failover was
# already tried against the same master by a given Sentinel, is two
# times the failover timeout.
# - The time needed for a slave replicating to a wrong master according
# to a Sentinel current configuration, to be forced to replicate
# with the right master, is exactly the failover timeout (counting since
# the moment a Sentinel detected the misconfiguration).
# - The time needed to cancel a failover that is already in progress but
# did not produced any configuration change (SLAVEOF NO ONE yet not
# acknowledged by the promoted slave).
# - The maximum time a failover in progress waits for all the slaves to be
# reconfigured as slaves of the new master. However even after this time
# the slaves will be reconfigured by the Sentinels anyway, but not with
# the exact parallel-syncs progression as specified.
# If the failover operation cannot be completed within this time (ms), the failover is considered to have failed
# Default is 3 minutes.
sentinel failover-timeout mymaster 180000
# sentinel notification-script and sentinel reconfig-script are used in order
# to configure scripts that are called to notify the system administrator
# or to reconfigure clients after a failover. The scripts are executed
# with the following rules for error handling:
# If script exits with "1" the execution is retried later (up to a maximum
# number of times currently set to 10).
# If script exits with "2" (or an higher value) the script execution is
# not retried.
# If script terminates because it receives a signal the behavior is the same
# as exit code 1.
# A script has a maximum running time of 60 seconds. After this limit is
# reached the script is terminated with a SIGKILL and the execution retried.
# sentinel notification-script <master-name> <script-path>
# Call the specified notification script for any sentinel event that is
# generated in the WARNING level (for instance -sdown, -odown, and so forth).
# This script should notify the system administrator via email, SMS, or any
# other messaging system, that there is something wrong with the monitored
# Redis systems.
# The script is called with just two arguments: the first is the event type
# and the second the event description.
# The script must exist and be executable in order for sentinel to start if
# this option is provided.
# Specify the alarm script to be called when sentinel detects the instance exception pointed to by the redis instance of the monitoring. This configuration item is optional, but very common.
# Example:
# sentinel notification-script mymaster /var/redis/notify.sh
# sentinel client-reconfig-script <master-name> <script-path>
# When the master changed because of a failover a script can be called in
# order to perform application-specific tasks to notify the clients that the
# configuration has changed and the master is at a different address.
# The following arguments are passed to the script:
# <master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port>
# <state> is currently always "failover"
# <role> is either "leader" or "observer"
# The arguments from-ip, from-port, to-ip, to-port are used to communicate
# the old address of the master and the new address of the elected slave
# (now a master).
# This script should be resistant to multiple invocations.
# Example:
# sentinel client-reconfig-script mymaster /var/redis/reconfig.sh

2.2 Redis cache penetration and avalanche

High availability of services!

The use of Redis cache has greatly improved the performance and efficiency of applications, especially in data query, but it will also bring some problems. Among them, the most crucial is the consistency of data. Strictly speaking, this problem has no solution. If the requirements for data consistency are high, cache cannot be used. There are also some typical problems: cache penetration, cache avalanche and cache breakdown. At present, the industry also has more popular solutions.

2.2.1 cache penetration

Can't find the cause


The user wants to query a data and finds that the redis memory database is missing, that is, the cache misses, so he queries the persistence layer database. No, so this query failed. When there are many users, the cache fails to hit, so they all request the persistence layer database. This will put a lot of pressure on the persistence layer database, which is equivalent to cache penetration.

Insert picture description here


  • Bloom filter

It is a data structure that stores all possible query parameters in the form of hash. It is verified at the control layer first and discarded if it does not meet the requirements, so as to avoid the query pressure on the underlying storage system

  • Cache empty objects

When the storage layer does not hit, even the returned empty object will be stored, and an expiration time will be set. After accessing this data, it will be obtained from the cache to protect the back-end data source

Two problems with caching empty objects:

  1. If null values can be cached, this means that the cache needs more space to store more keys, because there may be many null keys
  2. Even if the expiration time is set for the null value, there will still be inconsistency between the data of the cache layer and the storage layer in a certain time window, which will have an impact on the business that needs to maintain consistency

2.2.2 buffer breakdown

Too much, cache expired!


  • Cache breakdown refers to that there is no data in the cache but there is some data in the database (generally when the cache time expires). At this time, due to the large number of concurrent users, the data is not read in the read cache at the same time, and the data is fetched from the database at the same time, resulting in an instantaneous increase in the pressure on the database, resulting in excessive pressure

  • Cache breakdown refers to that a key is very hot and is constantly carrying large concurrency. Large concurrency focuses on accessing this point. When the key fails, the continuous large concurrency will break through the cache and directly request the database, which is like cutting a hole in a barrier

  • When a key expires, a large number of concurrent access requests will be made. This kind of data is generally hot data. Because the cache expires, the database will be accessed at the same time to query the latest data and write back to the cache, which will lead to excessive pressure on the database at the moment


  • Set hotspot data not to expire

From the cache level, no expiration time is set, so there will be no problems after the hot key expires

  • Add mutex

Distributed lock: ensure that there is only one thread for each key to query the back-end service at the same time, and other threads do not have access to the distributed lock, so they only need to wait. This method transfers the pressure of high concurrency to the distributed lock, because the test of distributed lock is very big

2.2.3 cache avalanche


It refers to a certain period of time when the cache set expires and redis goes down


  • redis high availability

redis cluster to ensure high availability of services (multi activity in different places)

  • Current limiting degradation

After the cache fails, the number of threads reading and writing to the database cache is controlled by locking and queuing. For example, for a key, only one thread is allowed to view data and write to the cache, while other threads wait

  • Data preheating

Before the formal deployment, access the possible data in advance, so that part of the data that may be accessed in large quantities will be loaded into the cache. Before a large concurrent access is about to occur, manually trigger the loading of different key s in the cache and set different expiration times to make the time point of cache invalidation as uniform as possible

Topics: Linux Operation & Maintenance Redis Middleware nosql