1, Redis installation
download
1. Decompress, compile and install
# Decompress, compile and install (odc) [mca@clu01 software]$ tar -xvf redis-6.2.6.tar.gz (odc) [mca@clu01 software]$ cd redis-6.2.6/ (odc) [mca@clu01 redis-6.2.6]$ make (odc) [mca@clu01 redis-6.2.6]$ cd src (odc) [mca@clu01 src]$ sudo make install
2. Directory settings
# For the convenience of operation and management, move the directory (odc) [mca@clu01 redis-6.2.6]$ mkdir bin (odc) [mca@clu01 redis-6.2.6]$ mkdir etc (odc) [mca@clu01 redis-6.2.6]$ mv redis.conf etc/ (odc) [mca@clu01 redis-6.2.6]$ cd src/ (odc) [mca@clu01 src]$ mv mkreleasdhdr.sh redis-benchmark redis-check-aof redis-check-dump redis-cli redis-server /home/mca/software/redis-6.2.6/bin/
3. Start
# Start (this is the foreground operation)
(odc) [mca@clu01 software]$ cd redis-6.2.6/
(odc) [mca@clu01 redis-6.2.6]$ cd bin
(odc) [mca@clu01 bin]$ ll
total 28200
-rwxrwxr-x. 1 mca mca 4829528 Dec 13 11:29 redis-benchmark
-rwxr-xr-x. 1 mca mca 9518928 Dec 13 11:29 redis-check-aof
-rwxrwxr-x. 1 mca mca 5003808 Dec 13 11:29 redis-cli
-rwxrwxr-x. 1 mca mca 9518928 Dec 13 11:29 redis-server
(odc) [mca@clu01 bin]$ ./redis-server
7497:C 13 Dec 2021 11:47:22.657 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
7497:C 13 Dec 2021 11:47:22.657 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=7497, just started
7497:C 13 Dec 2021 11:47:22.657 # Warning: no config file specified, using the default config. In order to specify a config file use ./redis-server /path/to/redis.conf
7497:M 13 Dec 2021 11:47:22.659 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
7497:M 13 Dec 2021 11:47:22.659 # Server can't set maximum open files to 10032 because of OS error: Operation not permitted.
7497:M 13 Dec 2021 11:47:22.659 # Current maximum open files is 4096. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
7497:M 13 Dec 2021 11:47:22.659 * monotonic clock: POSIX clock_gettime
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 6.2.6 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 7497
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | https://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
7497:M 13 Dec 2021 11:47:22.661 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
7497:M 13 Dec 2021 11:47:22.661 # Server initialized
7497:M 13 Dec 2021 11:47:22.661 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
7497:M 13 Dec 2021 11:47:22.661 * Ready to accept connections
# Start and run Redis in the background
(odc) [mca@clu01 redis-6.2.6]$ cd etc/
(odc) [mca@clu01 etc]$ vim redis.conf
take daemonize Property changed to yes(Indicates that it needs to run in the background)
# Start the redis service again and specify the startup service configuration file
(odc) [mca@clu01 bin]$ ./redis-server /home/mca/software/redis-6.2.6/etc/redis.conf
4. Check the port number
# After the server is started successfully, execute Redis cli to start the Redis client and check the port number. (odc) [mca@clu01 bin]$ redis-cli 127.0.0.1:6379> exit (odc) [mca@clu01 bin]$ netstat -tunpl | grep 6379 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 15083/./redis-serve tcp6 0 0 ::1:6379 :::* LISTEN 15083/./redis-serve
5. Password and connection configuration
Configure Ciphers
Method 1: through configuration file settings, the service needs to be restarted # redis set password access (odc) [mca@clu01 etc]$ vim redis.conf #requirepass foobared remove the comment and change foobared to your own password. I change it here to requirepass 123456 Method 2: set the password through the command This method is relatively simple and does not require restart redis Service. connect redis After that, set it through the command, as follows: root@9010eab8f082:/# redis-cli 127.0.0.1:6379> config set requirepass mca321 OK 127.0.0.1:6379> config get requirepass 1) "requirepass" 2) "mca321" 127.0.0.1:6379> exit root@9010eab8f082:/# redis-cli 127.0.0.1:6379> config get requirepass (error) NOAUTH Authentication required. 127.0.0.1:6379> auth mca321 OK
Configure remote connections
# The listening address of the port where redis is located is 127.0.0.1, which makes the external network unable to connect # bind 127.0.0.1 is bind 101.200.166.125 (true ip address) bind 10.3.1.151 #protected-mode yes # Safe mode. If you need remote connection, you need to make no protected-mode no
restart
# After configuration, restart (odc) [mca@clu01 bin]$ ./redis-server /home/mca/software/redis-6.2.6/etc/redis.conf
#Use the client to connect to Redis service. The command format is as follows:
$ redis-cli -h {redis_host} -p {redis_port} -a {password}
# After setting the password, you can connect without using the password; But you don't have permission to operate
(odc) [mca@clu01 redis-6.2.6]$ redis-cli -h 10.3.1.151 -p 6379
10.3.1.151:6379> config get requirepass
(error) NOAUTH Authentication required.
# Access with password connection
(odc) [mca@clu01 redis-6.2.6]$ redis-cli -h 10.3.1.151 -p 6379 -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.3.1.151:6379> keys*
(error) ERR unknown command `keys*`, with args beginning with:
10.3.1.151:6379> config get requirepass
1) "requirepass"
2) "123456"
10.3.1.151:6379> exit
# Or verify the password after connecting
(odc) [mca@clu01 redis-6.2.6]$ redis-cli -h 10.3.1.151 -p 6379
10.3.1.151:6379> auth 123456
OK
10.3.1.151:6379> config get requirepass
1) "requirepass"
2) "123456"
6. Close
(odc) [mca@clu01 bin]$ ./redis-cli 127.0.0.1:6379> shutdown not connected> not connected> exit (odc) [mca@clu01 bin]$ netstat -anp | grep 6379 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp 0 0 127.0.0.1:6379 127.0.0.1:49022 TIME_WAIT -
2, Using python to operate Redis
In python, we use the Redis py library to operate the Redis database, which will be highlighted below.
(base) [mca@clu01 ~]$ conda activate odc (odc) [mca@clu01 ~]$ pip install redis
1. Connection
The first way: ordinary
import redis # Import the redis module and operate redis through python. Redis can also directly operate the cache database on the server side of the redis host
r = redis.Redis(host='localhost', port=6379, decode_responses=True) # Host is the redis host. Both the redis server and client need to be started. The default port of redis is 6379
r.set('name', 'junxi') # Key is "foo" and value is "bar". Store the key value pair into redis cache
print(r['name'])
print(r.get('name')) # Take out the value corresponding to the key name
print(type(r.get('name')))
#redis_conn = redis.Redis(host='127.0.0.1', port= 6379, password= 'your pw', db= 0)
The second method: connection pool
Redis py uses the connection pool to manage all connections to a redis server, avoiding the overhead of establishing and releasing connections each time. By default, each redis instance maintains its own connection pool.
You can directly establish a connection pool and use it as a parameter Redis, so that multiple Redis instances can share a connection pool.
import redis # You can import the redis database directly through the redis server
pool = redis.ConnectionPool(host='localhost', port=6379, decode_responses=True) # Host is the redis host. Both the redis server and client need to play the role of redis. The default port of redis is 6379
r = redis.Redis(connection_pool=pool)
r.set('gender', 'male') # Key is "gender" and value is "male". Store the key value pair into redis cache
print(r.get('gender')) # gender takes out the value corresponding to the key male
# redis_pool = redis.ConnectionPool(host='127.0.0.1', port= 6379, password= 'your pw', db= 0)
The Conduit
Redis py creates (the connection pool applies for connection) and disconnects (returns the connection pool) once for each request by default. If you want to specify multiple commands in one request, you can use pipline to specify multiple commands in one request. By default, a pipline is an atomic operation.
#!/usr/bin/env python
# -*- coding:utf-8 -*-
import redis
pool = redis.ConnectionPool(host='192.168.22.132', port=6379)
r = redis.Redis(connection_pool=pool)
# pipe = r.pipeline(transaction=False)
pipe = r.pipeline(transaction=True)
pipe.set('name', 'root')
pipe.set('role', 'root')
pipe.execute()
2. Common data structures
Detailed explanation of using Redis in Python - Zhihu (zhihu.com)
Redis data type | rookie tutorial (runoob.com)
- String-character string
- List - List
- Hash hash
- Set set
- ZSet ordered set
- Bitmap bitmap
Application scenarios of various data types:
| type | brief introduction | characteristic | scene |
|---|---|---|---|
| String (string) | Binary security | It can contain any data, such as jpg pictures or serialized objects. One key can store up to 512M | — |
| Hash (Dictionary) | The set of key value pairs, that is, the Map type in the programming language | It is suitable for storing objects, and you can modify only one attribute value like an update attribute in the database (in Memcached, you need to take out the whole string and deserialize it into an object. After modification, serialize it and save it back) | Store, read and modify user attributes |
| List | Linked list (bidirectional linked list) | Fast addition and deletion. It provides an API for operating a certain section of elements | 1. Latest message ranking and other functions (such as the timeline of the circle of friends) 2. Message queue |
| Set | Hash table implementation, elements are not repeated | 1. The complexity of adding, deleting and searching is O(1) 2. It provides operations such as intersection, union and difference for the set | 1. Common friends 2. Make use of uniqueness to count all independent IP addresses visiting the website. 3. When friends recommend, find the intersection according to the tag. If it is greater than a certain threshold, you can recommend |
| Sorted set | Add a weight parameter score to the elements in the Set, and the elements are arranged in order by score | When the data is inserted into the collection, it has been sorted naturally | 1. Leaderboard 2. Message queue with weight |
Since the author mainly uses Zset data structure, the following only summarizes the use of Zset structure:
3.zset set operation
redis ordered collection:
Redis zset, like set, is also a collection of string elements, and duplicate members are not allowed.
The difference is that each element is associated with a score of type double. redis sorts the members of the collection from small to large through scores.
The members of zset are unique, but the score can be repeated.
The method provided by python redis is basically consistent with the command line. The specific function and meaning can be used by referring to the given command line, and the corresponding function name is basically consistent with the command name
Regular ZSET official operation commands:
1 ZADD key score1 member1 [score2 member2] Add one or more members to an ordered collection, or update the scores of existing members 2 ZCARD key Gets the number of members of an ordered collection 3 ZCOUNT key min max Calculates the number of members of a specified interval score in an ordered set 4 ZINCRBY key increment member Increments the score of a specified member in an ordered set increment 5 ZINTERSTORE destination numkeys key [key ...] Calculates the intersection of a given one or more ordered sets and stores the result set in a new ordered set key in 6 ZLEXCOUNT key min max Calculates the number of members in the specified dictionary interval in an ordered set 7 ZRANGE key start stop [WITHSCORES] The ordered set is returned through the index interval to synthesize the members in the specified interval 8 ZRANGEBYLEX key min max [LIMIT offset count] Returns the members of an ordered set through a dictionary interval 9 ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT] Returns the members in the specified interval of an ordered set through scores 10 ZRANK key member Returns the index of a specified member in an ordered collection 11 ZREM key member [member ...] Remove one or more members from an ordered collection 12 ZREMRANGEBYLEX key min max Removes all members of a given dictionary interval from an ordered set 13 ZREMRANGEBYRANK key start stop Removes all members of a given ranking interval from an ordered set 14 ZREMRANGEBYSCORE key min max Removes all members of a given score interval from an ordered set 15 ZREVRANGE key start stop [WITHSCORES] Returns the members in the specified interval in the ordered set. Through the index, the score is from high to low 16 ZREVRANGEBYSCORE key max min [WITHSCORES] Returns the members within the specified score range in the ordered set. The scores are sorted from high to low 17 ZREVRANK key member Returns the ranking of the specified members in the ordered set. The members of the ordered set are decremented by the score value(From big to small)sort 18 ZSCORE key member Returns the score value of a member in an ordered set 19 ZUNIONSTORE destination numkeys key [key ...] Calculate the union of one or more given ordered sets and store them in a new key in 20 ZSCAN key cursor [MATCH pattern] [COUNT count] Iterate the elements in the ordered set (including element members and element scores) 21 BZPOPMAX kes timeout BZPOPMAX Is an ordered set ZPOPMAX Blocking variants of primitives. When no member pops up from any given ordered set, It will block the connection timeout Is the timeout of blocking to prevent blocking all the time. For example: BZPOPMAX zset1 zset2 0 22 BZPOPMIN Blocking throw minimum 23 ZPOPMAX Throw maximum 24 ZPOPMIN Throw minimum 25 ZREVRANGEBYLEX When all elements in the sort set insert the same score, in order to force dictionary sorting, this command will return all elements in the sort set key Value is max Between and min
Zset in Python
Python redis provides 25 kinds of corresponding command support, and the corresponding names are basically consistent with the command line
The following is the usage of some functions:
def zadd(self, name, mapping, nx=False, xx=False, ch=False, incr=False)
#Add elements to the ordered collection corresponding to name
rediscli.zadd("me1", mapping={"x1": 1, "x2": 3})
1
def zcard(self, name)
#Gets the number of members of an ordered collection
rediscli.zcard("me1")
1
def zcount(self, name, min, max)
#Get the number of scores between [min,max] in the ordered set corresponding to name
rediscli.zcount("me1", 0, 5)
1
def zincrby(self, name, amount, value)
#Add increment to the score of the specified member in the ordered collection name
def zinterstore(self, dest, keys, aggregate=None)
#Calculate the intersection of one or more given ordered sets and store the result set in the new ordered set key
def zlexcount(self, name, min, max)
#Calculates the number of members in the specified dictionary interval in an ordered set
rediscli.zlexcount('me1', "[a", "[v")
1
def zpopmax(self, name, count=None)
#Throw maximum
def zpopmin(self, name, count=None)
#Throw minimum
def bzpopmax(self, keys, timeout=0)
#Throw maximum
def bzpopmin(self, keys, timeout=0)
def zrange(self, name, start, end, desc=False, withscores=False, score_cast_func=float)
#Get the elements of the ordered collection corresponding to name according to the index range
"""Parameters:
name，redis of name
start，Start position of ordered set (non fractional)
end，Ordered set index end position (non fractional)
desc，Sorting rule: sort by score from small to large by default
withscores，Whether to get the score of the element. By default, only the value of the element is obtained
score_cast_func，A function that converts scores into data"""
rediscli.zrange('me1', 0, 5)
Output: ['x1', 'x2']
rediscli.zrange('me1', 0, 5, withscores=True)
Output:
[('x1', 1.0), ('x2', 3.0)]
1
2
3
4
5
6
def zrangebylex(self, name, min, max, start=None, num=None)
#Returns the members of an ordered set through a dictionary interval
"""Parameters:
name，redis of name
min，Left interval (value). + Indicates positive infinity; - Indicates negative infinity; ( Indicates the open interval; [ Indicates a closed interval
min，Right interval (value)
start，The results are segmented and indexed
num，The results are segmented, and the data behind the index num Elements"""
zrangebylex('me1', '-', '(x')
1
def zrevrangebylex(self, name, max, min, start=None, num=None)
def zrangebyscore(self, name, min, max, start=None, num=None, withscores=False, score_cast_func=float)
rediscli.zrangebyscore('me1', 2, 5)
1
def zrank(self, name, value)
#Get the ranking of a value in the ordered collection corresponding to name (starting from 0)
def zrem(self, name, *values)
#Delete the ordered set corresponding to name. The value is a member of values
def zremrangebylex(self, name, min, max)
def zremrangebyrank(self, name, min, max)
#Delete according to ranking range
def zremrangebyscore(self, name, min, max)
#Delete according to score range
def zrevrange(self, name, start, end, withscores=False, score_cast_func=float)
def zrevrangebyscore(self, name, max, min, start=None, num=None, withscores=False, score_cast_func=float)
def zrevrank(self, name, value)
def zscore(self, name, value)
zscan(name, cursor=0, match=None, count=None, score_cast_func=float)
rediscli.scan()
Output: (0L, ['me1', 'm1', 'name', 'm2'])
1
2
3
zscan_iter(name, match=None, count=None,score_cast_func=float)
#List all keys in the form of generator, and add score compared with the string_ cast_ Func, used to operate on scores
def zinterstore(dest, keys, aggregate=None)
#Get the intersection of two ordered sets. If you encounter different scores with the same value, follow aggregate
aggregate The value of is: SUM MIN MAX
def zunionstore(dest, keys, aggregate=None)
#Get the union of two ordered sets. If you encounter different scores with the same value, follow aggregate
aggregate The value of is: SUM MIN MAX
Sample program:
# coding:utf-8
import redis
rediscli = redis.Redis(host='127.0.0.1', port=8888, db=13)
print rediscli.zadd("me1", mapping={"x1": 1, "x2": 3})
print rediscli.zadd("me2", mapping={"x1": 1, "x3": 5, "x4": 3})
print rediscli.zcard("me1")
print rediscli.zcount("me1", 0, 5)
print rediscli.zrange('me1', 0, 5, withscores=False)
print rediscli.zrange('me1', 0, 5, withscores=True)
print rediscli.zincrby("me1", amount=1, value="x1")
print rediscli.zinterstore(dest="me3", keys=['me1', 'me2'])
print rediscli.zrange('me3', 0, 5, withscores=True)
print rediscli.zlexcount('me1', "[a", "[v")
print rediscli.zrangebylex('me1', '-', '(x')
print rediscli.zrangebyscore('me1', 2, 5)
print rediscli.zrank('me1', 'x1')
print rediscli.zscore('me1', 'x1')
print rediscli.zadd('me1', mapping={"x4": 4})
print rediscli.zrem('me1', 'x4')
print rediscli.zremrangebyrank('me1', 7, 10)
print rediscli.zremrangebyscore('me1', 11, 15)
print rediscli.zrevrangebyscore('me1', 8, 3)
print rediscli.zrevrange('me1', 0, -1, withscores=True)
print rediscli.zrange('me1', 0, -1, withscores=True)
print rediscli.scan()
Output result:
2
3
2
2
['x1', 'x2']
[('x1', 1.0), ('x2', 3.0)]
2.0
1
[('x1', 3.0)]
0
[]
['x1', 'x2']
0
2.0
1
1
0
0
['x2']
[('x2', 3.0), ('x1', 2.0)]
[('x1', 2.0), ('x2', 3.0)]
(0L, ['me3', 'me2', 'me1'])
reference resources:
3, Redis usage and memory optimization
In fact, the memory management cost of redis is very high, that is, it takes up too much memory, which belongs to trading space for time. The author is also very clear about this, so he provides a series of parameters and means to control and save memory
It is not recommended to turn on the VM (virtual memory) option
VM option is a persistent strategy for Redis to store data beyond physical memory. It will seriously slow down the running speed of the system. Therefore, to turn off VM function, please check your Redis Conf file vm-enabled Is no.
Set maximum memory options
It's better to set Redis The maxmemory option in conf tells Redis how much is used physical memory Then start rejecting subsequent write requests. This parameter can protect your Redis from using too much physical memory to cause swap, which will seriously affect the performance and even crash.
Generally, you also need to set the memory saturation recovery strategy
- Volatile LRU: select the least recently used data from the data set with expiration time (server.db[i].expires)
- Volatile TTL: select the data to be expired from the data set (server.db[i].expires) with the expiration time set
- Volatile random: select any data from the data set (server.db[i].expires) for which the expiration time has been set
- Allkeys LRU: select the least recently used data from the data set (server.db[i].dict)
- Allkeys random: select any data from the data set (server.db[i].dict)
- no-enviction (expulsion): Prohibition of expulsion data
Parameters that control memory usage
Redis provides a set of parameters for different data types to control memory usage
- Hash
redis. The following 2 items in the conf configuration file
- **hash-max-zipmap-entries 64 **
It means that when there are no more than a few members in the value Map, it will be stored in a linear compact format. The default is 64, that is, if there are less than 64 members in the value, it will use the linear compact storage zipmap, and if it exceeds this value, it will be automatically converted to a real HashMap(ht).
- hash-max-zipmap-value 512
Hash Max zipmap value means that when the length of each member value in the Map does not exceed
How many bytes will use linear compact storage zipmap to save space.
If any of the above two conditions exceeds the set value, it will be converted into a real HashMap, which will no longer save memory, but the larger the better (the space and modification efficiency need to be weighed according to the actual situation)
- List
- list-max-ziplist-entries 512
The following nodes of the list data type will adopt the compact storage format ziplost without pointer - list-max-ziplist-value 64
list data type node value size less than how many bytes will use the compact storage format ziplost.
- Set
- set-max-intset-entries 512
If the internal data of set data type is all numeric and contains how many nodes, the following data will be stored in compact format
Redis internal optimization
- Redis's internal implementation does not optimize the memory allocation too much, and there will be memory fragmentation to a certain extent. However, in most cases, this will not become the performance bottleneck of redis.
- Redis caches a certain range of data Constant number As a resource sharing, if many data types are numeric, it can greatly reduce the memory overhead. The default is 1-10000. You can recompile and configure and modify a line of macro definition redis in the source code_ SHARED_ INTEGERS.
How to clean up memory fragments?
Restart Redis instance:
- If the data in Redis is not persistent, the data will be lost;
- Even if Redis data is persistent, we still need to recover through AOF or RDB. The recovery time depends on the size of AOF or RDB. If there is only one Redis instance, services cannot be provided in the recovery phase.
Fortunately, since version 4.0-RC3, Redis has provided a method to automatically clean up memory fragments:
Basic mechanism:
- Memory fragment cleaning, in short, is "moving to give way and merging space".
Redis specially sets parameters for automatic memory fragment cleaning mechanism:
- Control the start and end time of debris cleaning
- Percentage of CPU used
- So as to reduce the impact of fragment cleaning on the performance of Redis request processing
First, Redis needs to enable automatic memory fragment cleaning. You can set the activedefrag configuration item to yes. The command is as follows:
Conditions for triggering cleaning (to be met at the same time):
- Active defrag ignore bytes 100MB: indicates that when the number of bytes of memory fragments reaches 100MB, it starts to clean up;
- active-defrag-threshold-lower 10 : indicates that when the memory fragment space accounts for 10% of the total space allocated to Redis by the operating system, the cleaning is started.
In order to minimize the impact of fragment cleaning on the normal request processing of Redis, the automatic memory fragment cleaning function will also monitor the CPU time occupied by the cleaning operation during execution, and two parameters are set to control the upper and lower limits of the CPU time occupied by the cleaning operation respectively, so as to ensure the normal cleaning and avoid reducing the performance of Redis. These two parameters are as follows:
- active-defrag-cycle-min 25: indicates that the proportion of CPU time used in the automatic cleaning process shall not be less than 25%, so as to ensure that the cleaning can be carried out normally;
- active-defrag-cycle-max 75 : indicates that the proportion of CPU time used in the automatic cleaning process is no more than 75%. Once it exceeds, the cleaning will be stopped, so as to avoid a large number of memory copies blocking Redis during cleaning, resulting in an increase in response delay.