summary
Distributed locking has always been a topic that needs to be focused after the emergence of distributed systems. When it comes to distributed systems, we have to think of distributed locks and distributed transactions. In the past single application scenarios, using the built-in lock of jvm can solve the thread safety problem, but in the distributed scenario, It can't be done with the help of some components to ensure thread safety in distributed scenarios, such as using databases or some shared storage. However, with the wide use of Redis, in today's business scenarios, more and more companies use Redis to solve one of the middleware that provides system throughput in high concurrency scenarios, but Redis can do a lot of things, It is not only a middleware for data caching, but also can do some work such as subscription, publishing, distributed queue and distributed lock in distributed scenarios. Today, let's talk about Redis's distributed lock.
actual combat
We have a scenario. In the Internet scenario, the scenario of initiating an order to deduct inventory is very scenario and easy to understand. For example, a merchant does an activity and takes out a batch of goods. For example, there is 1000 inventory to do the activity. The program needs to ensure that the 1000 goods cannot be oversold, that is, the 1000 goods are sold to 1000 users, If there is oversold, there is a problem with the program, which is a very simple problem. However, in the distributed scenario, this is not a simple problem. The program should ensure thread safety, that is, ensure that all outbound actions should be executed in sequence, that is, serial execution, and there should be no concurrent deduction of inventory, In the case of a single machine, it can be completed simply through the built-in lock or AQS lock, but it becomes more complex in the distributed scenario. There are many scenarios to be considered. We will analyze them according to these scenarios.
Simple distributed lock
Let's look at a program to analyze the implementation of distributed locks
@RequestMapping("/deduct_stock")
public String deductStock() {
try {
synchronized (this){
int stock = Integer.parseInt(stringRedisTemplate.opsForValue().get("stock")); // jedis.get("stock")
if (stock > 0) {
int realStock = stock - 1;
stringRedisTemplate.opsForValue().set("stock", realStock + ""); // jedis.set(key,value)
System.out.println("Deduction successful, remaining inventory:" + realStock);
} else {
System.out.println("Deduction failed, insufficient inventory");
}
}
}catch (Exception e){
e.printStackTrace();
return "error";
}
return "end";
}
Obviously, this program is not a thread safe program. There are problems in distributed scenarios. If concurrency is not large, there may be no errors. However, as a programmer, the code written should be rigorous. This method must be problematic and can only solve the thread safety in single machine situations. We modify the program as follows:
@RequestMapping("/deduct_stock")
public String deductStock() {
String lockKey = "product:1001";
String value = Thread.currentThread().getId() + ":" + UUID.randomUUID().toString();
stringRedisTemplate.opsForValue().setIfAbsent(lockKey, value, 30, TimeUnit.SECONDS);
try {
int stock = Integer.parseInt(stringRedisTemplate.opsForValue().get("stock")); // jedis.get("stock")
if (stock > 0) {
int realStock = stock - 1;
stringRedisTemplate.opsForValue().set("stock", realStock + ""); // jedis.set(key,value)
System.out.println("Deduction successful, remaining inventory:" + realStock);
} else {
System.out.println("Deduction failed, insufficient inventory");
}
} catch (Exception e) {
e.printStackTrace();
return "error";
} finally {
if (stringRedisTemplate.opsForValue().get(lockKey).equals(value)) {
stringRedisTemplate.delete(lockKey);
}
}
return "end";
}
We use setnx of redis for distributed locking. Setnx can only succeed for the first time under the same key. After calling in java, if it is set successfully, it will return true. If the key already exists, it will return false
Format: setnx key value sets the value of the key to value if and only if the key does not exist. If the given key already exists, then
SETNX does nothing. SETNX is the abbreviation of "SET if Not eXists".
Therefore, the above code:
stringRedisTemplate.opsForValue().setIfAbsent(lockKey, value, 30, TimeUnit.SECONDS);
It means to set lockKey. value is a thread Id+uuid, ttl=30s generated by the method. It will be cleared automatically after 30s. Therefore, the whole sentence means that a ttl lock is added here. If a thread obtains the lock and the system hangs during execution, the key will not be cleared, and other distributed systems will never obtain the lock, This makes it impossible for this business to continue, so a 30s timeout is added, that is, when the system hangs up, it will unlock automatically for up to 30s. We should note that the setnx instruction is to set the key and set the timeout together, because these two actions must be atomic, otherwise other situations may occur, For example, when the timeout time is set after the key is set, it will hang up, so it needs to be executed together as an atomic instruction;
If the program is executed normally and unlocked in finally, other threads can get the lock, but is there really no problem with the program?
Question 1: the code in finally is the unlocking code. The unlocking code first obtains the key, and then determines whether the value corresponding to the key is the value set by my thread. If so, release the lock. At first glance, there is no problem, but we must have distributed thinking when we build a distributed system? There is nothing wrong with these two lines of code, but from the thinking of distributed scenarios, we can know that these two lines of code are not an atomic operation. Although there are very few exceptions in the execution of these two lines of code, there may still be problems from the perspective of the rigor of the program, so how can we turn it into an atomic operation? redis is a single thread task execution, but it does not provide a command to obtain the key and then judge whether the incoming value is equal. The solution is to use lua script to encapsulate and execute it as an atomic instruction;
Question 2: what is the problem from the timeout set above? Will there be a problem that the thread starts to execute after it gets the lock, and other threads are blocked, but the timeout you set is 30s? Will there be a problem that the thread that gets the lock does not complete execution after 30s? At this time, the lock fails, and other threads will get the lock and can execute. The problem is in this scenario, Data security can not be guaranteed, that is, in some cases with high concurrency, thread safety problems will occur in case of slow program execution. Although the probability of this situation will not be too high, as a rigorous developer, this is not intolerable.
Based on the above two problems, how can we solve them? What we need to solve are the atomic problem of instruction execution and the life extension of lock. If we can solve it and have high performance, we can solve the problem of thread safety in this distributed scenario. So how can we solve it?
1. For the atomic problem of lock, with the help of lua script, all the instructions in one thing can be executed in Lua script to ensure that all instructions are not executed or not executed. Anyway, it can not be a success or a failure;
2. The problem of lock renewal: This is to be implemented by developers themselves. After obtaining a lock, you can open a thread task to judge whether the task is completed. If the execution is not completed, the lock renewal will be carried out. If the execution is completed, the set lock will not exist, and other threads can get the lock.
Redisson implements distributed locks
Redisson solves the problem of distributed locks, which is a high encapsulation of Redis. Redisson is a Java in memory data grid based on Redis. On the Netty framework based on NIO, redison makes full use of a series of advantages provided by Redis key value database, and provides users with a series of common tool classes with distributed characteristics on the basis of common interfaces in Java Utility Kit. The toolkit originally used to coordinate single machine multithreaded concurrent programs has the ability to coordinate distributed multi machine multithreaded concurrent systems, which greatly reduces the difficulty of designing and developing large-scale distributed systems. At the same time, combined with various characteristic distributed services, it further simplifies the cooperation between programs in the distributed environment
It's very easy to use redission. The simpler the framework is used, the more the underlying encapsulation is. I think so. The program to become redission is as follows:
@RequestMapping("/deduct_stock")
public String deductStock() {
String lockKey = "product:1001";
RLock lock = redisson.getLock(lockKey);
lock.lock();
try {
int stock = Integer.parseInt(stringRedisTemplate.opsForValue().get("stock")); // jedis.get("stock")
if (stock > 0) {
int realStock = stock - 1;
stringRedisTemplate.opsForValue().set("stock", realStock + ""); // jedis.set(key,value)
System.out.println("Deduction successful, remaining inventory:" + realStock);
} else {
System.out.println("Deduction failed, insufficient inventory");
}
} catch (Exception e) {
e.printStackTrace();
return "error";
} finally {
lock.unlock();
}
return "end";
}
Three lines of code, and two lines of code for locking and unlocking. It is very simple, but it solves the problems we haven't solved above, the problem of atomic instructions and the problem of Lock life extension. Its use is the same as that of our Aqs Lock, because as a distributed Lock, it implements the standard interface Lock of jdk
The AQ Lock is implemented according to the same standard as the AQ Lock, so it should be implemented according to its own standard.
Redission source code analysis
Construction method
When we call getLock(lockKey) in the above program, we will enter the getLock method in Redisson, where name is the lockKey we import.
public RLock getLock(String name) {
return new RedissonLock(this.connectionManager.getCommandExecutor(), name);
}
//Construct a RedissonLock object, including the executor commandExecutor and the lockKey we passed in
public RedissonLock(CommandAsyncExecutor commandExecutor, String name) {
super(commandExecutor, name);
this.commandExecutor = commandExecutor;
this.id = commandExecutor.getConnectionManager().getId();
//A watchdog time, watch dog, is initialized here. It is the watch dog used to renew the life. The default value is 30s. This value can be changed and can be dynamically passed in when building the redsessionclient
//Lifetime of watch dog
this.internalLockLeaseTime = commandExecutor.getConnectionManager().getCfg().getLockWatchdogTimeout();
}
Get lock
org.redisson.RedissonLock#lock()
public void lock() {
try {
this.lockInterruptibly();
} catch (InterruptedException var2) {
Thread.currentThread().interrupt();
}
}
jdk is a standard that can be used to obtain a lock
public void lockInterruptibly() throws InterruptedException {
this.lockInterruptibly(-1L, (TimeUnit)null);
}
public void lockInterruptibly(long leaseTime, TimeUnit unit) throws InterruptedException {
long threadId = Thread.currentThread().getId();
//Here, you will call to try to obtain the lock, and pass in a thread ID with leaseTime of - 1 and time unit. After the method call is completed, a ttl will be returned
//If the ttl is empty, it means that the lock acquisition is successful and can be returned directly. If the lock acquisition fails, the returned ttl represents how long it will take to obtain the lock
//For example, if the lock acquisition fails, the remaining time of the last lock acquisition will be returned
Long ttl = this.tryAcquire(leaseTime, unit, threadId);
if (ttl != null) {
//Failed to obtain the lock. Subscribe to a channel. Wait until the channel has a message written. Unlock the semaphore after a message is written
RFuture<RedissonLockEntry> future = this.subscribe(threadId);
this.commandExecutor.syncSubscription(future);
try {
while(true) {
//When entering the loop, try to acquire the lock. If ttl returns null, it means the lock is acquired successfully. Otherwise, continue the loop
ttl = this.tryAcquire(leaseTime, unit, threadId);
if (ttl == null) {
return;
}
if (ttl >= 0L) {
//ttl is the returned value, which indicates how long it will take to release the lock. So here, getLatch returns a semaphore
//Lock, the tryAcquire of this call is the lock of jdk semaphore. If there is a lock here, it means there is a place to unlock
//getLatch() is a Semaphore
this.getEntry(threadId).getLatch().tryAcquire(ttl, TimeUnit.MILLISECONDS);
} else {
this.getEntry(threadId).getLatch().acquire();
}
}
} finally {
//Remove subscription feature
this.unsubscribe(future, threadId);
}
}
}
//Subscribe to a message in the channel
protected RFuture<RedissonLockEntry> subscribe(long threadId) {
return PUBSUB.subscribe(this.getEntryName(), this.getChannelName(), this.commandExecutor.getConnectionManager().getSubscribeService());
}
//Channel name
String getChannelName() {
return prefixName("redisson_lock__channel", this.getName());
}
org.redisson.pubsub.LockPubSub#onMessage subscription message notification method
protected void onMessage(RedissonLockEntry value, Long message) {
//It is to judge whether the published unlocking message is consistent with the subscribed message. The published message is a 0
if (message.equals(unlockMessage)) {
//If yes, get the Semaphore semaphore corresponding to the lockkey to unlock. The Semaphore is unlocked by calling the jdk code
value.getLatch().release();
while(true) {
Runnable runnableToExecute = null;
synchronized(value) {
Runnable runnable = (Runnable)value.getListeners().poll();
if (runnable != null) {
if (value.getLatch().tryAcquire()) {
runnableToExecute = runnable;
} else {
value.addListener(runnable);
}
}
}
if (runnableToExecute == null) {
return;
}
runnableToExecute.run();
}
}
}
private Long tryAcquire(long leaseTime, TimeUnit unit, long threadId) {
return (Long)this.get(this.tryAcquireAsync(leaseTime, unit, threadId));
}
private <T> RFuture<Long> tryAcquireAsync(long leaseTime, TimeUnit unit, final long threadId) {
if (leaseTime != -1L) {
return this.tryLockInnerAsync(leaseTime, unit, threadId, RedisCommands.EVAL_LONG);
} else {
//Acquire lock asynchronously
RFuture<Long> ttlRemainingFuture = this.tryLockInnerAsync(this.commandExecutor.getConnectionManager().getCfg().getLockWatchdogTimeout(), TimeUnit.MILLISECONDS, threadId, RedisCommands.EVAL_LONG);
//Add a listener to listen for lock acquisition
ttlRemainingFuture.addListener(new FutureListener<Long>() {
public void operationComplete(Future<Long> future) throws Exception {
if (future.isSuccess()) {
//The ttlRemaining returned is the one returned from obtaining the lock above. If null is returned, it means that the lock is obtained successfully. Otherwise, the lock is obtained
//Failed, so it can be seen from here that the listener is to renew the lock after it succeeds in obtaining the lock
Long ttlRemaining = (Long)future.getNow();
if (ttlRemaining == null) {
//Start a cycle scheduling for life renewal
RedissonLock.this.scheduleExpirationRenewal(threadId);
}
}
}
});
return ttlRemainingFuture;
}
}
//Execute lua script to acquire lock
<T> RFuture<T> tryLockInnerAsync(long leaseTime, TimeUnit unit, long threadId, RedisStrictCommand<T> command) {
//Calculate the lock time in milliseconds
this.internalLockLeaseTime = unit.toMillis(leaseTime);
//Execute lua script
return this.commandExecutor.evalWriteAsync(this.getName(), LongCodec.INSTANCE, command,
//The following lua script has three logics:
//The first if determines whether the passed key=lockKey exists. If it does not exist, needless to say, it directly uses the hash structure set, where
//Keys [1] = lockkey, argv [2] = redistribution uses uuid + thread id to generate a field with value as hash, and the value is 1. Why use it here
//What about 1? Because the re-entry lock is considered in the reission, the
//In the second if statement:
//redis.call('hexists', KEYS[1], ARGV[2]) == 1. This sentence means fieldARGV[2] of lockKey in hash table
// Whether it is equal to 1. If it is, it means the reentry of the same thread. Therefore, add 1 through hincrby, and then reset the expiration time of the lock. This is
// The reentrant lock logic of the lock, the number of reentry times of the lock + 1, and the failure time of the lock is reset, which is similar to the reentrant lock logic of aqs
The first two if Both can satisfy the logic of obtaining locks, so return null，stay redis Return in nil stay java Inside null，return null Represents acquiring a lock
Third if Failed to acquire the lock. Pass the command redis.call('pttl', KEYS[1])Returnable lockKey Remaining time to failure
"if (redis.call('exists', KEYS[1]) == 0)
then
redis.call('hset', KEYS[1], ARGV[2], 1);
redis.call('pexpire', KEYS[1], ARGV[1]);
return nil;
end;
if (redis.call('hexists', KEYS[1], ARGV[2]) == 1)
then
redis.call('hincrby', KEYS[1], ARGV[2], 1);
redis.call('pexpire', KEYS[1], ARGV[1]);
return nil;
end;
return redis.call('pttl', KEYS[1]);"
//The first parameter represents the array of key s, and the following parameters are value, corresponding to the above ARGV
Collections.singletonList(getName()), internalLockLeaseTime, getLockName(threadId));
}
//Lock life extension
private void scheduleExpirationRenewal(final long threadId) {
if (!expirationRenewalMap.containsKey(this.getEntryName())) {
//Use the timetask in netty to renew the lock
Timeout task = this.commandExecutor.getConnectionManager().newTimeout(new TimerTask() {
public void run(Timeout timeout) throws Exception {
RFuture<Boolean> future = RedissonLock.this.commandExecutor.evalWriteAsync(
RedissonLock.this.getName(), LongCodec.INSTANCE, RedisCommands.EVAL_BOOLEAN,
//Judge whether the lockKey is still there. If it is still there, it means that the lock has not been released, that is, the thread that obtains the lock has not completed execution. Pass
//pexpire renews its life, that is, reset the lock expiration time. pexpire sets the time in milliseconds and expire in seconds
"if (redis.call('hexists', KEYS[1], ARGV[2]) == 1)
//Reset lock failure time, life extension
then redis.call('pexpire', KEYS[1], ARGV[1]);
//1 returned by redis represents true in java and 0 represents false
return 1; end;
return 0;",
//getName() is the lockKey passed in from the outside. In our example, lockKey=product:1001
Collections.singletonList(RedissonLock.this.getName()), new Object[]{RedissonLock.this.internalLockLeaseTime, RedissonLock.this.getLockName(threadId)});
future.addListener(new FutureListener<Boolean>() {
public void operationComplete(Future<Boolean> future) throws Exception {
RedissonLock.expirationRenewalMap.remove(RedissonLock.this.getEntryName());
if (!future.isSuccess()) {
RedissonLock.log.error("Can't update lock " + RedissonLock.this.getName() + " expiration", future.cause());
} else {
//Returning true means that the lock renewal is successful. If the thread has not been completed, continue to call recursively for lock renewal. Otherwise, it is
//It's done. You can get the lock
if ((Boolean)future.getNow()) {
RedissonLock.this.scheduleExpirationRenewal(threadId);
}
}
}
});
}
//This scheduler is the time 3 of watch dog. If it is 30s, it is executed once in 10s
}, this.internalLockLeaseTime / 3L, TimeUnit.MILLISECONDS);
if (expirationRenewalMap.putIfAbsent(this.getEntryName(), task) != null) {
task.cancel();
}
}
}
Unlock
org.redisson.RedissonLock#unlock
public void unlock() {
Boolean opStatus = (Boolean)this.get(this.unlockInnerAsync(Thread.currentThread().getId()));
if (opStatus == null) {
throw new IllegalMonitorStateException("attempt to unlock lock, not locked by current thread by node id: " + this.id + " thread-id: " + Thread.currentThread().getId());
} else {
//opStatus=true indicates that the unlocking is successful
if (opStatus) {
//Here is the scheduler for canceling lock renewal
this.cancelExpirationRenewal();
}
}
}
void cancelExpirationRenewal() {
Timeout task = (Timeout)expirationRenewalMap.remove(this.getEntryName());
if (task != null) {
//Scheduler for unlocking renewal
task.cancel();
}
}
protected RFuture<Boolean> unlockInnerAsync(long threadId) {
return this.commandExecutor.evalWriteAsync(this.getName(), LongCodec.INSTANCE, RedisCommands.EVAL_BOOLEAN,
//Unlock logic
//The first if is to determine whether the lockKey exists. If it does not exist, the lock is invalid. Directly issue a message to the channel to tell the lock that it can be unlocked
//The second if is to judge whether the field of lockkey is the incoming ARGV[3], which is to judge whether the lock is added by your thread. If not, it cannot be unlocked
//Next, for lockkey-1, it is set to 1 when locking. if the re-entry is + 1, so subtract one here. if it is 0, it means there is no re-entry
//Unlock normally, delete the lockKey, and send a message to the channel to notify the lock to unlock. If it is not 0, it means re-entry lock and start again
//Reset lock expiration time
"if (redis.call('exists', KEYS[1]) == 0)" +
" then redis.call('publish', KEYS[2], ARGV[1]);" +
" return 1; " +
"end;" +
"if (redis.call('hexists', KEYS[1], ARGV[3]) == 0) then" +
" return nil;" +
"end; " +
"local counter = redis.call('hincrby', KEYS[1], ARGV[3], -1);" +
" if (counter > 0) then" +
" redis.call('pexpire', KEYS[1], ARGV[2]); " +
"return 0; else redis.call('del', KEYS[1]);" +
" redis.call('publish', KEYS[2], ARGV[1]); " +
"return 1; " +
"end; " +
"return nil;",
// this.getChannelName(): the name of the channel, corresponding to the subscription
//LockPubSub. The message released by unlockmessage is a 0
Arrays.asList(this.getName(), this.getChannelName()), new Object[]{LockPubSub.unlockMessage, this.internalLockLeaseTime, this.getLockName(threadId)});
}
Lock failure
Is there really no problem using Redisson? Now, many companies basically adopt redison in the scenario of using distributed locks. Redison is used because of the high performance of Redis, because Redis is a database based on memory, and most importantly, Redis is an AP architecture in the CAP architecture system, which ensures availability and sacrifices consistency to a certain extent. Therefore, from the perspective of Redis architecture, Can we reverse analyze whether Redis's distributed locks are really very reliable? In fact, this has little to do with redission. Redission is a framework based on Redis encapsulation, because Redis is an AP architecture, like Nacos, which is also the default AP architecture. Under the AP architecture, the availability is very good. Even if there are problems in the architecture, it can be adjusted in time, and then services can be provided, but its data can not be guaranteed and the consistency can not be fully guaranteed, Therefore, based on this question, we can think about why there are always some problems in Redis's distributed lock. However, these problems may occur in extreme cases, not necessarily. We just analyze the problems in AP architecture. We know that whether Reids is master-slave architecture, sentinel architecture or cluster architecture, there will be master and slave nodes, Slave nodes are basically designed for disaster recovery, as shown in the following figure:
As shown in the figure above, this kind of problem really occurs in some extreme cases when redis architecture makes distributed locks. We all know that there are two kinds of persistence schemes of redis, one is RDB, that is, how many commands are executed in how long will persist an RDB snapshot, and the other is aof, which has several options, However, for the sake of redis's high performance, aof will be executed in less than 1s, and aof will not be executed in every instruction. Therefore, for the aof persistence scheme, 1s of data will be lost at most. For systems with low concurrency, it is not a big problem. If the concurrency is particularly high, 1s of data may be distributed lock data, In the above figure, the thread can obtain the data from master node 1 when the master lock is not obtained from master node 2. Therefore, when the distributed thread 1 loses the data from master node 1, the thread can obtain the data from master node 2 when the master lock is not executed, In this way, a serial execution becomes a parallel execution, and the protection of shared resources has been lost. This situation can also be called distributed thread safety in distributed scenarios;
But it is still necessary to say that the probability of this situation is too small. It is only necessary to analyze the problem comprehensively and be strict when it comes to its realization. The point of rigor will always be correct. Analyze according to the business volume of its own, and do not over analyze and design. If the concurrency is very small, it is unnecessary to consider too much.
Red lock
RedLock says it can be solved online. I don't think so. Red lock is to engage in several redis unrelated nodes to do distributed locks. Only more than half of the nodes can be considered successful. However, it can't be done in extreme cases. For example, three unrelated redis nodes are used to do red locks, The condition for successful locking is that half of the three redis nodes are successfully locked, which means successful locking. However, if one node hangs after successful locking, and there are only two nodes when another thread locks, it may not succeed. However, if the restarted redis node data fails to persist due to some reasons, and the lock data is lost, The second thread can also lock successfully. Even if you add a slave node to redis, there will also be problems. In the business scenario of distributed lock, there is no 100% scheme, which can only be guaranteed as much as possible. In my opinion, it is better to directly use zookeeper as a distributed lock with redis's red lock. Zookeeper's natural CP architecture ensures the data consistency protocol, Its ZAB protocol can ensure the correctness of distributed lock. Because zookeeper will think it will succeed after more than half of the nodes are successful each time they write node data. Even if the leader is hung, the new leader node also has the previous node data. Therefore, in the distributed lock scenario, zookeeper is more stable than redis, but why is zookeeper used to do distributed lock?
1. Redis is a commonly used Middleware in the project, and the high performance of redis based on memory database is the first choice, mainly due to its high performance. This is the most important. It is acceptable even if there are occasional cache problems.
2. If zookeeper has no other use in the whole architecture and is only introduced to be a distributed lock, it will increase the maintenance cost of the system, and its performance is certainly not as good as Redis.
Analysis on lock implementation of high performance Redis
To be continued