1. What is distributed lock

Distributed lock is a way to control synchronous access to shared resources between distributed systems. In distributed systems, it is often necessary to coordinate their actions. If different systems or different hosts of the same system share one or a group of resources, when accessing these resources, they often need to be mutually exclusive to prevent interference with each other to ensure consistency. In this case, distributed locks need to be used.

2. Why use distributed locks

In order to ensure that a method or attribute can only be executed by the same thread at the same time in the case of high concurrency, in the case of single machine deployment of traditional single application, the API related to Java concurrency processing (such as ReentrantLock or Synchronized) can be used for mutual exclusion control. In the stand-alone environment, Java provides many APIs related to concurrent processing. However, with the needs of business development, after the system deployed by the original single machine is evolved into a distributed cluster system, because the distributed system is multi-threaded, multi process and distributed on different machines, this will invalidate the concurrency control lock strategy under the original single machine deployment, and the simple Java API can not provide the ability of distributed lock. In order to solve this problem, we need a cross JVM mutual exclusion mechanism to control the access of shared resources, which is the problem to be solved by distributed!

for instance:

Machine A and machine B are A cluster. The programs on both machines A and B are the same and have high availability.

A. Machine B has a scheduled task. It needs to execute a scheduled task at 2 a.m. every night, but this scheduled task can only be executed once, otherwise an error will be reported. When machines a and B are executing, they need to grab the lock. Whoever grabs the lock, who executes it, and who can't grab it, they don't have to execute it!

3. Treatment of lock

• Using locks in a single application: (single process multithreading)

synchronize distributed lock is a way to control the synchronous access of resources between distributed systems

Distributed lock is a way to control the synchronous sharing of resources between distributed systems

4. Implementation of distributed lock

• Data based optimistic lock to realize distributed lock
• Distributed lock based on zookeeper temporary node
• redis based distributed lock

5. redis distributed lock

• Acquire lock:

There are many options available in the set command to modify the behavior of the set command

redis 127.0.0.1:6379>SET KEY VALUE [EX seconds] [PX milliseconds] [NX|XX]    - EX seconds  Set the specified expiration time(In seconds)    - PX milliseconds Set the specified expiration time(Unit: ms)    - NX: Set keys only if they do not exist    - XX: Set only if the key already exists

Method 1: Promotion

    private static final String LOCK_SUCCESS = "OK";    private static final String SET_IF_NOT_EXIST = "NX";    private static final String SET_WITH_EXPIRE_TIME = "PX";        public static boolean getLock(JedisCluster jedisCluster, String lockKey, String requestId, int expireTime) {        // NX: ensure mutual exclusivity string result = jediscluster set(lockKey, requestId, SET_IF_NOT_EXIST, SET_WITH_EXPIRE_TIME, expireTime);         if (LOCK_SUCCESS.equals(result)) {            return true;        }         return false;    }

Mode 2:

public static boolean getLock(String lockKey,String requestId,int expireTime) {     Long result = jedis.setnx(lockKey, requestId);     if(result == 1) {         jedis.expire(lockKey, expireTime);         return true;     }     return false; }

Note: recommend mode 1, because setnx and expire in mode 2 are two operations, not an atomic operation. If setnx has a problem, it is a deadlock. Therefore, recommend mode 1

• Release lock:

Mode 1: del command implementation

public static void releaseLock(String lockKey,String requestId) {    if (requestId.equals(jedis.get(lockKey))) {        jedis.del(lockKey);    }}

Recommended method 2: redis+lua script implementation

public static boolean releaseLock(String lockKey, String requestId) {        String script = "if redis.call('get', KEYS[1]) == ARGV[1] then returnredis.call('del', KEYS[1]) else return 0 end";        Object result = jedis.eval(script, Collections.singletonList(lockKey),Collections.singletonList(requestId));        if (result.equals(1L)) {            return true;}        return false;    }

6. Distributed lock of zookeeper

6.1 principle of zookeeper to realize distributed lock

After understanding the principle of lock, you will find that Zookeeper is naturally the germ of a distributed lock.

First of all, each node of Zookeeper is a natural sequential signal generator.

When creating a child node under each node, as long as the selected creation type is ordered (temporary order or permanent order), an order number will be added after the new child node. This sequence number is the last generated sequence number plus one

For example, create a node "/ test/lock" for issuing signals, and then take it as the parent node. You can create a child node with the same prefix under the parent node. Assuming that the same prefix is "/ test/lock/seq -", when creating a child node, it also indicates that it is an ordered type. If it is the first child node created, the generated child node is / test/lock/seq-0000000000, the next node is / test/lock/seq-0000000001, and so on.

Secondly, the increment of Zookeeper node can specify that the one with the smallest node number can obtain the lock.

For a zookeeper distributed lock, first create a parent node, which should be a PERSISTENT node (PERSISTENT type) as far as possible, and then each thread that wants to obtain the lock will create a temporary sequential node under this node. Due to the increment of sequence number, the one with the smallest row number can be specified to obtain the lock. Therefore, before trying to occupy the lock, each thread first determines whether it is the current minimum or not. If so, it obtains the lock.

Third, the node monitoring mechanism of Zookeeper can ensure the orderly and efficient way of occupying locks.

Before each thread preempts the lock, it first grabs the number to create its own Znode. Similarly, when releasing the lock, you need to delete the number grabbing Znode. After the number scrambling is successful, if it is not the node with the smallest number, it will be in the state of waiting for notification. Waiting for whose notice? No one else is needed, just wait for the notice of the previous Znode. When a Znode is deleted, it's your turn to occupy the lock. The first informs the second, the second informs the third, beating the drum and passing flowers back in turn.

Zookeeper's node monitoring mechanism can be said to be very perfect to realize this kind of information transmission like beating drums and flowers. The specific method is that each Znode node waiting for notification only needs to monitor the node in front of linsten or watch, and the node immediately in front of itself. As long as the previous node is deleted, judge again to see if you are the node with the smallest sequence number. If so, you will obtain the lock.

Why is Zookeeper's node monitoring mechanism perfect?

One-stop head to tail, the back monitors the front, aren't you afraid of being cut off in the middle? For example, in a distributed environment, if the previous node cannot be successfully deleted by the program due to network reasons, server hangs up or other reasons, the latter node will wait forever?

In fact, the internal mechanism of Zookeeper can ensure that the following nodes can normally listen to delete and obtain locks. When creating a numbered node, try to create a temporary znode node instead of a permanent znode node. Once the client of the znode loses contact with the Zookeeper cluster server, the temporary znode will also be deleted automatically. The node behind it can also receive the deletion event and obtain the lock.

Zookeeper's node monitoring mechanism is perfect. There is another reason.

Zookeeper can avoid herding by connecting head to tail and monitoring the front. The so-called herding effect is that when each node hangs up, all nodes listen and respond, which will bring great pressure to the server. Therefore, with a temporary sequential node, when a node hangs up, only the node behind it will respond.

6.2 example of zookeeper implementing distributed lock

zookeeper implements distributed locking through temporary nodes

import org.apache.curator.RetryPolicy;import org.apache.curator.framework.CuratorFramework;import org.apache.curator.framework.CuratorFrameworkFactory;import org.apache.curator.framework.recipes.locks.InterProcessMutex;import org.apache.curator.retry.ExponentialBackoffRetry;import org.junit.Before;import org.junit.Test;/** * @ClassName ZookeeperLock * @Description TODO * @Author lingxiangxiang * @Date 2:57 PM * @Version 1.0 **/public class ZookeeperLock {    // Define shared resource private static int number = 10; Private static void printnumber() {/ / business logic: second kill System.out.println("******************************************* \ n"); System.out.println("current value:" + number); number --; try {thread. Sleep (2000);} catch (InterruptedException e) {            e.printStackTrace();        }         System.out.println("********** end of business method ***************** \ n");}// An error will be reported if @ Test is used here. public static void main(String[] args) {/ / define the side policy of retry. 1000 waiting time (MS) 10 times of retry. RetryPolicy policy = new ExponentialBackoffRetry(1000, 10); / / define the client of zookeeper. CuratorFramework client = CuratorFrameworkFactory.builder()                .connectString("10.231.128.95:2181,10.231.128.96:2181,10.231.128.97:2181")                .retryPolicy(policy)                .build();        //  Start the client start();        //  Define a lock in zookeeper. final InterProcessMutex lock = new InterProcessMutex(client, "/mylock")// Start is a thread for (int i = 0; I < 10; I + +) {new thread (New runnable() {@ override public void run() {try {/ / lock. Acquire(); printnumber();} obtained from the request catch (Exception e) {                        e.printStackTrace();                    }  Finally {/ / release the lock and try {lock. Release();} catch (Exception e) {                            e.printStackTrace();                        }                    }                }            }). start();        }    }}

7. Data based distributed lock

When discussing the use of distributed locks, we often first exclude the Database-based scheme, and instinctively feel that this scheme is not "advanced". From the perspective of performance, the performance of the database based scheme is indeed not excellent enough. The overall performance comparison: cache > zookeeper, etcd > database. Some people also put forward that the scheme based on database has many problems and is not reliable. The database scheme may not be suitable for frequent write operations

Let's take a look at the solutions based on database (MySQL), which are generally divided into three categories: table record based, optimistic lock and pessimistic lock.

7.1 record based on table

The simplest way to implement distributed locks is to create a lock table directly and then operate the data in the table. When we want to obtain the lock, we can add a record to the table. When we want to release the lock, we can delete this record.

For better demonstration, let's create a database table, as shown below:

CREATE TABLE `database_lock` (    `id` BIGINT NOT NULL AUTO_INCREMENT,    `resource` int NOT NULL COMMENT 'Locked resources',    `description` varchar(1024) NOT NULL DEFAULT "" COMMENT 'describe',    PRIMARY KEY (`id`),    UNIQUE KEY `uiq_idx_resource` (`resource`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='Database distributed lock table';

• Acquire lock

We can insert a piece of data:

INSERT INTO database_lock(resource, description) VALUES (1, 'lock');

Because the table database_ resource in lock is the only index, so when other requests are submitted to the database, an error will be reported, and the insertion will not succeed. Only one can be inserted When the insertion is successful, we get the lock

• Delete lock

INSERT INTO database_lock(resource, description) VALUES (1, 'lock');

This implementation method is very simple, but you should pay attention to the following points:

Once the lock cannot be released by other threads, the lock will fail to be released in the database. This defect is also easy to solve. For example, you can do a regular task to clean up regularly. The reliability of this lock depends on the database. It is recommended to set up a standby database to avoid single point and further improve reliability. This lock is non blocking, because an error will be reported directly after the failure of inserting data. If you want to obtain the lock, you need to operate again. If you need blocking, you can get a for loop or a while loop until INSERT succeeds. This lock is also non reentrant, because the same thread cannot obtain the lock again before releasing the lock, because the same record already exists in the database. To realize the reentrant lock, you can add some fields in the database, such as the host information and thread information of the lock. When you obtain the lock again, you can query the data first. If the current host information and thread information can be found, you can directly assign the lock to it.

7.2 optimistic lock

As the name suggests, the system believes that the update of data will not produce conflict in most cases, and the conflict detection of data will be carried out only when the database update operation is submitted. If the detection result is inconsistent with the expected data, the failure information is returned.

Optimistic locks are mostly implemented based on the recording mechanism of data version. What is the data version number? That is, add a version ID to the data. In the database table based version solution, you generally add a "version" field to the database table to read the data. When you take out the data, you will read the version number together, and then add 1 to this version number when you update it later. During the update process, the version number will be compared. If it is consistent and there is no change, the operation will be performed successfully; If the version numbers are inconsistent, the update fails.

In order to better understand the use of database optimistic lock in actual projects, here is an example of inventory that is often talked about in the industry.

An e-commerce platform will have an inventory of goods. When users buy, they will operate the inventory (inventory minus 1 means one has been sold). If only one user operates, the database itself can ensure the correctness of user operation, and some unexpected problems will occur in the case of Concurrency:

For example, when two users purchase a commodity at the same time, the actual operation at the database level should be to reduce the inventory by 2. However, due to the high concurrency, the first user reads the current inventory and reduces the inventory by 1 after completing the purchase, because this operation is not completely completed. The second user will purchase the same goods. At this time, the inventory found may be the inventory without subtracting 1 operation, which leads to the emergence of dirty data [thread unsafe operation]. Generally, if it is a single JVM, using the built-in lock in JAVA can ensure thread safety. If it is a multi JVM case, using the distributed lock can also achieve [later replenishment], This article focuses on the database level.

We are optimistic about the security of the above threads, and we usually do the same for the database:

select goods_num from goods where goods_name = "Little book";update goods set goods_num = goods_num -1 where goods_name = "Little book";

The above SQL is a group, and the current goods is usually queried first_ Num, and then goods_ The inventory is modified by subtracting 1 from num. in case of concurrency, this statement may cause a commodity with an original inventory of 3 to be purchased by two people, leaving 2 inventory, which will lead to oversold of commodities. So how is database optimistic lock implemented? First, define a version field to be used as a version number. Each operation will be like this:

select goods_num,version from goods where goods_name = "Little book";update goods set goods_num = goods_num -1,version =Inquired version Self increasing value where goods_name ="Little book" and version=Found out version；

In fact, optimistic locking can also be realized with the help of updated_at, which is similar to the way of using version field: the update operation front line obtains the current update time of the record, and when submitting the update, it is detected whether the current update time is equal to the update timestamp obtained at the beginning of the update.

7.3 lock

In addition to adding and deleting the records in the database table, we can also realize distributed locks with the help of the built-in locks in the database. Add FOR UPDATE after the query statement, and the database will add pessimistic lock, also known as exclusive lock, to the database table during the query. When a record is pessimistic locked, other threads can no longer add pessimistic locks to the line.

Pessimistic lock, contrary to optimistic lock, always assumes the worst case. It believes that the update of data will conflict in most cases.

While using pessimistic locks, we need to pay attention to the following lock levels. MySQL InnoDB causes that when locking, only those who explicitly specify the primary key (or index) will execute row locking (only lock the selected data), otherwise MySQL will execute table locking (lock the whole data form).

When using pessimistic lock, we must turn off the automatic submission property of MySQL database (refer to the following example), because MySQL uses autocommit mode by default, that is, when you perform an update operation, MySQL will submit the results immediately.

mysql> SET AUTOCOMMIT = 0;Query OK, 0 rows affected (0.00 sec)

In this way, after using FOR UPDATE to obtain the lock, you can execute the corresponding business logic, and then use COMMIT to release the lock.

We might as well follow the previous database_lock table to express the usage. Suppose A thread A needs to obtain A lock and perform corresponding operations, its specific steps are as follows:

STEP1 - acquire lock: SELECT * FROM database_lock WHERE id = 1 FOR UPDATE;.

STEP2 - execute business logic.

STEP3 - release lock: COMMIT.

last

Java core interview question bank of first-line Internet manufacturers

The specific steps are as follows:

STEP1 - acquire lock: SELECT * FROM database_lock WHERE id = 1 FOR UPDATE;.

STEP2 - execute business logic.

STEP3 - release lock: COMMIT.

last

Java core interview question bank of first-line Internet manufacturers

[external chain pictures are being transferred... (img-nwcx4j3z-162355525554)]

During the interview and job hopping season, I sorted out some real interview questions asked by big companies. Due to the length limit of the article, I only showed you some questions. More Java foundation, exception, collection, concurrent programming, JVM, Spring family bucket, MyBatis, Redis, database, middleware MQ, Dubbo, Linux, Tomcat, ZooKeeper, Netty, etc. have been sorted and uploaded on my website Tencent document [Java core interview question bank of first-line Internet manufacturers] can be obtained by clicking , and will continue to update... Interested friends can see and support a wave!