In daily development, there are many similar operations of reducing inventory, such as commodity inventory in e-commerce system, prize inventory in lottery system, etc.
Solution
- Using mysql database, use a field to store inventory, and update this field every time you deduct inventory.
- The database is still used, but the inventory is layered and stored in multiple records. When deducting the inventory, route it. This increases the concurrency, but it still can't avoid a lot of accessing the database to update the inventory.
- Put the inventory into redis and use the incrby feature of redis to deduct the inventory.
analysis
The first and second methods above are based on data to deduct inventory.
Single inventory based on Database
In the first way, all requests will wait for the lock here, and the lock can be obtained to deduct the inventory. It can be used when the concurrency is not high, but once the concurrency is large, a large number of requests will be blocked here, resulting in request timeout and then the whole system avalanche; Moreover, it will frequently access the database and occupy a lot of database resources, so this method is not applicable in the case of high concurrency.
Multi inventory based on Database
The second method is actually the optimized version of the first method, which improves the concurrency to a certain extent, but it will still update the database a lot and occupy a lot of database resources.
There are still some problems in reducing inventory based on database:
- In the way of deducting inventory from the database, the operation of deducting inventory must be executed in one statement, and selec cannot be update d first. In this way, over deduction will occur in case of concurrency. For example:
update number set x=x-1 where x > 0
- MySQL itself will have problems with the processing performance of high concurrency. Generally speaking, the processing performance of MySQL will rise with the rise of concurrent threads, but there will be an obvious inflection point after reaching a certain degree of concurrency, and then it will decline all the way, and finally it will be even worse than the performance of a single thread.
- When inventory reduction and high concurrency are encountered together, as the number of operating inventory is in the same line, there will be the problem of competing for InnoDB line lock, resulting in mutual waiting and even deadlock, which greatly reduces the processing performance of MySQL and eventually leads to timeout exception on the front-end page. Source: Code farmer Programming Advanced notes
redis based
To solve the above problems, we have a third solution. We put the inventory into the cache and reduce the inventory by using the incrby feature of redis, which solves the problems of over deduction and performance. However, once the cache is lost, the recovery scheme needs to be considered. For example, when the lottery system deducts the prize inventory, the initial inventory = the total inventory - the number of awards that have been issued. However, if the prize is issued asynchronously, you need to wait until the MQ message is consumed before restarting redis to initialize the inventory. Otherwise, there is also a problem of inconsistent inventory.
Implementation of inventory deduction based on redis
- We use the lua script of redis to reduce inventory
- In the distributed environment, a distributed lock is needed to control that only one service can initialize the inventory
- A callback function needs to be provided. When initializing inventory, call this function to obtain the initialization inventory
Initialize inventory callback function (IStockCallback)
/**
* Get inventory callback
* @author yuhao.wang
*/
public interface IStockCallback {
/**
* Get inventory
* @return
*/
int getStock();
}
Stock service deduction
/**
* Deduct inventory
*
* @author yuhao.wang
*/
@Service
public class StockService {
Logger logger = LoggerFactory.getLogger(StockService.class);
/**
* Unlimited inventory
*/
public static final long UNINITIALIZED_STOCK = -3L;
/**
* Redis client
*/
@Autowired
private RedisTemplate<String, Object> redisTemplate;
/**
* Execute stock deduction script
*/
public static final String STOCK_LUA;
static {
/**
*
* @desc Deduction inventory Lua script
* Stock - 1: indicates unlimited stock
* Stock 0: indicates that there is no stock
* Stock > 0: indicates the remaining stock
*
* @params Inventory key
* @return
* -3:Inventory not initialized
* -2:Insufficient inventory
* -1:Unlimited inventory
* Greater than or equal to 0: remaining inventory (inventory remaining after deduction)
* redis The cached Inventory (value) is - 1, which means unlimited inventory. 1 is returned directly
*/
StringBuilder sb = new StringBuilder();
sb.append("if (redis.call('exists', KEYS[1]) == 1) then");
sb.append(" local stock = tonumber(redis.call('get', KEYS[1]));");
sb.append(" local num = tonumber(ARGV[1]);");
sb.append(" if (stock == -1) then");
sb.append(" return -1;");
sb.append(" end;");
sb.append(" if (stock >= num) then");
sb.append(" return redis.call('incrby', KEYS[1], 0 - num);");
sb.append(" end;");
sb.append(" return -2;");
sb.append("end;");
sb.append("return -3;");
STOCK_LUA = sb.toString();
}
/**
* @param key Inventory key
* @param expire Inventory effective time, in seconds
* @param num Deduction quantity
* @param stockCallback Initialize inventory callback function
* @return -2:Insufficient inventory- 1: Unlimited inventory; Greater than or equal to 0: remaining inventory after deducting inventory
*/
public long stock(String key, long expire, int num, IStockCallback stockCallback) {
long stock = stock(key, num);
// Initialize inventory
if (stock == UNINITIALIZED_STOCK) {
RedisLock redisLock = new RedisLock(redisTemplate, key);
try {
// Acquire lock
if (redisLock.tryLock()) {
// Double verification to avoid repeated back to the source database during concurrency
stock = stock(key, num);
if (stock == UNINITIALIZED_STOCK) {
// Get initialization inventory
final int initStock = stockCallback.getStock();
// Set inventory to redis
redisTemplate.opsForValue().set(key, initStock, expire, TimeUnit.SECONDS);
// Adjust and deduct inventory at one time
stock = stock(key, num);
}
}
} catch (Exception e) {
logger.error(e.getMessage(), e);
} finally {
redisLock.unlock();
}
}
return stock;
}
/**
* Add Inventory (restore inventory)
*
* @param key Inventory key
* @param num Inventory quantity
* @return
*/
public long addStock(String key, int num) {
return addStock(key, null, num);
}
/**
* Add inventory
*Source: Code farmer Programming Advanced notes
* @param key Inventory key
* @param expire Expiration time (seconds)
* @param num Inventory quantity
* @return
*/
public long addStock(String key, Long expire, int num) {
boolean hasKey = redisTemplate.hasKey(key);
// Judge whether the key exists and update it directly
if (hasKey) {
return redisTemplate.opsForValue().increment(key, num);
}
Assert.notNull(expire,"Failed to initialize inventory. Inventory expiration time cannot be null");
RedisLock redisLock = new RedisLock(redisTemplate, key);
try {
if (redisLock.tryLock()) {
// After obtaining the lock, judge whether there is a key again
hasKey = redisTemplate.hasKey(key);
if (!hasKey) {
// Initialize inventory
redisTemplate.opsForValue().set(key, num, expire, TimeUnit.SECONDS);
}
}
} catch (Exception e) {
logger.error(e.getMessage(), e);
} finally {
redisLock.unlock();
}
return num;
}
/**
* Get inventory
*
* @param key Inventory key
* @return -1:Unlimited inventory; Greater than or equal to 0: remaining inventory
*/
public int getStock(String key) {
Integer stock = (Integer) redisTemplate.opsForValue().get(key);
return stock == null ? -1 : stock;
}
/**
* Deduct inventory
*Advanced Programming Notes: param * @
* @param num Deduct inventory quantity
* @return Remaining inventory after deduction [- 3: uninitialized inventory; - 2: insufficient inventory; - 1: unlimited inventory; greater than or equal to 0: remaining inventory after deducting inventory]
*/
private Long stock(String key, int num) {
// KEYS parameter in script
List<String> keys = new ArrayList<>();
keys.add(key);
// ARGV parameter in script
List<String> args = new ArrayList<>();
args.add(Integer.toString(num));
long result = redisTemplate.execute(new RedisCallback<Long>() {
@Override
public Long doInRedis(RedisConnection connection) throws DataAccessException {
Object nativeConnection = connection.getNativeConnection();
// Although cluster mode and stand-alone mode execute scripts in the same way, they have no common interface, so they can only be executed separately
// Cluster mode
if (nativeConnection instanceof JedisCluster) {
return (Long) ((JedisCluster) nativeConnection).eval(STOCK_LUA, keys, args);
}
// standalone mode
else if (nativeConnection instanceof Jedis) {
return (Long) ((Jedis) nativeConnection).eval(STOCK_LUA, keys, args);
}
return UNINITIALIZED_STOCK;
}
});
return result;
}
}
call
/**
* @author yuhao.wang
*/
@RestController
public class StockController {
@Autowired
private StockService stockService;
@RequestMapping(value = "stock", produces = MediaType.APPLICATION_JSON_UTF8_VALUE)
public Object stock() {
// Item ID
long commodityId = 1;
// Inventory ID
String redisKey = "redis_key:stock:" + commodityId;
long stock = stockService.stock(redisKey, 60 * 60, 2, () -> initStock(commodityId));
return stock >= 0;
}
/**
* Get initial inventory
*
* @return
*/
private int initStock(long commodityId) {
// TODO does some operations to initialize inventory here
return 1000;
}
@RequestMapping(value = "getStock", produces = MediaType.APPLICATION_JSON_UTF8_VALUE)
public Object getStock() {
// Item ID
long commodityId = 1;
// Inventory ID
String redisKey = "redis_key:stock:" + commodityId;
return stockService.getStock(redisKey);
}
@RequestMapping(value = "addStock", produces = MediaType.APPLICATION_JSON_UTF8_VALUE)
public Object addStock() {
// Item ID
long commodityId = 2;
// Inventory ID
String redisKey = "redis_key:stock:" + commodityId;
return stockService.addStock(redisKey, 2);
}
}