Database sub database sub table - distributed global Id required

Posted by sBForum on Fri, 24 Dec 2021 02:43:17 +0100

In the project development, with the continuous development of business, the data increases day by day. At this time, the database and table splitting operation of the data table will appear. Once the database and table are divided. The traditional ID loses its meaning. Therefore, a distributed global ID is required.

Characteristics of distributed global ID

1: Global uniqueness. Duplicate ID s are not allowed
2: Monotonically increasing to ensure that the next ID must be greater than the previous ID
3: The range trend is increasing. Within a time period, the generated ID tends to increase. For example, 202012120001 202012120002
The next day, count from 1. 202012130001 202012130002……
4: Security. In different fields, some of our businesses do not have continuous increment, which can well protect the data format and form. Because it is easy for competitors to obtain data.

2 and 4 above are mutually exclusive. Cannot be satisfied at the same time.

Implementation mode

UUID
Redis
Twitter snowflake algorithm
Meituan's Leaf algorithm

redis implements distributed globally unique ID s

The first is the concrete implementation of monotonic increasing:

In my opinion, redisAtomicLong has some limitations in doing distributed ID. It is a solution for redisAtomicLong data loss. If you have any good suggestions, we look forward to your continued sharing
Idea: call incr command
1: The incr command of Redis has the atomic operation of incr and get, that is, it adds the atomic operation of the result. This atomic operation is very convenient to realize the unique global distributed ID.
2: Redis itself is a single thread architecture. The INCR command does not have duplicate ID s

code:

1: Use the incr command of redis to add id from 1
2: Massive data storage is generally stored by database and table. Generally, there are 1024 tables at most. Generally, tables are created, such as product_0，product_1，product_2……product_1023 tables.

service

package com.kuangstudy.service.creatorid;
import lombok.extern.log4j.Log4j2;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.support.atomic.RedisAtomicLong;
import org.springframework.stereotype.Service;

@Service
@Log4j2
public class RedisCreatorIdService {
    @Autowired
    private RedisTemplate redisTemplate;
    /**
     * @Description Generate global id
     * @Date 18:50 2021/5/20
     **/
    public Long incrmentid() {
        RedisAtomicLong entityIdCounter = new RedisAtomicLong("id:creator:product", redisTemplate.getConnectionFactory());//0
        // Counter accumulation 
        Long increment = entityIdCounter.incrementAndGet();
		if (null == incr || incr.longValue() == 0) {// Initial set expiration time
				// Set expiration time 24 hours
				entityIdCounter.expire(24, TimeUnit.HOURS);
				// The second incr here starts from 1. By default, it starts from 0
				incr = entityIdCounter.getAndIncrement();
			}
			log.debug(keyName + "==========" + incr);
			return incr;

        // perhaps
        //Long increment = redisTemplate.opsForValue().increment(key);
        //return increment;
    }
}

controller

package com.kuangstudy.controller.creatorid;
import com.kuangstudy.entity.Product;
import com.kuangstudy.service.creatorid.RedisCreatorIdService;
import com.kuangstudy.vo.R;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class ProductController {
    @Autowired
    private RedisCreatorIdService redisCreatorIdService;
    @PostMapping("/product/creator")
    public R creatorIds() {
        // 1: Create a distributed global ID for the product
        Long productid = redisCreatorIdService.incrmentid();
        // 2: Create product
        Product product = new Product();
        product.setId(productid);
        product.setTitle("title:" + productid);
        System.out.println("The saved products are:" + product.toString());
        // 3: Store in the specified database
        int table = (int) (productid % 1024);
        System.out.println("The database tables saved to are: product_" + table);
        return R.ok().data("tablename", "product_" + table);
    }
}

Distributed global id based on range rule increment

For example, the system needs to generate a serial number generated according to the business type. It will be generated from 1 every day, and will be cleared the next day to continue from 0. The format of the serial number is: bizCode + date + incr, such as TT-201711230000100.

Idea:
1. Redis Incr is used to generate the serial number, and the date plus business code is used as the combined Key, so as to ensure that the serial number generated the next day starts from 1.
2. Because our business volume is not very large, we first judge whether the current key exists before generating the serial number. If it does not exist, set the expiration time of the key to 23:59:59 that night to avoid generating many expired keys.

service

package com.itheima.itheimadistributelock.redis;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.support.atomic.RedisAtomicLong;
import org.springframework.stereotype.Component;
import java.util.concurrent.TimeUnit;
@Component
public class CacheService {
    //There are other template s here, so the name doesn't look good
    @Autowired
    RedisTemplate redisTemplate;
    public Long getIncrementNum(String key) {
        // There are no key value pairs ready to create
        RedisAtomicLong entityIdCounter = new RedisAtomicLong(key, redisTemplate.getConnectionFactory());//0
        // Counter accumulation 
        Long counter = entityIdCounter.incrementAndGet();
        System.out.println("=========================>"+ counter);
        if ((null == counter || counter.longValue() == 1)) {// Initial set expiration time
            System.out.println("Set the expiration time to 1 day!");
            // The purpose of setting clearing is to make the counter start from 0 every day
            entityIdCounter.expire(1, TimeUnit.DAYS);// Unit day
        }
        return counter;
    }
}

utils

package com.kuangstudy.service.creatorid;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import java.text.SimpleDateFormat;
import java.util.Date;
/**
 * @Author Xu Ke
 * @Description Unified order serial number
 * @Date 19:29 2021/5/20
 * @Param
 * @return
 **/
@Component
public class SequenceUtils {
    @Autowired
    CacheService cacheService;
    // 1: The number of orders of the company every day, if it is several hundred, write three thousand, it will be more than forty-five thousand
    static final int DEFAULT_LENGTH = 3;
    public String getSequence() {
        // 1: Time prefix
        String currentDate = new SimpleDateFormat("yyyyMMdd").format(new Date());
        // 2: redis gets the daily increment quantity
        Long num = cacheService.getIncrementNum("id:generator:order:" + currentDate);
        // 3: Gets the increment length. Is it less than default_ If length is less than, fill in zero in front. If it is greater than, it can be incremented
        String str = String.valueOf(num);
        // 4: Zero filling operation
        int len = str.length();
        // 4-1: is it less than default_ If length is less than, fill in zero in front. If it is greater than, it can be incremented
        StringBuilder sb = new StringBuilder();
        // 5: Add time
        sb.append(currentDate);
        if (len >= DEFAULT_LENGTH) {
            sb.append(str);
            return sb.toString();
        }
        int rest = DEFAULT_LENGTH - len;
        for (int i = 0; i < rest; i++) {
            sb.append('0');
        }
        sb.append(str);
        // 6: Time + zero filling operation returns the order serial number
        return sb.toString();
    }
}