The leaf segment mode is an optimization to directly use the self increment ID of the database as the distributed ID to reduce the frequency of database operation. It is equivalent to obtaining self incrementing IDs from the database in batches. One number segment range is taken out from the database each time. For example, (11000] represents 1000 IDs. The business service will locally generate 1 ~ 1000 self incrementing IDs from the number segment and load them into memory.

The general process is shown in the figure below:

After the number segment is exhausted, go to the database to obtain a new number segment, which can greatly reduce the pressure of the database. Yes, max_ Update the ID field once, update max_id= max_ If Id + step and update are successful, the new number segment is successful. The range of the new number segment is (max)_ id ,max_id +step].

Due to the dependence on the database, we first design the following table structure:

CREATE TABLE `leaf_alloc` ( 
`biz_tag` varchar(128) NOT NULL DEFAULT '' COMMENT 'business key', 
 `max_id` bigint(20) NOT NULL DEFAULT '1' COMMENT 'Currently assigned maximum id',
  `step` int(11) NOT NULL COMMENT 'The initial step size is also the minimum step size for dynamic adjustment',  
`description` varchar(256) DEFAULT NULL COMMENT 'business key Description of',
  `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT 'Update time of database maintenance',  PRIMARY KEY (`biz_tag`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

Insert a test piece of business data in advance

INSERT INTO `leaf_alloc` (`biz_tag`, `max_id`, `step`, `description`, `update_time`) VALUES ('leaf-segment-test', '0', '10', 'test', '2020-02-28 10:41:03');

• biz_tag: biz is used for different business requirements_ The tag field is used for isolation. If you need to expand the capacity in the future, you only need to perform biz_tag sub database and sub table

• max_id: the maximum value of the current business number segment. It is used to calculate the next business number segment

• Step: step size, that is, the number of ID S obtained each time

• Description: there is nothing to say about the business description

Download the leaf project locally: https://github.com/Meituan-Dianping/Leaf

Modify the leaf.properties file in the project to add the database configuration

leaf.name=com.sankuai.leaf.opensource.test
leaf.segment.enable=true
leaf.jdbc.url=jdbc:mysql://127.0.0.1:3306/xin-master?useUnicode=true&characterEncoding=utf8leaf.jdbc.username=junkang
leaf.jdbc.password=junkang
leaf.snowflake.enable=false

Note: leaf.snowflake.enable and leaf.segment.enable cannot be started at the same time, otherwise the project cannot be started.

The configuration is quite simple. It's OK after directly starting the LeafServerApplication. Next, test it. leaf is a signaling service based on Http requests. There are only two methods in the LeafController, one is the number segment interface and the other is the snowflake interface. The key is the business biz pre inserted in the database_ tag.

@RestController
public class LeafController {
    private Logger logger = LoggerFactory.getLogger(LeafController.class);

    @Autowired
    private SegmentService segmentService;
    @Autowired
    private SnowflakeService snowflakeService;

    /**
     * Segment mode
     * @param key
     * @return
     */
    @RequestMapping(value = "/api/segment/get/{key}")
    public String getSegmentId(@PathVariable("key") String key) {
        return get(key, segmentService.getId(key));
    }

    /**
     * Snowflake algorithm mode
     * @param key
     * @return
     */
    @RequestMapping(value = "/api/snowflake/get/{key}")
    public String getSnowflakeId(@PathVariable("key") String key) {
        return get(key, snowflakeService.getId(key));
    }

    private String get(@PathVariable("key") String key, Result id) {
        Result result;
        if (key == null || key.isEmpty()) {
            throw new NoKeyException();
        }
        result = id;
        if (result.getStatus().equals(Status.EXCEPTION)) {
            throw new LeafServerException(result.toString());
        }
        return String.valueOf(result.getId());
    }
}

visit: http://127.0.0.1:8080/api/segment/get/leaf -Segment test, the result returned normally, I felt it was ok, but when I checked the data in the database table, I found a problem.

Usually, when using the number segment mode, the time to get the number segment is when the previous number segment is consumed, but just now I got an ID, but Max has been updated in the database_ ID, that is to say, leaf has acquired one more number segment. What kind of operation is this?

Why is Leaf designed like this?

Leaf hopes to achieve no blocking in the process of getting number segments in DB!

When the number segment is exhausted, go to the DB to remove a number segment. If the network jitters or the DB has a slow query, and the business system cannot get the number segment, the response time of the whole system will slow down. This is intolerable for businesses with huge traffic.

Therefore, when the current segment is consumed to a certain point, Leaf asynchronously loads the next segment into memory. Instead of waiting until the number segment is exhausted to update the number segment. This greatly reduces the risk of the system.

So when exactly is a certain point?

An experiment is done here. The length of the segment is set as step=10, max_id=1

When I took the first ID, I saw that the number segment increased by 1 / 10

When I take the third Id, I see that the number segment has increased again, 3 / 10

Leaf adopts double buffer mode, and its service has two segment buffer segments. When the current number segment has consumed 10%, and the next number segment has not been obtained, another update thread will be started to update the next number segment.

In short, Leaf guarantees that it will always cache two more number segments. Even if the database hangs at any time, it will ensure that the sending service can work normally for a period of time.

Generally, the length of the recommended segment is set to 600 times (10 minutes) of the QPS issued during the peak service period, so that even if the DB goes down, the Leaf can continue to issue numbers for 10-20 minutes without being affected.

advantage:

• The Leaf service can be easily linearly expanded, and its performance can fully support most business scenarios.
• High disaster tolerance: the Leaf service has an internal number segment cache. Even if the DB goes down, the Leaf can still provide external services normally in a short time.

If the DB goes down, the Leaf can still issue signals for 10-20 minutes without being affected.

advantage:

• The Leaf service can be easily linearly expanded, and its performance can fully support most business scenarios.
• High disaster tolerance: the Leaf service has an internal number segment cache. Even if the DB goes down, the Leaf can still provide external services normally in a short time.