Author: Little Fu
Blog: https://bugstack.cn

Sediment, sharing, growth, so that you and others can gain something!_

• Distributed task DcsSchedule middleware, Github address: https://github.com/fuzhengwei/schedule-spring-boot-starter
• Distributed Task DcsSchedule Console, Github Address: https://github.com/fuzhengwei/itstack-middleware-control
• Demo Video

Preface

@SpringBootApplication
@EnableScheduling
public class Application{
    public static void mian(String[] args){
        SpringApplication.run(Application.class,args);
    }
    
    @Scheduled(cron = "0/3 * * * * *")
    public void demoTask() {
        //...
    }
}

Click, you're familiar with the above code. It's SpringBoot's Schedule timer task. It's easy to use.Schedule is almost always used in our development when we need to do some timing or specify a time loop to execute logic.

However, if our tasks are relatively large, such as: fixed-time batch T+1 settlement, change of commodity status before second killing, refresh data preheating to cache, and so on, these fixed-time tasks have the same characteristics; large workload, real-time, high availability.At this point, Schedule alone is not enough to control.

Then our product needs come out, distributed DcsSchedule tasks;

1. Multi-machine deployment tasks
2. Unified Control Center Start and Stop
3. Downtime Preparedness, Autostart Execution
4. Real-time detection of task execution information: number of deployments, total tasks, number of successes, number of failures, execution time, etc.

Hmm?Someone has been talking for a while about using Quertz, but that's not the point of this article.Don't you want to see how a self-explanatory open source middleware was born and how it was pushed to the central Maven store?Like the picture below; it smells really bad!

Home page monitoring

task list

_Okay, let's start with how to use and develop this middleware!

Middleware use

1. Version Records

	Edition	Release date	Remarks
1	1.0.0-RELEASE	2019-12-07	Basic function implementation; Task access, distributed start-stop
2	1.0.1-RELEASE	2019-12-07	Upload Test Version

2. Environmental preparation

1. jdk1.8
2. StringBoot 2.x

3. Configuration Center zookeeper 3.4.14 {Prepare the zookeeper service, if windows debugging can be downloaded from here: https://www-eu.apache.org/dis...}

   1. Unzip after download, create folder data, logs in bin sibling path

   2. Modify conf/zoo.cfg, modify the configuration as follows;

      dataDir=D:\\Program Files\\apache-zookeeper-3.4.14\\data
      dataLogDir=D:\\Program Files\\apache-zookeeper-3.4.14\\logs

4. Packaged Deployment Control Platform

   1. Download address: https://github.com/fuzhengwei...
   2. Deployment access: http://localhost:7397

3. Configure POM

<dependency>
    <groupId>org.itstack.middleware</groupId>
    <artifactId>schedule-spring-boot-starter</artifactId>
    <version>1.0.0-RELEASE</version>
</dependency>

4. Introduce distributed task DcsSchedule @EnableDcsScheduling

1. Much like SpringBoot's Ceduling, his comment is: @EnableScheduling, make it as easy as possible to use
2. This comment is intended to provide an entry for our own middleware, as well(viii)That's what the pickup source finds {The code I've been saying is fussy}

@SpringBootApplication
@EnableDcsScheduling
public class HelloWorldApplication {

    public static void main(String[] args) {
        SpringApplication.run(HelloWorldApplication.class, args);
    }

}

5. Add notes to the task method

1. This comment is also similar to Schedule in SpringBoot, but with more desc descriptions and start-stop initialization controls
2. cron: execution plan
3. desc: Task description
4. autoStartup: Default startup state
5. If your task requires parameters, you can do so by introducing a service to call

@Component("demoTaskThree")
public class DemoTaskThree {
    
    @DcsScheduled(cron = "0 0 9,13 * * *", desc = "03 Timed Task Execution Test: taskMethod01", autoStartup = false)
    public void taskMethod01() {
        System.out.println("03 Timed Task Execution Test: taskMethod01");
    }

    @DcsScheduled(cron = "0 0/30 8-10 * * *", desc = "03 Timed Task Execution Test: taskMethod02", autoStartup = false)
    public void taskMethod02() {
        System.out.println("03 Timed Task Execution Test: taskMethod02");
    }

}

6. Start verification

1. Start the SpringBoot project, and autoStartup = true will automatically start the task (the task is multithreaded and executed in parallel)
2. Launch Control Platform: itstack-middleware-control, access: http://localhost 7397/Success interface as follows; can turn on/off verification!{Functionality is still in progress}
   

Middleware Development

It's also my first time to develop a middleware based on SpringBoot, since it's only about a month since I've been exposed to SpringBoot.Although SpringBoot has been around for a long time, it has always relied on no contact because our project development doesn't use a SpringBoot suite of things.Not until last month when I started thinking about domain-driven design, huh!Really good, then start to consolidate skills, learn ideas and use them in the project.

The model in my mind already exists to develop middleware for such a distributed task as my product needs.The other is to explore the knowledge tools that I need in the development process, including simply;

1. Read Yml Custom Configuration
2. Use zookeeper as the configuration center so that if a machine is down, it can be monitored through the temporary node
3. Through the Spring class; ApplicationContextAware, BeanPostProcessor, ApplicationListener, perform service startup, annotation scanning, node hanging
4. Distributed Task Unified Console to manage tasks

1. Engineering Models

schedule-spring-boot-starter
└── src
    ├── main
    │   ├── java
    │   │   └── org.itstack.middleware.schedule
    │   │       ├── annotation
    │   │       │    ├── DcsScheduled.java    
    │   │       │    └── EnableDcsScheduling.java
    │   │       ├── annotation    
    │   │       │    └── InstructStatus.java    
    │   │       ├── config
    │   │       │    ├── DcsSchedulingConfiguration.java    
    │   │       │    ├── StarterAutoConfig.java    
    │   │       │    └── StarterServiceProperties.java    
    │   │       ├── domain
    │   │       │    ├── DataCollect.java    
    │   │       │    ├── DcsScheduleInfo.java    
    │   │       │    ├── DcsServerNode.java    
    │   │       │    ├── ExecOrder.java    
    │   │       │    └── Instruct.java
    │   │       ├── export    
    │   │       │    └── DcsScheduleResource.java
    │   │       ├── service
    │   │       │    ├── HeartbeatService.java    
    │   │       │    └── ZkCuratorServer.java
    │   │       ├── task
    │   │       │    ├── TaskScheduler.java    
    │   │       │    ├── ScheduledTask.java    
    │   │       │    ├── SchedulingConfig.java    
    │   │       │    └── SchedulingRunnable.java    
    │   │       ├── util
    │   │       │    └── StrUtil.java    
    │   │       └── DoJoinPoint.java
    │   └── resources    
    │       └── META_INF
    │           └── spring.factories    
    └── test
        └── java
            └── org.itstack.demo.test
                └── ApiTest.java

2. Code explanation

1. Long, explaining only some of the key code blocks. If you'd like to participate in open source writing, you can apply with me
2. I said good code is fussy, so start with this part

2.1 Custom Notes

Annotation/EnableDcs Scheduling.java&Custom Notes

This annotates a bunch of Circle A configurations to start executing our middleware;

• Target Identity needs to be executed on a class
• Retention comments are recorded by the compiler in the class file and are retained by the VM at run time so they can be read reflectively
• Import introduces entry resources and executes into its own defined classes at program startup for our convenience; initializes configuration/services, starts tasks, and hangs on nodes
• ComponentScan tells the program where to scan

@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Import({DcsSchedulingConfiguration.class})
@ImportAutoConfiguration({SchedulingConfig.class, CronTaskRegister.class, DoJoinPoint.class})
@ComponentScan("org.itstack.middleware.*")
public @interface EnableDcsScheduling {
}

2.2 Scan for custom annotations, initialize configuration/services, start tasks, hang on nodes

Config/DcsSchEdulingConfiguration.java&Initialize configuration/services, start tasks, hang on nodes

• At this point, our custom notes are available, and we've written the methods, so how can we get them?
• Need to be implemented by BeanPostProcessor.postProcessAfterInitialization Scan each bean as it is instantiated
• There's an interesting problem here, one method gets twice, because there's a CGLIB for the agent, just like the real Monkey King, almost a hair.(viii)Scrape the source code to see, life notes notes notes are not.That's fine!method.getDeclaredAnnotations()
• We aggregate the scanned task information into Map and execute our middleware content when Spring initialization is complete.{Too early execution is a bit noisy!The main family won't let it. Throw you an exception.}

@Override
public Object postProcessAfterInitialization(Object bean, String beanName) throws BeansException {
    Class<?> targetClass = AopProxyUtils.ultimateTargetClass(bean);
    if (this.nonAnnotatedClasses.contains(targetClass)) return bean;
    Method[] methods = ReflectionUtils.getAllDeclaredMethods(bean.getClass());
    if (methods == null) return bean;
    for (Method method : methods) {
        DcsScheduled dcsScheduled = AnnotationUtils.findAnnotation(method, DcsScheduled.class);
        if (null == dcsScheduled || 0 == method.getDeclaredAnnotations().length) continue;
        List<ExecOrder> execOrderList = Constants.execOrderMap.computeIfAbsent(beanName, k -> new ArrayList<>());
        ExecOrder execOrder = new ExecOrder();
        execOrder.setBean(bean);
        execOrder.setBeanName(beanName);
        execOrder.setMethodName(method.getName());
        execOrder.setDesc(dcsScheduled.desc());
        execOrder.setCron(dcsScheduled.cron());
        execOrder.setAutoStartup(dcsScheduled.autoStartup());
        execOrderList.add(execOrder);
        this.nonAnnotatedClasses.add(targetClass);
    }
    return bean;
}

• Initialize service connection zookeeper configuration center
• Connecting creates our nodes and adds listeners, which are responsible for distributed message notifications and receives notifications to control task start and stop
• This includes cyclic creation of nodes and bulk node deletion, it seems!Interview Questions(viii)

private void init_server(ApplicationContext applicationContext) {
    try {
        //Get zk connection
        CuratorFramework client = ZkCuratorServer.getClient(Constants.Global.zkAddress);
        //Node assembly
        path_root_server = StrUtil.joinStr(path_root, LINE, "server", LINE, schedulerServerId);
        path_root_server_ip = StrUtil.joinStr(path_root_server, LINE, "ip", LINE, Constants.Global.ip);
        //Create Node & Recursively delete old content under this service IP
        ZkCuratorServer.deletingChildrenIfNeeded(client, path_root_server_ip);
        ZkCuratorServer.createNode(client, path_root_server_ip);
        ZkCuratorServer.setData(client, path_root_server, schedulerServerName);
        //Add Node & Listen
        ZkCuratorServer.createNodeSimple(client, Constants.Global.path_root_exec);
        ZkCuratorServer.addTreeCacheListener(applicationContext, client, Constants.Global.path_root_exec);
    } catch (Exception e) {
        logger.error("itstack middleware schedule init server error!", e);
        throw new RuntimeException(e);
    }
}

• Start the Schedule task marked True
• Scheduled defaults to single-threaded execution, which is expanded to multithreaded parallel execution

private void init_task(ApplicationContext applicationContext) {
    CronTaskRegister cronTaskRegistrar = applicationContext.getBean("itstack-middlware-schedule-cronTaskRegister", CronTaskRegister.class);
    Set<String> beanNames = Constants.execOrderMap.keySet();
    for (String beanName : beanNames) {
        List<ExecOrder> execOrderList = Constants.execOrderMap.get(beanName);
        for (ExecOrder execOrder : execOrderList) {
            if (!execOrder.getAutoStartup()) continue;
            SchedulingRunnable task = new SchedulingRunnable(execOrder.getBean(), execOrder.getBeanName(), execOrder.getMethodName());
            cronTaskRegistrar.addCronTask(task, execOrder.getCron());
        }
    }
}

• Hang on Task Node To zookeeper Hang On
• Depending on the scene, some content is suspended to the virtual machine node.{(viii)Another interview question, how do virtual node data hang, create permanent nodes, and how do virtual values add?}
• path_root_server_ip_clazz_method; this structure is: root directory, service, IP, class, method

private void init_node() throws Exception {
    Set<String> beanNames = Constants.execOrderMap.keySet();
    for (String beanName : beanNames) {
        List<ExecOrder> execOrderList = Constants.execOrderMap.get(beanName);
        for (ExecOrder execOrder : execOrderList) {
            String path_root_server_ip_clazz = StrUtil.joinStr(path_root_server_ip, LINE, "clazz", LINE, execOrder.getBeanName());
            String path_root_server_ip_clazz_method = StrUtil.joinStr(path_root_server_ip_clazz, LINE, "method", LINE, execOrder.getMethodName());
            String path_root_server_ip_clazz_method_status = StrUtil.joinStr(path_root_server_ip_clazz, LINE, "method", LINE, execOrder.getMethodName(), "/status");
            //Add Node
            ZkCuratorServer.createNodeSimple(client, path_root_server_ip_clazz);
            ZkCuratorServer.createNodeSimple(client, path_root_server_ip_clazz_method);
            ZkCuratorServer.createNodeSimple(client, path_root_server_ip_clazz_method_status);
            //Add Node Data [Temporary]
            ZkCuratorServer.appendPersistentData(client, path_root_server_ip_clazz_method + "/value", JSON.toJSONString(execOrder));
            //Add Node Data [Permanent]
            ZkCuratorServer.setData(client, path_root_server_ip_clazz_method_status, execOrder.getAutoStartup() ? "1" : "0");
        }
    }
}

2.3 zookeeper Control Service

Service/ZkCuratorServer.java& ZK Service

• Here is a collection of zk methods, the more important of which is adding listening
• zookeeper has a feature that it receives notifications when the content of the node changes after listening, and downtime is also received, which is the core trigger point for us to develop disaster preparedness later on.

public static void addTreeCacheListener(final ApplicationContext applicationContext, final CuratorFramework client, String path) throws Exception {
    TreeCache treeCache = new TreeCache(client, path);
    treeCache.start();
    treeCache.getListenable().addListener((curatorFramework, event) -> {
        //...
        switch (event.getType()) {
            case NODE_ADDED:
            case NODE_UPDATED:
                if (Constants.Global.ip.equals(instruct.getIp()) && Constants.Global.schedulerServerId.equals(instruct.getSchedulerServerId())) {
                    //Execute Command
                    Integer status = instruct.getStatus();
                    switch (status) {
                        case 0: //Stop Task
                            cronTaskRegistrar.removeCronTask(instruct.getBeanName() + "_" + instruct.getMethodName());
                            setData(client, path_root_server_ip_clazz_method_status, "0");
                            logger.info("itstack middleware schedule task stop {} {}", instruct.getBeanName(), instruct.getMethodName());
                            break;
                        case 1: //Start Task
                            cronTaskRegistrar.addCronTask(new SchedulingRunnable(scheduleBean, instruct.getBeanName(), instruct.getMethodName()), instruct.getCron());
                            setData(client, path_root_server_ip_clazz_method_status, "1");
                            logger.info("itstack middleware schedule task start {} {}", instruct.getBeanName(), instruct.getMethodName());
                            break;
                        case 2: //Refresh Task
                            cronTaskRegistrar.removeCronTask(instruct.getBeanName() + "_" + instruct.getMethodName());
                            cronTaskRegistrar.addCronTask(new SchedulingRunnable(scheduleBean, instruct.getBeanName(), instruct.getMethodName()), instruct.getCron());
                            setData(client, path_root_server_ip_clazz_method_status, "1");
                            logger.info("itstack middleware schedule task refresh {} {}", instruct.getBeanName(), instruct.getMethodName());
                            break;
                    }
                }
                break;
            case NODE_REMOVED:
                break;
            default:
                break;
        }
    });
}

2.4 Parallel Task Registration

• Since the default SpringBoot is single-threaded, it has been modified to support multithreaded parallel execution
• Includes add and delete tasks, that is, cancelFuture.cancel(true)

public void addCronTask(SchedulingRunnable task, String cronExpression) {
    if (null != Constants.scheduledTasks.get(task.taskId())) {
        removeCronTask(task.taskId());
    }
    CronTask cronTask = new CronTask(task, cronExpression);
    Constants.scheduledTasks.put(task.taskId(), scheduleCronTask(cronTask));
}
public void removeCronTask(String taskId) {
    ScheduledTask scheduledTask = Constants.scheduledTasks.remove(taskId);
    if (scheduledTask == null) return;
    scheduledTask.cancel();
}

2.5 Custom AOP to be extended

• The scan @ComponentScan that we initially configured ("Org.itstack.middleware"*"), mainly used here as a custom comment, otherwise it will not be scanned, that is, the effect of your custom slice failure
• At present, the function here has not been extended, basically just print execution time-consuming, follow-up perfect task execution time-consuming monitoring, etc., need to be improved here

@Pointcut("@annotation(org.itstack.middleware.schedule.annotation.DcsScheduled)")
public void aopPoint() {
}

@Around("aopPoint()")
public Object doRouter(ProceedingJoinPoint jp) throws Throwable {
    long begin = System.currentTimeMillis();
    Method method = getMethod(jp);
    try {
        return jp.proceed();
    } finally {
        long end = System.currentTimeMillis();
        logger.info("\nitstack middleware schedule method: {}.{} take time(m): {}", jp.getTarget().getClass().getSimpleName(), method.getName(), (end - begin));
    }
}

3. Jar package Publishing

You still need to publish the Jar package to the manven central repository after development, which takes a long time to write a separate blog. Publish Jar packages to the Maven Central Warehouse (in preparation for open source middleware development)

Summary

1. There's still a lot to do with development, not all of it in a weekend!And you need an idea ape/girlfriend to join!(viii)_(viii)
2. The distributed task middleware control platform itstack-middleware-control is not explained here because it is simpler to use the zk function interface of the middleware for presentation and operation.
3. Middleware development is a very interesting thing, unlike business, it is more like Yijinjing, temple monks, swords deflecting, galloping across, making a lot of noise.

Recommended reading

• Spring Boot Middleware Development (1) "Unified White List Verification of Service Governance Middleware"
• Publish Jar packages to the Maven Central Warehouse (in preparation for open source middleware development)
• Reasonably, as long as you are a troublesome programmer, you really don't need to spend any more money on training when you graduate to find a job!