ElasticJob3.0 integrate SpringBoot, elasticjob Lite [introduction to elasticjob]

Posted by bo0 on Mon, 24 Jan 2022 10:38:57 +0100

1, Foreword

1-1. What is ElasticJob

We used scheduled tasks in the SpringBoot project before. First, we started the scheduled task @ enableshcheduling, and then used @ Scheduled(cron = "*/1 * *?"), This is easy to use and has no problem.

But think about this scenario. If a service can't meet our needs, what should we do at this time? It's easy to think of clusters and deploy multiple copies.

Generally, we only need to add A load function before deploying multiple services. After all, each service provides the same service. But timing tasks are not good. For example, the fun method of service A runs half way, and then the fun method of service B starts running. This will certainly cause problems.

Our ElasticJob can solve this problem and support horizontal expansion / contraction. Of course, it also comes with other functions, such as good-looking and easy-to-use operation interface, exception notification (e-mail, enterprise wechat, nailing), etc.



1-2. Others

ElasticJob depends on zookeeper (multiple services must communicate to achieve the above functions). You can install it yourself. If you just test, you can install a windows version.

ElasticJob provides two modules, ElasticJob lite and ElasticJob cloud. We will only introduce the use of ElasticJob Lite below.

Official documents


2, Use

If the modified yml document does not take effect, there are two methods as follows

  • Change the name space of zookeeper
  • Add overwrite under the job in the configuration file: true

2-1. Operation

Homework is actually a scheduled task. Every job is a scheduled task.

There are many types of jobs, such as simple jobs, data flow jobs, script jobs and HTTP jobs (provided in 3.0.0-beta)


2-1-1. Ordinary operation

Ordinary jobs only need to implement the SimpleJob interface and then override the execute method (which is also the most used job)

import org.apache.shardingsphere.elasticjob.api.ShardingContext;
import org.apache.shardingsphere.elasticjob.simple.job.SimpleJob;
import org.springframework.stereotype.Component;

@Component
public class MyElasticJob implements SimpleJob {

    @Override
    public void execute(ShardingContext context) {
        System.out.println(context.getShardingTotalCount() + "  " + context.getShardingItem());
    }
}

2-1-2. Data flow operation

MyDataflowJob

import com.xdx97.elasticjob.bean.XdxBean;
import org.apache.shardingsphere.elasticjob.api.ShardingContext;
import org.apache.shardingsphere.elasticjob.dataflow.job.DataflowJob;
import org.springframework.stereotype.Component;
import java.util.ArrayList;
import java.util.List;

@Component
public class MyDataflowJob implements DataflowJob<XdxBean> {
    @Override
    public List<XdxBean> fetchData(ShardingContext shardingContext) {
        List<XdxBean> foos = new ArrayList<>();
        double random = Math.random();
        System.out.println("fetchData------ " + random);
        if (random > 0.5){
            XdxBean foo = new XdxBean();
            foo.setName("Xiaodaoxian");
            foos.add(foo);
        }
        return foos;
    }

    @Override
    public void processData(ShardingContext shardingContext, List<XdxBean> list) {
        System.out.println("coming processData------");
    }
}

XdxBean

public class XdxBean {

    private String name;

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }
}

Operation results:

fetchData------ 0.13745888666984807
fetchData------ 0.2922741337641118
fetchData------ 0.7834818165147507
 coming processData------
fetchData------ 0.8177868853353837
 coming processData------
fetchData------ 0.14076346085285385

Note: math The data generated by random() is between 0-1.

From the above running results, we can conclude that the so-called data flow job is actually a scheduled task, but when the scheduled task generates data, it will carry the data to call the processData() method


2-1-3. Script operation

I don't feel much use, so I don't study the relationship with time


2-1-4. HTTP job (provided in 3.0.0-beta)

I don't feel much use, so I don't study the relationship with time


2-2. Job scheduling (based on SpringBoot)

ElasticJob provides three usage methods: Java based, SpringBoot based and configuration file based. It is also troublesome to introduce all three. The principle is the same. I will only introduce SpringBoot here.


2-2-1. Import pom file

<!-- https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter</artifactId>
    <version>2.2.0.RELEASE</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter-web -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
    <version>2.2.0.RELEASE</version>
</dependency>
<dependency>
    <groupId>org.apache.shardingsphere.elasticjob</groupId>
    <artifactId>elasticjob-lite-spring-boot-starter</artifactId>
    <version>3.0.0-RC1</version>
</dependency>

There is a pit here. The official document only says that we are integrated with SpringBoot, but we do not provide a complete pom file, which leads to my card for a long time.

There are database related connections in this starter. We're just doing a simple test. If you don't want to configure the data source, just change the startup class annotation

@SpringBootApplication(exclude = { DataSourceAutoConfiguration.class })

2-2-2. Configuration file

server:
  port: 8085

elasticjob:
  regCenter:
    #zookeeper's ip:port
    serverLists: 127.0.0.1:2181
    #Name and life space, just define it yourself
    namespace: my-job4
  jobs:
    #Your scheduled task name, user-defined name
    myElasticJob:
      #The full pathname of the scheduled task
      elasticJobClass: com.elastic.job.MyElasticJob
      #cron expression for scheduled task execution
      cron: 0/5 * * * * ?
      #Number of slices
      shardingTotalCount: 10

2-2-3. Number of pieces

It is the core of horizontal expansion. For example, 10 slices are defined above (the corresponding slice names are 0-9). Suppose that our timing task is executed every 1 minute, and the timing method is execute.

When we have only one server, execute will be called ten times every minute (the fragment names (0-9) are different each time).

When we have two servers, servers A and B call execute five times every minute (the fragment names (A0-4, B5-9) are different each time)

Based on the above description, we can follow this analogy. When there are three servers, A (3), B (3) and C (4), it is easy to expand horizontally.

Based on the above slicing function, our homework also needs to be modified

public void execute(ShardingContext context) {
    System.out.println(context.getShardingTotalCount() + "  " + context.getShardingItem());
    switch (context.getShardingItem()) {
        case 0:
            // do something by sharding item 0
            break;
        case 1:
            // do something by sharding item 1
            break;
        case 2:
            // do something by sharding item 2
            break;
        // case n: ...
    }
}

If you copy five copies of the following test code and start it at one time (wait a minute and wait a while), you will find that the area allocated to each server is changing. This rule will also be found when the services are shut down in turn.


2-2-4. Manual call

For example, we need to call this scheduled task temporarily, but its interval must be a few hours or more.

The official operation is as follows, but I have tried for a long time and still haven't solved the error, so I implemented it according to the Java based calling method

The implementation code is as follows:

import org.apache.shardingsphere.elasticjob.api.JobConfiguration;
import org.apache.shardingsphere.elasticjob.lite.api.bootstrap.impl.OneOffJobBootstrap;
import org.apache.shardingsphere.elasticjob.reg.base.CoordinatorRegistryCenter;
import org.apache.shardingsphere.elasticjob.reg.zookeeper.ZookeeperConfiguration;
import org.apache.shardingsphere.elasticjob.reg.zookeeper.ZookeeperRegistryCenter;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class OneOffJobController {


    @GetMapping("/execute")
    public String executeOneOffJob() {
        OneOffJobBootstrap jobBootstrap = new OneOffJobBootstrap(createRegistryCenter(), new MyElasticJob(), createJobConfiguration());
        // One time scheduling can be called multiple times
        jobBootstrap.execute();
        return "success";
    }

    private static CoordinatorRegistryCenter createRegistryCenter() {
        CoordinatorRegistryCenter regCenter = new ZookeeperRegistryCenter(new ZookeeperConfiguration("127.0.0.1:2181", "my-job4"));
        regCenter.init();
        return regCenter;
    }

    private static JobConfiguration createJobConfiguration() {
        // Create job configuration
        return JobConfiguration.newBuilder("myElasticJob", 10).cron("").build();
    }
}

In fact, the above can be made general, and the parameters are passed externally. As for the job object, we can pass a string and create an object through reflection.

Class classType = Class.forName("com.elastic.job.MyElasticJob");
ElasticJob obj = (ElasticJob)classType.newInstance();

After the operation and maintenance interface is configured later, if you find that it doesn't need to be so troublesome, just click on the operation and maintenance interface (you don't need to provide an interface)


2-3. Configure error handling strategy

When an exception occurs in the execution of a scheduled task, what should we do to deal with it. The following six strategies are officially provided. I will demonstrate an email notification strategy here. Those who are interested or need to study other strategies.

Error handling policy nameexplainBuilt inDefaultIs additional configuration required
Logging policyRecord the job exception log without interrupting job executionyesyes
Throw exception strategyThrow a system exception and interrupt job executionyes
Ignore exception policyIgnore system exceptions without interrupting job executionyes
Mail notification policySend mail message notifications without interrupting job executionyes
Enterprise wechat notification strategySend enterprise wechat message notification without interrupting job executionyes
Nailing notification policySend nail message notification without interrupting job executionyes

2-3-1. Introduction of pom

<dependency>
    <groupId>org.apache.shardingsphere.elasticjob</groupId>
    <artifactId>elasticjob-error-handler-email</artifactId>
    <version>3.0.0-RC1</version>
</dependency>

2-3-2. Modify configuration file

server:
  port: 8085

elasticjob:
  regCenter:
    #zookeeper's ip:port
    serverLists: 127.0.0.1:2181
    #Name and life space, just define it yourself
    namespace: my-job5
  jobs:
    #Your scheduled task name, user-defined name
    oneSimpleJob:
      #The full pathname of the scheduled task
      elasticJobClass: com.xdx97.elasticjob.job.OneSimpleJob
      #cron expression for scheduled task execution
      cron: 0/30 * * * * ?
      #Number of slices
      shardingTotalCount: 1
      jobErrorHandlerType: EMAIL
      overwrite: true
      props:
        email:
          #Mail server address
          host: smtp.126.com
          #Mail server port
          port: 465
          #Mail server user name
          username: xxxxxxx
          #Mail server password
          password: xxxxx
          #Enable SSL encrypted transport
          useSsl: true
          #Mail subject
          subject: ElasticJob error message
          #Sender email address
          from: xxxxx@126.com
          #Recipient email address
          to: xxx@qq.com
          #CC email address
          cc: xxxxxx
          #Confidential email address
          bcc: xxxxx
          #	Enable debugging mode
          debug: false

2-3-3. Use

We only need to throw an exception in the above scheduled task and we will receive the email.


2-3-4. Others

There is a pit in it. According to the official documents, we think that props and jobs are of the same level, but they are not

If you want to configure multiple scheduled tasks and the notification objects of each scheduled task are different, you can configure multiple copies of the above. After all, each props belongs to each scheduled task.


2-4. Operation listener

Elasticjob Lite provides a job listener for listening before and after task execution. Listeners are divided into regular listeners executed by each job node and distributed listeners executed by only a single node in a distributed scenario. This section details how they are used.

After the job dependency (DAG) function is developed, you may consider deleting the job listener function.

The job listener document does not provide a springboot method. I tried it myself for a long time but failed. I didn't reply to the issue for the time being. Let's do it first.


3, Operation and maintenance platform configuration

If the above distributed fragment timing task makes me think it's awesome, then this operation and maintenance platform really brightens my eyes.


3-1. Installation and deployment

This ui is a separate interface, and it has nothing to do with our specific services. So we don't need to modify any previous code at all. We just need to download it and start it.


3-1-1. Download

https://www.apache.org/dyn/closer.cgi/shardingsphere/elasticjob-ui-3.0.0-RC1/apache-shardingsphere-elasticjob-3.0.0-RC1-lite-ui-bin.tar.gz


3-1-2. Startup

Here, if you decompress and start directly under the window, an error will be reported (the startup class cannot be found), which is due to the problem of decompression. (I also put the compressed package in the following source code)

My solution is to upload the compressed package to linux, unzip it and then pull it down.

tar zxvf apache-shardingsphere-elasticjob-3.0.0-RC1-lite-ui-bin.tar

3-1-3. Use

Browser open http://localhost:8088/ , username / password: root/root

Then go to global configuration > registry configuration and configure your zookeeper address

The specific operations are very simple, just click the button and don't say


4, Video address

https://www.bilibili.com/video/BV19L411p7qR


5, Source code acquisition

Pay attention to the following official account of WeChat, reply to key words: elasticJobLiteDemo

Topics: elastic-job