[source code analysis] Oozie history submission task

Posted by cowboy_x on Tue, 30 Jun 2020 05:26:29 +0200

0x00 summary

Oozie is an open source framework based on workflow engine contributed by Cloudera company to Apache. It is an open source workflow scheduling engine of Hadoop platform, which is used to manage Hadoop jobs. This article, the first in a series, introduces oozie's task submission phase.

0x01 problem

We deduce the implementation from the requirements, that is, if we implement the workflow engine from scratch, what parts do we need to implement? So we can ask a series of questions to explore in Oozie.

As a workflow engine, what parts need to be implemented? After thinking about it, I think it is necessary to have:

Task submission
Task persistence
The task is delegated to an actuator
task scheduling
Task callback is to notify the workflow engine when the task is completed by the executor
Support different tasks (synchronous, asynchronous)
Control the logical relationship between tasks (jump, wait...)
Status monitoring, monitoring task progress
......

Due to the limitation of space and energy, we can't study all the source codes and answer all the questions. So we'll sort out some questions and answer them one by one in Oozie source code analysis

Oozie is divided into several modules?
What is the function of each module?
How does Oozie submit tasks?
Where to submit the task? How to persist?
Is Oozie task synchronous and asynchronous?
How does Oozie handle synchronization tasks?
How does Oozie handle asynchronous tasks?
How to jump between Control Flow Nodes and Action Nodes of a task?
What types of tasks does Oozie support? Shell？ Java? Hive?
How does Oozie interact with Yarn?
How does Oozie know that Yarn's mission is complete?

Basic concepts of 0x02 Oozie

2.1 components

Oozie is composed of oozie client and Oozie Server. Oozie Server is a web application running in Java Servlet container (Tomcat). Oozie client is used to mention tasks to Oozie Server. Oozie client submits tasks through HTTP request.

In fact, Oozie Server is equivalent to a client of Hadoop. When a user needs to perform multiple related Mr tasks, he only needs to write the MR execution sequence workflow.xml And then submit the task using Oozie Server, which hosts the task flow.

Oozie Server operates workflow specifically, that is, oozie mainly maintains the execution of workflow / concatenation and jump of internal actions of workflow.

The specific Action is executed by Yan, who will assign the Action to the node with sufficient resources. The Action is executed asynchronously, so Oozie will be informed of the execution result through callback when the Action ends, and Oozie will also use polling to obtain the Action result (in order to improve the reliability).

The general submission process is as follows:

Oozie client ------> Oozie  Server -------> Yarn ------> Hadoop

2.2 features

Oozie has the following characteristics:

Oozie is not only used to configure multiple MR workflows. It can be a workflow with various programs mixed together. For example, after executing an MR1, a java script, then a shell script, then a Hive script, then a Pig script, and finally an MR2, can be easily completed with oozie. With oozie, if the previous task fails, the latter task will not be scheduled.
Oozie defines Control Flow Nodes (Control Flow Nodes) and Action Nodes (Action Nodes). The control flow node defines the beginning and end of the process, and the Execution Path of the control process, such as decision,fork,join, etc.; while the action node includes haoop map reduce Hadoop file system, Pig, SSH, HTTP, eMail and oozie subprocesses.
Oozie takes action as the basic unit, and can make multiple actions into a DAG diagram.
Oozie workflow must be a directed acyclic graph. In fact, oozie is equivalent to a client of Hadoop. When users need to perform multiple related Mr tasks, they only need to write the MR execution sequence workflow.xml And then submit the task using oozie, which hosts the task flow.

2.3 function module

Oozie is mainly composed of the following functional modules:

Workflow: this component is used to define and execute a specific sequence of mapreduce, hive and pig jobs. It is composed of every work that we need to deal with, and the requirements are streamed.
Coordinator: multiple workflows can be coordinated into one workflow for processing. Multiple workflows can form a coordinator. The output of the first several workflows can be used as the input of the next workflow. The trigger conditions of workflow can also be defined to trigger the timing.
Bundle Job: binding multiple coordinators to submit or trigger all coordinators together is an abstraction of a bunch of coordinators.
Oozie SLA (server level agreement): this component supports record tracking of workflow application execution.

Let's start from scratch and see how a workflow runs from submission to the end. Suppose the workflow starts and enters a hive action, which is configured to be executed by tez engine. Here's a simplified version of the code.

<workflow-app xmlns="uri:oozie:workflow:0.5" name="hive-wf">
  
    <start to="hive-node"/>

    <action name="hive-node">
        <hive xmlns="uri:oozie:hive-action:0.5">
            <script>hive.sql</script>
        </hive>
        <ok to="end"/>
        <error to="fail"/>
    </action>

    <kill name="fail">
       <message>Hive failed, error message</message>
    </kill>
    
    <end name="end"/>
</workflow-app>

0x03 Oozie client

Oozie Client is a way for users to submit tasks to Oozie Server. It can start tasks, stop tasks, submit tasks, start tasks, and view the execution of tasks. For example, the startup task is as follows:

oozie job -oozie oozie_url -config job.properties_address -run

3.1 program entry

Now that there is a startup script, we'll go straight inside to explore the program entry.

${JAVA_BIN} ${OOZIE_CLIENT_OPTS} -cp ${OOZIECPPATH} org.apache.oozie.cli.OozieCLI "${@}"

This shows the Client's entry class. Let's take a look.

public class OozieCLI {
      public static void main(String[] args) {
        if (!System.getProperties().containsKey(AuthOozieClient.USE_AUTH_TOKEN_CACHE_SYS_PROP)) {
            System.setProperty(AuthOozieClient.USE_AUTH_TOKEN_CACHE_SYS_PROP, "true");
        }
        System.exit(new OozieCLI().run(args));
    }
}

We can see that after verification, the program directly enters the run function from the main function.

public class OozieCLI {
     public synchronized int run(String[] args) {
        final CLIParser parser = getCLIParser();
        try {
            final CLIParser.Command command = parser.parse(args);
            String doAsUser = command.getCommandLine().getOptionValue(DO_AS_OPTION);

            if (doAsUser != null) {
                OozieClient.doAs(doAsUser, new Callable<Void>() {
                    @Override
                    public Void call() throws Exception {
                        processCommand(parser, command);
                        return null;
                    }
                });
            }
            else {
                processCommand(parser, command);
            }
            return 0;
        }
    }
}

It seems that the main content is in the processCommand, which will call the corresponding command method according to the command. adopt command.getName() we can clearly know what kind of tasks Oozie supports, such as JOB_CMD，JOBS_CMD，PIG_CMD，SQOOP_CMD，MR_CMD.

public void processCommand(CLIParser parser, CLIParser.Command command) throws Exception {
        switch (command.getName()) {
            case JOB_CMD:
                jobCommand(command.getCommandLine());
                break;
            case JOBS_CMD:
                jobsCommand(command.getCommandLine());
                break;
            case HIVE_CMD:
                scriptLanguageCommand(command.getCommandLine(), HIVE_CMD);
                break;
            ......
            default:
                parser.showHelp(command.getCommandLine());
        }
}

3.2 Hive as an example

Let's take hive as an example to see how to deal with it. Hive is a call to scriptLanguageCommand.

private void scriptLanguageCommand(CommandLine commandLine, String jobType){
    List<String> args = commandLine.getArgList();
    try {
        XOozieClient wc = createXOozieClient(commandLine);
        Properties conf = getConfiguration(wc, commandLine);
        String script = commandLine.getOptionValue(SCRIPTFILE_OPTION);
        List<String> paramsList = new ArrayList<>();
        ......
        System.out.println(JOB_ID_PREFIX + wc.submitScriptLanguage(conf, script,
                    args.toArray(new String[args.size()]),
                    paramsList.toArray(new String[paramsList.size()]), jobType));      
    }
}

The key code here is: wc.submitScriptLanguage So we need to see XOozieClient.submitScriptLanguage . The annotation indicates that the function is to submit Pig or Hive via HTTP.

public String submitScriptLanguage(Properties conf, String scriptFile, String[] args, String[] params, String jobType) throws IOException, OozieClientException {

    switch (jobType) {
        case OozieCLI.HIVE_CMD:
            script = XOozieClient.HIVE_SCRIPT;
            options = XOozieClient.HIVE_OPTIONS;
            scriptParams = XOozieClient.HIVE_SCRIPT_PARAMS;
            break;
        case OozieCLI.PIG_CMD:
            ......
    }

    conf.setProperty(script, readScript(scriptFile));
    setStrings(conf, options, args);
    setStrings(conf, scriptParams, params);

    return (new HttpJobSubmit(conf, jobType)).call();
}

HttpJobSubmit is to submit a job to Oozie Server, so we need to go to Oozie Server to explore.

private class HttpJobSubmit extends ClientCallable<String> {
    @Override
    protected String call(HttpURLConnection conn) throws IOException, OozieClientException {
        conn.setRequestProperty("content-type", RestConstants.XML_CONTENT_TYPE);
        writeToXml(conf, conn.getOutputStream());
        if (conn.getResponseCode() == HttpURLConnection.HTTP_CREATED) {
            JSONObject json = (JSONObject) JSONValue.parse(
                    new InputStreamReader(conn.getInputStream(), StandardCharsets.UTF_8));
            return (String) json.get(JsonTags.JOB_ID);
        }
        return null;
    }
}

0x04 Oozie Server

4.1 I am a web program

As we mentioned earlier, Oozie Server is a web application running in a Java Servlet container (Tomcat). Therefore, the specific startup configuration information is in the web.xml Medium. I haven't seen it for a long time web.xml I feel strange all of a sudden.

<!-- Servlets -->
<servlet>
    <servlet-name>callback</servlet-name>
    <display-name>Callback Notification</display-name>
    <servlet-class>org.apache.oozie.servlet.CallbackServlet</servlet-class>
    <load-on-startup>1</load-on-startup>
</servlet>

<servlet>
    <servlet-name>v1jobs</servlet-name>
    <display-name>WS API for Workflow Jobs</display-name>
    <servlet-class>org.apache.oozie.servlet.V1JobsServlet</servlet-class>
    <load-on-startup>1</load-on-startup>
</servlet>

......

4.2 initialization service

A lot of Ooize's basic work is done by Services, and each service is a singleton. The configuration information of these Services can be found in ooze-default.xml in

<property>
    <name>oozie.services</name>
    <value>
        org.apache.oozie.service.HadoopAccessorService,
        org.apache.oozie.service.LiteWorkflowAppService,
        org.apache.oozie.service.JPAService,
        org.apache.oozie.service.DBLiteWorkflowStoreService,
        org.apache.oozie.service.CallbackService,
        org.apache.oozie.service.ActionService,
        org.apache.oozie.service.CallableQueueService,
        org.apache.oozie.service.CoordinatorEngineService,
        org.apache.oozie.service.BundleEngineService,
        org.apache.oozie.service.DagEngineService,
        ......
    </value>
</property>

ServicesLoader is used to start and load all configured service s.

public class ServicesLoader implements ServletContextListener {
    private static Services services;
    /**
     * Initialize Oozie services.
     */
    public void contextInitialized(ServletContextEvent event) {
        services = new Services();
        services.init();
    }
}

The init function is used to initialize all configured Services. If there are Services of the same type, the latter will be stored.

public class Services {
      public void init() throws ServiceException {
 			 loadServices();	
      }  
  
    private void loadServices() throws ServiceException {
        try {
            Map<Class<?>, Service> map = new LinkedHashMap<Class<?>, Service>();
            Class<?>[] classes = ConfigurationService.getClasses(conf, CONF_SERVICE_CLASSES);
            Class<?>[] classesExt = ConfigurationService.getClasses(conf, CONF_SERVICE_EXT_CLASSES);

            List<Service> list = new ArrayList<Service>();
            loadServices(classes, list);
            loadServices(classesExt, list);

            //removing duplicate services, strategy: last one wins
            for (Service service : list) {
                if (map.containsKey(service.getInterface())) {
                      service.getClass());
                }
                map.put(service.getInterface(), service);
            }
            for (Map.Entry<Class<?>, Service> entry : map.entrySet()) {
                setService(entry.getValue().getClass());
            }
        } 
    }  
}

4.3 from Job to DAG

After the customer submits the job through oozie script, enter the org.apache.oozie.cli.OozieCLI . An OozieClient will be generated, and then use the JobCommand to submit the running information to the doPost interface of V1JosServlet. Oozier will call the submitJob() method in the doPos interface. A DAG object is generated, and then DAG.submitJon(JobConf,startJob).

We start with v1 JosServlet.doPost Start with. Here is the base class.

public abstract class BaseJobsServlet extends JsonRestServlet {
      protected void doPost(HttpServletRequest request,
            HttpServletResponse response) throws ServletException, IOException {
         JSONObject json = submitJob(request, conf);
      }
}

And then back to V1 JosServlet.submitJob

@Override
protected JSONObject submitJob(HttpServletRequest request, Configuration conf) throws XServletException,IOException {
    String jobType = request.getParameter(RestConstants.JOBTYPE_PARAM);

    if (jobType == null) {
            String wfPath = conf.get(OozieClient.APP_PATH);
 
            if (wfPath != null) {
                json = submitWorkflowJob(request, conf); // Our goal is here
            }
            else if (coordPath != null) {
                json = submitCoordinatorJob(request, conf);
            }
            else {
                json = submitBundleJob(request, conf);
            }
    }
    else { // This is a http submission job
       ......
    }
    return json;
}

And then called. DagEngine.submitJob . The DagEngine provides all the DAG engine functionality for WS calls.

private JSONObject submitWorkflowJob(HttpServletRequest request, Configuration conf) throws XServletException {
    try {
        String action = request.getParameter(RestConstants.ACTION_PARAM);
        DagEngine dagEngine = Services.get().get(DagEngineService.class).getDagEngine(user);
        if (action != null) {
            dryrun = (action.equals(RestConstants.JOB_ACTION_DRYRUN));
        }
        if (dryrun) {
            id = dagEngine.dryRunSubmit(conf);
        }
        else {
            id = dagEngine.submitJob(conf, startJob); // Here we are
        }
        json.put(JsonTags.JOB_ID, id);
    }
    return json;
}

0x06 core engine

Oozie has three core engines, all of which inherit the abstract class BaseEngine.

The three engines are:

Dagenine is responsible for workflow execution, and the above code will come here
Coordinator engine, responsible for coordinator execution
BundleEngine, responsible for bundle execution

Corresponding separately

org.apache.oozie.service.CoordinatorEngineService
org.apache.oozie.service.BundleEngineService
org.apache.oozie.service.DagEngineService

As we mentioned earlier, these belong to system Services, which are singletons and will be added to Services when Oozie is started. get when needed.

public class Services {
		private Map<Class<? extends Service>, Service> services = new LinkedHashMap<Class<? extends Service>, Service>();
  
		public <T extends Service> T get(Class<T> serviceKlass) {
    		return (T) services.get(serviceKlass);
		}
}

Specific examples are invoked in V1JosServlet:

String user = conf.get(OozieClient.USER_NAME);
DagEngine dagEngine = Services.get().get(DagEngineService.class).getDagEngine(user);

0x07 Command drives execution

Oozie abstracts all commands into commands, so that the program execution is summarized as driven by Command, similar to message driven.

Command is divided into synchronous and asynchronous. Its base class is XCommand. XCommand provides the following modes:

single execution: a command instance can be executed only once
eager data loading: loads data for eager precondition check
eager precondition check: verify precondition before obtaining lock
data loading: loads data for precondition check and execution
precondition check: verifies precondition for execution is still met
locking: obtains exclusive lock on key before executing the command
execution: command logic

public abstract class XCommand<T> implements XCallable<T> {
    ......
    private String key;
    private String name;
    private String type;
    private AtomicBoolean used = new AtomicBoolean(false);
    private Map<Long, List<XCommand<?>>> commandQueue;
    protected Instrumentation instrumentation;
    ......
}

Xcalable, the parent interface of XCommand, inherits java.util.concurrent.Callable. The ultimate goal is to arrange the execution plan of commands based on priority when executing asynchronously.

Therefore, several key functions of XCommand are: queue, call, execute:

queue: add a command to the commandQueue. The command is delay ed execution after the current command is executed. In the current execution sequence of single command (Series), it will be added in the current execution sequence.
call is the inherited Callable implementation function, which will be called to execute.
execute is a specific Command to implement its own specific business.

From our common SubmitXCommand, the inheritance relationship is as follows:

public class SubmitXCommand extends WorkflowXCommand<String> 
public abstract class WorkflowXCommand<T> extends XCommand<T> 
public abstract class XCommand<T> implements XCallable<T> 
public interface XCallable<T> extends Callable<T>

Another example is the inheritance relationship of TransitionXCommand

abstract class TransitionXCommand<T> extends XCommand<T> 
public abstract class SubmitTransitionXCommand extends TransitionXCommand<String>

As can be seen from the previous components, tasks have the concept of state machine, such as preparation, start, running, failure, end, etc., so the commands that operate on tasks need to deal with the changes of the state machine. oozie's commands for processing tasks need to inherit the abstract class TransitionXCommand, and the parent class of TransitionXCommand is XCommand.

0x08 engine processing commit

As mentioned earlier, doPost calls the ID= dagEngine.submitJob (conf, startJob);

Let's take a look at how dagenine handles submitted tasks.

First, submit the job by running its call() directly through SubmitXCommand.

public String submitJob(Configuration conf, boolean startJob) throws DagEngineException {
    validateSubmitConfiguration(conf);
    try {
        String jobId;
        SubmitXCommand submit = new SubmitXCommand(conf);
        jobId = submit.call();
        if (startJob) {
            start(jobId);
        }
        return jobId;
    }
}

Then start the Job through StartXCommand. From the comments, we can see that the execution is still synchronous (by actively executing the call() function).

public void start(String jobId) throws DagEngineException {
    // Changing to synchronous call from asynchronous queuing to prevent the
    // loss of command if the queue is full or the queue is lost in case of
    // failure.
    new StartXCommand(jobId).call();
}

8.1 SubmitXCommand

The submitted task of xmitcommand will be resolved after being submitted to the database.

The main business is to implement in execute.

Resolve the configuration and get the WorkflowApp
Create WorkflowInstance
Generating WorkflowJobBean
Save WorkflowJobBean to WF through JPA_ jobs

The code is summarized as follows:

protected String execute() throws CommandException {

    WorkflowAppService wps = Services.get().get(WorkflowAppService.class);
    try {
        HadoopAccessorService has = Services.get().get(HadoopAccessorService.class);
        FileSystem fs = has.createFileSystem(user, uri, fsConf);

        // Resolve the configuration and get the WorkflowApp
        WorkflowApp app = wps.parseDef(conf, defaultConf);

        // Create WorkflowInstance
        WorkflowInstance wfInstance;
        wfInstance = workflowLib.createInstance(app, conf);

        // Generating WorkflowJobBean
        WorkflowJobBean workflow = new WorkflowJobBean();
        workflow.setId(wfInstance.getId());
        workflow.setAppName(ELUtils.resolveAppName(app.getName(), conf));
        workflow.setAppPath(conf.get(OozieClient.APP_PATH));
        workflow.setConf(XmlUtils.prettyPrint(conf).toString());
        ......
        workflow.setWorkflowInstance(wfInstance);
        workflow.setExternalId(conf.get(OozieClient.EXTERNAL_ID));

        if (!dryrun) {
            workflow.setSlaXml(jobSlaXml);
            // Add to temporary list
            insertList.add(workflow); 
            JPAService jpaService = Services.get().get(JPAService.class);
            if (jpaService != null) {
                // Save WorkflowJobBean to wf_jobs
    BatchQueryExecutor.getInstance().executeBatchInsertUpdateDelete(insertList, null, null);
                }
            }
            return workflow.getId();
        }
}

Where insertList is used to temporarily store WorkflowJobBean

private List<JsonBean> insertList = new ArrayList<JsonBean>();

WorkflowJobBean corresponds to the table WF in the database_ JOBS.

public class WorkflowJobBean implements Writable, WorkflowJob, JsonBean {
    ......//Omitting other variables
    @Transient
    private List<WorkflowActionBean> actions;
}

In Oozie, in order to facilitate the management of user-defined actions and Workflow, the underlying uses JPA to store these data in the database. Specifically, execute batch insert update delete is called to insert into the database through JPA.

BatchQueryExecutor.getInstance().executeBatchInsertUpdateDelete(insertList, null, null);

The specific batchqueryexecutior code is as follows.

public class BatchQueryExecutor {
    public void executeBatchInsertUpdateDelete(Collection<JsonBean> insertList, Collection<UpdateEntry> updateList,Collection<JsonBean> deleteList) {
        List<QueryEntry> queryList = new ArrayList<QueryEntry>();
        JPAService jpaService = Services.get().get(JPAService.class);
        EntityManager em = jpaService.getEntityManager();

        if (updateList != null) {
            for (UpdateEntry entry : updateList) {
                Query query = null;
                JsonBean bean = entry.getBean();
                if (bean instanceof WorkflowJobBean) {
                    // Our program is here
                    query = WorkflowJobQueryExecutor.getInstance().getUpdateQuery(
                            (WorkflowJobQuery) entry.getQueryName(), (WorkflowJobBean) entry.getBean(), em);
                }
                else if (bean instanceof WorkflowActionBean) {
                    query = WorkflowActionQueryExecutor.getInstance().getUpdateQuery(
                            (WorkflowActionQuery) entry.getQueryName(), (WorkflowActionBean) entry.getBean(), em);
                }
                else if {
                  //Many other types are omitted here
                }
                queryList.add(new QueryEntry(entry.getQueryName(), query));
            }
        }
        // Insert the database here
        jpaService.executeBatchInsertUpdateDelete(insertList, queryList, deleteList, em);
    }  
}

The JPA digest code is as follows:

public class JPAService implements Service, Instrumentable {
  
    private OperationRetryHandler retryHandler;
  
    public void executeBatchInsertUpdateDelete(final Collection<JsonBean> insertBeans, final List<QueryEntry> updateQueryList, final Collection<JsonBean> deleteBeans, final EntityManager em) {
      
        try {
            retryHandler.executeWithRetry(new Callable<Void>() {

                public Void call() throws Exception {
                   ......
                    if (CollectionUtils.isNotEmpty(insertBeans)) {
                        for (final JsonBean bean : insertBeans) {
                            em.persist(bean);
                        }
                    }
                   ......
                }
            });
        }
    }
}

In this way, a Workflow Job is stored in the database.

8.2 workflow lifecycle

First, let's introduce the workflow lifecycle. Our code will use the PREP state.

Prep: the first time a workflow is created, it is in the prep state, which indicates that the workflow is created but has not been run.
Running: when a workflow job has been created, it is in the running state. It will not reach the end state, it can only end because of an error or be suspended.
Suspended: a workflow job in running state will become suspended state, and it will always be in this state, unless the workflow job is restarted or killed.
Killed: when a workflow job is in the state of being created, or in the state of running and suspended, the status of the workflow job changes to killed.
Failed: when an unexpected error of a workflow job fails and terminates, it will become a failed state.

8.3 StartXCommand

After processing SubmitXCommand, Oozie Server processes StartXCommand immediately.

The function of StartXCommand is to start a Command, which inherits SignalXCommand, so StartXCommand(jobId).call(); calls the call of SignalXCommand.

public class StartXCommand extends SignalXCommand

The relevant codes are as follows:

First, StartXCommand calls the base class constructor

public StartXCommand(String id) {
        super("start", 1, id);
        InstrumentUtils.incrJobCounter(getName(), 1, getInstrumentation());
}

Then, SignalXCommand gets the jobId, which is generated and returned by SubmitXCommand.

public SignalXCommand(String name, int priority, String jobId) {
    super(name, name, priority);
    this.jobId = ParamChecker.notEmpty(jobId, "jobId");
}

call() first calls to SignalXCommand.loadState . It reads Workflow job information from the database based on jobId.

protected void loadState() throws CommandException {
    try {
        jpaService = Services.get().get(JPAService.class);
        if (jpaService != null) {
            this.wfJob = WorkflowJobQueryExecutor.getInstance().get(WorkflowJobQuery.GET_WORKFLOW, jobId);
            if (actionId != null) {
                this.wfAction = WorkflowActionQueryExecutor.getInstance().get(WorkflowActionQuery.GET_ACTION_SIGNAL, actionId);
            }
        }
}

The SQL statement is as follows:

@NamedQuery(name = "GET_WORKFLOW", query = "select OBJECT(w) from WorkflowJobBean w where w.id = :id"),

call() is then called SignalXCommand.execute(), the specific operations are as follows:

1) In execute, because the state is PREP, the workflowInstance.start The corresponding instance here is LiteWorkflowInstance
- 1.1) LiteWorkflowInstance.start Call signal()
  - 1.1.1) signal() calls exit= nodeHandler.enter (context), the actual call is LiteActionHandler.enter
    - 1.1.1.1) call Lite WorkflowStoreService.liteExecute In this case, the WorkflowActionBean is generated and added to the temporary variable ACTIONS_TO_START
      - 1.1.1.1.1) WorkflowActionBean action = new WorkflowActionBean();
      - 1.1.1.1.2) action.setJobId(jobId); do various other settings
      - 1.1.1.1.3) List list = (List) context.getTransientVar(ACTIONS_TO_START);
      - 1.1.1.1.4) list.add(action); add to temporary list
- 1.2) return to signal(), because start is a synchronous operation, exit is true
  - 1.2.1) signal all new synch transitions. Traverse the pathsToStar. If there is a synchronous jump, the next step Action jump will be started, that is, signal (pathtostart, ":: Sync: 😊 ;
2) Go back to execute(), traverse WorkflowStoreService.getActionsToStart(workflowInstance), i.e. from ACTIONS_TO_START takes Action (because one was just put in before, so it is obtained this time)
- 2.1) traverse in the for loop, because it is submit, syncAction = newAction;
3)execute() will call BatchQueryExecutor.getInstance (). Executebackinsertupdatedelete writes the WorkflowActionBean to the database
4)execute() directly starts start command: new ActionStartXCommand(wfJob, syncAction.getId(), syncAction.getType()).call();

The code is as follows:

protected Void execute() throws CommandException {
    WorkflowInstance workflowInstance = wfJob.getWorkflowInstance();
    workflowInstance.setTransientVar(WorkflowStoreService.WORKFLOW_BEAN, wfJob);
    WorkflowJob.Status prevStatus = wfJob.getStatus();
    WorkflowActionBean syncAction = null;
    List<WorkflowActionBean> workflowActionBeanListForForked = new ArrayList<WorkflowActionBean>();

    if (wfAction == null) {
           if (wfJob.getStatus() == WorkflowJob.Status.PREP) {
                // For the above 1)
                completed = workflowInstance.start();
                wfJob.setStatus(WorkflowJob.Status.RUNNING);
                wfJob.setStartTime(new Date());
                wfJob.setWorkflowInstance(workflowInstance);
                generateEvent = true;
                queue(new WorkflowNotificationXCommand(wfJob));
            }
    }
    else { 
      ...... 
    }
    if (completed) {
      ......
    }
    else {
       // For the top outermost 2)
       for (WorkflowActionBean newAction : 
            WorkflowStoreService.getActionsToStart(workflowInstance)) {
         insertList.add(newAction);
         if (wfAction != null) { // null during wf job submit
            // The comment indicates that wf job is not submitted here
            .....
         }
         else {
            syncAction = newAction; // first action after wf submit should always be sync
         }
       }
    }
    // Write the WorkflowActionBean for the above 3)
    BatchQueryExecutor.getInstance().executeBatchInsertUpdateDelete(insertList, updateList, null);
    if (wfJob.getStatus() != WorkflowJob.Status.RUNNING && wfJob.getStatus() != WorkflowJob.Status.SUSPENDED) {
            ......
    }
    else if (syncAction != null) {
        // call() directly, corresponding to the above 4)
        new ActionStartXCommand(wfJob, syncAction.getId(), syncAction.getType()).call();
    }
}

8.4 database information

Workflow database information can be seen from the WorkflowActionBean. What we want to focus on here is the transition field. Oozie uses transition to record where the Action will jump next. The WorkflowActionBean is summarized as follows:

public class WorkflowActionBean implements Writable, WorkflowAction, JsonBean {
    @Id
    private String id;

    @Basic
    @Index
    @Column(name = "wf_id")
    private String wfId = null;

    @Basic
    @Index
    @Column(name = "status")
    private String statusStr = WorkflowAction.Status.PREP.toString();

    @Basic
    @Column(name = "execution_path", length = 1024)
    private String executionPath = null;

    @Basic
    @Column(name = "transition")
    private String transition = null;

    @Basic
    @Column(name = "data")
    @Lob
    @Strategy("org.apache.oozie.executor.jpa.StringBlobValueHandler")
    private StringBlob data;
}

8.5 synchronous execution

The jump of synchronous execution is mainly performed in LiteWorkflowInstance.signal This shows that if there is a synchronous jump after the end of the command, it will continue to execute.

public synchronized boolean signal(String executionPath, String signalValue) throws WorkflowException {
                    // signal all new synch transitions
                    for (String pathToStart : pathsToStart) {
                        signal(pathToStart, "::synch::");
                    }  
}

At this point, the submission of the program has been completed, and the subsequent execution phase of the program in Oozie begins with ActionStartXCommand.