Analysis of Nginx source code -- master process and worker process

Posted by caine on Tue, 07 Sep 2021 08:14:06 +0200

1, Explain

In the previous chapter, we had a general understanding of the startup of the nginx process. We didn't go deep into what the process did. In this chapter, we'll look at what the master process is doing.

2, Function analysis

./src/os/unix/ngx_process_cycle.c>ngx_master_process_cycle(ngx_cycle_t *)
 
ngx_new_binary = 0;
delay = 0;
sigio = 0;
live = 1;

for ( ;; ) {
    if (delay) {
        if (ngx_sigalrm) {
            sigio = 0;
            delay *= 2;
            ngx_sigalrm = 0;
        }

        ngx_log_debug1(NGX_LOG_DEBUG_EVENT, cycle->log, 0,
                       "termination cycle: %M", delay);

        itv.it_interval.tv_sec = 0;
        itv.it_interval.tv_usec = 0;
        itv.it_value.tv_sec = delay / 1000;
        itv.it_value.tv_usec = (delay % 1000 ) * 1000;

        if (setitimer(ITIMER_REAL, &itv, NULL) == -1) {
            ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                          "setitimer() failed");
        }
    }

    ngx_log_debug0(NGX_LOG_DEBUG_EVENT, cycle->log, 0, "sigsuspend");

    sigsuspend(&set);

    ngx_time_update();

    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, cycle->log, 0,
                   "wake up, sigio %i", sigio);

    if (ngx_reap) {
        ngx_reap = 0;
        ngx_log_debug0(NGX_LOG_DEBUG_EVENT, cycle->log, 0, "reap children");

        live = ngx_reap_children(cycle);
    }

    if (!live && (ngx_terminate || ngx_quit)) {
        ngx_master_process_exit(cycle);
    }

    if (ngx_terminate) {
        if (delay == 0) {
            delay = 50;
        }

        if (sigio) {
            sigio--;
            continue;
        }

        sigio = ccf->worker_processes + 2 /* cache processes */;

        if (delay > 1000) {
            ngx_signal_worker_processes(cycle, SIGKILL);
        } else {
            ngx_signal_worker_processes(cycle,
                                   ngx_signal_value(NGX_TERMINATE_SIGNAL));
        }

        continue;
    }

    if (ngx_quit) {
        ngx_signal_worker_processes(cycle,
                                    ngx_signal_value(NGX_SHUTDOWN_SIGNAL));
        ngx_close_listening_sockets(cycle);

        continue;
    }

    if (ngx_reconfigure) {
        ngx_reconfigure = 0;

        if (ngx_new_binary) {
            ngx_start_worker_processes(cycle, ccf->worker_processes,
                                       NGX_PROCESS_RESPAWN);
            ngx_start_cache_manager_processes(cycle, 0);
            ngx_noaccepting = 0;

            continue;
        }

        ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "reconfiguring");

        cycle = ngx_init_cycle(cycle);
        if (cycle == NULL) {
            cycle = (ngx_cycle_t *) ngx_cycle;
            continue;
        }

        ngx_cycle = cycle;
        ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx,
                                               ngx_core_module);
        ngx_start_worker_processes(cycle, ccf->worker_processes,
                                   NGX_PROCESS_JUST_RESPAWN);
        ngx_start_cache_manager_processes(cycle, 1);

        /* allow new processes to start */
        ngx_msleep(100);

        live = 1;
        ngx_signal_worker_processes(cycle,
                                    ngx_signal_value(NGX_SHUTDOWN_SIGNAL));
    }

    if (ngx_restart) {
        ngx_restart = 0;
        ngx_start_worker_processes(cycle, ccf->worker_processes,
                                   NGX_PROCESS_RESPAWN);
        ngx_start_cache_manager_processes(cycle, 0);
        live = 1;
    }

    if (ngx_reopen) {
        ngx_reopen = 0;
        ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "reopening logs");
        ngx_reopen_files(cycle, ccf->user);
        ngx_signal_worker_processes(cycle,
                                    ngx_signal_value(NGX_REOPEN_SIGNAL));
    }

    if (ngx_change_binary) {
        ngx_change_binary = 0;
        ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "changing binary");
        ngx_new_binary = ngx_exec_new_binary(cycle, ngx_argv);
    }

    if (ngx_noaccept) {
        ngx_noaccept = 0;
        ngx_noaccepting = 1;
        ngx_signal_worker_processes(cycle,
                                    ngx_signal_value(NGX_SHUTDOWN_SIGNAL));
    }
}
The old rule is to analyze it paragraph by paragraph, but the code looks easy to understand.

if (delay) {
    if (ngx_sigalrm) {
        sigio = 0;
        delay *= 2;
        ngx_sigalrm = 0;
    }

    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, cycle->log, 0,
                   "termination cycle: %M", delay);

    itv.it_interval.tv_sec = 0;
    itv.it_interval.tv_usec = 0;
    itv.it_value.tv_sec = delay / 1000;
    itv.it_value.tv_usec = (delay % 1000 ) * 1000;

    if (setitimer(ITIMER_REAL, &itv, NULL) == -1) {
        ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                      "setitimer() failed");
    }
}
Setimer means that after a specified time, the kernel will send a SIGALRM signal to the process. The specified time is itv. After receiving the instruction, ngx_sigalrm will become 1. When delay is not 0, it will continue to double. Delay is used to wait for the worker process to exit. After receiving the signal, the master process will send a signal to the worker process to exit. It takes time for the worker process to exit. Delay is the waiting time.
sigsuspend(&set);
At this time, the master process will be suspended. When the process receives a new signal, it will return to continue and complete the signal at the same time. As mentioned above, the processing and initiation signals are transmitted through ngx_init_signals execute NGX_ signal_ SIGALRM signal in handler. ngx_ signal_ The handler function will be based on the input
Set the following status

 

ngx_time_update();

When the master process processes information, it updates the cache time.

    if (ngx_reap) {
       ngx_reap = 0;
       ngx_log_debug0(NGX_LOG_DEBUG_EVENT, cycle->log, 0, "reap children");
       
       live = ngx_reap_children(cycle);
    }
As mentioned above, when the process is suspended, it will listen for the corresponding signal. Here, when the SIGCHLD signal is received, it enters this logic. SIGCHLD signal is a signal sent to the parent process when the child process terminates.

  ngx_ reap_ The children function traverses ngx_process array, check the status of each child process, and restart the process that exited abnormally. Only when all child processes exit normally, live will be 0, and the returned live at other times will be 1. This means that if it is an abnormal exit, the process will be pulled up again. If an exit command is received, it will exit normally.

if (!live && (ngx_terminate || ngx_quit)) {
    ngx_master_process_exit(cycle);
}
If live is 0, all child processes exit normally, and according to the entered command, if stop, exit the master process.

 

 

When the master exits, it will delete the pid file, release the link, release the memory space in the cycle object, delete the monitored events, and send a signal.
if (ngx_terminate) {
    if (delay == 0) {
        delay = 50;
    }

    if (sigio) {
        sigio--;
        continue;
    }

    sigio = ccf->worker_processes + 2 /* cache processes */;

    if (delay > 1000) {
        ngx_signal_worker_processes(cycle, SIGKILL);
    } else {
        ngx_signal_worker_processes(cycle,
                               ngx_signal_value(NGX_TERMINATE_SIGNAL));
    }

    continue;
}

When terminating, it was mentioned earlier to set the delay timeout. If there is no timeout, it will be executed

ngx_signal_worker_processes(cycle, ngx_signal_value(NGX_TERMINATE_SIGNAL));
Send a TERM signal and exit the worker process normally
If the exit times out, execute it and force the exit
ngx_signal_worker_processes(cycle, SIGKILL);
When closing through stop, it doesn't matter whether there are still requests to be processed. It will be closed directly. In this way, there will be a quit command// TODO   ngx_ signal_ worker_ After what the processes function does, fill the pit and when.. Let it be..
if (ngx_quit) {
    ngx_signal_worker_processes(cycle,
                                ngx_signal_value(NGX_SHUTDOWN_SIGNAL));
    ngx_close_listening_sockets(cycle);
    continue;
}
This is the quit command. Compared with the above, the socket will be closed only after there is no business. It is more elegant.
if (ngx_reconfigure) {
    ngx_reconfigure = 0;

    if (ngx_new_binary) {
        ngx_start_worker_processes(cycle, ccf->worker_processes,
                                   NGX_PROCESS_RESPAWN);
        ngx_start_cache_manager_processes(cycle, 0);
        ngx_noaccepting = 0;

        continue;
    }

    ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "reconfiguring");

    cycle = ngx_init_cycle(cycle);
    if (cycle == NULL) {
        cycle = (ngx_cycle_t *) ngx_cycle;
        continue;
    }

    ngx_cycle = cycle;
    ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx,
                                           ngx_core_module);
    ngx_start_worker_processes(cycle, ccf->worker_processes,
                               NGX_PROCESS_JUST_RESPAWN);
    ngx_start_cache_manager_processes(cycle, 1);

    /* allow new processes to start */
    ngx_msleep(100);


    live = 1;
    ngx_signal_worker_processes(cycle,
                                ngx_signal_value(NGX_SHUTDOWN_SIGNAL));
}
This corresponds to the reload command, which reads the configuration file again.

 

ngx_ new_ Don't look at the binary flag first, just look at the following. The subsequent actions are obvious. Initialize ngx_cycle_t structure, re read the configuration file. Start the new worker process and Cache process, wait for the start to complete, and then retire the old worker process.
Not in code order, let's talk about NGX first_ change_ Binary status
if (ngx_change_binary) {
    ngx_change_binary = 0;
    ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "changing binary");
    ngx_new_binary = ngx_exec_new_binary(cycle, ngx_argv);
}
The signal processed is USR2.
nginx provides the ability to upgrade smoothly. After smooth start, NGX_ new_ What is saved in binary is pid. So when reload above, ngx_new_binary judgment cannot be executed.
if (ngx_restart) {
    ngx_restart = 0;
    ngx_start_worker_processes(cycle, ccf->worker_processes,
                               NGX_PROCESS_RESPAWN);
    ngx_start_cache_manager_processes(cycle, 0);
    live = 1;
}

This is simply to restart. There is no corresponding signal, that is, NGX is set in the processing logic_ Restart is processed.

if (ngx_reopen) {
    ngx_reopen = 0;
    ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "reopening logs");
    ngx_reopen_files(cycle, ccf->user);
    ngx_signal_worker_processes(cycle,
                                ngx_signal_value(NGX_REOPEN_SIGNAL));
}
The corresponding command is reopen to reset the log file.
if (ngx_noaccept) {
    ngx_noaccept = 0;
    ngx_noaccepting = 1;
    ngx_signal_worker_processes(cycle,
    ngx_signal_value(NGX_SHUTDOWN_SIGNAL));
}

All processes are not receiving requests. Shut down the processes calmly.

3, Summary

Many of the problems to be solved are the problem of signal communication before the process. Before, I didn't have a special understanding of this. Many of them know only a little. I also hope to have a deeper understanding of this later.

Topics: Linux Operation & Maintenance Nginx