Introduction to supervisor

First, let's introduce supervisor. Supervisor( http://supervisord.org )It is a client/server service developed in Python. It is a process management tool under Linux/Unix system. It does not support Windows system. It can easily monitor, start, stop and restart one or more processes. For a process managed by supervisor, when a process is accidentally killed, supervisor will automatically pull it up after listening to the process death. It is very convenient to achieve the function of automatic recovery of the process, and there is no need to write your own shell script to control it
Environment: CentOS 7 four

Install supervisor

pip install supervisor

Because it is a library developed by python, it can be installed directly with pip, which is very convenient.
After supervisor is installed, three executors will be generated: supervisortd, supervisorctl and echo_supervisord_conf is the daemon service of Supervisor (used to receive process management commands), client (used to communicate with the daemon and send instructions to the management process), and the program for generating the initial configuration file.

Setting environment variables

$ vim ~/.bash_profile
#Add later: 
PATH=$PATH:$HOME/bin:/usr/local/python/bin
$ source ~/.bash_profile

Configure supervisor

Create directory and initialize configuration file

mkdir /etc/supervisor
echo_supervisord_conf > /etc/supervisor/supervisord.conf

Master profile parameters

[unix_http_server]
file=/tmp/supervisor.sock   ; UNIX socket Documents, supervisorctl Can use
;chmod=0700                 ; socket Document mode，The default is 0700
;chown=nobody:nogroup       ; socket Document owner，Format: uid:gid
  
;[inet_http_server]         ; HTTP Server, providing web Management interface
;port=127.0.0.1:9001        ; Web Manage background running IP And ports. If you open to the public network, you need to pay attention to security
;username=user              ; User name of login management background
;password=123               ; Password for login management background
  
[supervisord]
logfile=/tmp/supervisord.log ; Log file, default is $CWD/supervisord.log
logfile_maxbytes=50MB        ; Log file size, exceeding rotate，Default 50 MB
logfile_backups=10           ; The number of log file reserved backups is 10 by default
loglevel=info                ; Log level, default info，other: debug,warn,trace
pidfile=/tmp/supervisord.pid ; pid file
nodaemon=false               ; Whether to start in the foreground. The default is false，Namely daemon Start by
minfds=1024                  ; The minimum value of the file descriptor that can be opened. The default value is 1024
minprocs=200                 ; The minimum number of processes that can be opened. The default is 200
  
; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
  
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ;  Connect supervisor through UNIX socket, and the path is the same as UNIX_ http_ The file of the server part is consistent
;serverurl=http://127.0.0.1:9001 ;  Connect to supervisor via HTTP
  
; Contains additional configuration files
[include]
files = relative/directory/*.ini    ; Can be *.conf or *.ini

Manage a process

Put all managed process configuration files in the same directory and include the main configuration file.

mkdir /etc/supervisor/config.d

vim /etc/supervisor/supervisord.conf
[include]
files = /etc/supervisor/config.d/*.conf

Create a new profile

[program:sougou]   #souogu is the name of the program
command=scrapy crawl sougou #Commands to execute
directory=/home/mzj/desktop/sougou/sougou/wechat_name/wechat_name/spiders  #Directory of command execution
environment=ASPNETCORE__ENVIRONMENT=Production #environment variable
user=root #user
stopsignal=INT
autostart=true #Self start
autorestart=true #Auto restart
startsecs=3 #Automatic restart interval (s)
stderr_logfile=/home/mzj/desktop/sougou/sougou/wechat_name/wechat_name/spiders/ossoffical.err.log #Error log file
stdout_logfile=/home/mzj/desktop/sougou/sougou/wechat_name/wechat_name/spiders/ossoffical.out.log #Output log file

[program:nginx]
command = /usr/local/nginx/sbin/nginx -g 'daemon off;'
startsecs = 3
autostart = true
autorestart = true 
user = root
stdout_logfile = /etc/supervisor/logs/supervisord-nginx.log
stderr_logfile = /etc/supervisor/logs/supervisord-nginx-error.log

At the beginning, there was a pit, command = /usr/local/nginx/sbin/nginx. As a result, I started nginx all the time. Because in this case, nginx runs in the background. However, supervisor cannot manage the background running process, so turn off the nginx daemon off guard and let it block the bash running in the foreground to facilitate supervisor management.

Start supervisor

supervisord -c /etc/supervisor/supervisord.conf

Sometimes the connection needs to be released after the kill process

unlink /tmp/supervisor.sock

web interface management

Turn on web access

vi /etc/supervisor/supervisord.conf
[inet_http_server]        
port=0.0.0.0:9001       
username=user            
password=123

Note that the comments in front of [inet_http_server] should also be removed here

Usage of supervisorctl

• Supervisor: start supervisor
• Supervisor CTL reload: restart supervisor after modifying the configuration file
• supervisorctl status: view the process status supervised by the supervisor
• supervisorctl start process name: start XXX process
• supervisorctl stop process name: stop XXX process
• supervisorctl stop all: stop all processes. Note: start, restart and stop will not load the latest configuration file.
• supervisorctl update: start the process with new configuration or changes according to the latest configuration file. The process without configuration changes will not be affected and restarted

Introduction to superlance

Superlance is a set of command-line tools based on the event mechanism of supervisor. It implements many practical process monitoring and management features that supervisor itself does not implement, including memory monitoring, http interface monitoring, e-mail and SMS notification mechanism, etc. Similarly, superlance itself is written in python
Components of superlance
superlance is a collection of command line tools, including the following commands:

httpok

Make a GET request to an HTTP interface regularly. Determine whether a process is in normal state according to whether the request is successful. If not, restart the process.

crashmail

When a process exits unexpectedly, an email alarm is sent.

memmon

When the memory occupation of a process exceeds the set threshold, an email alarm is sent.

crashmailbatch

It is similar to the crashmail alarm, but the mail will be combined and sent over a period of time to avoid mail bombing.

fatalmailbatch

When a process fails to start successfully for many times, it will enter the FATAL state, and an email alarm will be sent at this time. Like crashmailbatch, synthetic alarms will be generated.

crashsms

When a process exits unexpectedly, a short message alarm is sent, which is also sent through the email gateway.

1. When supervisor is started, if our listener is configured as autostart=true, the listener will be started as a child process of supervisor.
2. After the listener is started, it will write a "READY" message to its stdout. At this time, the parent process, that is, supervisor, will think that the listener is READY after reading this message.
3. After the listener is in the ready state, when the event generated by supervisor is in the acceptable events configured by the listener, supervisor will send the event to the listener.
4. After the listener receives the event, we can do a series of processing according to the data in the head and body of the event. We judge, extract, alarm and so on according to the content of event.
5. After all the work is done, the listener needs to write a message "RESULTnOK" to his stdout. After the supervisor receives this message. I know that the listener has finished processing the event.

Event s supported by supervisor

PROCESS_STATE    The process state has changed
PROCESS_STATE_STARTING  Process state transitions from other state to starting(Supervisord There are startsecs Configuration item means that the program needs to run stably at least when the program is started x Seconds before the program runs normally, here x The program status is starting in seconds)
PROCESS_STATE_RUNNING   The process status changes from starting to running
PROCESS_STATE_BACKOFF   The process status changes from starting to failed
PROCESS_STATE_STOPPING   The process status changes from running to stopping
PROCESS_STATE_EXITED   The process status changes from running to exiting
PROCESS_STATE_STOPPED   The process status changes from being stopped to being stopped(exited and stopped The difference is exited The program exits by itself, and stopped It is artificially controlled to exit)
PROCESS_STATE_FATAL   The process status changes from running to failed
PROCESS_STATE_UNKNOWN   Unknown process state
REMOTE_COMMUNICATION   use Supervisord of RPC Interface and Supervisord Communicate
PROCESS_LOG   The process generates log output, including standard output and standard error output
PROCESS_LOG_STDOUT   The process produces standard output
PROCESS_LOG_STDERR   The process produces standard error output
PROCESS_COMMUNICATION   The log output of the process contains and
PROCESS_COMMUNICATION_STDOUT   The standard output of the process includes and
PROCESS_COMMUNICATION_STDERR   The standard error output of the process contains and
SUPERVISOR_STATE_CHANGE_RUNNING Supervisord  start-up
SUPERVISOR_STATE_CHANGE_STOPPING Supervisord  stop it
TICK_5   Triggered every 5 seconds
TICK_60   Triggered every 60 seconds
TICK_3600   Triggered every 3600
PROCESS_GROUP   Supervisord The process group has changed
PROCESS_GROUP_ADDED   Added Supervisord Process group for
PROCESS_GROUP_REMOVED   Deleted Supervisord Process group for

Install superlance

It is also a python program, which can be installed directly through pip

pip install superlance

superlance actual combat

Before actually configuring the supervisor to use superlance, first install sendmail. Superlance uses this command-line tool to send email. It can be easily installed by using various package management tools:

wget http://caspian.dotconf.net/menu/Software/SendEmail/sendEmail-v1.56.tar.gz
tar zxvf sendEmail-v1.56.tar.gz
cp sendEmail-v1.56/sendEmail /usr/bin/sendemail

Send an email to test it

sendemail -f Sender mailbox -t Recipient mailbox -s Sender mailbox smtp The server -u "sendEmail" -m "haha" -xu Sender mailbox -xp Sender mailbox password 
 
-f Sender
-t addressee
-s Sender smtp The server
-u theme
-m content
-xu Sender user name
-xp Sender password

What we need to do now is to configure a supervisor configuration file to monitor the status changes of the process all the time

[eventlistener:crashmail]
command=crashmail -a -s "sendemail -f dataalarm@wjoyxt.com -t alarm@wjoyxt.com -s smtp.mxhichina.com -u 'Process is crashed' -xu dataalarm@wjoyxt.com -xp Bigdata@2019@ -m >/dev/null 2>&1" -m alarm@wjoyxt.com
events=PROCESS_STATE_EXITED
redirect_stderr=false

or

[eventlistener:crashmail]
command=crashmail -a -s "mail -s 'Process is crashed' alarm@wjoyxt.com" -m alarm@wjoyxt.com
events=PROCESS_STATE_EXITED
redirect_stderr=false

The - p parameter in the command parameter is configured with crashmail, which will only respond to the monitoring item named redis, while the - m parameter is configured with the address to which the crash mail will be sent, and - a monitors all items

Crashmail is a supervisor "event listener" for subscribing to PROCESS_STATE_EXITED event. When crashmail receives the event and the conversion is "not expected", crashmail will send an email notification to the configured address

events=PROCESS_STATE_EXITED,PROCESS_STATE_STOPPED,PROCESS_STATE_FATAL,PROCESS_LOG_STDERR multiple events are separated by commas

PROCESS_STATE_EXITED is an event that will be triggered when the process corresponding to a supervisor's monitoring item exits unexpectedly, which will notify crashmail when a process exits unexpectedly.

crashmail.py [-p processname] [-a] [-o string] [-m mail_address]
             [-s sendmail]website

Options:

-p  - appoint supervisor process_name. This process sends mail
      Unexpected conversion to EXITED Status. If this process is
      Part of a group that can be used
      'process_name: group_name'Grammar.

-a  - When supervisord Send mail when any child transitions
      Left unexpectedly EXITED Status. Cover any-p
      In the same crashmail Parameters passed in a process call.

-o  - Specifies the parameter to use as a prefix in the subject header of the message.

-s  - For sending e-mail sendmail command
      (For example“ / usr / sbin / sendmail - t - i "). Must be an accepted command
      stdin Header and message data on and send mail. The default is
      " / usr / sbin / sendmail - t - i ". 

-m  - Specify an e-mail address. The script will send a message to this
      crashmail The address at which the process crash was detected. If there is no email
      If an address is specified, no e-mail will be sent.

You can specify multiple times-p Options, allow
 Specification of multiple processes. appoint-a Will overwrite any
 choice-p. 

Example call:

crashmail.py -p program1 -p group1: program2 -m dev@example.com

After adding a process configuration file

supervisorctl reload

Restart it

Overall sample code

[program:eacloud]
command=/root/python/bin/uwsgi --ini uwsgi.ini ; Program start command
directory=/root/tyy
autostart=true       ; stay supervisord It also starts automatically when starting
startsecs=10         ; If there is no abnormal exit after 10 seconds of startup, it means that the process is started normally. The default is 1 second
autorestart=true     ; Automatic restart after program exit,Optional values:[unexpected,true,false]，Default to unexpected，Indicates that the process was restarted after being killed unexpectedly
startretries=3       ; The number of automatic retries after startup failure. The default is 3
user=root            ; Which user is used to start the process? The default is root
priority=999         ; The process startup priority is 999 by default. The process with a small value will be started first
redirect_stderr=true ; hold stderr Redirect to stdout，default false
stdout_logfile_maxbytes=20MB  ; stdout Log file size, default 50 MB
stdout_logfile_backups = 20   ; stdout The number of log file backups. The default is 10
; stdout The log file cannot be started normally when the specified directory does not exist, so you need to create the directory manually( supervisord (log files are created automatically)
stdout_logfile=/root/tyy/eacloud.log
stopasgroup=false     ;Default to false,Whether to send to this process group when the process is killed stop Signals, including child processes
killasgroup=false     ;Default to false，Send to process group kill Signals, including child processes
[eventlistener:crashmail-exited]
command=/root/python/bin/crashmail -p eacloud -m 1524701427@qq.com
events=PROCESS_STATE
redirect_stderr=false