Supervisor multi process management exception automatic restart visual management

Posted by konky on Mon, 07 Feb 2022 11:58:25 +0100

1, Preface

Supervisor is a multi process management tool. The processes associated in Docker can be managed through supervisor.

The microservice project development stage can be used for the start-up management of microservice sub projects.

Support web visual management, which can greatly help developers to monitor and restart the management of project status.

2, Installation and use

(1) Installation and configuration

1. Service installation

Before installing the service, it is recommended to update the python version. Using a newer version is conducive to service expansion. If the managed service depends on the newer Python version, you need to reinstall the service again.

yum install -y epel-release
yum install -y supervisor

View version number

supervisord -v

2. Configuration file

The configuration file path is / etc / Supervisor Conf, in English; Indicates a comment. Back up the configuration file and form a new configuration file after filtering the annotation configuration.

# Backup profile
mv /etc/supervisord.conf /etc/supervisord.example.conf
# Keep the non annotation configuration and initialize it as a new configuration file
cat /etc/supervisord.example.conf | grep -v '^;' | tr -s "\n" > /etc/supervisord.conf

Use the command echo_supervisord_conf view default configuration

[unix_http_server]
file=/var/run/supervisor/supervisor.sock

; visualization web Monitoring module (no direct comments required)
[inet_http_server]
port=0.0.0.0:9001
username=root
password=root

[supervisord]
logfile=/var/log/supervisor/supervisord.log
; Maximum single log file size
logfile_maxbytes=50MB
; Maximum number of reserved log files
logfile_backups=10
loglevel=info
pidfile=/var/run/supervisord.pid
; If it is set as system service, it needs to be set as false
nodaemon=false
minfds=1024
minprocs=200

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/run/supervisor/supervisor.sock

; Sub process configuration storage directory
[include]
files = supervisord.d/*.ini

View the version number through the following command:

supervisord -v

(2) Start

1. Background start

Set the supervisor to start automatically to ensure that the services entrusted to it can start normally. It is recommended to start this way.

# Set startup and self startup
systemctl enable supervisord

# Start main service
systemctl start supervisord

2. Foreground start

When writing a Docker image, you need to manage multiple services in one image at the same time, and you need to use the foreground to start. The default startup mode of supervisor is daemon. If you want to configure it as foreground startup, you need to modify the configuration file / etc / Supervisor The value of nodaemon attribute in conf is true.

# Replace with script
sed -i 's/nodaemon=false/nodaemon=true/g' /etc/supervisord.conf

The foreground start command is as follows

supervisord -c /etc/supervisord.conf

3. Start parameters

Usually, parameters are added to the configuration file. In some scenarios, it is cumbersome to modify the configuration file (such as the generated image). At this time, it is more convenient to add runtime parameters on the command line.

parameter	purpose	Default value
-c	Specify the configuration file path	/etc/supervisord.conf
-s	The URL that the supervisor server listens on	http://localhost:9001
-u	The user name used to authenticate with the server	user
-p	The password used to authenticate with the server	123

3, Service management

Service management includes the management of main services and sub services; Sub services are divided into single management and batch (grouping) management.

1. View main service status

If you do not specify a sub service name, you will re view the status of all sub services by default. Specify the sub service name and only view the current sub service status.

# View service status
supervisorctl status

Main process management

#Common commands of process management
systemctl stop supervisord
systemctl start supervisord
systemctl restart supervisord

2. Visual interface management

The visual interface adopts different strategies in different development stages of the software. In the project development and testing stages, in order to improve the development efficiency, the visual interface is often opened. When the project is delivered after development, the visual interface is usually closed to ensure the safety of the server.

To start the Web visualization service, you need to add INET in the configuration file_ http_ Server module.

(1) Single service management

Single service management refers to the management of a single sub service. All sub services are in the default group, but are not displayed.

1. Refresh sub service list

When adding new sub services, the list needs to be refreshed before the main service can be included in the management scope.

(1)reload

If the sub service name is not specified, the list of all sub services will be restarted by default. Specify the sub service name. Only restart the current sub service, and other services will not be affected.

All sub services are restarted regardless of whether the configuration is modified or not.

# Refresh service list
supervisorctl reload

(2)update

Restart all sub services with changed configuration (including new sub services). Sub services without changed configuration will not be restarted.

# Refresh service list
supervisorctl update

2. Process management run

In this way, the granularity of management sub processes is smaller.

# Start the specified service
supervisorctl start program_name
# Stop the specified service
supervisorctl stop program_name
# Restart the specified service
supervisorctl restart program_name
# Start all services
supervisorctl start all
# Stop all services
supervisorctl stop all

(2) Group management

When there are associated sub services, grouping management can be adopted. Once grouping is set and sub services are added, the sub service name will change: from the original program_name becomes group_name:program_name, such as redis:redis80

The main service configuration file needs to be modified for group management.

1. View the list of grouped sub services

View the sub service list of the specified group name,

# View the list of grouped sub services
supervisorctl status group_name:

2. Group subprocess management

Manage sub processes in groups, including starting services, stopping services and restarting services.

# Start the service under the specified group name
supervisorctl start group_name:
# Stop the service under the specified group name
supervisorctl stop group_name:
# Restart the service under the specified group name
supervisorctl restart group_name:

Note the colon after the group name:.

3. Grouping application

It is convenient for a group of associated processes to delegate processes to the Supervisor for management and grouping, such as Redis master-slave service, ES cluster, ZK cluster and Kafka cluster. They are a set of sub services with high correlation.

4, Write sub process running configuration file

The configuration file of supervisor main process is / etc / Supervisor conf

In the directory / etc / Supervisor D create a new one ini is the suffix configuration file, and each configuration file represents a child process. Execute the following command to add sub process configuration.

Shortcut script Portal

(1) Parameter interpretation

1,directory

When the subprocess startup command cannot be read from the environment variable, use this parameter to switch to the specified working directory, and then run the entry command.

2,priority

The higher the priority parameter, the lower the priority.

3,environment

If the sub application cannot obtain the system environment variable, it can explicitly indicate the path of the specific environment.

environment=JAVA_HOME=/usr/local/java

(2) Log management

1. View sub process log

After the sub process is managed by the Supervisor, it will generate corresponding operation logs, including access logs and error logs.

; Access log
stdout_logfile=/var/log/park/access.log
; Error log
stderr_logfile=/var/log/park/error.log

Add log configuration in the sub process configuration file to view the sub process log without using the visual interface. The visual Web interface is convenient to view the log, but the defect is that you can't view the error log.

tail -f /var/log/park/access.log

Add parameter stdout to subprocess configuration file_ Logfile and stderr_ The log file of logfile will be automatically included in the main process log management, and the log rotation operation will be carried out automatically without user intervention.

When the child process does not display the indicated log file path, the default log file exists under the / tmp path.

Supervisord will be based on logfile_maxbytes and logfile_backups rotates logs. The former limits the size of a single log file, while the latter limits the number of log backups. This configuration exists with the master configuration file, not the child process configuration file.

(3) Common component configuration

1,Nginx

cat <<EOF> /etc/supervisord.d/nginx.ini
[program:nginx]
directory=/usr/local/nginx/sbin
command=/usr/local/nginx/sbin/nginx -g 'daemon off;'
; Automatically start the current sub service when the main service starts
autostart=true
; The sub service exits abnormally and restarts automatically
autorestart=true
; Sub service startup time (consistent with the time as far as possible)
startsecs=5
startretries=3
redirect_stderr=true
stdout_logfile=/usr/local/nginx/logs/access.log
stderr_logfile=/usr/local/nginx/logs/error.log
EOF

The execution mode of the child process is the previous execution mode of the background process, which must not be used.

2,Redis

cat <<EOF> /etc/supervisord.d/redis.ini
[program:redis]
command=/usr/local/redis/bin/redis-server /usr/local/redis/conf/redis.conf --daemonize no
autostart=true
autorestart=true
startsecs=5
startretries=3
redirect_stderr=true
EOF

The commands of the subprocess must be executed in the way of previous running, and cannot be executed in the way of background running.

3,Nacos

cat <<EOF> /etc/supervisord.d/nacos.ini
[program:nacos]
command=sh /usr/local/nacos/bin/startup.sh -m standalone
autostart=true
autorestart=true
startsecs=8
startretries=3
redirect_stderr=true
environment=JAVA_HOME=/usr/local/java
priority=1
EOF