1 - [high performance Nginx server] - 8 HTTP dynamic load balancing

Posted by BigMonkey on Wed, 17 Nov 2021 02:06:55 +0100

1 what is dynamic load balancing

In traditional load balancing, if the Upstream parameters change, the nginx.conf file needs to be reloaded every time, so the scalability is not very high. Therefore, we can adopt dynamic load balancing to realize the configurability and dynamics of Upstream without manually reloading nginx.conf.

This is similar to a distributed configuration center.

2. Implementation scheme of dynamic load balancing

reload nginx and restart Nginx every time a configuration change is found.

  1. Consul + Consul-template
  2. Consul + OpenResty realizes dynamic load balancing without reload
  3. Consul + upsync + Nginx realizes dynamic load balancing without reload

3 common server registration and discovery framework

Common service discovery frameworks Consul, Eureka, ZooKeeper and Etcd.

ZooKeeper is one of the oldest projects of this type. It originated from Hadoop. It is very mature and reliable, and is used by many large companies (YouTube, eBay, Yahoo, etc.).

etcd is a key / value pair storage system using HTTP protocol. It is a distributed and functional hierarchical configuration system, which can be used to build a service discovery system. It is easy to deploy, install and use, and provides reliable data persistence features. It is secure and well documented.

4 Consul quick start

Consul is an open source distributed service registration and discovery system. It can make service registration and discovery very simple through HTTP API. It supports the following features.

  • Service registration: the service implementer can register the service to consult through HTTP API or DNS.
  • Service discovery: service consumers can obtain the service IP and PORT from Consul through HTTP API or DNS.
  • Fault detection: it supports health check mechanisms such as TCP and HTTP, so that it can be automatically removed when there is a fault in the service.
  • K/V storage: use K/V storage to realize dynamic configuration center, which uses HTTP long polling to realize change triggering and configuration change.
  • Multiple data centers: support multiple data centers. You can register and discover services according to the data center, that is, you can only consume local computer room services. Using multiple data center clusters can also avoid single point of failure of a single data center.
  • Raft algorithm: Consul t uses raft algorithm to achieve cluster data consistency.

Consul can manage service registration and discovery. Next, an Agent deployed on the same machine as Nginx is required to implement Nginx configuration change and Nginx restart functions. We have two choices: Confd or consult template, and the consult template is officially provided by consult, so we choose it. It uses HTTP long polling for change triggering and configuration changes (using the watch command of consul). That is, we use the consult template to implement the configuration template, and then pull the consult configuration rendering template to generate the actual configuration of Nginx.

5 principle of nginx + consul + upsync

ConsulServer is used to store load balancing configuration

ConsulWeb serves as a visual configuration file modification interface

Nginx will read the configuration file on ConsulServer through the UpSync interval and cache a configuration file locally to avoid service unavailability due to the downtime of the configuration center.

6. Construction of consul environment

1. Download consumer_ 0.7.5_ linux_ amd64.zip

wget https://releases.hashicorp.com/consul/0.7.5/consul_0.7.5_linux_amd64.zip

2. Unzip the consumer_ 0.7.5_ linux_ amd64.zip

unzip consul_0.7.5_linux_amd64.zip

If this error occurs during decompression

-bash: unzip: Command not found

terms of settlement

yum -y install unzip

3. Execute the following. / consumer. The following information indicates that the installation is successful

[root@weaver-1 ~]# ./consul
usage: consul [--version] [--help] <command> [<args>]

Available commands are:
    agent          Runs a Consul agent
    configtest     Validate config file
    event          Fire a new event
    exec           Executes a command on Consul nodes
    force-leave    Forces a member of the cluster to enter the "left" state
    info           Provides debugging information for operators
    join           Tell Consul agent to join cluster
    keygen         Generates a new encryption key
    keyring        Manages gossip layer encryption keys
    kv             Interact with the key-value store
    leave          Gracefully leaves the Consul cluster and shuts down
    lock           Execute a command holding a lock
    maint          Controls node or service maintenance mode
    members        Lists the members of a Consul cluster
    monitor        Stream logs from a Consul agent
    operator       Provides cluster-level tools for Consul operators
    reload         Triggers the agent to reload configuration files
    rtt            Estimates network round trip time between nodes
    snapshot       Saves, restores and inspects snapshots of Consul server state
    version        Prints the Consul version
    watch          Watch for changes in Consul

[root@weaver-1 ~]# 

4. Start consumer

./consul agent -dev -ui -node=consul-dev -client=

5. Temporarily close the firewall

systemctl stop firewalld

6. Browser access:

7. Use PostMan to register Http services
  • Datacenter: Specifies the data center
  • Address: Specifies the service IP
  • Service.Id: Specifies the unique ID of the service
  • Service.Service: Specifies the service grouping
  • Service.tags: Specifies the service tag (such as test environment, advance environment, etc.)
  • Service.Port: Specifies the service port.

7. Discover Http services

7 nginx-upsync-module

Note: clear the previous Nginx environment and reinstall.

7.1 introduction to nginx upsync module

Upsync is an open-source three-party module of sina Weibo to realize dynamic configuration based on Nginx. The function of Nginx upsync module is to pull the list of consul's backend server s and dynamically update the routing information of Nginx. This module does not depend on any third-party modules. Consul, as the DB of Nginx, uses consul's KV service. Each Nginx Work process independently pulls the configuration of each upstream and updates its route.

7.2 nginx upsync module installation

Download File

cd /usr/local/

1. Download Nginx

wget http://nginx.org/download/nginx-1.9.10.tar.gz

Function: realize reverse proxy and load library

2. Download consumer

wget https://releases.hashicorp.com/consul/0.7.1/consul_0.7.1_linux_amd64.zip

Function: register the dynamic load balancing configuration

3. Download nginx upsync module

wget https://github.com/weibocom/nginx-upsync-module/archive/master.zip

Function: nginx dynamically obtains the latest upstream information

Decompression installation

unzip master.zip

unzip consul_0.7.1_linux_amd64.zip

If this error occurs during decompression

-bash: unzip: Command not found

terms of settlement

yum -y install unzip

Installing Nginx

Unzip Nginx

tar -zxvf nginx-1.9.10.tar.gz

Configure Nginx

groupadd nginx
useradd -g nginx -s /sbin/nologin nginx
mkdir -p /var/tmp/nginx/client/
mkdir -p /usr/local/nginx

Compile Nginx

cd nginx-1.9.10
./configure   --prefix=/usr/local/nginx   --user=nginx   --group=nginx   --with-http_ssl_module   --with-http_flv_module   --with-http_stub_status_module   --with-http_gzip_static_module   --with-http_realip_module   --http-client-body-temp-path=/var/tmp/nginx/client/   --http-proxy-temp-path=/var/tmp/nginx/proxy/   --http-fastcgi-temp-path=/var/tmp/nginx/fcgi/   --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi   --http-scgi-temp-path=/var/tmp/nginx/scgi   --with-pcre --add-module=../nginx-upsync-module-master
make && make install

The compilation is error reporting

./configure: error: SSL modules require the OpenSSL library.

terms of settlement

yum -y install openssl openssl-devel

Upstream dynamic configuration

## Dynamically go to the consumer to obtain the registered real reverse proxy address
upstream test {
    upsync upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off;
    upsync_dump_path /usr/local/nginx/conf/servers/servers_test.conf;

server {
    listen       80;
    server_name  localhost;

    location / {
        proxy_pass http://test;
        index  index.html index.htm;
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
  • The upsync instruction specifies which path to pull the upstream server configuration from the consumer;
  • upsync_timeout configures the timeout time configured by the upstream server pulled from the consumer;
  • upsync_interval configures the interval between pulling the upstream server configuration from the consumer;
  • upsync_type specifies to use consumer to configure the server;
  • strong_dependency configures whether nginx is forced to depend on the configuration server during startup. If the configuration is on, nginx startup also fails when pulling configuration fails.
  • upsync_dump_path specifies the location where the upstream server pulled from the consumer will be persisted, so that even if the consumer server fails, there will be a local backup.

Note: replace the consumer registry address

Create upsync_dump_path

mkdir /usr/local/nginx/conf/servers/

upsync_dump_path specifies the location where the upstream server pulled from the consumer will be persisted, so that even if the consumer server fails, there will be a local backup.

Start consumer

Temporarily turn off the firewall

systemctl stop firewalld
/usr/local/consul agent -dev -ui -node=consul-dev -client=

Start two Tomcat services


Start Nginx


Add nginx Upstream service

  1. Send put request using postmen


Modify load balancing information parameters

{"weight":1, "max_fails":2, "fail_timeout":10, "down":0}
{"weight":2, "max_fails":2, "fail_timeout":10, "down":0}

After the test, you can find that the weight has taken effect.

Topics: Nginx server http