1 what is dynamic load balancing
In traditional load balancing, if the Upstream parameters change, the nginx.conf file needs to be reloaded every time, so the scalability is not very high. Therefore, we can adopt dynamic load balancing to realize the configurability and dynamics of Upstream without manually reloading nginx.conf.
This is similar to a distributed configuration center.
2. Implementation scheme of dynamic load balancing
reload nginx and restart Nginx every time a configuration change is found.
- Consul + Consul-template
- Consul + OpenResty realizes dynamic load balancing without reload
- Consul + upsync + Nginx realizes dynamic load balancing without reload
3 common server registration and discovery framework
Common service discovery frameworks Consul, Eureka, ZooKeeper and Etcd.
ZooKeeper is one of the oldest projects of this type. It originated from Hadoop. It is very mature and reliable, and is used by many large companies (YouTube, eBay, Yahoo, etc.).
etcd is a key / value pair storage system using HTTP protocol. It is a distributed and functional hierarchical configuration system, which can be used to build a service discovery system. It is easy to deploy, install and use, and provides reliable data persistence features. It is secure and well documented.
4 Consul quick start
Consul is an open source distributed service registration and discovery system. It can make service registration and discovery very simple through HTTP API. It supports the following features.
- Service registration: the service implementer can register the service to consult through HTTP API or DNS.
- Service discovery: service consumers can obtain the service IP and PORT from Consul through HTTP API or DNS.
- Fault detection: it supports health check mechanisms such as TCP and HTTP, so that it can be automatically removed when there is a fault in the service.
- K/V storage: use K/V storage to realize dynamic configuration center, which uses HTTP long polling to realize change triggering and configuration change.
- Multiple data centers: support multiple data centers. You can register and discover services according to the data center, that is, you can only consume local computer room services. Using multiple data center clusters can also avoid single point of failure of a single data center.
- Raft algorithm: Consul t uses raft algorithm to achieve cluster data consistency.
Consul can manage service registration and discovery. Next, an Agent deployed on the same machine as Nginx is required to implement Nginx configuration change and Nginx restart functions. We have two choices: Confd or consult template, and the consult template is officially provided by consult, so we choose it. It uses HTTP long polling for change triggering and configuration changes (using the watch command of consul). That is, we use the consult template to implement the configuration template, and then pull the consult configuration rendering template to generate the actual configuration of Nginx.
5 principle of nginx + consul + upsync
ConsulServer is used to store load balancing configuration
ConsulWeb serves as a visual configuration file modification interface
Nginx will read the configuration file on ConsulServer through the UpSync interval and cache a configuration file locally to avoid service unavailability due to the downtime of the configuration center.
6. Construction of consul environment
1. Download consumer_ 0.7.5_ linux_ amd64.zip
wget https://releases.hashicorp.com/consul/0.7.5/consul_0.7.5_linux_amd64.zip
2. Unzip the consumer_ 0.7.5_ linux_ amd64.zip
unzip consul_0.7.5_linux_amd64.zip
If this error occurs during decompression
-bash: unzip: Command not found
terms of settlement
yum -y install unzip
3. Execute the following. / consumer. The following information indicates that the installation is successful
./consul
[root@weaver-1 ~]# ./consul
usage: consul [--version] [--help] <command> [<args>]
Available commands are:
agent Runs a Consul agent
configtest Validate config file
event Fire a new event
exec Executes a command on Consul nodes
force-leave Forces a member of the cluster to enter the "left" state
info Provides debugging information for operators
join Tell Consul agent to join cluster
keygen Generates a new encryption key
keyring Manages gossip layer encryption keys
kv Interact with the key-value store
leave Gracefully leaves the Consul cluster and shuts down
lock Execute a command holding a lock
maint Controls node or service maintenance mode
members Lists the members of a Consul cluster
monitor Stream logs from a Consul agent
operator Provides cluster-level tools for Consul operators
reload Triggers the agent to reload configuration files
rtt Estimates network round trip time between nodes
snapshot Saves, restores and inspects snapshots of Consul server state
version Prints the Consul version
watch Watch for changes in Consul
[root@weaver-1 ~]#
4. Start consumer
./consul agent -dev -ui -node=consul-dev -client=192.168.153.11
5. Temporarily close the firewall
systemctl stop firewalld
6. Browser access: http://192.168.153.11:8500/
7. Use PostMan to register Http services
http://192.168.153.11:8500/v1/catalog/register
{
"Datacenter":"dc1",
"Node":"tomcat",
"Address":"192.168.153.11",
"Service":{
"Id":"192.168.153.11:8001",
"Service":"test",
"tags":[
"dev"
],
"Port":8001
}
}
{
"Datacenter":"dc1",
"Node":"tomcat",
"Address":"192.168.153.11",
"Service":{
"Id":"192.168.153.11:8002",
"Service":"test",
"tags":[
"dev"
],
"Port":8002
}
}
- Datacenter: Specifies the data center
- Address: Specifies the service IP
- Service.Id: Specifies the unique ID of the service
- Service.Service: Specifies the service grouping
- Service.tags: Specifies the service tag (such as test environment, advance environment, etc.)
- Service.Port: Specifies the service port.
7. Discover Http services
http://192.168.153.11:8500//v1/catalog/service/item_jd_tomcat
7 nginx-upsync-module
Note: clear the previous Nginx environment and reinstall.
7.1 introduction to nginx upsync module
Upsync is an open-source three-party module of sina Weibo to realize dynamic configuration based on Nginx. The function of Nginx upsync module is to pull the list of consul's backend server s and dynamically update the routing information of Nginx. This module does not depend on any third-party modules. Consul, as the DB of Nginx, uses consul's KV service. Each Nginx Work process independently pulls the configuration of each upstream and updates its route.
7.2 nginx upsync module installation
Download File
cd /usr/local/
1. Download Nginx
wget http://nginx.org/download/nginx-1.9.10.tar.gz
Function: realize reverse proxy and load library
2. Download consumer
wget https://releases.hashicorp.com/consul/0.7.1/consul_0.7.1_linux_amd64.zip
Function: register the dynamic load balancing configuration
3. Download nginx upsync module
wget https://github.com/weibocom/nginx-upsync-module/archive/master.zip
Function: nginx dynamically obtains the latest upstream information
Decompression installation
unzip master.zip
unzip consul_0.7.1_linux_amd64.zip
If this error occurs during decompression
-bash: unzip: Command not found
terms of settlement
yum -y install unzip
Installing Nginx
Unzip Nginx
tar -zxvf nginx-1.9.10.tar.gz
Configure Nginx
groupadd nginx
useradd -g nginx -s /sbin/nologin nginx
mkdir -p /var/tmp/nginx/client/
mkdir -p /usr/local/nginx
Compile Nginx
cd nginx-1.9.10
./configure --prefix=/usr/local/nginx --user=nginx --group=nginx --with-http_ssl_module --with-http_flv_module --with-http_stub_status_module --with-http_gzip_static_module --with-http_realip_module --http-client-body-temp-path=/var/tmp/nginx/client/ --http-proxy-temp-path=/var/tmp/nginx/proxy/ --http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi --http-scgi-temp-path=/var/tmp/nginx/scgi --with-pcre --add-module=../nginx-upsync-module-master
make && make install
The compilation is error reporting
./configure: error: SSL modules require the OpenSSL library.
terms of settlement
yum -y install openssl openssl-devel
Upstream dynamic configuration
## Dynamically go to the consumer to obtain the registered real reverse proxy address
upstream test {
server 127.0.0.1:11111;
upsync 192.168.153.11:8500/v1/kv/upstreams/test upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off;
upsync_dump_path /usr/local/nginx/conf/servers/servers_test.conf;
}
server {
listen 80;
server_name localhost;
location / {
proxy_pass http://test;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
- The upsync instruction specifies which path to pull the upstream server configuration from the consumer;
- upsync_timeout configures the timeout time configured by the upstream server pulled from the consumer;
- upsync_interval configures the interval between pulling the upstream server configuration from the consumer;
- upsync_type specifies to use consumer to configure the server;
- strong_dependency configures whether nginx is forced to depend on the configuration server during startup. If the configuration is on, nginx startup also fails when pulling configuration fails.
- upsync_dump_path specifies the location where the upstream server pulled from the consumer will be persisted, so that even if the consumer server fails, there will be a local backup.
Note: replace the consumer registry address
Create upsync_dump_path
mkdir /usr/local/nginx/conf/servers/
upsync_dump_path specifies the location where the upstream server pulled from the consumer will be persisted, so that even if the consumer server fails, there will be a local backup.
Start consumer
Temporarily turn off the firewall
systemctl stop firewalld
/usr/local/consul agent -dev -ui -node=consul-dev -client=192.168.153.11
Start two Tomcat services
/root/tomcat-1/bin/startup.sh
/root/tomcat-2/bin/startup.sh
Start Nginx
/usr/local/nginx/sbin/nginx
Add nginx Upstream service
- Send put request using postmen
http://192.168.153.11:8500/v1/kv/upstreams/test/192.168.153.11:8001
http://192.168.153.11:8500/v1/kv/upstreams/test/192.168.153.11:8002
Test:
Modify load balancing information parameters
{"weight":1, "max_fails":2, "fail_timeout":10, "down":0}
{"weight":2, "max_fails":2, "fail_timeout":10, "down":0}
After the test, you can find that the weight has taken effect.