In the process of supporting customers, EMQ X learns that customers use Nginx for load balancing and that the Docker container manually joins the cluster to run the EMQ cluster. The main process is now recorded.
Business Requirements
- Use Nginx as a reverse proxy
- Nginx needs to assign the address of the proxy server in advance
- Run EMQ using Docker container
- EMQ Auto Restart
- Automatic Clustering after EMQ Restart
To configure
Nginx Configuration
$ cat /etc/nginx/tcpstream.conf## tcp LB and SSL passthrough for backend ##stream { upstream mqtt_broker { server 127.0.0.1:21871; #max_fails=5 fail_timeout=30s; server 127.0.0.1:21872; #max_fails=5 fail_timeout=30s; server 127.0.0.1:21873; #max_fails=5 fail_timeout=30s; server 127.0.0.1:21874; #max_fails=5 fail_timeout=30s; server 127.0.0.1:21875; #max_fails=5 fail_timeout=30s; server 127.0.0.1:21881; #max_fails=5 fail_timeout=30s; server 127.0.0.1:21891; #max_fails=5 fail_timeout=30s; server 127.0.0.1:21882; #max_fails=5 fail_timeout=30s; server 127.0.0.1:21892; #max_fails=5 fail_timeout=30s; server 127.0.0.1:21883; #max_fails=5 fail_timeout=30s; server 127.0.0.1:21893; #max_fails=5 fail_timeout=30s; server 127.0.0.1:21884; #max_fails=5 fail_timeout=30s; server 127.0.0.1:21894; #max_fails=5 fail_timeout=30s; server 127.0.0.1:21885; #max_fails=5 fail_timeout=30s; server 127.0.0.1:21895; #max_fails=5 fail_timeout=30s; } log_format basic '$proxy_protocol_addr - $remote_addr [$time_local] ' '$protocol $status $bytes_sent $bytes_received ' '$session_time "$upstream_addr" ' '"$upstream_bytes_sent" "$upstream_bytes_received" "$upstream_connect_time"'; access_log /var/log/nginx/access.log basic; error_log /var/log/nginx/error.log; server { listen 8884 ssl; # proxy_protocol; proxy_next_upstream on; #proxy_bind $remote_addr transparent; proxy_ssl off; proxy_pass mqtt_broker; proxy_protocol on; #ssl_on; # adding some extra proxy settings proxy_timeout 350s; #proxy_buffer_size 128k; #ssl_certificate /etc/nginx/certs/solace.pem; #ssl_certificate_key /etc/nginx/certs/solace.pem; ssl_certificate /etc/nginx/certs/cert.pem; ssl_certificate_key /etc/nginx/certs/key.pem; #ssl_verify_client off; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers HIGH:!aNULL:!MD5; } }
Docker Configuration
Customer-compiled Docker image s are not an official mirror provided by EMQ.
The Dockerfile directory is as follows:
$ ll /opt/Docker/Total usage 28 -rw-r--r-- 1 alexeyp emq 620 10 February 2217:26 Dockerfile lrwxrwxrwx 1 alexeyp emq 13 10 24/13:59 emqttd -> emqttd.2.3.11 drwxr-xr-x 10 alexeyp emq 110 10 24/24:27 emqttd.2.3.11 -rwxr-xr-x 1 alexeyp emq 3463 10 February 2605:03 StartEmqInstance.sh -rwxr-xr-x 1 alexeyp alexeyp 270 10 February 2510:46 status.sh
Dockerfile:
$ cat DockerfileFROM centos:latest RUN yum -y update EXPOSE 60000-65000 WORKDIR /opt/emqttd ADD ./emqttd /opt/emqttd ADD ./vsparc.rpm /tmp/vsparc.rpm ADD ./StartEmqInstance.sh /opt/emqttd/StartEmqInstance.sh RUN yum install -y epel-release RUN yum install -y which less sed net-tools telnet gtest /tmp/vsparc.rpm ENV TZ Australia/Melbourne CMD bash /opt/emqttd/StartEmqInstance.sh && bash
You can see that the Docker container executes a script called StartEmqInstance.sh when it is started to view the script:
$ cat StartEmqInstance.sh#!/bin/bashDIR=$(dirname $0) HOSTNAME=$(hostname -s) function adjust_instance() { local INST=$1 local INST_ROOT=$2 cat $INST_ROOT/etc/emq.conf | \ sed -re "s/^node\.name\s*=.*$/node.name = emq$INST@127.0.0.1/" | \ #sed -re "s/^cluster\.name\s*=.*$/cluster.name = $HOSTNAME/" | \ sed -re "s/^listener\.tcp\.external\s*=.*$/listener.tcp.external = 0.0.0.0:6188$INST/" | \ sed -re "s/^listener\.tcp\.external1\s*=.*$/listener.tcp.external1 = 0.0.0.0:6189$INST/" | \ sed -re "s/^listener\.tcp\.external2\s*=.*$/listener.tcp.external2 = 0.0.0.0:6187$INST/" | \ sed -re "s/^listener\.tcp\.internal\s*=.*$/listener.tcp.internal = 127.0.0.1:6298$INST/" | \ sed -re "s/^listener\.ssl\.external\s*=.*$/listener.ssl.external = 6288$INST/" | \ sed -re "s/^listener\.ws\.external\s*=.*$/listener.ws.external = 6208$INST/" | \ sed -re "s/^listener\.wss\.external\s*=.*$/listener.ws.external = 6308$INST/" | \ sed -re "s/^listener\.api\.mgmt\s*=.*$/listener.api.mgmt = 6408$INST/" | \ sed -re "s/^(##\s)?listener\.tcp\.external\.proxy_protocol\s=.*$/listener.tcp.external.proxy_protocol = on/" | \ sed -re "s/^(##\s)?listener\.tcp\.external1\.proxy_protocol\s=.*$/listener.tcp.external1.proxy_protocol = on/" | \ sed -re "s/^(##\s)?listener\.tcp\.external2\.proxy_protocol\s=.*$/listener.tcp.external2.proxy_protocol = on/" | \ sed -re "s/^(##\s)?listener\.tcp\.external\.proxy_protocol_timeout\s=.*$/listener.tcp.external.proxy_protocol_timeout = 30s/" | \ sed -re "s/^(##\s)?listener\.tcp\.external1\.proxy_protocol_timeout\s=.*$/listener.tcp.external1.proxy_protocol_timeout = 30s/" | \ sed -re "s/^(##\s)?listener\.tcp\.external2\.proxy_protocol_timeout\s=.*$/listener.tcp.external2.proxy_protocol_timeout = 30s/" | \ sed -re "s/^(##\s)?node.dist_listen_min\s*=.*$/node.dist_listen_min = 6000$INST/" | \ sed -re "s/^(##\s)?node.dist_listen_max\s*=.*$/node.dist_listen_max = 6000$INST/" | \ cat - > $INST_ROOT/etc/emq.conf.new mv $INST_ROOT/etc/emq.conf.new $INST_ROOT/etc/emq.conf } function cluster_instance() { local INST=$1 for DEST in 1 2 3 4 5; do if [ $DEST == $INST ]; then continue; fi DEST_NODE="emq$DEST@127.0.0.1" RESULT=$(/opt/emqttd/bin/emqttd_ctl cluster join $DEST_NODE 2>&1) echo "$RESULT" echo "$RESULT" | grep -E 'successfully|already' > /dev/null RC=$? [ $RC == 0 ] && break done } cd "$DIR" if [ "$EMQ_INSTANCE_NUMBER" == "" ]; then echo "Environment variable EMQ_INSTANCE_NUMBER(1..10) is not set." echo "eMQ instance name is not configured." exit 1 else adjust_instance $EMQ_INSTANCE_NUMBER $DIR fi function run_application() { local CMD="$1" local RC=1 while [ $RC != 0 ]; do $CMD RC=$? echo "### Exited: $CMD" echo "### rc = $RC" #[ $RC != 0 ] && sleep 3 RC=1 done echo "### Done: $CMD" } function start_node() { bin/emqttd start STARTED=0 while [ $STARTED == 0 ]; do sleep 1 /opt/emqttd/bin/emqttd_ctl status | grep "is running" [ $? == 0 ] && break done cluster_instance $EMQ_INSTANCE_NUMBER > /tmp/cluster_instance.log } start_node sleep 5 run_application "/usr/local/bin/emqtt-stats-collector" &#waitIDLE_TIME=0 while [[ $IDLE_TIME -lt 5 ]] do IDLE_TIME=$((IDLE_TIME+1)) if [[ ! -z "$( /opt/emqttd/bin/emqttd_ctl status|grep 'is running'|awk '{print $1}')" ]]; then IDLE_TIME=0 else echo "['$(date -u +"%Y-%m-%dT%H:%M:%SZ")']:emqttd not running, waiting for recovery in $((60-IDLE_TIME*5)) seconds" fi sleep 5 done echo "['$(date -u +"%Y-%m-%dT%H:%M:%SZ")']:emqttd exit abnormally" exit 1
The script is slightly more complex and needs to be viewed in conjunction with the start.sh script and etc/emq.conf
$ cat start.sh#!/bin/bashfor INST in 1 2 3 4 5 do docker ps | grep -E "\sinstance_$INST$" if [ $? != 0 ]; then #docker run -itd ---ulimit nofile=1048576 -restart=always -v /opt/Docker/emqtt/emq$INST/data/mnesia:/opt/emqttd/data/mnesia -e EMQ_INSTANCE_NUMBER=$INST --name=instance_$INST --network host emq:test & docker run -itd --ulimit nofile=1048576 -e EMQ_INSTANCE_NUMBER=$INST --name=instance_$INST --network host emq:latest & fi done wait
EMQ Configuration
etc/emq.conf`The full text will not be posted out, mainly by adding two tcp Listen on port and close`listener.tcp.external.tune_buffer $ cat etc/emq.conf...... ##-------------------------------------------------------------------- listener.tcp.external = 0.0.0.0:21881 listener.tcp.external.acceptors = 16 listener.tcp.external.max_clients = 512000 listener.tcp.external.access.1 = allow all listener.tcp.external.proxy_protocol = on listener.tcp.external.proxy_protocol_timeout = 30s listener.tcp.external.backlog = 1024 listener.tcp.external.send_timeout = 15s listener.tcp.external.send_timeout_close = on ## listener.tcp.external.tune_buffer = on listener.tcp.external.nodelay = true listener.tcp.external.reuseaddr = true ##-------------------------------------------------------------------- listener.tcp.external1 = 0.0.0.0:21891 listener.tcp.external1.acceptors = 16 listener.tcp.external1.max_clients = 512000 listener.tcp.external1.access.1 = allow all listener.tcp.external1.proxy_protocol = on listener.tcp.external1.proxy_protocol_timeout = 30s listener.tcp.external1.backlog = 1024 listener.tcp.external1.send_timeout = 15s listener.tcp.external1.send_timeout_close = on ## listener.tcp.external1.tune_buffer = on listener.tcp.external1.nodelay = true listener.tcp.external1.reuseaddr = true ##-------------------------------------------------------------------- listener.tcp.external2 = 0.0.0.0:21871 listener.tcp.external2.acceptors = 16 listener.tcp.external2.max_clients = 512000 listener.tcp.external2.access.1 = allow all listener.tcp.external2.proxy_protocol = on listener.tcp.external2.proxy_protocol_timeout = 30s listener.tcp.external2.backlog = 1024 listener.tcp.external2.send_timeout = 15s listener.tcp.external2.send_timeout_close = on ## listener.tcp.external2.tune_buffer = on listener.tcp.external2.nodelay = true listener.tcp.external2.reuseaddr = true ......
Business Analysis
Docker container initialization
After the Docker container is created, StartEmqInstance.sh executes adjust_instance() to modify the port listened on in etc/emq.conf to the proxy server for Nginx
sed -re "s/^node\.name\s*=.*$/node.name = emq$INST@127.0.0.1/" | \ sed -re "s/^listener\.tcp\.external\s*=.*$/listener.tcp.external = 0.0.0.0:6188$INST/" sed -re "s/^listener\.tcp\.external1\s*=.*$/listener.tcp.external1 = 0.0.0.0:6189$INST/" sed -re "s/^listener\.tcp\.external2\s*=.*$/listener.tcp.external2 = 0.0.0.0:6187$INST/" sed -re "s/^listener\.tcp\.internal\s*=.*$/listener.tcp.internal = 127.0.0.1:6298$INST/"
Cluster functionality is implemented through the join command
function cluster_instance() { local INST=$1 for DEST in 1 2 3 4 5; do if [ $DEST == $INST ]; then continue; fi DEST_NODE="emq$DEST@127.0.0.1" RESULT=$(/opt/emqttd/bin/emqttd_ctl cluster join $DEST_NODE 2>&1) echo "$RESULT" echo "$RESULT" | grep -E 'successfully|already' > /dev/null RC=$? [ $RC == 0 ] && break done }
Loop checks the status of EMQ and exits the container when EMQ is stopped
IDLE_TIME=0 while [[ $IDLE_TIME -lt 5 ]] do IDLE_TIME=$((IDLE_TIME+1)) if [[ ! -z "$( /opt/emqttd/bin/emqttd_ctl status|grep 'is running'|awk '{print $1}')" ]]; then IDLE_TIME=0 else echo "['$(date -u +"%Y-%m-%dT%H:%M:%SZ")']:emqttd not running, waiting for recovery in $((60-IDLE_TIME*5)) seconds" fi sleep 5 done echo "['$(date -u +"%Y-%m-%dT%H:%M:%SZ")']:emqttd exit abnormally" exit 1
Visit
Clients connect to <Nginx IP:8884>addresses via SSL, and Nginx loads connections to EMQ nodes via TCP.
PS: Refer to EMQ X message server Nginx reverse proxy for setting up how Nginx reverse proxy tcp and ssl
Auto restart and auto cluster
The state of EMQ is queried by StartEmqInstance.sh script after the container is started. When EMQ stops, it exits the container and cooperates with--restart=always to restart the container.
EMQ stores cluster information in data/mnesia, maps directories in containers to hosts, reads the related directories mapped by hosts when containers are restarted, and implements automatic clustering after restart.
Problem
- Docker's host network mode uses the host's network and is prone to port conflicts when the host has other operations running
Solution
- Modify/proc/sys/net/ipv4/ip_local_port_range to specify that the system assigns a port of 1024 60000, and then assign EMQ's business port to a port after 60000
Practice Cases
It is recommended that you use kubernetes to organize docker containers:
- EMQ can achieve automatic clustering through kube-apiserver.
- The client currently only deploys a docker cluster on a single machine, which can be easily deployed between multiple nodes using kubernetes.
- The deployment of kubernetes can monitor the status of emqx pod, achieve automatic restart, elastic expansion, and other functions.
- Each emqx pod has its own virtual IP, so there is no port conflict.
- kubernetes'Service implements fixed IP and load balancing requirements. In requests created by Service, you can specify your own cluster IP address by setting the spec.cluster IP field, set Nginx's proxy server to cluster IP, and Service can achieve load balancing on its own.
For more information please visit our website emqx.io Or follow our open source projects github.com/emqx/emqx , please visit the detailed documentation Official Documents.
