Practical test of deploying rabbitMQ image cluster

Posted by JayBachatero on Tue, 30 Jun 2020 03:06:29 +0200

Deploying rabbitMQ image cluster

Version information

rabbit MQ: 3.8.5
 Erlang: Official recommended minimum 21.3 recommended 22.x
         Here's 23

Environmental preparation

Host planning

host node Disk node Memory node Disk node
Memory node:
    The memory node stores all the metadata definitions of queues, switches, bindings, users, permissions and vhost s in memory. The advantage is that it can make operations such as switch and queue declaration faster. The exception is: the contents of the persistent queue will be saved to disk.

Disk node:
    The metadata is stored in the disk, and only disk type nodes are allowed in single node system to prevent the loss of system configuration information when RabbitMQ is restarted.

    1. The performance of memory node is higher than that of disk node because it does not read and write to disk.
    2. There can be multiple disk nodes in a cluster. The more disk nodes, the better the availability of the whole cluster. However, the overall performance of the cluster will not increase linearly, which needs to be considered.
    3. RabbitMQ requires at least one disk node in the cluster, and all other nodes can be memory nodes. When a node joins or leaves the cluster, it must notify at least one disk node of the change. If the only disk node in the cluster crashes, the cluster can still keep running, but no other operations (add, delete, modify, check) can be performed until the node recovers.
    4. Set up two disk nodes, at least one of which is available to save metadata changes.

Download offline package

Official website installation manual(

rabbit MQ: binary version
➜  wget

Erlang: no dependencies -- the package strips off Erlang modules and dependencies that are not necessary to run RabbitMQ.
➜  wget

Install offline package

# Install erlang
yum install -y yum install erlang-22.2.8-1.el7.x86_64.rpm 

# Decompress rabbitmq
xz -d rabbitmq-server-generic-unix-3.8.5.tar.xz 
tar -xvf rabbitmq-server-generic-unix-3.8.5.tar -C /opt

hosts file    MQ1    MQ2    MQ3

configuration file

We need to create rabbitmq ourselves in $Home/etc/rabbitmq-env.conf , see for more information Official configuration description

# Create persistent directory
mkdir -p /data/rabbitmq/store
mkdir -p /data/rabbitmq/logs 

# create profile
vim /opt/rabbitmq_server-3.8.5/etc/rabbitmq/rabbitmq-env.conf
# Specify the name of the node. The default is rabbit @ ${host name} 
# Specify the port, the default is 5672 
# Configure persistent directory 
# Configuration log directory default file name: ${NODENAME}.log can be modified with configuration 

Enable service

Common commands

sbin/rabbitmq-server                          # Start the server 
sbin/rabbitmq-server -detached                # Start server in the background 
sbin/rabbitmqctl status                       # View node status 
sbin/rabbitmqctl shutdown                     # Stopped nodes 
sbin/rabbitmqctl stop_app                     # Stop mq service
sbin/rabbitmqctl start_app                    # Start mq service
sbin/rabbitmqctl cluster_status               # View cluster status 
sbin/rabbitmqctl set_cluster_name rabbit@MQ1  # Modify cluster name 
sbin/rabbitmqctl join_cluster <cluster_name>  # Join the cluster 
sbin/rabbitmqctl change_cluster_node_type --node <node_name> [ disk | ram ]  # Modify node type

Start rabbit

cd /opt/rabbitmq_server-3.8.5/ 
sbin/rabbitmq-server -detached   # Start the app in the background
sbin/rabbitmqctl status          # View node status 


Erlang nodes can communicate with each other by authenticating Erlang cookies. Because rabbitmqctl uses Erlang OTP communication mechanism to communicate with Rabbit nodes, the machine running rabbitmqctl and the Rabbit node to be connected must use the same Erlang cookie. Otherwise you will get a mistake.

cat /root/.erlang.cookie 
IJPCAHDPWVYSDERZDUPG # Keep cookie s consistent

scp /root/.erlang.cookie       n218:/root/.erlang.cookie # You need to restart the mq service after copying
sbin/rabbitmqctl shutdown                     # Stopped nodes 
sbin/rabbitmq-server -detached                # Start the app in the background

scp /root/.erlang.cookie       n219:/root/.erlang.cookie # You need to restart the mq service after copying
sbin/rabbitmqctl shutdown                     # Stopped nodes 
sbin/rabbitmq-server -detached                # Start the app in the background

Now there are the same Erlang cookie s on three machines. Let's start building a cluster.


Basic concepts

There are two types of RabbitMQ clusters:

  • General cluster
  • Mirror cluster (upgrade of normal cluster)

General cluster:

Take two nodes (rabbit01 and rabbit02) as examples to illustrate.
Rabbit01 and rabbit02 have the same metadata, that is, the structure of the queue, but the message entity only exists in one of the nodes rabbit01 (or rabbit02).

When the message enters the Queue of rabbit01 node and the consumer consumes from rabbit02 node, RabbitMQ will temporarily transmit messages between rabbit01 and rabbit02, and take out the message entity in A and send it to the consumer through B.

Therefore, the consumer should try to connect to each node and get messages from it. That is, for the same logical Queue, physical queues should be established in multiple nodes. Otherwise, no matter whether the consumer connects rabbit01 or rabbit02, the outlet will always be rabbit01, which will cause bottleneck. When rabbit01 node fails, rabbit02 node cannot get the message entity that has not been consumed in rabbit01 node.

If message persistence is done, it must wait for rabbit01 node to recover before it can be consumed; if there is no persistence, message loss will occur.

Image cluster:

On the basis of ordinary clusters, the required queues are made into mirror queues, and the message entities will actively synchronize among the mirror nodes, instead of pulling data temporarily when the client fetches data, that is, how many messages from nodes will be backed up.

The side effects of this mode are also obvious. In addition to reducing the system performance, if the number of image queues is too large and a large number of messages enter, the network bandwidth within the cluster will be greatly consumed by this synchronous communication.

Therefore, it is suitable for high reliability requirements. Due to the automatic synchronization of messages between mirror queues and the internal election of master mechanism, even if the master node is down, it will not affect the use of the whole cluster, achieve the purpose of decentralization, and effectively prevent the loss of messages and service unavailability.

General cluster

Cluster name

Change the cluster name to rabbit@MQ1

# Modify cluster name 
sbin/rabbitmqctl set_cluster_name rabbit@MQ1 
Setting cluster name to rabbit@MQ1 ... 

# View cluster status 
sbin/rabbitmqctl cluster_status   
Cluster status of node 
rabbit@MQ1 ... 

Basics Cluster name: 

Disk Nodes 

Running Nodes 

Join the cluster

It is executed on 218 and 219 nodes

sbin/rabbitmqctl stop_app 
sbin/rabbitmqctl join_cluster rabbit@MQ1 
sbin/rabbitmqctl start_app

Modify node type

View cluster status

sbin/rabbitmqctl cluster_status 
Cluster status of node 
rabbit@MQ1 ... 

Basics Cluster name: 

Disk Nodes    # All three disk nodes are now 
rabbit@MQ1    # We can see that all nodes are disk type, which does not conform to our default architecture 
rabbit@MQ2    # We need to change the architecture 

Running Nodes 

Change 218 node to memory node

# Stop node 
sbin/rabbitmqctl stop_app 
# Delete the communication node from the cluster 
sbin/rabbitmqctl reset 
# Rejoin the cluster in RAM mode 
sbin/rabbitmqctl join_cluster rabbit@MQ1 --ram 
# Start node 
sbin/rabbitmqctl start_app 
sbin/rabbitmqctl cluster_status 
Cluster status of node rabbit@MQ1 ... 
Basics Cluster name: 

Disk Nodes 

RAM Nodes 

Running Nodes 

When the node is in a stand-alone state, the reset command clears the state of the node and restores it to a blank state. When a node is part of a cluster, the command also communicates with the disk nodes in the cluster, telling them that the node is leaving the cluster.

This is very important. Otherwise, the cluster will think that the node has failed and expect it to recover eventually. Before the node comes back, the cluster will prohibit new nodes from joining.

Mirror cluster (HA)

Above, we have successfully deployed a normal cluster, which is not highly available. Next, we upgrade it to a mirror cluster based on the normal cluster Official HA scheme

sbin/rabbitmqctl set_policy <name> [-p <vhost>] <pattern> <definition> [--apply-to <apply-to>]    
Name: policy name    
vhost: specifies vhost, the default value/    
pattern: match which needs to be mirrored through regular expression, ^ for all    
Ha mode: indicates the mode of the mirror queue. The valid values are all/exactly/nodes
	All means to mirror on all nodes in the cluster without setting ha params            
	exactly means to mirror on the specified number of nodes, the number of nodes is specified by HA params             
	nodes means to mirror on the specified node, and the node name is specified by HA params
Parameters needed for HA mode mode mode         
Ha sync mode: the synchronization mode of messages in the mirror queue. The valid values are automatic, manually    
Apply to: strategy object. Three optional values, all by default        
	exchanges means mirror exchange (I don't know the meaning)        
	queues means mirror queue all       
	Represents the image exchange and queue ➜  

#Sample command
sbin/rabbitmqctl set_policy admin "^" '{"ha-mode":"all","ha-sync-mode":"automatic"}'
ha-mode ha-params function
all empty The mirror queue is replicated across the cluster. When a new node is added, a copy will also be copied on the node.
exactly count The mirror queue will replicate count copies on the cluster. If the number of clusters is less than count, the queue will be copied to all nodes. If it is larger than the count cluster, after a node crash es, the newly entered node will not do a new image.
nodes node name The mirror queue is replicated in node name. If the name is not one of the clusters, this will not trigger an error. If no node is online in the node list, the queue will be declared in the node connected by the client.

WEB Management

Enable WEB Management plug-in

#Start the web management plug-in 
sbin/rabbitmq-plugins enable rabbitmq_management 

#Add users 
sbin/rabbitmqctl add_user admin admin 

#Configure user permissions
sbin/rabbitmqctl set_permissions -p / admin ".*" ".*" ".*"
Example: we give admin the configuration and read-write permissions for all resources under '/'. Note that the '/ "represents that the virtual host is' /", which is different from the root directory in Linux. It is not true that the virtual host can be accessed if the virtual host is' / ", so understand the' /" as a string.

#Setting up user roles 
sbin/rabbitmqctl set_user_tags admin administrator

Access management interface

Open browser access http://nodeip:15672 Log in using the user created above

ss -tnlp | grep 5672 LISTEN     0      128          *:25672                    *:*                   users:(("beam.smp",pid=3593,fd=77)) LISTEN     0      128          *:15672                    *:*                   users:(("beam.smp",pid=3593,fd=93)) LISTEN     0      128         :::5672                    :::*                   users:(("beam.smp",pid=3593,fd=92))

load balancing

Here we use haproxy to balance the load. The haproxy startup configuration is omitted. This place is responsible for the internal and external load balancing. If you need to start the Internet, you need to forward it.

Increase VIP

ip addr add dev eth0:mq

ip a
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 5a:fd:bf:c3:43:ec brd ff:ff:ff:ff:ff:ff
    inet brd scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet scope global secondary eth0
       valid_lft forever preferred_lft forever

configuration file

➜  vim /opt/rabbitmq_server-3.8.5/etc/haproxy.cnf
    log  local0 info
    log  local1 notice
    maxconn 4096

    log     global
    mode    tcp
    option  tcplog
    option  dontlognull
    retries 3
    option  abortonclose
    maxconn 4096
    timeout connect  5000ms
    timeout client  3000ms
    timeout server  3000ms
    balance roundrobin

listen private_monitoring
    mode    http
    option  httplog
    stats   refresh  5s
    stats   uri  /stats
    stats   realm   Haproxy
    stats   auth  admin:admin

listen rabbitmq_cluster
    mode    tcp
    option  tcplog
    balance roundrobin
    server  MQ1  check  inter  5000  rise  2  fall  3
    server  MQ2  check  inter  5000  rise  2  fall  3
    server  MQ3  check  inter  5000  rise  2  fall  3

listen rabbitmq_admin
    server  MQ1
    server  MQ2
    server  MQ3

Start haproxy

➜  haproxy -f /opt/rabbitmq_server-3.8.5/etc/haproxy.cnf

Reference link:

Reference link:
1,Official binary Handbook
2,Official cluster manual

Topics: RabbitMQ Erlang Unix RPM