Deploying rabbitMQ image cluster

Version information

rabbit MQ: 3.8.5
 Erlang: Official recommended minimum 21.3 recommended 22.x
         Here's 23

Environmental preparation

Host planning

Memory node:
    The memory node stores all the metadata definitions of queues, switches, bindings, users, permissions and vhost s in memory. The advantage is that it can make operations such as switch and queue declaration faster. The exception is: the contents of the persistent queue will be saved to disk.

Disk node:
    The metadata is stored in the disk, and only disk type nodes are allowed in single node system to prevent the loss of system configuration information when RabbitMQ is restarted.


Note:
    1. The performance of memory node is higher than that of disk node because it does not read and write to disk.
    2. There can be multiple disk nodes in a cluster. The more disk nodes, the better the availability of the whole cluster. However, the overall performance of the cluster will not increase linearly, which needs to be considered.
    3. RabbitMQ requires at least one disk node in the cluster, and all other nodes can be memory nodes. When a node joins or leaves the cluster, it must notify at least one disk node of the change. If the only disk node in the cluster crashes, the cluster can still keep running, but no other operations (add, delete, modify, check) can be performed until the node recovers.
    4. Set up two disk nodes, at least one of which is available to save metadata changes.

Download offline package

Official website installation manual( https://www.rabbitmq.com/install-generic-unix.html)

rabbit MQ: binary version
➜  wget https://github.com/rabbitmq/rabbitmq-server/releases/download/v3.8.5/rabbitmq-server-generic-unix-3.8.5.tar.xz

Erlang: no dependencies -- the package strips off Erlang modules and dependencies that are not necessary to run RabbitMQ.
➜  wget https://github.com/rabbitmq/erlang-rpm/releases/download/v22.2.8/erlang-22.2.8-1.el7.x86_64.rpm

Install offline package

# Install erlang
yum install -y yum install erlang-22.2.8-1.el7.x86_64.rpm 

# Decompress rabbitmq
xz -d rabbitmq-server-generic-unix-3.8.5.tar.xz 
tar -xvf rabbitmq-server-generic-unix-3.8.5.tar -C /opt

hosts file

172.16.14.3    MQ1
172.16.14.4    MQ2
172.16.14.5    MQ3

configuration file

We need to create rabbitmq ourselves in $Home/etc/rabbitmq-env.conf , see for more information Official configuration description

# Create persistent directory
mkdir -p /data/rabbitmq/store
mkdir -p /data/rabbitmq/logs 

# create profile
vim /opt/rabbitmq_server-3.8.5/etc/rabbitmq/rabbitmq-env.conf
# Specify the name of the node. The default is rabbit @ ${host name} 
NODENAME=rabbit@MQ1 
# Specify the port, the default is 5672 
NODE_PORT=5672 
# Configure persistent directory 
MNESIA_BASE=/ahdata/rabbitmq/store
# Configuration log directory default file name: ${NODENAME}.log can be modified with configuration 
LOG_BASE=/ahdata/rabbitmq/logs

Enable service

Common commands

sbin/rabbitmq-server                          # Start the server 
sbin/rabbitmq-server -detached                # Start server in the background 
sbin/rabbitmqctl status                       # View node status 
sbin/rabbitmqctl shutdown                     # Stopped nodes 
sbin/rabbitmqctl stop_app                     # Stop mq service
sbin/rabbitmqctl start_app                    # Start mq service
sbin/rabbitmqctl cluster_status               # View cluster status 
sbin/rabbitmqctl set_cluster_name rabbit@MQ1  # Modify cluster name 
sbin/rabbitmqctl join_cluster <cluster_name>  # Join the cluster 
sbin/rabbitmqctl change_cluster_node_type --node <node_name> [ disk | ram ]  # Modify node type

Start rabbit

cd /opt/rabbitmq_server-3.8.5/ 
sbin/rabbitmq-server -detached   # Start the app in the background
sbin/rabbitmqctl status          # View node status 

erlang.cookie

Erlang nodes can communicate with each other by authenticating Erlang cookies. Because rabbitmqctl uses Erlang OTP communication mechanism to communicate with Rabbit nodes, the machine running rabbitmqctl and the Rabbit node to be connected must use the same Erlang cookie. Otherwise you will get a mistake.

cat /root/.erlang.cookie 
IJPCAHDPWVYSDERZDUPG # Keep cookie s consistent

scp /root/.erlang.cookie       n218:/root/.erlang.cookie # You need to restart the mq service after copying
sbin/rabbitmqctl shutdown                     # Stopped nodes 
sbin/rabbitmq-server -detached                # Start the app in the background


scp /root/.erlang.cookie       n219:/root/.erlang.cookie # You need to restart the mq service after copying
sbin/rabbitmqctl shutdown                     # Stopped nodes 
sbin/rabbitmq-server -detached                # Start the app in the background

Now there are the same Erlang cookie s on three machines. Let's start building a cluster.

colony

Basic concepts

There are two types of RabbitMQ clusters:

• General cluster
• Mirror cluster (upgrade of normal cluster)

General cluster:

Take two nodes (rabbit01 and rabbit02) as examples to illustrate.
Rabbit01 and rabbit02 have the same metadata, that is, the structure of the queue, but the message entity only exists in one of the nodes rabbit01 (or rabbit02).

When the message enters the Queue of rabbit01 node and the consumer consumes from rabbit02 node, RabbitMQ will temporarily transmit messages between rabbit01 and rabbit02, and take out the message entity in A and send it to the consumer through B.

Therefore, the consumer should try to connect to each node and get messages from it. That is, for the same logical Queue, physical queues should be established in multiple nodes. Otherwise, no matter whether the consumer connects rabbit01 or rabbit02, the outlet will always be rabbit01, which will cause bottleneck. When rabbit01 node fails, rabbit02 node cannot get the message entity that has not been consumed in rabbit01 node.

If message persistence is done, it must wait for rabbit01 node to recover before it can be consumed; if there is no persistence, message loss will occur.

Image cluster:

On the basis of ordinary clusters, the required queues are made into mirror queues, and the message entities will actively synchronize among the mirror nodes, instead of pulling data temporarily when the client fetches data, that is, how many messages from nodes will be backed up.

The side effects of this mode are also obvious. In addition to reducing the system performance, if the number of image queues is too large and a large number of messages enter, the network bandwidth within the cluster will be greatly consumed by this synchronous communication.

Therefore, it is suitable for high reliability requirements. Due to the automatic synchronization of messages between mirror queues and the internal election of master mechanism, even if the master node is down, it will not affect the use of the whole cluster, achieve the purpose of decentralization, and effectively prevent the loss of messages and service unavailability.

General cluster

Cluster name

Change the cluster name to rabbit@MQ1

# Modify cluster name 
sbin/rabbitmqctl set_cluster_name rabbit@MQ1 
Setting cluster name to rabbit@MQ1 ... 

# View cluster status 
sbin/rabbitmqctl cluster_status   
Cluster status of node 
rabbit@MQ1 ... 

Basics Cluster name: 
rabbit@MQ1 

Disk Nodes 
rabbit@MQ1 

Running Nodes 
rabbit@MQ1

Join the cluster

It is executed on 218 and 219 nodes

sbin/rabbitmqctl stop_app 
sbin/rabbitmqctl join_cluster rabbit@MQ1 
sbin/rabbitmqctl start_app

Modify node type

View cluster status

sbin/rabbitmqctl cluster_status 
Cluster status of node 
rabbit@MQ1 ... 

Basics Cluster name: 
rabbit@MQ1

Disk Nodes    # All three disk nodes are now 
rabbit@MQ1    # We can see that all nodes are disk type, which does not conform to our default architecture 
rabbit@MQ2    # We need to change the architecture 
rabbit@MQ3 

Running Nodes 
rabbit@MQ1 
rabbit@MQ2 
rabbit@MQ3

Change 218 node to memory node

# Stop node 
sbin/rabbitmqctl stop_app 
# Delete the communication node from the cluster 
sbin/rabbitmqctl reset 
# Rejoin the cluster in RAM mode 
sbin/rabbitmqctl join_cluster rabbit@MQ1 --ram 
# Start node 
sbin/rabbitmqctl start_app 
sbin/rabbitmqctl cluster_status 
Cluster status of node rabbit@MQ1 ... 
Basics Cluster name: 
rabbit@MQ1

Disk Nodes 
rabbit@MQ1 
rabbit@MQ3 

RAM Nodes 
rabbit@MQ2 

Running Nodes 
rabbit@MQ1 
rabbit@MQ2 
rabbit@MQ3

When the node is in a stand-alone state, the reset command clears the state of the node and restores it to a blank state. When a node is part of a cluster, the command also communicates with the disk nodes in the cluster, telling them that the node is leaving the cluster.

This is very important. Otherwise, the cluster will think that the node has failed and expect it to recover eventually. Before the node comes back, the cluster will prohibit new nodes from joining.

Mirror cluster (HA)

Above, we have successfully deployed a normal cluster, which is not highly available. Next, we upgrade it to a mirror cluster based on the normal cluster Official HA scheme

sbin/rabbitmqctl set_policy <name> [-p <vhost>] <pattern> <definition> [--apply-to <apply-to>]    
Name: policy name    
vhost: specifies vhost, the default value/    
pattern: match which needs to be mirrored through regular expression, ^ for all    
definition:         
Ha mode: indicates the mode of the mirror queue. The valid values are all/exactly/nodes
	All means to mirror on all nodes in the cluster without setting ha params            
	exactly means to mirror on the specified number of nodes, the number of nodes is specified by HA params             
	nodes means to mirror on the specified node, and the node name is specified by HA params
ha-params: 
Parameters needed for HA mode mode mode         
Ha sync mode: the synchronization mode of messages in the mirror queue. The valid values are automatic, manually    
Apply to: strategy object. Three optional values, all by default        
	exchanges means mirror exchange (I don't know the meaning)        
	queues means mirror queue all       
	Represents the image exchange and queue ➜  

#Sample command
sbin/rabbitmqctl set_policy admin "^" '{"ha-mode":"all","ha-sync-mode":"automatic"}'

WEB Management

Enable WEB Management plug-in

#Start the web management plug-in 
sbin/rabbitmq-plugins enable rabbitmq_management 

#Add users 
sbin/rabbitmqctl add_user admin admin 

#Configure user permissions
sbin/rabbitmqctl set_permissions -p / admin ".*" ".*" ".*"
Example: we give admin the configuration and read-write permissions for all resources under '/'. Note that the '/ "represents that the virtual host is' /", which is different from the root directory in Linux. It is not true that the virtual host can be accessed if the virtual host is' / ", so understand the' /" as a string.

#Setting up user roles 
sbin/rabbitmqctl set_user_tags admin administrator

Access management interface

Open browser access http://nodeip:15672 Log in using the user created above

ss -tnlp | grep 5672 LISTEN     0      128          *:25672                    *:*                   users:(("beam.smp",pid=3593,fd=77)) LISTEN     0      128          *:15672                    *:*                   users:(("beam.smp",pid=3593,fd=93)) LISTEN     0      128         :::5672                    :::*                   users:(("beam.smp",pid=3593,fd=92))

load balancing

Here we use haproxy to balance the load. The haproxy startup configuration is omitted. This place is responsible for the internal and external load balancing. If you need to start the Internet, you need to forward it.

Increase VIP

ip addr add 192.168.100.242/24 dev eth0:mq

ip a
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 5a:fd:bf:c3:43:ec brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.217/24 brd 192.168.100.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet 192.168.100.242/24 scope global secondary eth0
       valid_lft forever preferred_lft forever

configuration file

➜  vim /opt/rabbitmq_server-3.8.5/etc/haproxy.cnf
global
    log     127.0.0.1  local0 info
    log     127.0.0.1  local1 notice
    daemon
    maxconn 4096

defaults
    log     global
    mode    tcp
    option  tcplog
    option  dontlognull
    retries 3
    option  abortonclose
    maxconn 4096
    timeout connect  5000ms
    timeout client  3000ms
    timeout server  3000ms
    balance roundrobin

listen private_monitoring
    bind    192.168.100.242:8100
    mode    http
    option  httplog
    stats   refresh  5s
    stats   uri  /stats
    stats   realm   Haproxy
    stats   auth  admin:admin

listen rabbitmq_cluster
    bind    192.168.100.242:8101
    mode    tcp
    option  tcplog
    balance roundrobin
    server  MQ1  192.168.100.217:5672  check  inter  5000  rise  2  fall  3
    server  MQ2  192.168.100.218:5672  check  inter  5000  rise  2  fall  3
    server  MQ3  192.168.100.219:5672  check  inter  5000  rise  2  fall  3

listen rabbitmq_admin
    bind    192.168.100.242:8102
    server  MQ1  192.168.100.217:15672
    server  MQ2  192.168.100.218:15672
    server  MQ3  192.168.100.219:15672

Start haproxy

➜  haproxy -f /opt/rabbitmq_server-3.8.5/etc/haproxy.cnf

Reference link: