RabbitMQ Work Model for Distributed Message Communication

Posted by DevilsAdvocate on Sat, 11 May 2019 12:26:50 +0200

This chapter focuses on:

1. Introduction of three main switches

2. SpringBook Integrates RabbitMQ Switches

3. Dead letter queue

4. Priority queues and messages

5. Server Flow Control

6. Current Limitation at Consumer End

Features of RabbitMQ 
RabbitMQ is written in Erlang language and messages are stored in a Mnesia database.

Reliability RabbitMQ uses some mechanisms to ensure reliability, such as persistence, transmission confirmation, publication confirmation. 
2. Flexible Routing routes messages through Exchange before they enter the queue. For typical routing functions, RabbitMQ has provided some built-in Exchange implementations. For more complex routing functions, you can bind multiple Exchanges together and implement your Exchange through plug-in mechanisms.
3. Clustering. Multiple RabbitMQ servers can form a cluster to form a logical Broker. 
4. Highly Available Queues can be mirrored on machines in the cluster, making the queues still available when some nodes are in trouble.
5. Multi-protocol RabbitMQ supports multiple message queuing protocols, such as AMQP, STOMP, MQTT and so on.
6. Many Clients RabbitMQ supports almost all common languages, such as Java,. NET, Ruby, PHP, C#, JavaScript, etc. 
7. Management UI RabbitMQ provides an easy-to-use user interface that enables users to monitor and manage messages and nodes in the cluster.
8. Plugin System RabbitMQ provides many plug-ins to extend from many aspects. Of course, you can also write your own plug-ins.

Work model

Introduction to RabbitMQ Terminology

Broker: An entity server for RabbitMQ. Provides a transmission service, maintains a transmission line from producer to consumer, guarantees that message data can be transmitted in a specified way.
Exchange: Message Exchange. Specifies which rules messages are routed to which queue Queue.
Queue: Message queue. The carrier of a message, each message is delivered to one or more queues.
Binding: Binding. The purpose is to bind Exchange and Queue to some routing rule.
Routing Key: Routing keyword. Exchange delivers messages based on Routing Key. The keyword specified when defining the binding is called Binding Key.
Vhost: Virtual host. A Broker can have multiple virtual hosts, which can be used to separate the rights of different users. A virtual host holds a set of Exchange s, Queue s, and Binding.
Producer: Message producer. Mainly deliver messages to the corresponding Exchange. Generally, it is an independent procedure.
Consumer: News consumers. The recipient of a message is usually a separate program.
Connection: TCP long connection between Producer and Constumer and Broker.
Channel: Message channel, also known as channel. Multiple Channels can be established in each connection of the client, and each Channel represents a session task. In the RabbitMQ Java Client API, a large number of programming interfaces are defined on channel.

Three main switches

  1. Direct Exchange Direct Switch
  2. Topic Exchange Theme Switch
  3. Fanout Exchange Broadcasting Switch

RabbitMQ basically uses:

This article uses SpringBoot directly for development.

Message consumers:

Project structure:

Adding pom dependencies

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-amqp</artifactId>
 </dependency>

Add connection configuration application.properties

spring.application.name=spring-boot-rabbitmq
spring.rabbitmq.host=127.0.0.1
spring.rabbitmq.port=5672
spring.rabbitmq.username=guest
spring.rabbitmq.password=guest

Create a configuration class: RabbitConfig

Three types of switches, four queues, are created here.

package com.zbb.rabbitmq_consumer.config;
 
import org.springframework.amqp.core.*;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
 
/**
 * Configuration class, need to use the configuration annotation
 */
@Configuration
public class RabbitConfig {
    //Define three switches
 
    /**
     * Direct Switch
     * @return
     */
    @Bean
    public DirectExchange directExchange(){
        return new DirectExchange("DIRECT_EXCHANGE");
    }
 
    /**
     * Subject Switch
     * @return
     */
    @Bean
    public TopicExchange topicExchange(){
        return new TopicExchange("TOPIC_EXCHANGE");
    }
 
    /**
     * Broadcasting Switch
     * @return
     */
    @Bean
    public FanoutExchange fanoutExchange(){
        return new FanoutExchange("FANOUT_EXCHANGE");
    }
    //Define four queues
    @Bean
    public Queue firsQueue(){
        return new Queue("FIST_QUEUE");
    }
 
    @Bean
    public Queue secondQueue(){
        return new Queue("SECOND_QUEUE");
    }
 
    @Bean
    public Queue thirdQueue(){
        return new Queue("THIRD_QUEUE");
    }
 
    @Bean
    public Queue fourthQueue(){
        return new Queue("FOURTH_QUEUE");
    }
 
    //Define four binding relationships
    @Bean
    public Binding bindFirst(@Qualifier("firsQueue") Queue queue,
                             @Qualifier("directExchange") DirectExchange directExchange){
        return BindingBuilder.bind(queue).to(directExchange).with("zbb.test");
    }
 
    @Bean
    public Binding bindSecond(@Qualifier("secondQueue") Queue queue,
                              @Qualifier("topicExchange") TopicExchange topicExchange) {
        return BindingBuilder.bind(queue).to(topicExchange).with("*.zbb.*");
    }
 
    @Bean
    public Binding bindThird(@Qualifier("thirdQueue") Queue queue,
                             @Qualifier("fanoutExchange") FanoutExchange fanoutExchange) {
        return BindingBuilder.bind(queue).to(fanoutExchange);
    }
 
    @Bean
    public Binding bindFourth(@Qualifier("fourthQueue") Queue queue,
                              @Qualifier("fanoutExchange") FanoutExchange fanoutExchange) {
        return BindingBuilder.bind(queue).to(fanoutExchange);
    }
 
}

Create consumers, four, monitor four queues

1,FirstConsumer
 

package com.zbb.rabbitmq_consumer.consumer;
 
import org.springframework.amqp.rabbit.annotation.RabbitHandler;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.stereotype.Component;
 
@Component
@RabbitListener(queues = "FIST_QUEUE")
public class FirstConsumer {
    @RabbitHandler
    public void process(String msg) {
        System.out.println("As soon as the consumer receives the message:"+msg);
    }
}

2,SecondConsumer
 

package com.zbb.rabbitmq_consumer.consumer;
 
import org.springframework.amqp.rabbit.annotation.RabbitHandler;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.stereotype.Component;
 
@Component
@RabbitListener(queues = "SECOND_QUEUE")
public class SecondConsumer {
    @RabbitHandler
    public void process(String msg) {
        System.out.println("Consumer 2 receives the message:"+msg);
    }
}

3,ThirdConsumer

package com.zbb.rabbitmq_consumer.consumer;
 
import org.springframework.amqp.rabbit.annotation.RabbitHandler;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.stereotype.Component;
 
@Component
@RabbitListener(queues = "THIRD_QUEUE")
public class ThirdConsumer {
    @RabbitHandler
    public void process(String msg) {
        System.out.println("Consumer 3 receives news:"+msg);
    }
}

4,FourthConsumer
 

package com.zbb.rabbitmq_consumer.consumer;
 
import org.springframework.amqp.rabbit.annotation.RabbitHandler;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.stereotype.Component;
 
@Component
@RabbitListener(queues = "FOURTH_QUEUE")
public class FourthConsumer {
    @RabbitHandler
    public void process(String msg) {
        System.out.println("Consumer 4 receives news:"+msg);
    }
}

Message producers:

Introducing pom dependencies
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-amqp</artifactId>
</dependency>

Create a producer class to send messages to three switches

package com.zbb.rabbitmq_producer.producer;
 
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
 
@Component
public class MyProducer {
 
    @Autowired
    RabbitTemplate rabbitTemplate;
 
    public void send(){
        //Direct connection
        rabbitTemplate.convertAndSend("DIRECT_EXCHANGE", "zbb.test",
                "this is a direct msg");
        //theme
        rabbitTemplate.convertAndSend("TOPIC_EXCHANGE","kaifa.zbb.IT", 
                "this is a Topic msg");
 
        //Radio broadcast
        rabbitTemplate.convertAndSend("FANOUT_EXCHANGE", "",
                "this is a fanout msg");
    }
}

Modify the test class:

package com.zbb.rabbitmq_producer;
 
import com.zbb.rabbitmq_producer.producer.MyProducer;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;
 
@RunWith(SpringRunner.class)
@SpringBootTest
public class RabbitmqProducerApplicationTests {
 
    @Autowired
    MyProducer myProducer;
 
    @Test
    public void contextLoads() {
        myProducer.send();
    }
 
}
 
  1. Let's start the message producer first. Opening the management interface, we can see that there is a message stack in four queues.

2. Now when you start the message consumer, you will find that the console prints out the message and the management interface changes:

This is the simple use of RabbitMQ in java. Specifically, the use of RabbitMQ in the project needs to be combined with business to send and receive messages.

Here's an important part of RabbitMQ: Exchange

Only three main switches are introduced here.

Direct Exchange Direct Switch

When a directly connected switch binds to a queue, it is necessary to specify a binding key.
Routing rules: When sending a message to a directly connected switch, the bound queue can receive the message only if the routing key matches the binding key exactly.

Queue:

@Bean
public Queue firsQueue(){
    return new Queue("FIST_QUEUE");
}
//Switchboard:

@Bean
public DirectExchange directExchange(){
    return new DirectExchange("DIRECT_EXCHANGE");
}
//Binding:

@Bean
public Binding bindFirst(@Qualifier("firsQueue") Queue queue,
                         @Qualifier("directExchange") DirectExchange directExchange){
    return BindingBuilder.bind(queue).to(directExchange).with("zbb.test");
}
//Send a message:

rabbitTemplate.convertAndSend("DIRECT_EXCHANGE", "zbb.test",
                "this is a direct msg");


Topic Exchange Theme Switch

Definition: When a Topic-type switch is bound to a queue, a routing key matching by pattern can be specified.
There are two wildcards, * for matching a word. # Represents matching zero or more words. Separation between words.
Routing rules: When a message is sent to a Topic-type switch, the bound queue receives the message only if the routing key conforms to the binding key mode.

Queue:

@Bean
public Queue secondQueue(){
    return new Queue("SECOND_QUEUE");
}
//Switchboard:

@Bean
public TopicExchange topicExchange(){
    return new TopicExchange("TOPIC_EXCHANGE");
}
//Binding:

@Bean
public Binding bindSecond(@Qualifier("secondQueue") Queue queue,
                          @Qualifier("topicExchange") TopicExchange topicExchange) {
    return BindingBuilder.bind(queue).to(topicExchange).with("*.zbb.*");
}
//Send a message:

//theme
rabbitTemplate.convertAndSend("TOPIC_EXCHANGE","kaifa.zbb.IT",
                              "this is a Topic msg");


Fanout Exchange Broadcast Switch, bound to two queues thirdQueue fourthQueue

Definition: When a broadcasting type switch binds to a queue, there is no need to specify binding key.
Routing rules: When a message is sent to a broadcasting type switch, no routing key is required, and all queues bound to it receive the message.

Queue:

@Bean
public Queue thirdQueue(){
    return new Queue("THIRD_QUEUE");
}
@Bean
public Queue fourthQueue(){
    return new Queue("FOURTH_QUEUE");
}
//Switchboard:

@Bean
public FanoutExchange fanoutExchange(){
    return new FanoutExchange("FANOUT_EXCHANGE");
}
//Send a message:

//Radio broadcast
rabbitTemplate.convertAndSend("FANOUT_EXCHANGE", "",
                "this is a fanout msg");

Advanced Knowledge 1, TTL (Time To Live) a, expiration time of messages
There are two settings:
Set message expiration time by queue property:

Map<String, Object> argss = new HashMap<String, Object>(); 
                argss.put("x-message-ttl",6000);
 
                channel.queueDeclare("TEST_TTL_QUEUE", false, false, false, argss);

Set the expiration time of a single message:

AMQP.BasicProperties properties = new AMQP.BasicProperties.Builder()
                                 // Persistent messages
                                 .deliveryMode
                                 .contentEncoding("UTF-8")       
                                 .expiration("10000") // TTL        
                                 .build();
channel.basicPublish("", "TEST_TTL_QUEUE", properties, msg.getBytes());

b. Queue expiration time:

Map<String, Object> argss = new HashMap<String, Object>(); 
argss.put("x-message-ttl",6000);
 
channel.queueDeclare("TEST_TTL_QUEUE", false, false, false, argss);

The expiration time of the queue determines how long the queue can survive without any consumers.

2. Dead Letter Queue
There are three scenarios where messages enter the Dead Letter Exchange Dead Letter Switch.

  1. (NACK || Reject ) && requeue == false
  2. Message expiration
  3. Maximum queue length (first-in messages are sent to DLX)

A Dead Letter Queue can be set up to bind to DLX, that is, Dead Letter can be stored, and consumers can listen to the queue to retrieve messages.

Map<String,Object> arguments = new HashMap<String,Object>(); 
arguments.put("x-dead-letter-exchange","DLX_EXCHANGE"); 
// Designated a Dead Message Switch for this queue 
channel.queueDeclare("TEST_DLX_QUEUE", false, false, false, arguments); 
// Statement Dead Message Switch 
channel.exchangeDeclare("DLX_EXCHANGE","topic", false, false, false, null); 
// Statement Dead Letter Queue 
channel.queueDeclare("DLX_QUEUE", false, false, false, null); 
// binding 
channel.queueBind("DLX_QUEUE","DLX_EXCHANGE","#");

3. Priority queue
Set the maximum priority of a queue:

Map<String, Object> argss = new HashMap<String, Object>(); 
argss.put("x-max-priority",10);  
// Queue Maximum Priority
channel.queueDeclare("ORIGIN_QUEUE", false, false, false, argss);

When sending a message, specify the current priority of the message:

AMQP.BasicProperties properties = new AMQP.BasicProperties.Builder()        
.priority(5)  
// Message Priority  
.build();
channel.basicPublish("", "ORIGIN_QUEUE", properties, msg.getBytes());

Messages with high priority can be consumed first, but priority makes sense only when messages are stacked (messages are sent faster than consumers are consumed).
4. Delay queue
RabbitMQ itself does not support delayed queues. Delayed delivery of messages can be achieved by combining TTL with DLX, that is, binding DLX to a queue. When the message expires, it will be routed from DLX to the queue from which consumers can retrieve messages. Another way is to use the rabbitmq-delayed-message-exchange plug-in.
Of course, the information that will be sent will be saved in the database, and it can also be achieved by scanning and sending it using task scheduling system.

5,RPC 
RabbitMQ implements the principle of RPC: After the server processes the message, it sends the response message to a response queue, and the client retrieves the result from the response queue.
Question: How does Client know which message to reply to when it receives a message? So you have to have a unique ID to associate, correlation Id.


6. Flow Control
 

RabbitMQ detects the physical memory value of the machine at startup. By default, when MQ occupies more than 40% of memory, MQ actively throws a memory warning and blocks all Connections.

The memory threshold can be adjusted by modifying the rabbitmq.con enabling file. The default value is 0.4, as follows: [{rabbit,[{vm_memory_high_watermark, 0.4}]]]. By default, if the remaining disk space is less than 1GB, RabbitMQ actively blocks all producers. This threshold is also adjustable.
Note that the queue length is meaningful only in the case of message accumulation, and will delete the incoming messages, which can not achieve server-side flow restriction.

7. Current Limitation at Consumer End
In the case of false AutoACK, if a certain number of messages (by setting the value of Qos based on consumer or channel) are not confirmed, no new messages are consumed.
 

channel.basicQos(2); 
// If more than two messages do not send ACK, the current consumer no longer accepts queue messages 
channel.basicConsume(QUEUE_NAME, false, consumer);

Use of UI Management Interface
The management plug-in provides a simpler way to manage. Enable management plug-ins
Windows Enables Management Plug-ins

cd C:\Program Files\RabbitMQ Server\rabbitmq_server-3.6.6\sbin 
rabbitmq-plugins.bat enable rabbitmq_management

Linux Enables Management Plug-ins

cd /usr/lib/rabbitmq/bin 
./rabbitmq-plugins enable rabbitmq_management

Management Interface Access Port:

The default port is 15672, default user guest, password guest. Guest users can only access locally by default.  

Topics: RabbitMQ Spring Java Database