1 what is MQ
MQ (message queue): translated into message queue. Through the typical producer and consumer model, producers constantly produce messages to the message queue, and consumers constantly get messages from the queue.
Because the production and consumption of messages are asynchronous, and only care about the sending and receiving of messages, without the intrusion of business logic, it is easy to realize the decoupling of system problems.
Through the use of efficient and reliable messaging mechanism for platform independent data exchange, and based on data communication for distributed system integration.
Different MQ features
- ActiveMQ is the most popular and powerful open source message bus produced by Apache. It is a message middleware that fully supports JMS specification.
Rich API s and various cluster architecture modes make ActiveMQ an old message middleware in the industry, which is very popular in small and medium-sized enterprises - kafka is an open-source distributed publish subscribe messaging system of LinkedIn, which is currently a top-level project of Apache. kafka's main feature is to handle message consumption based on Pull mode. In pursuit of high throughput, the initial purpose is to use
Mobile log and transmission. Version 0.8 starts to support load, does not support transactions, and has no strict requirements on message repetition, loss and error. It is suitable for producing a large amount of data and Internet data collection business - RocketMQ is Alibaba's open source message middleware. It is developed in pure java. It has the characteristics of high throughput, high availability and is suitable for large-scale distributed system applications. The idea of RocketMQ originates from Kafka, but it is not a Copy of Kafka,
He has optimized reliable message transmission and transactions. At present, it is widely used in Alibaba Group in trading, recharge, stream computing, message push, log streaming processing, binglog distribution and other scenarios - RabbitMQ is an open source message queue system developed in Erlang language and implemented based on AMQP protocol. The main features of AMQP are message oriented, queue, routing (including point-to-point, publish and subscribe), reliability and security.
AMQP protocol is more used in scenarios where enterprise systems require high data consistency, stability and reliability, followed by performance and throughput requirements.
Introduction of RABBIT2
Open source message broker software (also known as message oriented middleware) that implements advanced message queuing protocol (AMQP). RabbitMQ server is written in Erlang language, and clustering and failover are built on the framework of open telecommunications platform.
All major programming languages have client libraries that communicate with proxy interfaces.
AMQP, or Advanced Message Queuing Protocol, is an application layer standard Advanced Message Queuing Protocol that provides unified messaging services. It is an open standard of application layer protocol and is designed for message oriented middleware.
The client and message middleware based on this protocol can deliver messages, which is not limited by different products and development languages of the client / middleware. Implementations in Erlang include RabbitMQ and so on.
2.1 installing MQ
Many of them don't demonstrate on the Internet. I'll try to install them myself when I'm free in the later stage
2.2 the first model (direct connection)
In the model above, there are the following concepts:
- P producer, that is, the program to send messages
- C consumers, the recipients of the message, will always wait for the message to arrive
- Queue message queue, shown in red. Similar to a mailbox, messages can be cached; Producers deliver messages to them, and consumers get messages from them.
producer
public class Provider {
@Test
public void testSendMessage() throws IOException, TimeoutException {
//Create connection mq connection factory object
ConnectionFactory connectionFactory = new ConnectionFactory();
//Connect host port virtual host user name and password of virtual host
connectionFactory.setHost("47.110.39.229");
connectionFactory.setPort(3302);
//connectionFactory.setVirtualHost("");
connectionFactory.setUsername("admin");
connectionFactory.setPassword("xxx");
//Create connection acquisition channel
Connection connection = connectionFactory.newConnection();
Channel channel = connection.createChannel();
/**
* Message queue corresponding to channel binding
* String queue,Parameter 1 queue name automatically created if the queue does not exist
* boolean durable,Parameter 2 is used to define whether the queue feature needs to be persisted. true persistent queue false not persistent
* boolean exclusive, Parameter 3 exclusive queue true exclusive queue false non exclusive
* boolean autoDelete, Delete queue automatically after consumption true delete queue automatically false do not delete queue automatically
* Map<String, Object> arguments Additional parameters
*/
channel.queueDeclare("hello",true,false,false,null);
//Release news
//Parameter 1: switch name, parameter 2: queue name, BasicProperties parameter 3: deliver message. Additionally, set body parameter 4: specific content of message
channel.basicPublish("","hello",null,"hello rabbitmq".getBytes());
channel.close();
connection.close();
}
}
consumer
public class Customer {
public static void main(String[] args)throws IOException, TimeoutException {
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setHost("47.110.39.229");
connectionFactory.setPort(3302);
//connectionFactory.setVirtualHost("");
connectionFactory.setUsername("admin");
connectionFactory.setPassword("xxx");
Connection connection = connectionFactory.newConnection();
Channel channel = connection.createChannel();
channel.queueDeclare("hello",true,false,false,null);
/**
* Consumption news
*Parameter 1 message queue name of the queue to consume
*Parameter 2 start consumption automatic confirmation mechanism
*Parameter 3 callback interface during consumption
*/
channel.basicConsume("hello",true,new DefaultConsumer(channel){
//Last parameter: message in message queue
public void handleDelivery(String consumerTag, Envelope envelope,AMQP.BasicProperties properties,byte[] body){
System.out.println("body======>"+new String(body));
}
});
}
}
Tool encapsulation
public class RabbitMQUtils {
private static ConnectionFactory connectionFactory;
//Static code block, which is executed only once when the class is loaded
static {
connectionFactory = new ConnectionFactory();
connectionFactory.setHost("47.110.39.229");
connectionFactory.setPort(3302);
connectionFactory.setUsername("admin");
connectionFactory.setPassword("xxx");
}
public static Connection getConnection() {
try {
return connectionFactory.newConnection();
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
public static void closeConnectionAndChannel(Connection con, Channel channel){
try {
if(channel!=null){
channel.close();
}
if(con!=null){
con.close();
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
Source code analysis
String queue, queue name, automatically created if the queue does not exist
boolean durable, used to determine whether the queue needs to be persisted. true persistent queue false not persistent
boolean exclusive: whether to monopolize the queue. true: monopolize the queue. false: not monopolize the queue
boolean autoDelete: whether to automatically delete the queue after consumption. true: automatically delete false: not automatically delete
Map < string, Object > arguments additional parameters
public com.rabbitmq.client.AMQP.Queue.DeclareOk queueDeclare(String queue, boolean durable, boolean exclusive, boolean autoDelete, Map<String, Object> arguments) throws IOException {
com.rabbitmq.client.AMQP.Queue.DeclareOk ok = this.delegate.queueDeclare(queue, durable, exclusive, autoDelete, arguments);
this.recordQueue(ok, queue, durable, exclusive, autoDelete, arguments);
return ok;
}
String exchange, switch name
String routingKey, queue name
BasicProperties props, additional settings for delivering messages
byte[] body message details
public void basicPublish(String exchange, String routingKey, BasicProperties props, byte[] body) throws IOException {
this.delegate.basicPublish(exchange, routingKey, props, body);
}
2.3 second mode (working)
Work queues, also known as task queues, is a task model. When message processing is time-consuming, the speed of message production may be much faster than that of message consumption.
In the long run, more and more messages will accumulate and cannot be processed in time. At this point, you can use the work model; Let multiple consumers bind to a queue and consume messages in the queue together.
Once the messages in the queue are consumed, they will disappear, so the task will not be repeated.
Role:
- P producer: publisher of the task
- C1 consumer 1, receive the task and complete the task. It is assumed that the completion speed is slow
- C2 consumer 2, receive the task and complete the task. It is assumed that the completion speed is fast
producer
public class Provider {
public static void main(String[] args) throws IOException {
Connection connection = RabbitMQUtils.getConnection();
Channel channel = connection.createChannel();
channel.queueDeclare("work",true,false,false,null);
for (int i = 0; i < 10; i++) {
channel.basicPublish("","work",null,("hello work queue " +i).getBytes());
}
RabbitMQUtils.closeConnectionAndChannel(connection,channel);
}
}
Consumer 1
public class Customer1 {
public static void main(String[] args) throws IOException {
Connection connection = RabbitMQUtils.getConnection();
Channel channel = connection.createChannel();
channel.queueDeclare("work",true,false,false,null);
channel.basicConsume("work",true,new DefaultConsumer(channel){
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties,byte[] body){
System.out.println("consumer-1: "+new String(body));
}
});
}
}
Consumer 2
public class Customer1 {
public static void main(String[] args) throws IOException {
Connection connection = RabbitMQUtils.getConnection();
Channel channel = connection.createChannel();
channel.queueDeclare("work",true,false,false,null);
channel.basicConsume("work",true,new DefaultConsumer(channel){
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties,byte[] body){
System.out.println("consumer-1: "+new String(body));
}
});
}
}
Information printing
Summary:
After testing, we found that by default, rabbitMQ will send each message to the next consumer in order. On average, every consumer receives the same number of messages. This way of distributing messages is called a loop.
Automatic confirmation mechanism of message
Turn off automatic confirmation to realize that those who can do more work
Producer unchanged ~ consumer 3
public class Customer3 {
public static void main(String[] args) throws IOException {
Connection connection = RabbitMQUtils.getConnection();
final Channel channel = connection.createChannel();
channel.basicQos(1);//Only one message can be consumed at a time
channel.queueDeclare("work",true,false,false,null);
//Parameter 2 auto confirm false turn off auto confirm true turn on auto confirm
channel.basicConsume("work",false,new DefaultConsumer(channel){
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties,byte[] body) throws IOException {
System.out.println("consumer-3: "+new String(body));
//Manual confirmation ~ parameter 1 confirms the specific message in the queue. Parameter 2 whether to enable multiple messages to be confirmed at the same time
channel.basicAck(envelope.getDeliveryTag(),false);
}
});
}
}
Producer unchanged ~ consumer 4
public class Customer4 {
public static void main(String[] args) throws IOException {
Connection connection = RabbitMQUtils.getConnection();
final Channel channel = connection.createChannel();
channel.basicQos(1);//Only one message can be consumed at a time
channel.queueDeclare("work",true,false,false,null);
//Parameter 2 auto confirm false turn off auto confirm true turn on auto confirm
channel.basicConsume("work",false,new DefaultConsumer(channel){
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties,byte[] body) throws IOException {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("consumer-4: "+new String(body));
//Manual confirmation ~ parameter 1 confirms the specific message in the queue. Parameter 2 whether to enable multiple messages to be confirmed at the same time
channel.basicAck(envelope.getDeliveryTag(),false);
}
});
}
}
Information printing
2.4 the third mode (fanout) - broadcast mode
Fanout, also known as broadcast model
In broadcast mode, the message sending process is as follows:
There can be multiple consumers
- Each consumer has its own queue
- Each queue must be bound to its own Exchange (switch)
- When the producer sends a message, it only needs to send it to the switch. The switch decides which queue to send to, but the producer cannot decide.
- The switch sends messages to all queues that have been bound
- Consumers in the queue can get consumption. Realize that a message is consumed by multiple consumers
producer
public class Provider {
public static void main(String[] args) throws IOException {
Connection connection = RabbitMQUtils.getConnection();
Channel channel = connection.createChannel();
//Assign the channel declaration to switch parameter 1 switch name parameter 2 switch type fanout broadcast mode
channel.exchangeDeclare("logs","fanout");
//send message
channel.basicPublish("logs","",null,"fanout type message".getBytes());
RabbitMQUtils.closeConnectionAndChannel(connection,channel);
}
}
Consumer 1
public class Customer1 {
public static void main(String[] args) throws IOException {
Connection connection = RabbitMQUtils.getConnection();
Channel channel = connection.createChannel();
//Channel bound switch
channel.exchangeDeclare("logs","fanout");
//Temporary queue
String queueName = channel.queueDeclare().getQueue();
//Binding messages and queues
channel.queueBind(queueName,"logs","");
//consumption
channel.basicConsume(queueName,true,new DefaultConsumer(channel){
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body){
System.out.println("fanout consumer-1: "+new String(body));
}
});
}
}
Consumer 2
public class Customer2 {
public static void main(String[] args) throws IOException {
Connection connection = RabbitMQUtils.getConnection();
Channel channel = connection.createChannel();
//Channel bound switch
channel.exchangeDeclare("logs","fanout");
//Temporary queue
String queueName = channel.queueDeclare().getQueue();
//Binding messages and queues
channel.queueBind(queueName,"logs","");
//consumption
channel.basicConsume(queueName,true,new DefaultConsumer(channel){
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body){
System.out.println("fanout consumer-2: "+new String(body));
}
});
}
}
Consumer 3
public class Customer3 {
public static void main(String[] args) throws IOException {
Connection connection = RabbitMQUtils.getConnection();
Channel channel = connection.createChannel();
//Channel bound switch
channel.exchangeDeclare("logs","fanout");
//Temporary queue
String queueName = channel.queueDeclare().getQueue();
//Binding messages and queues
channel.queueBind(queueName,"logs","");
//consumption
channel.basicConsume(queueName,true,new DefaultConsumer(channel){
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body){
System.out.println("fanout consumer-3: "+new String(body));
}
});
}
}
Test all consumers output results
2.5 the fourth model (Routing) ~ Routing mode
2.5.1 subscription model of routing direct
In the fanout model, a message is consumed by all subscribed queues. However, in some scenarios, we want different messages to be consumed by different queues. In this case, exchange of Direct type will be used
Under the direct model:
The binding between the queue and the switch cannot be arbitrary, but a routingkey needs to be specified
When sending a message to exchange, the sender of the message must also specify the routingKey of the message.
Exchange cannot deliver messages to each bound queue, but judges them according to the routing key of the message. Only when the routing key of the queue is completely consistent with the routing key of the message can the message be received
Producer P sends a message to exchange. When sending a message, it will specify a routing key
X exchange (exchange), which receives the producer's message, and then submits the message to the queue whose routing key exactly matches
C1 consumer, whose queue specifies the message whose routing key needs to be error
C2 consumer, whose queue specifies the message whose routing key needs to be info, error and warning
producer
public class DirectProvider {
public static void main(String[] args) throws IOException {
Connection connection = RabbitMQUtils.getConnection();
Channel channel = connection.createChannel();
//Declare switch parameter 1 switch name parameter 2 direct routing mode
String exchange = "logs_direct";
String type = "direct";
channel.exchangeDeclare(exchange,type);
//send message
String routingKey = "info";
String message = "This is direct Model publishing based on rout key:["+routingKey+"]Messages sent";
channel.basicPublish(exchange,routingKey,null,message.getBytes());
RabbitMQUtils.closeConnectionAndChannel(connection,channel);
}
}
Error ~ consumer
public class DirectErrorCustomer {
public static void main(String[] args) throws IOException {
Connection connection = RabbitMQUtils.getConnection();
Channel channel = connection.createChannel();
//Declare switch and switch type
String exchange = "logs_direct";
String type = "direct";
channel.exchangeDeclare(exchange,type);
//Create a temporary queue
String queue = channel.queueDeclare().getQueue();
//Binding queue and switch based on route key
String routingKey = "error";
channel.queueBind(queue,exchange,routingKey);
channel.basicConsume(queue,true,new DefaultConsumer(channel){
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body){
System.out.println("Direct Error Customer consumer:"+new String(body));
}
});
}
}
All ~ consumer
public class DirectAllCustomer {
public static void main(String[] args) throws IOException {
Connection connection = RabbitMQUtils.getConnection();
Channel channel = connection.createChannel();
//Declare switch and switch type
String exchange = "logs_direct";
String type = "direct";
channel.exchangeDeclare(exchange,type);
//Create a temporary queue
String queue = channel.queueDeclare().getQueue();
//Binding queue and switch based on route key
String routingKey1 = "info";
String routingKey2 = "warning";
String routingKey3 = "error";
channel.queueBind(queue,exchange,routingKey1);
channel.queueBind(queue,exchange,routingKey2);
channel.queueBind(queue,exchange,routingKey3);
channel.basicConsume(queue,true,new DefaultConsumer(channel){
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body){
System.out.println("Direct All Customer consumer:"+new String(body));
}
});
}
}
test result
When it is error, it is output. When it is info, only all is output:
2.5.2 subscription model of routing topic
Compared with direct, Topic type exchange can route messages to different queues according to the routingkey.
However, the exchange of topic type allows the queue to use wildcards when Binding routing key s!
This model routingkey is generally composed of one or more words, with "." between multiple words division. Such as: item insert
wildcard
(star) no more, no less, exactly one word
#(hash) matches zero or more words
For example:
audit.# Match audit irs. Cor or audit IRS et al
audit. Can only match audit irs
producer
public class TopicProvider {
public static void main(String[] args) throws IOException {
Connection connection = RabbitMQUtils.getConnection();
Channel channel = connection.createChannel();
//Declare switch parameter 1 switch name parameter 2 direct routing mode
String exchangeName = "exchange_topics";
String type = "topic";
channel.exchangeDeclare(exchangeName,type);
//send message
String routingKey = "user";
String message = "This is topic Model publishing based on rout key:["+routingKey+"]Messages sent";
channel.basicPublish(exchangeName,routingKey,null,message.getBytes());
RabbitMQUtils.closeConnectionAndChannel(connection,channel);
}
}
Star consumer
public class StarCustomer {
public static void main(String[] args) throws IOException {
Connection connection = RabbitMQUtils.getConnection();
Channel channel = connection.createChannel();
//Declare switch and switch type
String exchangeName = "exchange_topics";
String type = "topic";
channel.exchangeDeclare(exchangeName,type);
//Create a temporary queue
String queue = channel.queueDeclare().getQueue();
//Binding queue and switch based on route key
String routingKey = "user.*";
channel.queueBind(queue,exchangeName,routingKey);
channel.basicConsume(queue,true,new DefaultConsumer(channel){
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body){
System.out.println("topic *(star) Customer consumer:"+new String(body));
}
});
}
}
Hash consumer
public class HashCustomer {
public static void main(String[] args) throws IOException {
Connection connection = RabbitMQUtils.getConnection();
Channel channel = connection.createChannel();
//Declare switch and switch type
String exchangeName = "exchange_topics";
String type = "topic";
channel.exchangeDeclare(exchangeName,type);
//Create a temporary queue
String queue = channel.queueDeclare().getQueue();
//Binding queue and switch based on route key
String routingKey = "user.#";
channel.queueBind(queue,exchangeName,routingKey);
channel.basicConsume(queue,true,new DefaultConsumer(channel){
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body){
System.out.println("topic #(hash) Customer: "+ new String(body));
}
});
}
}
Test: * can only match one, # match one or more
2. Springboot integration
Subsequent updates continuous updates
Add dependency and configuration
RabbitTemplate
3 application scenarios
3.1 asynchronous processing
3.2 application decoupling
3.3 flow peak elimination
4. Common interview questions
4.1 how does rabbitmq ensure that messages are not lost?
First: the producer loses the data. When the producer sends the data to RabbitMQ, the data may be lost on the way because of network problems. (enable confirm mode)
Second: rabbitMQ lost data. MQ has not been persisted, and it hangs up (rabbitMQ persistence is enabled)
Third: the consumer loses data. It's just been consumed and hasn't been processed yet. As a result, the process hangs up, such as restarting. (turn off rabbitMQ automatic ACK)
4.2 Rabbitmq ensures that messages are not consumed repeatedly
1. When you get this message, do the database insert operation. That's easy. Make a unique primary key for this message,
Even if repeated consumption occurs, it will lead to primary key conflict and avoid dirty data in the database.
2. When you get this message and do the set operation of redis, it's easy. There's no need to solve it, because no matter how many times you set, the result is the same. The set operation is originally an idempotent operation.
3. If the above two situations do not work, prepare a third-party storage to make consumption records. Take redis as an example, assign a global id to the message,
As long as the message is consumed, the < ID, message > is written to redis in the form of K-V. Before consumers start spending, they can check whether there are consumption records in redis.