This article has been included in the github repository, which is used to share Java related knowledge summary, including Java basics, MySQL, Spring Boot, MyBatis, Redis, RabbitMQ, computer network, data structure and algorithms, etc. Welcome to pr and star!
GitHub address:https://github.com/Tyson0314/Java-learning
If github is not accessible, you can access the gitee repository.
Gitee address:https://gitee.com/tysondai/Java-learning
Article Directory:
brief introduction
RabbitMQ is a message queue developed by erlang. Message queues are used for asynchronous collaboration between applications.
Basic concepts
Message: consists of a message header and a message body. The message body is opaque, while the message header consists of a series of optional attributes, including routing-key, priority, delivery-mode (whether persistent storage) and so on.
Publisher: The producer of the message.
Exchange: Receives messages and routes them to one or more Queues. default exchange is the default direct switch with an empty string name. Each new Queue is automatically bound to the default switch with the same routing key name as the Queue name.
Binding: Associate Exchange with Queue through Binding so that Exchange knows which Queue to route messages to.
Queue: Stores messages and queues are first in, first out. A message can be distributed to one or more queues.
Virtual host: Each Vhost is essentially a mini version of RabbitMQ server with its own queues, switches, bindings, and permissions mechanisms. Vhost is the basis of the AMQP concept and must be specified at connection time. RabbitMQ defaults to Vhost /. When multiple different users use the same RabbitMQWhen servers provide services, they can be divided into multiple vhosts, and each user creates exchange s and queues in his own vhost.
Broker: Message Queuing server entity.
When to use MQ
For operations that do not require immediate effect, they can be split up, executed asynchronously, and implemented using message queues.
For example, in a common order system, the business logic after a user clicks on an order button may include: deducting inventory, generating the corresponding document, sending a text message notification. In this scenario, you can use MQ. Place the text message notification into the MQ asynchronous execution, and the main process of the order (such as deducting inventory, generating the corresponding document)Send a message to MQ when finished, allowing the main process to finish quickly, while other threads consume MQ messages.
Advantages and disadvantages
Disadvantages: erlang implementation is not conducive to secondary development and maintenance; performance is worse than kafka, single-machine throughput of production and consumption messages is about 120,000 when persistent messages and ACK confirm, and single-machine throughput of Kafka is at 100,000 level.
Advantages: management interface, easy to use, high reliability, rich functionality, support for message persistence, message confirmation mechanism, multiple message distribution mechanisms.
Exchange type
Exchange distributes messages according to different distribution strategies for different types. There are currently four types: direct, fanout, topic, headers. The headers mode routes messages based on the headers of the message. In addition, the headers and direct switches are identical, but their performance is much worse.
Exchange rules.
| Type Name | Type Description |
|---|---|
| fanout | Route all messages sent to this Exchange to all Queue s bound to it |
| direct | Routing Key==Binding Key |
| topic | Fuzzy Matching |
| headers | Exchange does not rely on routing key and binding key matching rules to route messages, but instead matches based on the header attribute in the message content being sent. |
direct
The direct switch routes messages to a queue where the binding key and routing key match exactly. It is a fully matched, unicast mode.
fanout
All messages sent to a fanout-type switch are routed to all queues bound to that switch. A fanout-type forwarding message is the fastest.
topic
topic switches use routing key s and binding keys for fuzzy matching, and successful matching sends messages to the corresponding queues. Roug keys and binding keys are periods'. 'delimited strings. There can be two special characters'*' and'#'in binding keys for fuzzy matching, where'*' matches one word and'#'matches multiple words.
headers
The headers switch is routed based on the headers property in the message content sent. A set of key-value pairs is specified when binding a Queue to Exchange; when a message is sent to Exchange, RabbitMQ takes the headers of the message (also in the form of a key-value pair)Compares whether the key-value pairs match exactly the key-value pairs specified when the Queue is bound to Exchange; if they match exactly, messages will be routed to the Queue; otherwise, they will not be routed to the Queue.
Message Loss
Message Loss Scenarios: Producer production message to RabbitMQ Server message is lost, RabbitMQ Server stored message is lost, and RabbitMQ Server to consumer message is lost.
Message loss can be solved in three ways: producer confirmation mechanism, consumer manual confirmation message and persistence.
Producer Confirmation Mechanism
The producer sends the message to the queue and cannot ensure that the message successfully reaches the server.
Solution:
- Transaction mechanism. Blocks the sender after a message has been sent and waits for a response from RabbitMQ before continuing to send the next message. Poor performance.
- Turn on the producer confirmation mechanism and RabbitMQ will send an ack to the producer as long as the message is successfully sent to the switch (even if the message is not received by Queue). If the message is not successfully sent to the switch, a nack message will be sent indicating that the send failed.
In Springboot, confirm mode is set through the publisher-confirms parameter:
spring:
rabbitmq:
#Open confirm confirmation mechanism
publisher-confirms: true
Provides a callback method on the production side. When the service side confirms one or more messages, the producer calls back this method, and subsequently processes the messages based on the specific results, such as resending, logging, and so on.
// Whether the message was successfully sent to Exchange
final RabbitTemplate.ConfirmCallback confirmCallback = (CorrelationData correlationData, boolean ack, String cause) -> {
log.info("correlationData: " + correlationData);
log.info("ack: " + ack);
if(!ack) {
log.info("exception handling....");
}
};
rabbitTemplate.setConfirmCallback(confirmCallback);
Routing Unreachable Messages
The producer validation mechanism only ensures that messages arrive at the switch correctly, and messages that fail to route from the switch to Queue are discarded, resulting in message loss.
There are two ways to handle non-routable messages: the Return message mechanism and the backup switch.
Return message mechanism
The Return message mechanism provides a callback function, ReturnCallback, which calls back messages when they fail to route from the switch to the Queue. mandatory needs to be set to true to listen for messages whose routes are unreachable.
spring:
rabbitmq:
#Triggering ReturnCallback must set mandatory=true, otherwise Exchange will discard the message if it does not find the Queue and will not trigger ReturnCallback
template.mandatory: true
Listen for route unreachable messages through ReturnCallback.
final RabbitTemplate.ReturnCallback returnCallback = (Message message, int replyCode, String replyText, String exchange, String routingKey) ->
log.info("return exchange: " + exchange + ", routingKey: "
+ routingKey + ", replyCode: " + replyCode + ", replyText: " + replyText);
rabbitTemplate.setReturnCallback(returnCallback);
When a message fails to route from the switch to Queue, it returns return exchange:, routingKey: MAIL, replyCode: 312, replyText: NO_ROUTE.
Backup switch
The backup switch alternate-exchange is a common exchange. When you send a message to the corresponding exchange, if it does not match a queue, it is automatically transferred to the queue corresponding to the backup switch so that the message is not lost.
Consumer Manual Message Confirmation
It is possible that the consumer will go down before they have time to process the MQ service, causing the message to be lost. Because the messenger defaults to automatic ack, once the consumer receives the message, they will notify MQ Server that the message has been processed, and MQ will remove the message.
Solution: The consumer is set to confirm the message manually. The consumer replies ack to the broker after processing the logic, indicating that the message has been successfully consumed and can be deleted from the broker. When the message consumer fails to consume, reply nack to the broker and decide whether to rejoin or remove from the broker or to enter the dead letter queue based on configuration. As long as the consumer is confiscatedAcknowledgment, the broker keeps this message, but it doesn't requeue or distribute it to other consumers.
Consumer set manual ack:
#Set up consumer manual ack spring.rabbitmq.listener.simple.acknowledge-mode=manual
When the message is processed, confirm manually:
@RabbitListener(queues = RabbitMqConfig.MAIL_QUEUE)
public void onMessage(Message message, Channel channel) throws IOException {
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
long deliveryTag = message.getMessageProperties().getDeliveryTag();
//Manual ack; the second parameter, multiple, is set to true, meaning that messages before the deliveryTag serial number (including itself) have been received, and false means that a message has been received
channel.basicAck(deliveryTag, true);
System.out.println("mail listener receive: " + new String(message.getBody()));
}
When the message consumption fails, the consumer replies to the broker nack. If the consumer sets the requeue to false, the broker deletes the message or enters the dead-letter queue after the nack, otherwise the message is re-queued.
Persistence
If the RabbitMQ service is restarted unexpectedly, the message will be lost. RabbitMQ provides a persistence mechanism to persist messages in memory to the hard disk, even if RabbitMQ is restarted, the message will not be lost.
Message persistence requires the following:
- Message set persistence. Before publishing a message, set delivery mode to 2, indicating that the message needs to be persisted.
- Queue settings are persisted.
- Switch settings are persisted.
When a message is published to a switch, Rabbit writes the message to the persistence log before sending a response to the producer. Once a message is consumed from the queue and confirmed, RabbitMQ removes the message from the persistence log. Before consuming the message, if RabbitMQ restarts, the server automatically rebuilds the switch and queue to load persistenceEnsure that messages in the log are not lost by placing them in the appropriate queue or switch.
Mirrored Queues
When MQ fails, the service will be unavailable. A mirroring queue mechanism of RabbitMQ is introduced to mirror queue over other nodes in the cluster. If one node in the cluster fails, it can automatically switch to another node in the mirror to ensure service availability.
Usually each mirror queue contains one master and multiple slaves, corresponding to different nodes. All messages sent to the mirror queue are always sent directly to the master and all slaves. All actions except publish are sent to the master only, and then the master broadcasts the results of the commands to the slave. Consumer actions from the mirror queue are actually in the form ofExecuted on the master.
Reconsumption
There are two reasons for duplicate messages: 1. duplicate messages in production and 2. duplicate messages in consumption.
The producer sends a message to MQ, there is network fluctuation during MQ confirmation, the producer does not receive confirmation, then the producer sends this message again, causing MQ to receive duplicate messages.
After successful consumer consumption, there is network fluctuation when confirming to MQ. MQ does not receive confirmation. In order to ensure that the message is not lost, MQ will continue to deliver the previous message to the consumer. At this time, the consumer receives two identical messages. Because duplicate messages are caused by network reasons, it is unavoidable.
Solution: When sending messages, let each message carry a global unique ID. When consuming messages, first judge if the message has been consumed, and then guarantee the idempotency of message consumption logic. The specific consumption process is:
- Consumers query redis/db for the presence of a message based on id after getting it
- If it doesn't exist, then normal consumption, write redis/db after consumption is complete
- If it exists, it proves that the message has been consumed and discarded directly
Consumer End Limit
When the RabbitMQ server is backing up a large number of messages, messages in the queue will flood into the consumer side, possibly causing the consumer side server to burst. In this case, the consumer side needs to be restricted.
Spring RabbitMQ provides the parameter prefetch to set the number of messages processed by a single request. If the message processed by the consumer at the same time reaches the maximum value, the consumer will block and will not consume new messages until there is a message ack.
Turn on consumer current limiting:
#Number of messages processed in a single request, maximum number of unack s spring.rabbitmq.listener.simple.prefetch=2
Native RabbitMQ also provides prefetchSize and global s, which Spring RabbitMQ does not have.
//Single message size limit, 0 means no limit //global: Restrict whether the current limiting function is channel level or consumer level. When set to false, consumer level, the current limiting function takes effect and true does not have current limiting function, because channel level is not yet implemented. void basicQos(int prefetchSize, int prefetchCount, boolean global) throws IOException;
Dead Letter Queue
Queue where messages with consumption failure are stored.
Reasons for message consumption failure:
- Message rejected and not requeued (requeue=false)
- Message Timeout Not Consumed
- Maximum queue length reached
Set exchange and queue for the dead letter queue, then bind:
@Bean
public DirectExchange dlxExchange() {
return new DirectExchange(RabbitMqConfig.DLX_EXCHANGE);
}
@Bean
public Queue dlxQueue() {
return new Queue(RabbitMqConfig.DLX_QUEUE, true);
}
@Bean
public Binding bindingDeadExchange(Queue dlxQueue, DirectExchange deadExchange) {
return BindingBuilder.bind(dlxQueue).to(deadExchange).with(RabbitMqConfig.DLX_QUEUE);
}
Add two parameters to the normal queue to bind the normal queue to the dead letter queue. When message consumption fails, messages are routed to the dead letter queue.
@Bean
public Queue sendSmsQueue() {
Map<String,Object> arguments = new HashMap<>(2);
// Bind the queue to a private switch
arguments.put("x-dead-letter-exchange", RabbitMqConfig.DLX_EXCHANGE);
arguments.put("x-dead-letter-routing-key", RabbitMqConfig.DLX_QUEUE);
return new Queue(RabbitMqConfig.MAIL_QUEUE, true, false, false, arguments);
}
Producer full code:
@Component
@Slf4j
public class MQProducer {
@Autowired
RabbitTemplate rabbitTemplate;
@Autowired
RandomUtil randomUtil;
@Autowired
UserService userService;
final RabbitTemplate.ConfirmCallback confirmCallback = (CorrelationData correlationData, boolean ack, String cause) -> {
log.info("correlationData: " + correlationData);
log.info("ack: " + ack);
if(!ack) {
log.info("exception handling....");
}
};
final RabbitTemplate.ReturnCallback returnCallback = (Message message, int replyCode, String replyText, String exchange, String routingKey) ->
log.info("return exchange: " + exchange + ", routingKey: "
+ routingKey + ", replyCode: " + replyCode + ", replyText: " + replyText);
public void sendMail(String mail) {
//Appearing thread insecurity range 100000 - 999999
Integer random = randomUtil.nextInt(100000, 999999);
Map<String, String> map = new HashMap<>(2);
String code = random.toString();
map.put("mail", mail);
map.put("code", code);
MessageProperties mp = new MessageProperties();
//Instead of using Message in a production environment, you use tools such as fastJson to convert objects to json format for sending
Message msg = new Message("tyson".getBytes(), mp);
msg.getMessageProperties().setExpiration("3000");
//If the consumer side is set to a manual ACK, the production side must send the correlationData when sending the message, which is globally unique to uniquely identify the message.
CorrelationData correlationData = new CorrelationData("1234567890"+new Date());
rabbitTemplate.setMandatory(true);
rabbitTemplate.setConfirmCallback(confirmCallback);
rabbitTemplate.setReturnCallback(returnCallback);
rabbitTemplate.convertAndSend(RabbitMqConfig.MAIL_QUEUE, msg, correlationData);
//Save in redis
userService.updateMailSendState(mail, code, MailConfig.MAIL_STATE_WAIT);
}
}
Consumer Complete Code:
@Slf4j
@Component
public class DeadListener {
@RabbitListener(queues = RabbitMqConfig.DLX_QUEUE)
public void onMessage(Message message, Channel channel) throws IOException {
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
long deliveryTag = message.getMessageProperties().getDeliveryTag();
//Manual ack
channel.basicAck(deliveryTag,false);
System.out.println("receive--1: " + new String(message.getBody()));
}
}
When there is dead letter in the normal queue, RabbitMQ will automatically republish this message to the set dead letter switch and route it to the dead letter queue. You can listen for messages in the dead letter queue and process them accordingly.
Other
pull mode
pull mode mainly obtains messages through channel.basicGet method, the sample code is as follows:
GetResponse response = channel.basicGet(QUEUE_NAME, false); System.out.println(new String(response.getBody())); channel.basicAck(response.getEnvelope().getDeliveryTag(),false);
Message expiration time
You can set an expiration time for messages in milliseconds (ms) when they are sent from the production side
Message msg = new Message("tyson".getBytes(), mp);
msg.getMessageProperties().setExpiration("3000");
You can also specify the ttl of the queue at the time of queue creation, starting from the time the message enters the queue, beyond which the message will be removed.
Reference Links
Springboot Integration RabbitMQ
Message Persistence for RabbitMQ
Finally, share a github warehouse with over 200 classic computer books, including C Language, C++, Java, Python, Front End, Database, Operating System, Computer Network, Data Structure and Algorithms, Machine Learning, Programming Life, etc. You can star t by searching directly on it next time, the warehouse is continuously updated~
GitHub address:https://github.com/Tyson0314/java-books
If github is not accessible, you can access the gitee repository.
Gitee address:https://gitee.com/tysondai/java-books