Catalog
Preface
When it comes to the job hunting phase, you must have been asked how MQ guarantees the reliability of messages or how MQ guarantees that messages are not lost. We all know that MQ sending messages is generally divided into three phases: producer sending messages to MQ, MQ storing messages to memory or hard disk, and consumer consuming messages. However, all three processes can cause message loss for a variety of reasons. For example, during the producer sending phase, MQ messages may be lost due to network latency; During the storage phase, Broker puts the message in memory first, then persists it to the hard disk according to the brush policy. However, if you just receive a message and it has not been persisted to the hard disk server down, the message will be lost. When consuming on the consumer side, MQ loses the message due to network reasons in the transmission process. At this time, MQ also deletes the message from the queue, or consumer consumption failure message is lost, and so on. So how does MQ ensure that messages are not lost? Here's a summary of how rocketmq, rabbitmq, and kafka ensure that messages are not lost.
RocketMQ
Producer guarantees message is not lost
1. RocketMQ sends messages in three modes: synchronous, asynchronous and one-way.
- Synchronization blockages when sending messages synchronously and waits for Broker to return the sending result. SendResult is the most reliable way to send a message if the sending fails.
- Sending messages asynchronously can tell the result of sending in a callback method.
- One-way sending is the most unreliable way to send a message regardless of whether it is successful or unsuccessful.
/**
* @description: One-way Send
* This approach is primarily used in scenarios where you don't care very much about sending results, such as log sending.
* @param:
* @return: void
* @author xiaojie
* @date: 2021/11/9 23:39
*/
public void sendMq() {
for (int i = 0; i < 10; i++) {
rocketMQTemplate.convertAndSend("xiaojie-test", "Test Send Message","" + i);
}
}
/***********************************************************************************/
/**
* @description: Synchronized Send
* This reliable synchronous delivery method is widely used, such as: important message notification, SMS notification.
* @param:
* @return: void
* @author xiaojie
* @date: 2021/11/10 22:25
*/
public void sync() {
SendResult sendResult = rocketMQTemplate.syncSend("xiaojie-test", "sync Send a message.");
log.info("Send results{}", sendResult);
}
/***********************************************************************************/
/**
* @description: Asynchronous Send
* Asynchronous messages are often used in response time-sensitive business scenarios where the sender cannot tolerate waiting for a Broker's response for a long time.
* @param:
* @return: void
* @author xiaojie
* @date: 2021/11/10 22:29
*/
public void async() {
String msg = "Send message asynchronously..........";
log.info(">msg:<<" + msg);
rocketMQTemplate.asyncSend("xiaojie-test", msg, new SendCallback() {
@Override
public void onSuccess(SendResult var1) {
log.info("Asynchronous Send Successful{}", var1);
}
@Override
public void onException(Throwable var1) {
//Send failures can be retried
log.info("Asynchronous send failed{}", var1);
}
});
}2. Retry mechanism for producers
mq provides producers with a failure retry mechanism. Synchronous and asynchronous sending defaults to both failed retries. Of course, you can modify the number of retries. If it fails or fails multiple times, you can record this information and then manually adopt a compensation mechanism.
Broker guarantees message is not lost
1. Brush Disk Policy
There are two strategies for RocketMq persistent messages, synchronous and asynchronous. By default, the asynchronous brush disk is used. In this mode, when a producer sends a message to the broker and the message is stored in memory, the message is considered successful and the result of the successful message is returned to the producer. However, if the message is not persisted to the hard disk and the server is down, the message will be lost. Synchronized brush discs are the result of successful message sending when Broker receives a message and persists it to the hard disk. This guarantees that the message will not be lost, but they are less efficient than asynchronous brush discs, about 10% lower, depending on your business needs.
-
Modify brush disk mode in profile, ASYNC_FLUSH=asynchronous brush disc, SYNC_FLUSH = synchronous brush disc
2. Cluster mode
The rocketmq cluster mode ensures that rocketMQ is highly available. High availability of rocketmq is guaranteed with multiple master and slave nodes.
#Master-Slave Replication ASYNC_MASTER Asynchronous Replication, SYNC_MASTER synchronous replication brokerRole=SYNC_MASTER #Brush Disk Mode, ASYNC_FLUSH=asynchronous brush disc, SYNC_FLUSH = synchronous brush disc flushDiskType=SYNC_FLUSH
This mode is a broker configuration that ensures that messages are not lost, master-slave replication synchronizes replication, and brush disk mode synchronizes the brush disk, but performance decreases in this mode. The cluster is built as follows
Consumer guarantees message is not lost
1. Manual ack
/**
* @author xiaojie
* @version 1.0
* @description: Consumer Confirms Message Consumer Success
* @date 2022/3/8 23:23
*/
@Component
@Slf4j
public class MqConsumerAck implements MessageListenerConcurrently {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
for (MessageExt msg:msgs){
log.info("The message received is>>>>>>>{}",new String(msg.getBody()));
}
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
}2. Retry Mechanism of Consumer Consumption Failure
Consumers who fail to consume will automatically retry, and if they fail without a manual ack, they will automatically retry 15 times.
RabbitMQ
Producer guarantees message is not lost
1. Rabbit MQ introduces transaction mechanism and confirm mechanism
- When a transaction mechanism is opened, it is equivalent to synchronous execution, which will inevitably degrade the performance of the system, so we generally do not use this method.
- Indeed, when mq receives a message from the producer, it returns an ack to inform the producer that it received the message, and if it does not, it takes a retry mechanism to compensate the latter in other ways.
Transaction mode
public static void main(String[] args) {
try {
System.out.println("Producer Start Successfully..");
// 1. Create a connection
connection = MyConnection.getConnection();
// 2. Create Channels
channel = connection.createChannel();
channel.queueDeclare(QUEUE_NAME, false, false, false, null);
String msg = "Test the transaction mechanism to ensure the reliability of message sending.";
channel.txSelect(); //Open Transaction
channel.basicPublish("", QUEUE_NAME, null, msg.getBytes(StandardCharsets.UTF_8));
//When an exception occurs, no new messages are queued in mq
//int i=1/0;
//No exceptions occurred, commit transaction
channel.txCommit();
System.out.println("Producer sent message successfully:" + msg);
} catch (Exception e) {
e.printStackTrace();
//Roll back transaction if an exception occurs
try {
if (channel != null) {
channel.txRollback();
}
} catch (IOException ioException) {
ioException.printStackTrace();
}
} finally {
try {
if (channel != null) {
channel.close();
}
if (connection != null) {
connection.close();
}
} catch (IOException e) {
e.printStackTrace();
} catch (TimeoutException e) {
e.printStackTrace();
}
}
}confirm mode
#Turn on producer confirmation mode publisher-confirm-type: correlated # Open the message and return it if delivery fails publisher-returns: true #publisher-confirm-type has three values #NONE value is disabled publish acknowledgement mode and is the default #The CORRELATED value is a callback method that triggers when a message is successfully published to the exchanger #SIMPLE values are tested for two effects, one of which triggers the callback method as well as the CORELATED value
Callback function method class
@Component
@Slf4j
public class ConfirmCallBackListener implements RabbitTemplate.ConfirmCallback, RabbitTemplate.ReturnsCallback {
@Autowired
private RabbitTemplate rabbitTemplate;
@PostConstruct
public void init() {
//Specify ConfirmCallback
rabbitTemplate.setConfirmCallback(this);
rabbitTemplate.setReturnsCallback(this);
}
@Override
public void confirm(CorrelationData correlationData, boolean ack, String cause) {
log.info("correlation>>>>>>>{},ack>>>>>>>>>{},cause>>>>>>>>{}", correlationData, ack, cause);
if (ack) {
//Confirm receipt of message
} else {
//Receiving a message fails, you can customize the retry mechanism or save the failure to compensate
}
}
/*
*
* @param returnedMessage
* Whether messages are routed from Exchange to Queue or not, this method is callback only if messages fail to route from Exchange to Queue
* @author xiaojie
* @date 2021/9/29 13:53
* @return void
*/
@Override
public void returnedMessage(ReturnedMessage returnedMessage) {
log.info("Returned Information is""{}", returnedMessage.getMessage());
log.info("replyCode>>>>>>{}", returnedMessage.getReplyCode());
log.info("replyText>>>>>>{}", returnedMessage.getReplyText());
log.info("exchange>>>>>>{}", returnedMessage.getExchange());
log.info("routingKey>>>>>>>{}", returnedMessage.getRoutingKey());
}
}
2. Retry mechanism
Rabbit MQ also sets the retry mechanism for producers, defaulting to 3 times, and can also modify the number of retries, exceeding the maximum number of retries limit using manual compensation mechanism.
Broker guarantees message is not lost
1. Rabbit Mq Persistence Mechanism
After the message arrives at mq, the MQ is down, and then the message is not persisted, the message is lost. Turn on the persistence mechanism of mq, message queue, switch and message.
Turn on persistence operations Refer to RabbitMq Confirmation Mechanism & SpringBoot Integration RabbitMQ_ Well-known Snail Blog - CSDN Blog
2. Use mirror clusters
Rabbit MQ Cluster Setup Reference Message Middleware Introduction & RabitMQ Environment Setup (Linux)_ Well-known Snail Blog - CSDN Blog
3. If the queue is full, you can use a dead letter queue to guarantee that messages will not be discarded when extra messages are sent to Broker
Consumer guarantees message is not lost
1. Open the manual ack on the consumer side
- Manual-manual ack
- Auto auto
- none does not use ack
Manual ack code
@Component
@Slf4j
public class SnailConsumer {
@RabbitListener(queues = "snail_direct_queue")
public void process(Message message, @Headers Map<String, Object> headers, Channel channel) throws Exception {
// Get message Id
String messageId = message.getMessageProperties().getMessageId();
String msg = new String(message.getBody(), "UTF-8");
log.info("Get the message>>>>>>>{},news id>>>>>>{}", msg, messageId);
try {
int result = 1 / 0;
System.out.println("result" + result);
// //manual ack
Long deliveryTag = (Long) headers.get(AmqpHeaders.DELIVERY_TAG);
// Manual signing
channel.basicAck(deliveryTag, false);
} catch (Exception e) {
//Reject consumer messages (lost messages) to dead letter queues
channel.basicNack(message.getMessageProperties().getDeliveryTag(), false, false);
}
}
}2. Consumers can also use the retry mechanism. If the retry exceeds the maximum number of times and is not successful, the manual compensation mechanism can be used.
Kafka
Producer guarantees message is not lost
1. ack mechanism of producer
There are three kafka producer confirmation mechanisms with values of 0, 1 and -1 (all)
- acks = 0 If set to zero, the producer will not wait for any confirmation from the server, and the record will be immediately added to the socket buffer and treated as sent. In this case, it is not guaranteed that the server has received the record, and the retry configuration will not take effect (because the client usually does not know any failure).
- acks = 1 means that leader writes records to its local log, but does not have to wait for full confirmation from all Follwer servers to respond. In this case, there is a possibility of data loss when leader has not synchronized data to Follwer downtime.
- acks = -1 is the strongest guarantee that all partition copies are backed up without losing data. However, this pattern is often relatively inefficient.
2. producer retry mechanism
Broker guarantees message is not lost
kafka's broker uses a replica mechanism to ensure data reliability. The number of partitions in each broker is usually set to replication. When a producer writes, he or she first writes to the leader according to the distribution policy (partition by partition, key by key, no polling). The follower synchronizes the data with the leader so that he or she has a backup, which also ensures that the message data is not lost.
kafka cluster building reference
Consumer guarantees message is not lost
1. Manual ack
/*
*
* @param message
* @param ack
* @Manually submit ack
* containerFactory Manually submit message ack
* errorHandler Consumer Exception Handler
* @author xiaojie
* @date 2021/10/14
* @return void
*/
@KafkaListener(containerFactory = "manualListenerContainerFactory", topics = "xiaojie-topic",
errorHandler = "consumerAwareListenerErrorHandler"
)
public void onMessageManual(List<ConsumerRecord<?, ?>> record, Acknowledgment ack) {
for (int i=0;i<record.size();i++){
System.out.println(record.get(i).value());
}
ack.acknowledge();//Submit offset directly
}2,offset commit
Consumers use offset commit to ensure data is not lost. kafka records the offset value for each consumption, and the next time it continues to consume, it will follow the last offset to consume. Unlike other message queues, kafka does not delete data from the queue after consuming the message, but maintains a log file for log deletion policies with time and storage size. If offset is not submitted, the program will continue to consume from where it was last consumed after it was started, and there may be a case of duplicate consumption.
Offset Reset three modes
- earliest: When there are submitted offsets under each partition, start spending from the submitted offsets; Consume from scratch when no offset is submitted.
- Latest: When there are submitted offsets under each partition, start consumption from the submitted offsets; When no offset is submitted, the newly generated data under this partition is consumed.
- None: Consumption begins after offset when submitted offsets exist in each topic partition; An exception is thrown as long as one partition does not have a committed offset.
The above complete code is available for your own use Please click on me! Go away from you
Finally, I wish you all the best jobs!!!
