It should have been updated yesterday, but when using the browser to access the web port of the server's RabbitMQ, chrome shows that it is not a private link and does not allow you to log in to the Edge. Baidu has found many problems and still can't solve them. The reason is that the server doesn't install SSL certificate and uses ip directly to access.
1. Persistence
When the RabbitMQ service is stopped, the messages sent by the message producer will not be lost. By default, queues and messages are ignored when RabbitMQ exits or crashes. In order to ensure that messages are not lost, both queues and messages need to be marked as persistent.
1.1 achieving persistence
- Queue persistence: when creating a queue, channel queueDeclare(); The second parameter is changed to true.
- Message persistence: when sending messages using the channel basicPublish(); Change the third parameter to: messageproperties PERSISTENT_ TEXT_ Plan represents a persistent message.
/**
* @Description Persistent MQ
* @date 2022/3/7 9:14
*/
public class Producer3 {
private static final String LONG_QUEUE = "long_queue";
public static void main(String[] args) throws Exception {
Channel channel = RabbitMQUtils.getChannel();
// Persistent queue
channel.queueDeclare(LONG_QUEUE,true,false,false,null);
Scanner scanner = new Scanner(System.in);
int i = 0;
while (scanner.hasNext()){
i++;
String msg = scanner.next() + i;
// Persistent message
channel.basicPublish("",LONG_QUEUE, MessageProperties.PERSISTENT_TEXT_PLAIN,msg.getBytes(StandardCharsets.UTF_8));
System.out.println("Send message:'" + msg + "'success");
}
}
}
However, there is also a cache interval point for storing messages. There is no real write to disk. The persistence guarantee is not strong enough, but it is more than enough for a simple queue.
1.2 unfair distribution
The polling distribution method is not applicable when the processing efficiency of consumers is different. Therefore, real fairness should follow the premise that those who can do more work.
Modify channel at the consumer basicQos(1); Indicates that unfair distribution is enabled
/**
* @Description Unfair distribution to consumers
* @date 2022/3/7 9:27
*/
public class Consumer2 {
private static final String LONG_QUEUE = "long_queue";
public static void main(String[] args) throws Exception {
Channel channel = RabbitMQUtils.getChannel();
DeliverCallback deliverCallback = (consumerTag, message) -> {
// Simulate concurrent sleep for 30 seconds
try {
Thread.sleep(30000);
System.out.println("thread B Receive message:"+ new String(message.getBody(), StandardCharsets.UTF_8));
channel.basicAck(message.getEnvelope().getDeliveryTag(),false);
} catch (InterruptedException e) {
e.printStackTrace();
}
};
// Set unfair distribution
channel.basicQos(1);
channel.basicConsume(LONG_QUEUE,false,deliverCallback,
consumerTag -> {
System.out.println(consumerTag + "Consumer cancels consumption");
});
}
}
1.3 unfair distribution of tests
Test purpose: whether it can be realized, those who can do more work.
Test method: two consumers sleep different events to simulate different processing events. If the processing time (sleep time) is short and can process multiple messages, it means that the purpose is achieved.
First start the producer to create a queue, and then start two consumers respectively.
The producer sends four messages in sequence:
Thread A with short sleep time received three messages
Thread B with long sleep time only receives the second message:
Because thread B takes A long time to process messages, other messages are allocated to thread A.
The experiment is successful!
1.4 pre value
Both message sending and manual confirmation are completed asynchronously, so there is a buffer for unconfirmed messages. Developers hope to limit the size of the buffer to avoid unlimited unconfirmed messages in the buffer.
The expected value here is the above method channel basicQos(); If there is a message equal to the parameter on the current channel, the message will not be consumed in the current channel.
1.4.1 code test
Test method:
- Create two different consumers and give 5 2 expected values respectively.
- Specify 5 for those with long sleep time and 2 for those with short sleep time.
- If the message is obtained according to the specified expected value, it indicates that the test is successful, but it does not mean that it will be allocated according to 5 and 2. This is similar to the judgment of weight.
The code can modify the expected value according to the above code.
2. Release confirmation
Publish confirmation is the process in which the producer is notified to the producer after the queue confirmation is persisted after the producer publishes the message to the queue. This ensures that messages are not lost.
It should be noted that queue persistence needs to be turned on before confirmation publishing can be used.
Opening method: channel confirmSelect();
2.1 single confirmation release
It is a synchronous publishing method, that is, after sending a message, the subsequent messages will continue to be published only after confirming its publishing. If there is no confirmation within the specified time, an exception will be thrown. The disadvantage is that it is very slow.
/**
* @Description Confirmation release - single confirmation
* @date 2022/3/7 14:49
*/
public class SoloProducer {
private static final int MESSAGE_COUNT = 100;
private static final String QUEUE_NAME = "confirm_solo";
public static void main(String[] args) throws Exception {
Channel channel = RabbitMQUtils.getChannel();
// Generate queue
channel.queueDeclare(QUEUE_NAME,true,false,false,null);
// Open confirmation release
channel.confirmSelect();
// Record start time
long beginTime = System.currentTimeMillis();
for (int i = 0; i < MESSAGE_COUNT; i++) {
String msg = ""+i;
channel.basicPublish("",QUEUE_NAME, MessageProperties.PERSISTENT_TEXT_PLAIN,msg.getBytes(StandardCharsets.UTF_8));
// Single release confirmation
boolean flag = channel.waitForConfirms();
if (flag){
System.out.println("Send message:" + i);
}
}
// Record end time
long endTime = System.currentTimeMillis();
System.out.println("send out" + MESSAGE_COUNT + "Message consumption:"+(endTime - beginTime) + "millisecond"); }
}
2.2 batch confirmation release
Batch by batch confirmation release can improve the throughput of the system. However, the disadvantage is that when a failure causes a problem in publishing, you need to save the whole batch in memory and republish it later.
/**
* @Description Release - batch confirmation
* @date 2022/3/7 14:49
*/
public class BatchProducer {
private static final int MESSAGE_COUNT = 100;
private static final String QUEUE_NAME = "confirm_batch";
public static void main(String[] args) throws Exception {
Channel channel = RabbitMQUtils.getChannel();
// Generate queue
channel.queueDeclare(QUEUE_NAME,true,false,false,null);
// Open confirmation release
channel.confirmSelect();
// Set the number of batches to be confirmed once.
int batchSize = MESSAGE_COUNT / 10;
// Record start time
long beginTime = System.currentTimeMillis();
for (int i = 0; i < MESSAGE_COUNT; i++) {
String msg = ""+i;
channel.basicPublish("",QUEUE_NAME, MessageProperties.PERSISTENT_TEXT_PLAIN,msg.getBytes(StandardCharsets.UTF_8));
// Batch release confirmation
if (i % batchSize == 0){
if (channel.waitForConfirms()){
System.out.println("Send message:" + i);
}
}
}
// Record end time
long endTime = System.currentTimeMillis();
System.out.println("send out" + MESSAGE_COUNT + "Message consumption:"+(endTime - beginTime) + "millisecond");
}
}
Obviously, the efficiency is much higher than that of a single confirmation release.
2.3 asynchronous confirmation Publishing
The programming is more complex than the above two, but the cost performance is very high. The reliability and efficiency are much better. The callback function is used to achieve the reliability of message transmission.
/**
* @Description Confirmation release - asynchronous confirmation
* @date 2022/3/7 14:49
*/
public class AsyncProducer {
private static final int MESSAGE_COUNT = 100;
private static final String QUEUE_NAME = "confirm_async";
public static void main(String[] args) throws Exception {
Channel channel = RabbitMQUtils.getChannel();
// Generate queue
channel.queueDeclare(QUEUE_NAME,true,false,false,null);
// Open confirmation release
channel.confirmSelect();
// Record start time
long beginTime = System.currentTimeMillis();
// Confirm successful callback
ConfirmCallback ackCallback = (deliveryTab,multiple) ->{
System.out.println("Confirmation success message:" + deliveryTab);
};
// Confirmation failure callback
ConfirmCallback nackCallback = (deliveryTab,multiple) ->{
System.out.println("Unacknowledged message:" + deliveryTab);
};
// Message listener
/**
* addConfirmListener:
* 1. Confirm the successful message;
* 2. Confirm the message of failure.
*/
channel.addConfirmListener(ackCallback,nackCallback);
for (int i = 0; i < MESSAGE_COUNT; i++) {
String msg = "" + i;
channel.basicPublish("",QUEUE_NAME, MessageProperties.PERSISTENT_TEXT_PLAIN,msg.getBytes(StandardCharsets.UTF_8));
}
// Record end time
long endTime = System.currentTimeMillis();
System.out.println("send out" + MESSAGE_COUNT + "Message consumption:"+(endTime - beginTime) + "millisecond");
}
}
2.4 processing unconfirmed messages
The best way to handle it is to put the unacknowledged messages into a memory based queue that can be accessed by the publishing thread.
For example, the ConcurrentLinkedQueue can transfer messages between the confirm callbacks of the confirmation queue and the publishing thread.
Treatment method:
- Record all messages to be sent;
- Confirm that the deletion is successful at the publishing place;
- Print unconfirmed messages.
Using a hash table to store messages has the following advantages:
- You can associate needs with messages;
- Easily delete entries in batch;
- Support high concurrency.
ConcurrentSkipListMap<Long,String > map = new ConcurrentSkipListMap<>();
/**
* @Description Asynchronously publish confirmation and process messages that are not successfully published
* @date 2022/3/7 18:09
*/
public class AsyncProducerRemember {
private static final int MESSAGE_COUNT = 100;
private static final String QUEUE_NAME = "confirm_async_remember";
public static void main(String[] args) throws Exception {
Channel channel = RabbitMQUtils.getChannel();
// Generate queue
channel.queueDeclare(QUEUE_NAME,true,false,false,null);
// Open confirmation release
channel.confirmSelect();
// Thread safe and orderly a hash table, suitable for high concurrency
ConcurrentSkipListMap< Long, String > map = new ConcurrentSkipListMap<>();
// Record start time
long beginTime = System.currentTimeMillis();
// Confirm successful callback
ConfirmCallback ackCallback = (deliveryTab, multiple) ->{
//2. Delete at the confirmation of successful publishing;
// Batch delete
if (multiple){
ConcurrentNavigableMap<Long, String> confirmMap = map.headMap(deliveryTab);
confirmMap.clear();
}else {
// Delete separately
map.remove(deliveryTab);
}
System.out.println("Confirmation success message:" + deliveryTab);
};
// Confirmation failure callback
ConfirmCallback nackCallback = (deliveryTab,multiple) ->{
// 3. Print unconfirmed messages.
System.out.println("Unacknowledged message:" + map.get(deliveryTab) + ",Marking:" + deliveryTab);
};
// Message listener
/**
* addConfirmListener:
* 1. Confirm the successful message;
* 2. Confirm the message of failure.
*/
channel.addConfirmListener(ackCallback,nackCallback);
for (int i = 0; i < MESSAGE_COUNT; i++) {
String msg = "" + i;
channel.basicPublish("",QUEUE_NAME, MessageProperties.PERSISTENT_TEXT_PLAIN,msg.getBytes(StandardCharsets.UTF_8));
// 1. Record all messages to be sent;
map.put(channel.getNextPublishSeqNo(),msg);
}
// Record end time
long endTime = System.currentTimeMillis();
System.out.println("send out" + MESSAGE_COUNT + "Message consumption:"+(endTime - beginTime) + "millisecond");
}
}
2.5 summary
Obviously, in addition to some trouble in coding, asynchronous processing is much better than single processing and batch processing in processing time, efficiency and availability.