Preface
Previous Message Middleware - RabbitMQ (7) Advanced features are all here! (top) We describe how messages can be delivered 100% successfully., idempotency concept in detail, how to avoid the problem of repeated consumption of messages during the peak business period of mass order generation?Confirm confirmation message, Return return message.Let's introduce the following.
- Custom consumers
- Limit the flow of messages (to prevent excessive memory usage and node downtime)
- ACK of Message and Return Queue
- TTL message
- Dead Letter Queue
1. Custom consumers
1.1 Custom listening on the consumer side
We typically write a while loop in our code, use the consumer.nextDelivery method to get the next message, and then do consumer processing!
But this kind of round training is certainly not good and the code is low.
- We use Custom Constumer for greater convenience, greater decoupling, and the most common way to use it in practice!
1.2 Code Demo
1.2.1 Producers
/** * * @ClassName: Producer * @Description: Producer * @author Coder programming * @date2019 July 30, 2003 23:15:51 p.m. * */ public class Producer { public static void main(String[] args) throws Exception { //1 Create ConnectionFactory Connection connection = ConnectionUtils.getConnection(); Channel channel = connection.createChannel(); String exchange = "test_consumer_exchange"; String routingKey = "consumer.save"; String msg = "Hello RabbitMQ Consumer Message"; for(int i =0; i<5; i ++){ channel.basicPublish(exchange, routingKey, true, null, msg.getBytes()); } } }
1.2.2 Consumers
/** * * @ClassName: Consumer * @Description: Consumer * @author Coder programming * @date2019 July 30, 2003 23:13:51 p.m. * */ public class Consumer { public static void main(String[] args) throws Exception { // Create ConnectionFactory Connection connection = ConnectionUtils.getConnection(); Channel channel = connection.createChannel(); String exchangeName = "test_consumer_exchange"; String routingKey = "consumer.#"; String queueName = "test_consumer_queue"; channel.exchangeDeclare(exchangeName, "topic", true, false, null); channel.queueDeclare(queueName, true, false, false, null); channel.queueBind(queueName, exchangeName, routingKey); //Implement your own MyConsumer() channel.basicConsume(queueName, true, new MyConsumer(channel)); } }
1.2.3 Custom class: MyConsumer
/** * * @ClassName: MyConsumer * @Description: TODO * @author Coder programming * @date 2019 July 30, 2003 23:11:55 p.m. * */ public class MyConsumer extends DefaultConsumer { public MyConsumer(Channel channel) { super(channel); } //Rewrite the method you need as needed. @Override public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException { System.err.println("-----------consume message----------"); //Consumer Label System.err.println("consumerTag: " + consumerTag); //This object contains a lot of key information System.err.println("envelope: " + envelope); System.err.println("properties: " + properties); System.err.println("body: " + new String(body)); } }
1.3 Print results
2. Current Limitation at Consumer End
2.1 What is consumer-side restriction?
- Imagine a scenario where, first, we have tens of thousands of unprocessed messages on our Rabbitmq server and we open a consumer client at random. The following will happen:
- Huge amounts of messages are being pushed all at once, but we can't process so much data at once on a single client!This time it is easy to cause the server to crash and fail.
Why not limit the flow on the production side?
Because the number of customers is very large in high concurrency, it is difficult to restrict on the production side.So we can use MQ to limit the flow on the consumer side.
- RabbitMQ provides a QoS (Quality of Service Assurance) function that does not consume new messages until a certain number of messages (by setting Qos values based on consume or channel) are acknowledged without automatically acknowledging them.
In the case of current limit, do not set automatic signing, but set manual signing.
- void BasicQos(uint prfetchSize,ushort prefetchCount,bool global);
Explanation of parameters:
prefetchSize:0
PreetchCount: Tells RabbitMQ not to push more than N messages to a consumer at the same time, that is, once N messages have not been ack, the consumer will block out until there is a ack.
Global: whether truefalse applies the above settings to channels, in short, whether the above restrictions are channel or consumer.
PreetchSize and global are two values, rabbitmq is not implemented and prefetch_count is not studied for the time being to take effect in the case of no_ask = false, that is, they are not valid in the case of an automatic response.
First parameter: message limit size, how many megabytes of message.Normally no restrictions, set to 0
Second parameter: how many items to process at a time, set to 1 in practice
Third parameter: On what does the current limiting strategy apply?There are generally two application levels in RabbitMQ: 1. Channel 2.Consumer level.Typically set to false, true indicates channel level, false indicates consumer level
2.2 Code Demo
2.2.1 Producers
/** * * @ClassName: Producer * @Description: Producer * @author Coder programming * @date2019 July 30, 2003 23:15:51 p.m. * */ public class Producer { public static void main(String[] args) throws Exception { //1 Create ConnectionFactory Connection connection = ConnectionUtils.getConnection(); Channel channel = connection.createChannel(); String exchange = "test_qos_exchange"; String routingKey = "qos.save"; String msg = "Hello RabbitMQ QOS Message"; for(int i =0; i<5; i ++){ channel.basicPublish(exchange, routingKey, true, null, msg.getBytes()); } } }
2.2.2 Consumers
/** * * @ClassName: Consumer * @Description: Consumer * @author Coder programming * @date2019 July 30, 2003 23:13:51 p.m. * */ public class Consumer { public static void main(String[] args) throws Exception { //1 Create ConnectionFactory Connection connection = ConnectionUtils.getConnection(); Channel channel = connection.createChannel(); String exchangeName = "test_qos_exchange"; String queueName = "test_qos_queue"; String routingKey = "qos.#"; channel.exchangeDeclare(exchangeName, "topic", true, false, null); channel.queueDeclare(queueName, true, false, false, null); channel.queueBind(queueName, exchangeName, routingKey); //1 The first thing about current limiting is that autoAck is set to false //Set to 1 to represent one piece of data processing channel.basicQos(0, 1, false); channel.basicConsume(queueName, false, new MyConsumer(channel)); } }
2.2.3 Custom class: MyConsumer
/** * * @ClassName: MyConsumer * @Description: TODO * @author Coder programming * @date 2019 July 30, 2003 23:11:55 p.m. * */ public class MyConsumer extends DefaultConsumer { private Channel channel ; public MyConsumer(Channel channel) { super(channel); this.channel = channel; } @Override public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException { System.err.println("-----------consume message----------"); System.err.println("consumerTag: " + consumerTag); System.err.println("envelope: " + envelope); System.err.println("properties: " + properties); System.err.println("body: " + new String(body)); //Need to sign in, false says it does not support bulk signing channel.basicAck(envelope.getDeliveryTag(), false); } }
2.2.4 Test Results
Let's comment out: channel.basicAck(envelope.getDeliveryTag(), false); then start Consumer.
View Exchange
View Queues
Then start Producer.View print results:
We'll find the consumer side and only get one message.Why is that?
The first point is because we're in consumer
channel.basicConsume(queueName, false, new MyConsumer(channel));
The second parameter is set to false to sign manually.
The second point is to set the qos to accept only one message.If this message is not answered by Broker Ack, Broker will assume that you have not consumed the message and will not continue sending messages.
channel.basicQos(0, 1, false);
You can see the console, unack=1, Ready=4, total=5.
Next, we release the comment channel.basicAck(envelope.getDeliveryTag(), false); and sign the message.Restart the service.
3.1 Print results
You can see five results printed normally
4. Consumer ACK and Queue Return
4.1 Consumer-side Manual ACK and NACK
When consumers consume, we can log and compensate for business anomalies!
If there are serious problems such as server downtime, then we need to do ACK manually to ensure successful consumer consumption!
4.2 Consumer Return Queue
The consumer requeued to relay the message to the Broker for messages that were not processed successfully!
Normally, in practice, we close the requeue, which is set to False.
4.3 Code Demo
4.3.1 Producers
/** * * @ClassName: Producer * @Description: Producer * @author Coder programming * @date2019 July 30, 2003 23:15:51 p.m. * */ public class Producer { public static void main(String[] args) throws Exception { //1 Create ConnectionFactory Connection connection = ConnectionUtils.getConnection(); Channel channel = connection.createChannel(); String exchange = "test_ack_exchange"; String routingKey = "ack.save"; for(int i =0; i<5; i ++){ Map<String, Object> headers = new HashMap<String, Object>(); headers.put("num", i); //Add properties that will be used later AMQP.BasicProperties properties = new AMQP.BasicProperties.Builder() .deliveryMode(2) //Delivery mode, persistence .contentEncoding("UTF-8") .headers(headers) .build(); String msg = "Hello RabbitMQ ACK Message " + i; channel.basicPublish(exchange, routingKey, true, properties, msg.getBytes()); } } }
4.3.2 Consumers
/** * * @ClassName: Consumer * @Description: Consumer * @author Coder programming * @date2019 July 30, 2003 23:13:51 p.m. * */ public class Consumer { public static void main(String[] args) throws Exception { //1 Create ConnectionFactory Connection connection = ConnectionUtils.getConnection(); Channel channel = connection.createChannel(); String exchangeName = "test_ack_exchange"; String queueName = "test_ack_queue"; String routingKey = "ack.#"; channel.exchangeDeclare(exchangeName, "topic", true, false, null); channel.queueDeclare(queueName, true, false, false, null); channel.queueBind(queueName, exchangeName, routingKey); // Manual signing must close autoAck = false channel.basicConsume(queueName, false, new MyConsumer(channel)); } }
4.3.3 Custom class: MyConsumer
/** * * @ClassName: MyConsumer * @Description: TODO * @author Coder programming * @date 2019 July 30, 2003 23:11:55 p.m. * */ public class MyConsumer extends DefaultConsumer { private Channel channel ; public MyConsumer(Channel channel) { super(channel); this.channel = channel; } @Override public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException { System.err.println("-----------consume message----------"); System.err.println("body: " + new String(body)); try { Thread.sleep(2000); } catch (InterruptedException e) { e.printStackTrace(); } if((Integer)properties.getHeaders().get("num") == 0) { //Nack three parameters Second parameter: batch or not, Third parameter: Queue or not (need to be aware that duplicate consumption may occur, causing an endless loop) channel.basicNack(envelope.getDeliveryTag(), false, true); } else { channel.basicAck(envelope.getDeliveryTag(), false); } } }
5.1 Print results:
Be careful:
You can see that the problem of duplicate consumption leading to a dead cycle can occur when you go back to the queue. It is best to set the number of retries, such as discarding messages if they fail or fail after more than three times.
6. TTL Queue/Message
6.1 TTL
- TTL is short for Time To Live, which means life time
- RabbitMQ supports the expiration time of messages, which can be specified when the message is sent
- RabbitMQ supports the expiration time of a queue, which is calculated from the time the message enters the queue. As long as the timeout configuration of the queue is exceeded, the message is automatically cleared
6.2 Code Demo
6.2.1 Demonstrate directly from the console
Create a queue from the console
Maximum size of x-max-length queue
The x-message-ttl is set for 10 seconds and will be cleared if the message has not been consumed.
Add exchange
Queue is bound to Exchange
Click test_ttl_exchange to bind
Check to see if the binding was successful
Send messages through the console
Message unprocessed automatic cleanup
Set expiration time on production side
AMQP.BasicProperties properties = new AMQP.BasicProperties.Builder() .deliveryMode(2) .contentEncoding("UTF-8") .expiration("10000") .headers(headers) .build();
These two attributes are different, one corresponding to the message body and the other to the expiration of the queue.
7. Dead Letter Queue
7.1 Conceptual understanding
Dead letter queue: DLX,Dead-Letter-Exchange
RabbitMQ's dead letter team is closely related to Exchange
- With DLX, when a message becomes a dead message in one queue, it can be re publish ed to another Exchange, which is DLX
There are several situations when a message becomes dead letter
- Message rejected (basic.reject/basic.nack) and requeue=false
- Message TTL expires
- Queue reaches maximum length
DLX is also a normal Exchange, no different from normal Exchange, it can be specified on any queue, in fact, set the properties of a queue
When there are dead letters in this queue, RabbitMQ automatically republishes this message to the set Exchange and routes it to another queue.
The ability to listen for messages in this queue and process them accordingly compensates for the immediate parameter previously supported by RabbitMQ3.0.
7.2 Code Demo
- Dead Letter Queue Settings:
- First you need to set exchange and queue for the dead letter queue, then bind:
Exchange:dlx.exchange
Queue:dlx.queue
RoutingKey:# - We then declare switches, queues, bindings as normal, but we need to add a parameter to the queue: arguments.put("x-dead-letter-exchange","dlx.exchange");
- This allows messages to be routed directly to the dead letter queue when they expire, requeue, and queue reach their maximum length!
7.2.1 Producers
/** * * @ClassName: Producer * @Description: Producer * @author Coder programming * @date2019 July 30, 2003 23:15:51 p.m. * */ public class Producer { public static void main(String[] args) throws Exception { //Create ConnectionFactory Connection connection = ConnectionUtils.getConnection(); Channel channel = connection.createChannel(); String exchange = "test_dlx_exchange"; String routingKey = "dlx.save"; String msg = "Hello RabbitMQ DLX Message"; for(int i =0; i<1; i ++){ AMQP.BasicProperties properties = new AMQP.BasicProperties.Builder() .deliveryMode(2) .contentEncoding("UTF-8") .expiration("10000") .build(); channel.basicPublish(exchange, routingKey, true, properties, msg.getBytes()); } } }
7.2.2 Consumers
/** * * @ClassName: Consumer * @Description: Consumer * @author Coder programming * @date2019 July 30, 2003 23:13:51 p.m. * */ public class Consumer { public static void main(String[] args) throws Exception { //Create ConnectionFactory Connection connection = ConnectionUtils.getConnection(); Channel channel = connection.createChannel(); // This is a normal switch, queue and route String exchangeName = "test_dlx_exchange"; String routingKey = "dlx.#"; String queueName = "test_dlx_queue"; channel.exchangeDeclare(exchangeName, "topic", true, false, null); Map<String, Object> agruments = new HashMap<String, Object>(); agruments.put("x-dead-letter-exchange", "dlx.exchange"); //This agruments property, to be set on the declaration queue channel.queueDeclare(queueName, true, false, false, agruments); channel.queueBind(queueName, exchangeName, routingKey); //To make a declaration of the dead letter queue: channel.exchangeDeclare("dlx.exchange", "topic", true, false, null); channel.queueDeclare("dlx.queue", true, false, false, null); channel.queueBind("dlx.queue", "dlx.exchange", "#"); channel.basicConsume(queueName, true, new MyConsumer(channel)); } }
7.2.3 Custom class: MyConsumer
/** * * @ClassName: MyConsumer * @Description: TODO * @author Coder programming * @date 2019 July 30, 2003 23:11:55 p.m. * */ public class MyConsumer extends DefaultConsumer { public MyConsumer(Channel channel) { super(channel); } @Override public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException { System.err.println("-----------consume message----------"); System.err.println("consumerTag: " + consumerTag); System.err.println("envelope: " + envelope); System.err.println("properties: " + properties); System.err.println("body: " + new String(body)); } }
7.2.4 Test Results
Run Consumer, view console
View Exchanges
View queue
You can see that test_dlx_queue has an additional DLX identity, indicating that when a dead letter appears in the queue, the message will be sent to the dead letter queue dlx_queue
Close Consumer and run Producer only
After 10 seconds, the message expires
In our work, the dead letter queue is very important for messages that have no consumers and are dead letter.We can use compensation mechanisms.
Summary
This paper mainly introduces the advanced features of RabbitMQ. First, it describes how the Internet factory can guarantee 100% message delivery success and idempotency in actual use, as well as the confirmation message, return message, ACK and return queue, flow restriction of message, and the use of timeout and dead-letter queue for RabbitMQ.
end of document
Welcome to Personal WeChat Public Number: Coder Programming
Get the latest original technical articles and free learning materials, more boutique mind maps, interview materials, PMP reference materials and so on, to help you learn technical knowledge anytime, anywhere!
A new qq group has been created: 315211365. Welcome to the group to exchange and learn together.Thank you!You can also introduce to friends who need it.
Articles are included in
Github: https://github.com/CoderMerli...
Gitee: https://gitee.com/573059382/c...
Welcome to your attention and star~
Reference article:
RabbitMQ Message Middleware Essentials
Recommended articles:
Message Middleware - RabbitMQ (6) Understand the core concepts of Exchange switches!
Message Middleware - RabbitMQ (7) Advanced features are all here! (top)