Applicable scenario: for tasks with priority, send messages to the corresponding queue according to the priority of the task, so that more resources can be assigned to deal with high priority queues

Theme switch

Routing of direct connected switch_ The key scheme is very simple. If we want to send a message to multiple queues, the switch needs to be bound with a lot of routing_key, suppose that each switch is bound with a bunch of routing_key is connected to each queue. Then the management of messages will be extremely difficult.

Therefore, RabbitMQ provides a topic switch. The messages sent to the topic switch need to carry the routing of the specified rules_ Key, the subject switch will send data to the corresponding (multiple) queues according to this rule.

Routing of subject switch_ The key needs to have certain rules for binding switches and queues_ Key needs to use *. #. * Format for each part Separate, where:

• *Represents a word
• #Represents any number (zero or more) of words

When the binding key of a queue is #, the queue will ignore the message routing key and receive all messages

Head switch

The first switch ignores routing_ A routing method of key. The routing rules of routers and switches are exchanged through Headers, which is a bit like HTTP Headers.

When a switch is declared as the head switch and a queue is bound, a hash data structure is defined. When a message is sent, a group of hash data structure information will be carried. When the hash content matches, the message will be written to the queue.

When Binding switches and queues, the Hash structure is required to carry a key x-match. The Value of this key can be any or all, which indicates whether the Hash carried by the message needs to match all or only one key

Compared with the direct connected switch, the advantage of the head switch is that the matching rules are not limited to strings

• any: as long as there is a pair of key value pairs carried when publishing the message, the headers can match one of the multiple parameter arguments defined by the queue. Note that this is a complete match of key value pairs. It is impossible to match only the keys, but the values are different;
• All: all entries carried when publishing the message must exactly match all entries bound to the queue

Binding

For the virtual connection between exchange and queues, exchange will generate a routing table after binding with multiple Message queues. The routing table stores the Binding Key, which is the limiting condition of the messages required by Message queues. When exchange receives a Message, it will parse its Header to get the Routing Key. Exchange routes the Message to the Message Queue according to the Routing Key and Exchange Type. The Binding Key is specified by the Consumer when Binding Exchange and Message Queue, while the Routing Key is specified by the Producer when sending Message. The matching method of the two is determined by the Exchange Type

Routing Key

A routing rule that a virtual machine can use to determine how to route a particular message.

Queue

Also known as Message Queue, Message Queue saves messages and forwards them to consumers.

Message publishing process:

1. The producer establishes a TCP connection with the Broker.
2. Establish channels between producers and brokers.
3. The producer sends the message to the Broker through the channel, and Exchange forwards the message.
4. Exchange forwards messages to the specified Queue

Message receiving process:

1. The consumer establishes a TCP connection with the Broker.
2. Establish channels between consumers and brokers.
3. Consumer listens to the specified Queue
4. When a message arrives at the Queue, the Broker pushes the message to the consumer by default.
5. Consumer received message.

Message flow process

The producer produces a Message and delivers it to Exchange

An Exchange can bind multiple message queues. It routes to the specified queue according to the routing key. Finally, the consumer listens to the queue

Working mode

Queue mode:

When the task is too heavy or there are many tasks, using work queue can improve the speed of task processing.

1. A message will only be received by one consumer;

2. rabbitmq uses polling to send messages to consumers on average;

3. Consumers will not receive the next message until they have processed a message.

Publish subscribe mode:

1. Each consumer listens to their own queue.

2. The producer sends the message to the broker, and the switch forwards the message to each queue bound to the switch, and each queue bound to the switch will receive the message

Corresponding to the fan out type in the switch

Routing mode:

1. Each consumer listens to their own queue and sets the routing key.

2. The producer sends the message to the switch, and the switch forwards the message to the specified queue according to the routingkey.

Corresponding to the direct type in the switch

Wildcard pattern:

Corresponding topics type in the switch

Header transponder mode:

Corresponding to the header type in the switch

Remote procedure call mode:

RPC is the method by which the client remotely calls the server. The asynchronous call of RPC can be realized by using MQ, which is based on the Direct switch. The process is as follows:

1. The client is the producer and consumer. It sends RPC call messages to the RPC request queue and listens to the RPC response queue.
2. The server listens to the message of the RPC request queue. After receiving the message, it executes the server's method to get the result returned by the method.
3. The server sends the result of the RPC method to the RPC response queue.
4. The client (RPC caller) listens to the RPC response queue and receives the RPC call result.

Basic use

Basic command line operations

About service operation:

1. Service start: rabbitmqctl start_ app / rabbitmq-server start &

2. Service stop: rabbitmqctl stop_app / rabbitmq-server stop

3. Service restart: service rabbitmq server restart

4. Node status: rabbitmqctl status

About user actions:

1. Add user: rabbitmqctl add_user username password
2. List all users: rabbitmqctl list_users
3. Delete user: rabbitmqctl delete_user username
4. Clear user permissions: rabbitmqctl clear_permissions -p vhostpath username
5. List user permissions: rabbitmqctl list_user_permissions username
6. Change Password: rabbitmqctl change_password username newpassword
7. Set user permissions: rabbitmqctl set_permissions -p vhostpath username ".*" ".*" ".*"

About virtual host operations:

1. Create virtual host: rabbitmqctl add_vhost vhostpath
2. List all virtual hosts: rabbitmqctl list_vhost
3. List all permissions on the virtual host: rabbitmqctl list_permissions -p vhostpath
4. Delete virtual host: rabbitmqctl delete_vhost vhostpath

About message queuing:

1. View all queue information: rabbitmqctl list_queues
2. Clear messages in the queue: rabbitmqctl -p vhostpath purge_queue blue

Advanced command

1. rabbitmqctl reset: to remove all data, click rabbitmqctl stop_ Use after app
2. Cluster composition command: rabbitmqctl join_cluster [--ram] (RAM memory level storage, disc disk)
3. View cluster status: rabbitmqctl cluster_status
4. Modify the storage form of cluster nodes: rabbitmqctl change_cluster_node_type disc | ram
5. Forget (remove) node: rabbitmqctl forget_cluster_node [--offline] (when the offline service is not started)
6. Modify node name: rabbitmqctl rename_cluster_node oldnode1 newnode1 [oldnode2] [newnode2 ...]

In case of cluster configuration failure and failover, the failed node can be removed. It can remove nodes without starting

Getting started

Introducing RabbitMQ dependency:

<dependency>
	<groupId>com.rabbitmq</groupId>
	<artifactId>amqp-client</artifactId>
	<version>3.6.5</version>
</dependency>

Create a producer

public class Producer {

    private static final String QUEUE_NAME = "test01";

    public static void main(String[] args) throws IOException, TimeoutException {
        // 1\.  Create connection factory and configure
        ConnectionFactory connectionFactory = new ConnectionFactory();
        connectionFactory.setHost("192.168.58.129");
        connectionFactory.setPort(5672);
        // Set up virtual machine
        connectionFactory.setVirtualHost("/test");
        // 2\.  Establish connection through connection factory
        Connection connection = connectionFactory.newConnection();

        // 3\.  Create a Channel through connection
        Channel channel = connection.createChannel();

        // 4\.  Send data through Channel (exchange, routingKey, props, body)
        // When Exchange is not specified, the switch defaults to AMQP default. At this time, it depends on RoutingKey
        // RoutingKey must be equal to the queue name to be routed, otherwise the message will be deleted
        for (int i = 0; i < 5; i++) {
            String msg = "Learn For RabbitMQ-" + i;
            channel.basicPublish("", QUEUE_NAME, null, msg.getBytes());
            System.out.println("Send message : " + msg);
        }

        // 5. Close the connection
        channel.close();
        connection.close();
    }
}

Create a consumer

public class Consumer {

    private static final String QUEUE_NAME = "test01";

    public static void main(String[] args) throws IOException, TimeoutException {
        // 1\.  Create connection factory and configure
        ConnectionFactory connectionFactory = new ConnectionFactory();
        connectionFactory.setHost("192.168.58.129");
        connectionFactory.setPort(5672);
        // Set up virtual machine
        connectionFactory.setVirtualHost("/test");
        // 2\.  Establish connection through connection factory
        Connection connection = connectionFactory.newConnection();

        // 3\.  Create a Channel through connection
        Channel channel = connection.createChannel();

        // 4\.  Declare queue (queue, durable, exclusive, autoDelete, args)
        channel.queueDeclare(QUEUE_NAME, true, false, false, null);

        // 5\.  Create consumer
        DefaultConsumer defaultConsumer = new DefaultConsumer(channel) {
            /**
             * Get message (called when listening for a message)
             * @param consumerTag For consumer tags, autoAck can be set to false when listening to the queue, that is, manual confirmation (to avoid message loss) and message uniqueness processing
             * @param envelope envelope
             * @param properties Properties of the message
             * @param body Content of the message
             * @throws IOException
             */
            @Override
            public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
                String msg = new String(body, "utf-8");
                System.out.println("Received message : " + msg);
            }
        };

        // 6\.  Set the Channel and listen to the queue (String queue, boolean autoAck,Consumer callback)
        channel.basicConsume(QUEUE_NAME, true, defaultConsumer);

    }
}

Parameters:

• Queue: queue name
• durable: persistent, true. Even if the service is restarted, it will not be deleted
• Exclusive: exclusive, true. The queue can only use one connection. The connection is disconnected and the queue is deleted
• autoDelete: Auto delete. true is disconnected from Exchange. That is, when the queue is not associated with Exchange, it will be automatically deleted
• arguments: extended parameters
• autoAck: Auto sign in (receipt)

When Exchange is not specified, the switch defaults to AMQP default. At this time, it depends on the RoutingKey. The RoutingKey must be equal to the queue name to be routed, otherwise the message will be deleted

Switch properties

Name: switch name

Type: switch type - direct, topic, fanout, header

Durability: whether persistence is required. true means persistence

Auto Delete: when the last queue bound to Exchange is deleted, that is, there is no queue bound on Exchange, the exhrange is automatically deleted

Internal: whether the current Exchange is used for RabbitMQ internal use. Most use False by default

Arguments: extension parameter, used to extend the customization of AMQP protocol

Direct Exchange:

 // Consumer
 // Claim switch: 
 // (String exchange, String type, boolean durable, boolean autoDelete, boolean internal, Map<String, Object) arguments)
 channel.exchangeDeclare("exchangeName", BuiltinExchangeType.DIRECT, true, false, false, null);

 // Declaration queue (string queue, Boolean durable, Boolean exclusive, Boolean autodelete, map < string, object) args)
 channel.queueDeclare("queueName", true, false, false, null);

 // Establish binding relationship:
 channel.queueBind("queueName", "exchangeName", "routingKey");
 // ===================================================================
 // Producer
 // Send message (String exchange, String routingKey, BasicProperties props, Bytes[] body)
 channel.basicPublish("exchangeName", "routingKey", null, "msg".getBytes());

Topic Exchange:

 // Consumer
 // Claim switch: 
 // (String exchange, String type, boolean durable, boolean autoDelete, boolean internal, Map<String, Object) arguments)
 channel.exchangeDeclare("exchangeName", BuiltinExchangeType.TOPIC, true, false, false, null);

 // Declaration queue (string queue, Boolean durable, Boolean exclusive, Boolean autodelete, map < string, object) args)
 channel.queueDeclare("queueName", true, false, false, null);

 // Establish binding relationship:
 channel.queueBind("queueName", "exchangeName", "routingKey.#");
 // ===================================================================
 // Producer
 // Send message (String exchange, String routingKey, BasicProperties props, Bytes[] body)
 channel.basicPublish("exchangeName", "routingKey.hi", null, "msg".getBytes());
 channel.basicPublish("exchangeName", "routingKey.save", null, "msg".getBytes());
 channel.basicPublish("exchangeName", "routingKey.save.hi", null, "msg".getBytes());

Because the "#" of fuzzy matching is used, three messages sent can be matched. Therefore, three messages can be received

Fanout Exchange:

 // Consumer
 // Claim switch: 
 // (String exchange, String type, boolean durable, boolean autoDelete, boolean internal, Map<String, Object) arguments)
 channel.exchangeDeclare("exchangeName", BuiltinExchangeType.FANOUT, true, false, false, null);

 // Declaration queue (string queue, Boolean durable, Boolean exclusive, Boolean autodelete, map < string, object) args)
 channel.queueDeclare("queueName", true, false, false, null);

 // Establish binding relationship: 
 //(routingKey is not set, and this field is empty)
 channel.queueBind("queueName", "exchangeName", "");
 // ===================================================================
 // Producer
 // Send message (String exchange, String routingKey, BasicProperties props, Bytes[] body)
 // Similarly, the routingKey is empty (it can also be any string, because fanout does not rely on the routingKey)
 channel.basicPublish("exchangeName", "", null, "msg".getBytes());

Advanced features

Reliable delivery

What is reliable delivery on the production side

• Ensure the successful sending of the message
• Ensure that the MQ node receives successfully
• The sender receives an MQ node (Broker) acknowledgement (received)
• Improve the message compensation mechanism

Scheme I of reliable delivery

The message is stored in the database (persisted to the database), and the message status is marked. If the message does not respond, the polling operation is carried out

1. Drop the business message into the database, regenerate it into a message, and drop it into the message DB for recording (for example, the message has just been created and is being sent. status: 0)

Disadvantages: the database is persisted twice

2. The production end sends a message.

3. After receiving the message, the Broker will respond to the production terminal. Confirm Listener asynchronously listens to the Broker's response.

4. After the response indicates that the message is delivered successfully, grab the specified message record in the message DB and update the status, such as status: 1

5. If the network is unstable in 3, the Listener does not receive the response of successful message confirmation.

Then the status in the message database is still 0, and the Broker may be the state of receiving the message.

Therefore, set a rule (scheduled task), for example, if the message is still in the state of 0 after 5 minutes (timeout), the record will be extracted.

6. Redelivery

7. Limit the number of retries, such as 3. If it is greater than 3, the delivery fails. Update the value of status

Use manual compensation mechanism to query the cause of message failure

Delayed delivery of messages in high concurrency scenarios, secondary confirmation and callback check

Upstream service: production side

Downstream service: consumer side

1: After the business message is stored in the database, send the message to the Broker.

2: The second delay (set delay time) check message is then sent.

3: The consumer listens to the specified queue and processes the received message

4: After processing, a response message is generated and sent to the Broker.

5: The Callback service listens to the response message, and after receiving the response message, it is persisted to the message DB (recording success status).

6: When the delay time comes, the delayed message is also monitored by the listener of the Callback service, and then check the message DB. If no successful status is found, the Callback service needs to compensate, initiate RPC communication and ask the production side to resend. The production end queries the service message in the database through the id carried in the received command, and then resend it, that is, jump to 1.

The scheme reduces the storage of database and ensures the performance

Consumer idempotency

Avoid repeated consumption of messages

The consumer realizes idempotency, receives multiple identical messages, but does not consume repeatedly, that is, receives multiple identical messages.

Scheme:

1. Unique ID + fingerprint code mechanism

• Unique ID + fingerprint code (business rules, timestamp, etc.) mechanism, using the database primary key to remove duplication
• SELECT COUNT(1) FROM T_ORDER WHERE ID = unique ID + fingerprint code. insert the message before query. If yes, it indicates that the message has been processed, and failure is returned
• Advantages: simple implementation
• Disadvantages: high throughput and database write performance bottleneck
• Solution: divide databases, tables and routing algorithm according to ID

2. Utilize the atomicity of Redis

Issues to consider:

• Whether to drop the database, for example, how to achieve data consistency between the database and cache
• How to set the timing synchronization strategy (reliability guarantee) when the database is not dropped and the data is stored in the cache

Confirm confirmation message

After the producer delivers the message, if the Broker receives the message, it will give the producer a reply.

Write at the end

Learning technology is a slow, long and hard road. We can't learn it by passion or by staying up for a few days and nights. We must develop the habit of studying hard at ordinary times. So: insist!

Finally, I will share some interview questions of BATJ and other large factories in the past 20 or 21 years. These technical points are sorted into videos and PDF s (in fact, it took a lot of energy than expected), including knowledge context + many details. Due to the limited space, the above is only a part of them in the form of pictures.

Collection method: You can get it for free by poking here

Mybatis interview topic

MySQL interview topic

Confirm confirmation message

After the producer delivers the message, if the Broker receives the message, it will give the producer a reply.

Write at the end

Learning technology is a slow, long and hard road. We can't learn it by passion or by staying up for a few days and nights. We must develop the habit of studying hard at ordinary times. So: insist!

Finally, I will share some interview questions of BATJ and other large factories in the past 20 or 21 years. These technical points are sorted into videos and PDF s (in fact, it took a lot of energy than expected), including knowledge context + many details. Due to the limited space, the above is only a part of them in the form of pictures.

Collection method: You can get it for free by poking here

[external chain picture transferring... (img-h2MY3VkP-1628396048402)]

Mybatis interview topic

[external chain picture transferring... (img-pQTLbFBJ-1628396048402)]

MySQL interview topic

[external chain picture transferring... (img-12MBUNrg-1628396048403)]

Concurrent programming interview topic