MQ related concepts
What is mq
Message queue is literally a queue. First in, first out, only messages are stored. It is also a cross process communication mechanism for upstream and downstream message delivery. In the Internet architecture, mq is a common upstream and downstream "logical decoupling + physical decoupling" message communication service. Message queue is a container that holds messages during transmission. It is typical: producer consumer model. Producers continue to produce messages to the message queue, and consumers continue to get messages from the queue. Because the production and consumption of messages are asynchronous, and only care about the sending and receiving of messages without the intrusion of business logic, the decoupling between producers and consumers is realized.
Application scenario
1. Task asynchronous processing:
In the high concurrency environment, requests are often blocked because there is no time for synchronous processing. For example, a large number of requests such as insert and update arrive at MySQL at the same time, which directly leads to countless row locks and table locks, and even too many requests will accumulate in the end, triggering the too many connections error. By using message queuing, we can process requests asynchronously, so as to relieve the pressure of the system. The message queue notifies the message receiver of operations that do not need synchronous processing and take a long time to process asynchronously. Reduces application response time. (synchronous: the next request can only be made after one request is completed, asynchronous: other operations can be performed during the request). Example: traditional: order request inventory operation - > Save order information to order table - > response to successful order, which takes a long time to process information and users wait a long time. After the lock is released, the order is returned directly to the order success. The order information is placed in the queue and distributed to the module that saves the order information. The main process is more time-consuming and allows it to be carried out alone.
2. Application decoupling:
MQ is equivalent to an intermediary. The producer interacts with the consumer through MQ, which decouples the application. Prevent a module from failing and the system from crashing. Different modules correspond to different queues. For example, a message needs to be distributed by SMS, email and weixin. There is no need to write these methods together to share the message. Instead, as consumers, they correspond to their respective queues and send messages to these queues.
3. Flow peak elimination
When the concurrency is too high, the processing limit is exceeded. The message queue is used as a cache to put messages in the queue and process them after a period of time
Common MQ products
- ActiveMQ: JMS based
- RabbitMQ : Based on AMQP protocol, erlang language development, good stability
- RocketMQ: Based on JMS, Alibaba product, which is currently handed over to Apache foundation
- Kafka: distributed message system, high throughput
Basic concepts
Producer - > MQ (switch, queue) - > consumer
Producer: an application that sends messages.
Consumer: the application that receives the message.
Queue: a cache that stores messages.
Message: the information sent by the producer to the consumer through RabbitMQ.
Connection: a TCP connection between RabbitMQ and the application server.
Channel: a virtual channel in a connection. When you send or receive messages through the message queue, this operation is carried out through the channel.
Exchange: the exchange is responsible for receiving messages from the producer and distributing them to the corresponding message queue according to the exchange type. To receive messages, a queue must be bound to a switch.
Binding: binding is an associated connection between a queue and a switch.
Routing Key: a Routing Key is a key for the switch to view and decide how to distribute messages to the queue according to the key. The Routing Key can be said to be the destination address of the message.
RabbitMQ message sending mode
RabbitMQ includes five queue modes: simple queue, work queue, publish / subscribe, routing, topic, rpc, etc.
install
centos system
systemctl start rabbitmq-server
#Or
Rabbitmq server - detached # background daemon mode
Other related operations
#Start service
systemctl start rabbitmq-server.service
#Set startup
systemctl enable rabbitmq-server.service
#Stop startup
systemctl disable rabbitmq-server.service
#Restart service
systemctl restart rabbitmq-server.service
#View the current status of the service
systemctl status rabbitmq-server.service
#View all started services
systemctl list-units --type=service
systemctl stop rabbitmq-server.service
#Shut down service
/Location / service rabbitmq server stop
#Install plug-ins
rabbitmq-plugins enable rabbitmq_management
Enter the web plug-in
http: / / server ip:15672
rabbitmqctl add_user admin 123456 # add user name and password
rabbitmqctl set_user_tags admin administrator # add user role
rabbitmqctl set_permissions -p "/" admin ".*" ".*" ".*" #Modify permissions
principle
component:
Broker: Message Queuing service process, which consists of two parts: Exchange and Queue
Exchange: message queue switch, which routes and forwards messages to a queue according to certain rules and filters messages.
Queue: message queue, the queue that stores messages. Messages arrive at the queue and are forwarded to the specified
Producer: the message producer, that is, the producer client, which sends the message
Consumer: the message consumer, that is, the consumer client, receives messages forwarded by MQ.
AMQP protocol
Message sending process of producer:
1. The producer establishes a TCP connection with the Broker.
2. Establish channels between producers and brokers.
3. The producer sends the message to the Broker through the channel, and the Exchange forwards the message.
4. Exchange forwards messages to the specified Queue according to the mode
Consumer message receiving process:
1. The consumer establishes a TCP connection with the Broker
2. Establishing channels between consumers and brokers
3. The consumer listens to the specified Queue
4. When a message arrives at the Queue, the Broker pushes the message to the consumer by default.
5. Consumer received message.
6. ack reply (confirmation is a reliable message)
Simple mode
(1) The producer sends the message to the queue, and the consumer gets the message from the queue.
(2) A queue corresponds to a consumer.
Simple and working modes have default switches that push producer messages into queues
work
(1) One producer, multiple consumers.
(2) When a message is sent to the queue, it can only be obtained by one consumer.
(3) Multiple consumers process messages in parallel to improve message processing speed.
Publish / subscribe mode
Routing mode with empty routing name
Switch type: fanout
Send the message to the switch, and the queue obtains the message from the switch. The queue needs to be bound to the switch.
(1) One producer, multiple consumers.
(2) Each consumer has its own queue.
(3) Instead of sending the message directly to the queue, the producer sends it to the switch.
(4) Each queue must be bound to the switch.
(5) The message sent by the producer reaches one destination through multiple queues.
(6) The switch type is "fanout".
example
application.yml
server:
port: 8081 #Port number
spring:
rabbitmq:
host: 121.40.242.138
port: 5672
virtual-host: /
username: admin
password: 123456
Configuration class (create switch, queue, binding relationship)
@Configuration
public class RabbitMqConfiguration {
@Bean
public FanoutExchange fanoutExchange(){
return new FanoutExchange("fanout_order_exchange",true,false);
}
@Bean
public Queue smsQueue()
{
return new Queue("sms.fanout.queue",true);
}
@Bean
public Queue duanxinQueue()
{
return new Queue("duanxin.fanout.queue",true);
}
@Bean
public Queue emailQueue()
{
return new Queue("email.fanout.queue",true);
}
@Bean
public Binding smsbinding()
{
return BindingBuilder.bind(smsQueue()).to(fanoutExchange());
}
@Bean
public Binding duanxinbinding()
{
return BindingBuilder.bind(duanxinQueue()).to(fanoutExchange());
}
@Bean
public Binding emailbinding()
{
return BindingBuilder.bind(emailQueue()).to(fanoutExchange());
}
}
Producer (send data through switch)
@Service
public class OrderService {
@Autowired
private RabbitTemplate rabbitTemplate;
public void makeOrder(String userId,String productId,int num)
{
//1. Query the inventory of goods
//2. Save order
//3. Distribute through message queue
//Parameter 1: switch parameter 2: route name / queue name parameter 3: content name
String orderId="001";
String exchangeName="fanout_order_exchange";
String routeKey="";
rabbitTemplate.convertAndSend(exchangeName,routeKey,orderId );
}
}
Consumer (listening queue)
@Service
@RabbitListener(queues = {"sms.fanout.queue"})
public class ListenerService {
@RabbitHandler
public void getMessage(String message) {
System.out.println("Receiver Consumer receives message : " + message);
}
}
Test (controller)
@RestController
@RequestMapping("mq/fanout")
public class OrderController {
@Autowired
OrderService orderService;
@RequestMapping("order")
public void contextLoads()
{
orderService.makeOrder("1","1",12);
}
}
main program
@SpringBootApplication(exclude= {DataSourceAutoConfiguration.class})
public class FanoutApplication
{
public static void main(String[] args) {
SpringApplication.run(FanoutApplication.class, args);
}
}
Routing mode
Switch type: direct
Routing mode is a special case of publish / subscribe mode.
(1) The switch type of routing mode is "direct".
(2) When binding the queue to the switch, specify the key, that is, the routing key. A queue can specify multiple routing keys.
(3) When the producer sends a message, it specifies the routing key. At this time, the message will only be sent to the corresponding queue of the bound key
When the switch is created, the type is changed to routing mode
Set the route with() when the switch is bound to a queue
@Bean
public Binding emailbinding()
{
return BindingBuilder.bind(emailQueue()).to(directExchange()).with("routingKey");
}
The switch in the producer distributes messages, and the route name is filled in the second parameter in convertAndSend
rabbitTemplate.convertAndSend(exchangeName,routeKey,orderId );
Topic mode
Match the routing key to a pattern. At this point, the queue needs to be bound to a pattern.
The symbol "#" matches one or more words, and "*" matches no more than one word.
Wildcard pattern matching is performed when binding the queue to the specified key of the switch.
Expiration time TTL
ttl queue
@Bean
public Queue smsQueue()
{
Map<String,Object>args=new HashMap<>();
args.put("x-message-ttl",5000);
return new Queue("sms.fanout.queue",true,false,false,args);
}
Set the expiration time when creating the queue. If it is not received by the consumer within the expiration time, it will disappear automatically
Set expiration time for messages
Change at producer
MessagePostProcessor messagePostProcessor=new MessagePostProcessor() {
@Override
public Message postProcessMessage(Message message) throws AmqpException {
message.getMessageProperties().setExpiration("5000");
message.getMessageProperties().setContentEncoding("UTF-8");
return message;
}
};
rabbitTemplate.convertAndSend(exchangeName,routeKey,orderId,messagePostProcessor);
Dead letter queue
Expired messages, rejected messages and messages exceeding the length can be transferred to the dead letter queue
@Bean
public FanoutExchange deadExchange(){
return new FanoutExchange("dead_exchange",true,false);}
@Bean
public Queue deadQueue()
{
return new Queue("dead_queue",true);
}
@Bean
public Binding deadBinding()
{
return BindingBuilder.bind(deadQueue()).to(deadExchange());
}
Add args to ttl queue creation put(“x-dead-letter-exchange”,“dead_exchange”);
Expired messages are distributed to the dead letter switch
@Bean
public Queue smsQueue()
{
Map<String,Object>args=new HashMap<>();
args.put("x-message-ttl",5000);
args.put("x-dead-letter-exchange","dead_exchange");
//args.put("x-dead-letter-routing-key","key1"); When dead letter switch is in routing mode
return new Queue("sms.fanout.queue",true,false,false,args);
}
The queue reaches the maximum length and is forwarded to the dead letter queue
@Bean
public Queue smsQueue()
{
Map<String,Object>args=new HashMap<>();
//args.put("x-message-ttl",5000);
args.put("x-max-length",6);//Set the maximum length. If the length exceeds, it will be transferred to the dead letter queue
args.put("x-dead-letter-exchange","dead_exchange");
//args.put("x-dead-letter-routing-key","key1"); When dead letter switch is in routing mode
return new Queue("sms.fanout.queue",true,false,false,args);
}
Delay queue
ttl queue + dead letter queue
ttl expiration completes the function of delay, and the dead letter queue receives expired information. Achieve the effect of delay