According to personal analysis and understanding, does the project need MQ technology
Order item - > order specification - > fulfillment set - > fulfillment process - > OPU
The above is the order framework process structure abstracted by the company according to the order business.
The OPU here is the smallest execution unit, which can be synchronous or asynchronous.
Since the project does not require second kill, MQ is used for cross system asynchronous communication.
Two technologies have been used for asynchrony here. At first, it is multithreading (thread pool) + scheduled task (Quartz), and later it is changed to RabbitMQ.
Due to the confidentiality regulations, pseudo code is used here to show how to realize asynchronous communication with peripheral systems without MQ:
There are several points to pay attention to first.
- When using a thread pool, pay attention to the number of thread pools. Apply the formula U * U * (1 + w/c): number of CPU cores * expected CPU utilization * (1 + waiting time of a single thread / calculation time of a single thread)
- The cycle period of scheduled tasks needs to be determined. Once it was set to 1 minute in the project, then it was changed to 5 minutes according to the daily order quantity and the asynchronous communication quantity in the order
- The retry mechanism for message sending failure should be well thought out. At the beginning of the project, in order to support this business, two important tables are mainly prepared: (1) quartz for the fields such as order number, stored message, interface information and retry times_ info_ Request table (2) records the log information of all internal and external calls or external calls_ info_ Log, which contains fields such as order number and interface information
/**
* @Description:Thread pool configuration
* @author: Old street layman
*/
@Configuration
public class ThreadPoolConfig {
@Bean
public ThreadPoolExecutor threadPoolExecutor{
//Number of core threads
int corePoolSize = 50;
//Maximum number of threads
int maximumPoolSize = 100;
//The maximum idle time of threads exceeds the number of corePoolSize threads
long keepAliveTime =2;
//In seconds
TimeUnit unit = TimeUnit.SECONDS;
//Used to store submitted pending tasks
BlokckingQueue<Runnable> workQueue = new ArrayBlockingQueue<Runnable>(40);
ThreadPoolExecutor threadPoolExecutor = new ThreadPoolExecutor(
corePoolSize,maximumPoolSize,keepAliveTime,unit,workQueue,
//The default policy is to throw RejectedExecutionException when rejecting a task
new ThreadPoolExecutor.AbortPolicy());
);
return threadPoolExecutor;
}
}
/**
* @Description:Asynchronous request
* @author: Old street layman
*/
public class SendService{
@Autowired
ThreadPoolExecutor threadPoolExecutor;
public void asyncSend(){
threadPoolExecutor.submit(()->{
//1. Call the corresponding interface and send the request
//2. Record the corresponding information in quartz_ info_ In the request table, the number of retries is set to 0 and the success is set to false
})
}
}
/**
* @Description:Responsible for scanning quartz in the scheduled task_ info_ Request and quartz_info_log table and compare which interfaces need to be retried
* @author: Old street layman
*/
public class Sendjob{
public void retry(){
//Scan quartz_info_request and quartz_info_log table
//Match according to order number
//quartz_ info_ If the log does not match the information returned from the third-party response interface and has timed out, retry the interface;
//If the response interface has reported an error, the quartz_ info_ Set the retry times of the data corresponding to the request as the maximum times, and enter the current order information into the exception table for reconciliation and manual document revision
//......
//In fact, there are still many steps to consider. Here are several main steps for simplicity and clarity
}
}
In this way, thread pool + scheduled tasks can also achieve the results of asynchronous communication. As for the reconciliation and repair of orders, it is all based on the consideration of designing the message sending failure mechanism.
Obviously not so easy to control
Next, let's look at the impact of using RabbitMQ.
The first is the code:
/**
* @Description:MQ Configuration, direct connection switch is used here
* @author: Old street layman
*/
@Configuration
public class RabbitConfig {
//The name of the queue is DirectQueue
@Bean
public Queue DirectQueue(){
//durable sets whether to persist. The default value is false. true will be stored on the hard disk
return new Queue("DirectQueue",true);
}
//The name of the switch is MyDirectExchange
@Bean
public DirectExchange MyDirectExchange(){
//durable sets whether to persist. The default value is false. true will be stored on the hard disk
return new DirectExchange("MyDirectExchange",true,false);
}
//Bind, bind queue to switch
@Bean
public Binding bindingDirect(){
//durable sets whether to persist. The default value is false. true will be stored on the hard disk
return BindingBuilder.bind(DirectQueue()).to(MyDirectExchange()).with("MyDirectRouting");
}
/**
* @Description:Configuration of message confirmation mechanism
* @author: Old street layman
*/
@Bean
public RabbitTemplate createRabbitTemplate(ConnectionFactory connectionFactory){
RabbitTemplate rabbitTemplate = new RabbitTemplate();
rabbitTemplate.setConnectionFactory(connectionFactory);
//Open the Mandatory and trigger the callback function
rabbitTemplate.setMandatory(true);
//This is a measure for the message delivery failure of producer - > broker (host)
rabbitTemplate.setConfirmCallback(new RabbitTemplate.ConfirmCallback(){
@Override
public void confirm(CorrelationData correlationData, boolean ack, String cause) {
System.out.println("ConfirmCallback: "+"correlationData: "+correlationData);
System.out.println("ConfirmCallback: "+"Confirmation:"+correlationData);
System.out.println("ConfirmCallback: "+"cause: "+correlationData);
//Record the correlationData in the exception information table to facilitate the repair of orders (the callback function in the confirmation mode may generally be a network problem or the disk is full. These situations rarely occur. With the log monitoring system, the operation and maintenance personnel can repair them immediately)
}
});
//This is for the measures of switch - > queue message delivery failure
rabbitTemplate.setReturnCallback(new RabbitTemplate.ReturnCallback(){
@Override
public void returnedMessage(Message message, int replyCode, String replyText, String exchange, String routingKey){
System.out.println("ReturnCallback: "+"Message:"+message);
System.out.println("ReturnCallback: "+"Response code:"+replyCode);
System.out.println("ReturnCallback: "+"Response information:"+replyText);
System.out.println("ReturnCallback: "+"Switch:"+exchange);
System.out.println("ReturnCallback: "+"Routing key:"+routingKey);
//Record the message in the exception information table to facilitate order repair (in this case, the callback function may be routingkey error or the queue cannot be found. These situations rarely occur, and the production environment is basically impossible)
}
});
return rabbitTemplate;
}
}
Next is the message confirmation mechanism code manually confirmed by the consumer
@Configuration
public class RabbitConfig2 {
@Autowired
private CachingConnectionFactory connectionFactory;
@Autowired
private AckReceiver ackReceiver;//Message receiving and processing class
@Bean
public SimpleMessageListenerContainer simpleMessageListenerContainer(){
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer(connectionFactory);
container.setConcurrentConsumers(1);
container.setMaxConcurrentConsumers(1);
container.setAcknowledgeMode(AcknowledgeMode.MANUAL); // RabbitMQ defaults to automatic confirmation, which is changed to manual confirmation
//Set up a queue
container.setQueueNames("DirectQueue");
container.setMessageListener(ackReceiver);
return container;
}
}
Consumer code
/**
* @Description:The consumer sends the request according to the Message. If there is an abnormal failure, they can choose to resend or give up according to the specific situation
* @author: Old street layman
*/
@Component
public class AckReceiver implements ChannelAwareMessageListener {
@Override
public void onMessage(Message message, Channel channel) throws Exception {
long deliveryTag = message.getMessageProperties().getDeliveryTag();
try {
//msg is the message delivered
String msg = message.toString();
//Get the corresponding information from msg, call the corresponding interface and send the request
//Multiple messages can be sent with the for loop
channel.basicAck(deliveryTag, true); //When the parameter is true, the delivery can be confirmed at one time_ All messages with tag less than or equal to the incoming value
} catch (Exception e) {
//Exception s can be divided in more detail according to different exceptions
//You can choose to put it back in the queue, such as timeout or network problems
//If it fails and does not resend, it shall be recorded in the abnormal order table to facilitate reconciliation and document revision
//channel.basicReject(deliveryTag, true);// When the parameter is true, the message will be put back to the queue, so you need to judge when to use rejection according to the business logic
channel.basicReject(deliveryTag, false);
e.printStackTrace();
}
}
}
Producer code
/**
* @Description:Asynchronous request
* @author: Old street layman
*/
public class SendService{
@Autowired
RabbitTemplate rabbitTemplate;
public String asyncSend(){
Map<String,Object> map=new HashMap<>();
//Add multiple key value pairs, store relevant information and send it to the broker
//For example, order number, request message, interface name, etc
rabbitTemplate.convertAndSend("TestDirectExchange", "TestDirectRouting", map);
return "ok";
}
}
To sum up:
After using MQ, it is obvious that the message confirmation mechanism is more perfect, which is equivalent to using MQ to replace the Database-based scheduled task, which is more efficient. Needless to say, it is easier for developers to troubleshoot in case of message exception or loss.
Looking only at the asynchronous communication function in the distributed environment, the introduction of MQ is still necessary.
Based on personal understanding of the company's project, if you have a better view, please communicate~