I haven't shared my learning experience for a long time. After reading the release records, the last article is from 2020-12-10. Today, I take time to sort out the next technology sharing I want to sort out very early. By the way, as an aside, because I haven't had time to sort it out, and the development partners don't understand Mq enough, I opened several development partners to deal with this matter, so I hope this article can bring you some help.
Background description
Mq (message queue) is often used as a peak elimination tool. Our commonly used Mq is mainly divided into the following four types:
- ActiveMQ
- RabbitMq
- Kafka
- RocketMq
Today, I'm mainly talking about RabbitMq. There are many reasons for choosing RabbitMq in business scenarios. I won't talk about it in detail today. Today is mainly about how to dynamically create queues and realize dynamic monitoring.
Demand background
As a CRM-SAAS platform, a large amount of customer information will be entered into the platform every day, so we need to send these data to sales in an efficient way. Here are the following issues to consider:
- Timeliness of data distribution
- Data grouping
- The receiver belongs to different groups and different levels
- The data does not meet the distribution conditions (for example, the receiver is busy and may need to be retransmitted for a period of time)
Technical scheme
- In order to ensure the timeliness of data, the consumption queue is promoted in time after the data enters the system
- For different packets and different levels of data packets and receivers, and if they need to be controlled manually, set different queues for listening and consumption. We can also make the queue name meaningful and obtain some necessary information we need from the queue name, such as which packet the data belongs to and the group to which the data should be distributed.
Based on the above considerations, we choose RabbitMq to implement this scheme. Since different queues consume different data, the first step is to consider how to dynamically create queues, because there is also a person to be controllable, that is, personnel can be managed, so it is accompanied by the deletion and reconstruction of queues.
How queues are created
Annotation based usage
@Bean
public Queue syncCdrQueue(){
return new Queue(CrmMqConstant.SYNC_CDR_TO_CRM_Q,true,false,false);
}Non annotation configuration
Channel channelForm = connectionFactory().createConnection().createChannel(false); channelForm.queueDeclare(nameForm, true, false, false, null);
RabbitAdmin based
rabbitAdmin.declareQueue(queue); rabbitAdmin.declareExchange(fanoutExchange); rabbitAdmin.declareBinding(BindingBuilder.bind(queue).to(fanoutExchange));
In terms of the flexibility of creating queues, it must be weakened in turn:
- Annotation method: the queue name is defined in advance, which is generally defined by constants. Of course, the variable method is also supported, but the requirements for loading sequence are high. For example, here we use a dynamic IP as the queue name
private final String QUEUE_NAME="crm.websocket."+ IPUtils.getLocalhostIp(); @Bean public Queue queue(){ return new Queue(QUEUE_NAME,true,false,false); } //monitor @RabbitListener(queues = "#{queue.name}") - Non annotation method: this method is actually used to obtain channels and create queues through ConnectionFactory. This is more suitable for establishing links, so it is generally more suitable for batch initialization of queues
@Bean
public List<String> mqMsgQueues() throws IOException {
List<String> queueNames = new ArrayList<String>();
List<Map<String,Object>> engineList = autoAssignEngineService.getAllAutoAssignEngine(-1,-1);
logger.info("engineList:{}", JsonUtils.toJson(engineList));
if(engineList != null && engineList.size() > 0) {
for(Map<String,Object> engine : engineList) {
String groupId = String.valueOf(engine.get("orgId"));
String semAdType = String.valueOf(engine.get("semAdType"));
logger.info("groupId:{},semAdType:{}", groupId,semAdType);
createQueue(queueNames, groupId,semAdType,"1");
createQueue(queueNames, groupId,semAdType,"2");
}
}
return queueNames;
}
private void createQueue(List<String> queueNames, String groupId, String semType, String level) throws IOException {
String nameForm = queue +"."+ groupId+"."+semType + "." + level;
logger.info("nameForm:{}",nameForm);
Channel channelForm = connectionFactory().createConnection().createChannel(false);
channelForm.queueDeclare(nameForm, true, false, false, null);
channelForm.exchangeDeclare(topicExchange, BuiltinExchangeType.TOPIC,true);
channelForm.queueBind(nameForm,topicExchange,routingKey + "."+groupId+"."+semType+"."+level);
queueNames.add(nameForm);
}- RabbitAdmin based method: this method is relatively flexible and supports the creation of queues at any time. Then simply encapsulate:
public void createMqQueue(String queueName,String exName,String rk,String type){ Properties properties = rabbitAdmin.getQueueProperties(queueName); if(properties==null) { Queue queue = new Queue(queueName, true, false, false, null); if(BuiltinExchangeType.DIRECT.getType().equals(type)) { DirectExchange directExchange = new DirectExchange(exName); rabbitAdmin.declareQueue(queue); rabbitAdmin.declareExchange(directExchange); rabbitAdmin.declareBinding(BindingBuilder.bind(queue).to(directExchange).with(rk)); }else if(BuiltinExchangeType.FANOUT.getType().equals(type)){ FanoutExchange fanoutExchange = new FanoutExchange(exName); rabbitAdmin.declareQueue(queue); rabbitAdmin.declareExchange(fanoutExchange); rabbitAdmin.declareBinding(BindingBuilder.bind(queue).to(fanoutExchange)); }else{ TopicExchange topicExchange = new TopicExchange(exName); rabbitAdmin.declareQueue(queue); rabbitAdmin.declareExchange(topicExchange); rabbitAdmin.declareBinding(BindingBuilder.bind(queue).to(topicExchange).with(rk)); } } }After we know how to dynamically create a queue, let's find a way to solve the problem of dynamic consumption monitoring:
Dynamic consumption monitoring
RabbitMq's abstract listening class is AbstractMessageListenerContainer. There are three implementation classes below. Here we use the SimpleMessageListenerContainer class for a simple description.
Method 1 (I'm a senior):
Initialize the queue, store it in the static cache, and use different bean s to load listening:
private List<Map<String,String>> groupOrgIds = new ArrayList<Map<String,String>>()
@PostConstruct
public void init() {
if (logger.isDebugEnabled()) {
logger.debug("initbean...");
}
List<AutoAssignEngine> engineList = autoAssignEngineService.getAllAutoAssignEngine();
if (engineList != null && engineList.size() > 0) {
for(AutoAssignEngine engine : engineList) {
createQueueList(engine.getOrgId(),engine.getSemAdType(),"1");
createQueueList(engine.getOrgId(),engine.getSemAdType(),"2");
}
}
}
private void createQueueList(String orgId,String semType,String userLevel) {
Map<String,String> feed = new HashMap<String, String>();
feed.put("orgId", orgId);
feed.put("type", semType);
feed.put("userLevel", userLevel);
groupOrgIds.add(feed);
}
public SimpleMessageListenerContainer setContainer() {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(rabbitConfig.connectionFactory());
container.setAcknowledgeMode(AcknowledgeMode.AUTO);
container.setMaxConcurrentConsumers(8);
container.setConcurrentConsumers(5);
container.setPrefetchCount(10);
return container;
}
public SimpleMessageListenerContainer queueMethod(SimpleMessageListenerContainer container) {
Map<String,String> orgIdMap = groupOrgIds.get(0);
String orgId = orgIdMap.get("orgId");
String sourceType = orgIdMap.get("type");
String userLevel = orgIdMap.get("userLevel");
String queueNames=queueName + "." + orgId+"."+sourceType+"."+userLevel;
container.addQueueNames(queueNames);
excute(orgId,sourceType, container,queueNames);
groupOrgIds.remove(0);
return container;
}
public SimpleMessageListenerContainer excute(String orgId,String semAdType, SimpleMessageListenerContainer container,String queneName) {
container.setMessageListener(new ChannelAwareMessageListener() {
@Override
public void onMessage(Message message, Channel channel) throws Exception {
}
});
return container;
}
/**
* Create multiple queue listeners and consume groupOrgIds by using the initialization sequence of beans
*/
@Bean
public SimpleMessageListenerContainer container1() {
SimpleMessageListenerContainer container = setContainer();
queueMethod(container);
return container;
}
@Bean
public SimpleMessageListenerContainer container2() {
SimpleMessageListenerContainer container = setContainer();
queueMethod(container);
return container;
}
.....Well, this method can indeed dynamically monitor different queues and consumption, but because it uses the initialization method of Bean, the Bean must be reloaded every time the queue content to be loaded is changed, that is, the service needs to be restarted.
Mode 2: real dynamic monitoring
@Bean
public SimpleMessageListenerContainer simpleMessageListenerContainer() {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(rabbitConfig.connectionFactory());
container.setAcknowledgeMode(AcknowledgeMode.AUTO);
container.setMaxConcurrentConsumers(8);
container.setConcurrentConsumers(5);
container.setPrefetchCount(10);
// Query how many allocation engines there are and one queue for each allocation engine
List<AutoAssignEngine> engineList = autoAssignEngineService.getAllAutoAssignEngine();
if (engineList != null && engineList.size() > 0) {
for(AutoAssignEngine engine : engineList) {
mqService.addNewListener(engine.getOrgId(),engine.getSemAdType(),"1",container);
mqService.addNewListener(engine.getOrgId(),engine.getSemAdType(),"2",container);
}
}
return container;
}
public Boolean addNewListener(String orgId,String semType,String userLevel,SimpleMessageListenerContainer container ){
String queueNames=queueName + "." + orgId+"."+semType+"."+userLevel;
container.addQueueNames(queueNames);
container.setMessageListener(new ChannelAwareMessageListener() {
@Override
public void onMessage(Message message, Channel channel) throws Exception {
}
});
return true;
}Problem 1: when receiving messages (within the onMessage method) here, do not use methods to pass parameters, which will lead to concurrency problems.
Solution 1:
String receiveQueueName = message.getMessageProperties().getConsumerQueue();
The queue name is obtained by parsing. I can use it.
Solution 2:
Use the final variable to receive the parameters again, but this needs to be tested and may not be used again.
Question 2: This is still loaded when the Bean is initialized. What if you want to add listening after the service is started
Complete dynamic queue creation and listening (business process implementation)
After we know how to create queues and listen, we begin to solve problem 2.
Requirement: change the existing queue.
The conversion requirements are: delete the existing queue and monitor, create a new queue and add monitor
Problem: push and consumption are no longer unified services.
Solution: expose the interface and use http request to realize synchronization.
Code implementation:
Consumer side
public Boolean updateListener(String orgId,String semType,String oldOrg){
logger.info("================================Consumer starts processing");
String newFirstQueueName = queueName+"."+orgId+"."+semType+"."+1;
String newFirstRk = routingKey+"."+orgId+"."+semType+"."+1;
String newSecondQueueName = queueName+"."+orgId+"."+semType+"."+2;
String newSecondRk = routingKey+"."+orgId+"."+semType+"."+2;
createMqQueue(newFirstQueueName,topicExchange,newFirstRk, BuiltinExchangeType.TOPIC.getType());
createMqQueue(newSecondQueueName,topicExchange,newSecondRk, BuiltinExchangeType.TOPIC.getType());
logger.info("================================Create queue");
SimpleMessageListenerContainer container = SpringCtxUtils.getBean(SimpleMessageListenerContainer.class);
String oneQueueNames=queueName + "." + orgId+"."+semType+"."+1;
String twoQueueNames=queueName + "." + orgId+"."+semType+"."+2;
if(!"NO".equals(oldOrg)) {
String oneOldQueueNames = queueName + "." + oldOrg + "." + semType + "." + 1;
String twoOldQueueNames = queueName + "." + oldOrg + "." + semType + "." + 2;
container.removeQueueNames(oneOldQueueNames);
container.removeQueueNames(twoOldQueueNames);
logger.info("================================Successfully deleted listening");
}
container.addQueueNames(oneQueueNames);
container.addQueueNames(twoQueueNames);
logger.info("================================Listening added successfully");
container.setMessageListener(new ChannelAwareMessageListener() {
@Override
public void onMessage(Message message, Channel channel) throws Exception {
}
});
return true;
}
public void createMqQueue(String queueName,String exName,String rk,String type){
Properties properties = rabbitAdmin.getQueueProperties(queueName);
if(properties==null) {
Queue queue = new Queue(queueName, true, false, false, null);
if(BuiltinExchangeType.DIRECT.getType().equals(type)) {
DirectExchange directExchange = new DirectExchange(exName);
rabbitAdmin.declareQueue(queue);
rabbitAdmin.declareExchange(directExchange);
rabbitAdmin.declareBinding(BindingBuilder.bind(queue).to(directExchange).with(rk));
}else if(BuiltinExchangeType.FANOUT.getType().equals(type)){
FanoutExchange fanoutExchange = new FanoutExchange(exName);
rabbitAdmin.declareQueue(queue);
rabbitAdmin.declareExchange(fanoutExchange);
rabbitAdmin.declareBinding(BindingBuilder.bind(queue).to(fanoutExchange));
}else{
TopicExchange topicExchange = new TopicExchange(exName);
rabbitAdmin.declareQueue(queue);
rabbitAdmin.declareExchange(topicExchange);
rabbitAdmin.declareBinding(BindingBuilder.bind(queue).to(topicExchange).with(rk));
}
}
}
Exposed interface:
@GetMapping("/add-listener/{orgId}/{semType}/{oldOrg}")
public ComResponse addListener(@PathVariable("orgId") String orgId,@PathVariable("semType") String semType,@PathVariable("oldOrg") String oldOrg){
mqService.updateListener(orgId,semType,oldOrg);
return ComResponse.successResponse();
}Note: execute the sequence, create a new queue, delete listening and add listening
Push end
//Add new listener
String requestUrl = consumerUrl+"/"+newOrg+"/"+semType+"/"+oldOrgId;
String result = restTemplateService.getWithNoParams(requestUrl,String.class);
log.info("End of request:{}",result);
if(!"NO".equals(oldOrgId)) {
String firstQueueName = queue + "." + oldOrgId + "." + semType + "." + 1;
String secondQueueName = queue + "." + oldOrgId + "." + semType + "." + 2;
mqService.deleteMqQueue(firstQueueName);
mqService.deleteMqQueue(secondQueueName);
log.info("End of delete queue");
}
//Add a new queue
String newFirstQueueName = queue+"."+newOrg+"."+semType+"."+1;
String newFirstRk = routingKey+"."+newOrg+"."+semType+"."+1;
String newSecondQueueName = queue+"."+newOrg+"."+semType+"."+2;
String newSecondRk = routingKey+"."+newOrg+"."+semType+"."+2;
mqService.createMqQueue(newFirstQueueName,topicExchange,newFirstRk, BuiltinExchangeType.TOPIC.getType());
mqService.createMqQueue(newSecondQueueName,topicExchange,newSecondRk, BuiltinExchangeType.TOPIC.getType());
log.info("Add queue end");Note: execution sequence: change listening, delete queue and add new queue
Here, the dynamic queue creation and dynamic listening are basically realized. If you have something you don't quite understand, you can leave a message and take time to sort it out, so it's written in a careless way. Let's make do with it.