Premise background
When using RocketMQ to send messages, we generally must specify topic. In addition, topic must be established in advance. However, the setting of topic creation (automatic or manual) has a switch autoCreateTopicEnable. This part will be set mainly in the configuration file of the broker node. The default setting autoCreateTopicEnable = true will be used in the running environment, However, this will make the setting of topic difficult to regulate and manage. Therefore, in the production environment, the parameter autoCreateTopicEnable = false will be set in the broker. If there is a slight deviation in this parameter or the topic is not created manually in advance, the error No route info of this topic will occur frequently. Next, let's explore the causes of this problem and how the system creates the topic.
No route info of this topic
It is believed that the partners who have worked in the RocketMQ project may be familiar with No route info of this topic at all. The meaning of the explanation is that the topic cannot be parsed or routed at first, but there are many reasons.
NameServer service is not configured
When the Broker is started, we do not configure the NameSrv address, and the sender will report an error: No route info of this topic. But when we add the NameSrv address, we can start again and send messages normally.
autoCreateTopicEnable=true was not created and the topic was not created
When autoCreateTopicEnable=false, defaultmqproducerimpl Senddefaultimpl: when sending a message, you must first obtain some information about the topic. For example, there are several message queues, which are ordered topics from time to time, and there is a Broker list of the topic. When you can't obtain the correct information, an exception will be thrown
The client version of RocketMQ is inconsistent with the server version
RocketMQ Java client calls No route info of this topic error (the reason is that the version is inconsistent). At this time, it is useless to set autoCreateTopicEnable=true even when starting the broker. Suppose that the version of rocketmq used is 4.9.0 and the version of java client is 4.3.0
In rocketmq version 4.3.0 auto create (autoCreateTopicEnable), the client passes the auto used_ CREATE_ TOPIC_ KEY_ Topic is "auto"_ CREATE_ TOPIC_ Key ", the new version of client, and the default auto passed by the client_ CREATE_ TOPIC_ KEY_ Topic is "TBW102".
org.apache.rocketmq.client.producer.DefaultMQProducer#createTopicKey org.apache.rocketmq.common.MixAll#AUTO_CREATE_TOPIC_KEY_TOPIC public static final String AUTO_CREATE_TOPIC_KEY_TOPIC = "TBW102";
Actual code
>Version 4.4.0
< = version 4.3.0
Scheme 1: otherwise, adjust the version of the client version
<dependency>
<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-client</artifactId>
<version>4.5.1</version>
</dependency>
Scheme 2: adjust the automatic creation code to AUTO_CREATE_TOPIC_KEY
DefaultMQProducer producer = new DefaultMQProducer("please_rename_unidcque_group_name");
//Set the key value for automatically creating topic
producer.setCreateTopicKey("AUTO_CREATE_TOPIC_KEY");
The Topic has not been created before. The Broker does not configure the NameSrv address and cannot send it. After configuring the NameSrv, it can be sent normally. There are two questions:
1. How is topic automatically created?
2. How do the Broker and NameSrv cooperate during the automatic creation of topic?
Analyze the following flow of how to automatically create the source code of topic
RocketMQ basic routing rules
-
When starting, the Broker registers the routing information stored on the Nameserver, sends heartbeat packets to the Nameserver every 30s, and updates the routing information.
Nameserver scans the routing table every 10s. If it detects that the Broker service is down, it removes the corresponding routing information. -
Every 30s, the message producer will pull the routing information of Topic again from the Nameserver and update the local routing table; Before sending the message, if there is no routing message of the corresponding Topic in the local routing table, it will actively pull the message of the Topic from the Nameserver.
-
If autoCreateTopicEnable is set to true, the message sender will try to use a system default topic name (MixAll.AUTO_CREATE_TOPIC_KEY_TOPIC) when the route message of the query topic from the NameServer returns null. At this time, the route information obtained by the message sender is:
How is the routing information of the default Topic created?
Nameserver？ broker？ When autoCreateTopicEnable=false, defaultmqproducerimpl Senddefaultimpl: when sending a message, you must first obtain some information about the topic. For example, there are several message queues, which are ordered topics from time to time, and there is a broker list of the topic. When you can't obtain the correct information, an exception will be thrown
private SendResult sendDefaultImpl(
Message msg,
final CommunicationMode communicationMode,
final SendCallback sendCallback,
final long timeout
) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
this.makeSureStateOK();
Validators.checkMessage(msg, this.defaultMQProducer);
final long invokeID = random.nextLong();
long beginTimestampFirst = System.currentTimeMillis();
long beginTimestampPrev = beginTimestampFirst;
long endTimestamp = beginTimestampFirst;
// If you get the routing information of topic, send it. Otherwise, throw an exception
TopicPublishInfo topicPublishInfo = this.tryToFindTopicPublishInfo(msg.getTopic());
if (topicPublishInfo != null && topicPublishInfo.ok()) {
... ...
}
List<String> nsList = this.getmQClientFactory().getMQClientAPIImpl().getNameServerAddressList();
if (null == nsList || nsList.isEmpty()) {
throw new MQClientException(
"No name server address, please set it." + FAQUrl.suggestTodo(FAQUrl.NAME_SERVER_ADDR_NOT_EXIST_URL), null).setResponseCode(ClientErrorCode.NO_NAME_SERVER_EXCEPTION);
}
throw new MQClientException("No route info of this topic, " + msg.getTopic() + FAQUrl.suggestTodo(FAQUrl.NO_TOPIC_ROUTE_INFO),
null).setResponseCode(ClientErrorCode.NOT_FOUND_TOPIC_EXCEPTION);
}
tryToFindTopicPublishInfo is the key to sending. If the topic information is obtained, it will be sent, otherwise it will be abnormal; Therefore, the previous exception of No route info of this topic is that the Producer cannot obtain the topic information, resulting in the sending failure.
First get from the topicPublishInfoTable cache
private TopicPublishInfo tryToFindTopicPublishInfo(final String topic) {
// topicPublishInfoTable is the topic information table cached locally by Producer
// After the Producer starts, the default topic: TBW102 will be added
TopicPublishInfo topicPublishInfo = this.topicPublishInfoTable.get(topic);
if (null == topicPublishInfo || !topicPublishInfo.ok()) {
this.topicPublishInfoTable.putIfAbsent(topic, new TopicPublishInfo());
// Not obtained. Get the topic information from NameSrv
this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic);
topicPublishInfo = this.topicPublishInfoTable.get(topic);
}
// If obtained, return
if (topicPublishInfo.isHaveTopicRouterInfo() || topicPublishInfo.ok()) {
return topicPublishInfo;
} else {
// If it is not obtained, it can be obtained from NameSrv in another way
// If you can't get it again, you can't send it later
this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic, true, this.defaultMQProducer);
topicPublishInfo = this.topicPublishInfoTable.get(topic);
return topicPublishInfo;
}
}
-
There is no topic information in the Producer's local topicPublishInfoTable variable, only TBW102 is cached.
-
Try to get information about topic from NameSrv. Failed to obtain. There is no topic in NameSrv at all, because this topic is set when the Producer sends it and is not synchronized to NameSrv.
-
Another way is to obtain it from NameSrv. If it is obtained, the sending process can be executed. If it is still not obtained, the exception of No route info of this topic will be thrown.
Then get it from the NameServer service
public boolean updateTopicRouteInfoFromNameServer(final String topic) {
return updateTopicRouteInfoFromNameServer(topic, false, null);
}
-
During the first acquisition, isDefault passes false and defaultMQProducer passes null. Therefore, in updateTopicRouteInfoFromNameServer, the else branch will be taken and Topic will be used to obtain
-
During the second acquisition, isDefault passes true, and defaultMQProducer also passes a value. Therefore, it will go through the if branch to convert the topic of the input parameter to the default TBW102 and obtain the information of TBW102
-
No matter whether the Broker is configured with NameSrv address or not, obtaining Topic information must fail
-
Get TBW102 information:
- 2.1 the broker configured NameSrv address successfully
- 2.2 Broker failed to configure NameSrv address
The producer first queries the NameServer for routing information. Because it is a non-existent topic, the returned routing information is empty. RocketMQ will use the default topic to look for it again. Since automatic creation of routing information is enabled, NameServer will return the routing information of the default topic to the producer.
Then select a queue from the returned routing information (default polling). After the message sender obtains the queue information of the default Topic from the Nameserver, will the number of queues change?
Get from NameServer. Note that isDefault=false and defaultMQProducer=null
Warm tip: when the message sender receives the default routing information, the number of queues will be defaultmqproducer #defaulttopicqueueuenums and the number of queues returned by Nameserver. The default value of defaultmqproducer #defaulttopicquenums is 4, so the number of queues for automatically created topics is 4 by default.
Get the topic information corresponding to the message
Send remotingcommand request = remotingcommand Createrequestcommand (requestcode. Get_routeinto_by_topic, requestheader), but since no Broker has information about this topic, namesrv will return that the topic does not exist, and the code for processing the request is in the default requestprocessor.
case RequestCode.GET_ROUTEINTO_BY_TOPIC: return this.getRouteInfoByTopic(ctx, request);
That is, the response code responsecode TOPIC_ NOT_ Exist, and then throw new mqclientexception (response. Getcode()), response getRemark()); Exit after being captured and return false.
Get relevant Topic information data from NameServer
updateTopicRouteInfoFromNameServer will eventually send a get to NameSrv_ ROUTEINTO_ BY_ Topic request
public boolean updateTopicRouteInfoFromNameServer(final String topic, boolean isDefault,
DefaultMQProducer defaultMQProducer) {
try {
if (this.lockNamesrv.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) {
try {
TopicRouteData topicRouteData;
if (isDefault && defaultMQProducer != null) {
topicRouteData = this.mQClientAPIImpl.getDefaultTopicRouteInfoFromNameServer(defaultMQProducer.getCreateTopicKey(),
1000 * 3);
if (topicRouteData != null) {
for (QueueData data : topicRouteData.getQueueDatas()) {
int queueNums = Math.min(defaultMQProducer.getDefaultTopicQueueNums(), data.getReadQueueNums());
data.setReadQueueNums(queueNums);
data.setWriteQueueNums(queueNums);
}
}
} else {
topicRouteData = this.mQClientAPIImpl.getTopicRouteInfoFromNameServer(topic, 1000 * 3);
}
} catch (Exception e) {
if (!topic.startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX) && !topic.equals(MixAll.DEFAULT_TOPIC)) {
log.warn("updateTopicRouteInfoFromNameServer Exception", e);
}
} finally {
this.lockNamesrv.unlock();
}
} else {
log.warn("updateTopicRouteInfoFromNameServer tryLock timeout {}ms", LOCK_TIMEOUT_MILLIS);
}
} catch (InterruptedException e) {
log.warn("updateTopicRouteInfoFromNameServer Exception", e);
}
return false;
}
Because the if condition is not satisfied, get the default topic information. Note that isDefault=true, defaultMQProducer=defaultMQProducer
if (topicPublishInfo.isHaveTopicRouterInfo() || topicPublishInfo.ok()) {
return topicPublishInfo;
} else {
this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic, true, this.defaultMQProducer);
topicPublishInfo = this.topicPublishInfoTable.get(topic);
return topicPublishInfo;
}
The default topic is "TBW102". At this time, if there is still no information about this topic in namesrv, an exception No route info of this topic will be thrown.
autoCreateTopicEnable=true.
The Broker starts the process and automatically creates a topic
-
In the Broker startup process, the TopicConfigManager object will be built. In its construction method, first judge whether automatic topic creation is enabled. If automatic topic creation is enabled, add the routing information of the default topic to the topicConfigTable.
-
When the Broker is started, the TopicConfigManager initializes. Here, it will judge the identity, create TBW102topic, and update the information to namesrv in the subsequent heartbeat, so that nonexistent exceptions will not be thrown when sending messages.
// MixAll.DEFAULT_TOPIC
if (this.brokerController.getBrokerConfig().isAutoCreateTopicEnable()) {
String topic = MixAll.DEFAULT_TOPIC;
TopicConfig topicConfig = new TopicConfig(topic);
this.systemTopicList.add(topic);
topicConfig.setReadQueueNums(this.brokerController.getBrokerConfig()
.getDefaultTopicQueueNums());
topicConfig.setWriteQueueNums(this.brokerController.getBrokerConfig()
.getDefaultTopicQueueNums());
int perm = PermName.PERM_INHERIT | PermName.PERM_READ | PermName.PERM_WRITE;
topicConfig.setPerm(perm);
this.topicConfigTable.put(topicConfig.getTopicName(), topicConfig);
}
All routing information in the topicConfigTable will be sent to the Nameserver along with the Broker. After receiving these information, the Nameserver will update the routing information table of the corresponding Topic.
The defaultTopicQueueNum of BrokerConfig defaults to 8. Both Broker servers will run the above process, so the routing information about the default topic in the final Nameserver will contain 8 queue information for each of the two brokers.
TopicConfigManager construction method
When finding out Topic related information from namesrv, set the number of message queues info. Info in topicRouteData2TopicPublishInfo getMessageQueueList(). add(mq);， Call the updateTopicPublishInfo method to update the cached topicPublishInfoTable
// Update Pub info
{
TopicPublishInfo publishInfo = topicRouteData2TopicPublishInfo(topic, topicRouteData);
publishInfo.setHaveTopicRouterInfo(true);
Iterator<Entry<String, MQProducerInner>> it = this.producerTable.entrySet().iterator();
while (it.hasNext()) {
Entry<String, MQProducerInner> entry = it.next();
MQProducerInner impl = entry.getValue();
if (impl != null) {
impl.updateTopicPublishInfo(topic, publishInfo);
}
}
}
Then if (topicpublishinfo! = null & & topicpublishinfo. Ok()) will meet the condition, and the exception will not be thrown.
When autoCreateTopicEnable=false
- Create the class UpdateTopicSubCommand() of topic, set the corresponding information, and finally call defaultMQAdminExt.. createAndUpdateTopicConfig(addr, topicConfig);
- Send a message requestcode UPDATE_ AND_ CREATE_ Topic, AdminBrokerProcessor processing message case requestcode UPDATE_ AND_ CREATE_ TOPIC: return this. updateAndCreateTopic(ctx, request);
- Synchronize to other brokers
this.brokerController.getTopicConfigManager().updateTopicConfig(topicConfig); this.brokerController.registerBrokerAll(false, true);
Processing flow after the Broker receives the message
The processor of the server receiving the message sending is SendMessageProcessor. When processing the message sending, it will call super Msgcheck method:
AbstractSendMessageProcessor#msgCheck
On the Broker side, firstly, the TopicConfigManager will be used to query the routing information according to the topic. If the routing configuration (routing information) of the topic does not exist on the Broker side, then if the routing configuration information of the default topic exists in the Broker, the routing information of a new topic will be created in the Broker according to the number of queues in the message sending request. In this way, the routing information of the subject will exist on the Broker server.
The routing information in the topic configuration manager on the broker side will send a heartbeat packet to the Nameserver and report to the Nameserver. On the other hand, there will be a scheduled task, which is regularly stored on the broker side. The specific path is ${rock_home} / store / config / topics JSON, so that after the broker is closed and restarted, the routing information will not be lost.
What is TBW102?
TBW102 is the default topic that will be automatically created when the Broker is started and the configuration of autoCreateTopicEnable is true.
public TopicConfigManager(BrokerController brokerController) {
this.brokerController = brokerController;
// ...
{
// MixAll.AUTO_CREATE_TOPIC_KEY_TOPIC
if (this.brokerController.getBrokerConfig().isAutoCreateTopicEnable()) {
String topic = MixAll.AUTO_CREATE_TOPIC_KEY_TOPIC;
TopicConfig topicConfig = new TopicConfig(topic);
this.systemTopicList.add(topic);
topicConfig.setReadQueueNums(this.brokerController.getBrokerConfig()
.getDefaultTopicQueueNums());
topicConfig.setWriteQueueNums(this.brokerController.getBrokerConfig()
.getDefaultTopicQueueNums());
int perm = PermName.PERM_INHERIT | PermName.PERM_READ | PermName.PERM_WRITE;
topicConfig.setPerm(perm);
this.topicConfigTable.put(topicConfig.getTopicName(), topicConfig);
}
}
// ...
}
The default value of autoCreateTopicEnable is true. You can synchronize the external configuration file and load it when the Broker starts to change this value. I understand that the role of TBW102 is to enable the function of automatically creating topic, and unconfigured topic is used when sending. This topic can inherit the default TBW102 configuration to send messages.
Summary and analysis
-
Firstly, the client does not cache the routing information of the corresponding topic locally, and then go to the nameserver to find it. There is no routing information of this topic in the nameserver, and then return it to the client. After receiving the return, the client requests the routing information with topic tbw102 from the nameserver.
-
If autocreateTopic is set for a broker, the broker will create the corresponding topicconfig in the topicManager when starting, send it to the nameserver through heartbeat, and the nameserver will save it. The nameserver returns the previously saved routing information of tbw102 to the requesting client.
-
After the client gets the routing information with tbw102 as the topic, the client returns it. According to the returned tbw102 routing information (including all brokers with autocreateTopic set to true, by default, each broker will create DefaultTopicQueueNums=4 read-write queues locally, assuming that there are 8 queues for you to choose between two brokers), the client caches it in the local topicPublishInfoTable table first, key is topic, value is topicRouteData, and a queue is selected for sending by polling.
Send the topic message according to the broker corresponding to the selected queue.
After receiving this message, broker will call the createTopicInSendMessageMethod method in the msgcheck method to create topicConfig information into the topicConfigTable table, and then send the same message as the flow that sent the created topic.
Meanwhile, the topicConfigTable will send the new topicConfigTable information to the nameserver through heartbeat.
After receiving the message, the nameserver will update the routing information of the topic. If the broker that received the message before does not cover it all, because the broker will send a heartbeat to the nameserver in 30S, and the heartbeat package contains topicconfig. The covered broker will send the automatically created topicconfig information to the nameserver, so that the new topic information will be registered after receiving it from the nameserver, Because consumers also go to the nameserver to update the local topicalrouteinfo every 30S, and send the request to the nameserver to get the latest topic routing information after the heartbeat packet sent by the previously covered broker is updated, then the uncovered broker will never join the load balance, which will cause the load balance to fail to meet the expectation, That is, not all brokers that can automatically create topics can participate.
reference material
https://www.cnblogs.com/dingwpmz/p/11809404.html