1. General
Let's first look at the polling mechanism in the consumption process of RocketMQ. First, we need to add some pre knowledge related to consumption.
1.1 message consumption mode
RocketMQ supports a variety of consumption modes, including Push mode and Pull mode
- Pull mode: users pull messages and update consumption progress
- Push mode: the Broker automatically sends new messages to users for consumption
1.2 Push consumption mode
We generally use Push mode when using RocketMQ, because it is more convenient and does not need to manually pull messages and update consumption progress.
So have you ever thought about how Push mode can consume new news immediately?
1.2.1 principle of push mode
In fact, when pushing consumption, consumers are constantly polling the Broker to ask whether there are new messages available for consumption. Once a new message arrives, Pull the message immediately. In other words, the Push mode also uses the Pull message mode internally, so that the latest messages can be consumed immediately.
1.3 how to poll?
So how to query messages in Push mode or Pull mode?
The stupid way to think of is to send a query request to the Broker every certain time (such as 1ms), and return it immediately if there is no new message. It can be imagined that this method is a waste of network resources.
In order to improve the network performance, RocketMQ will not return immediately if there is no new message when pulling messages. Instead, it will suspend the query request for a period of time and then retry the query. If there is no new message, it will not return until the polling time exceeds the set threshold.
According to the timeout threshold set by polling, RocketMQ has two polling methods: long polling (default) and short polling.
1.4 long polling and short polling
The Broker side parameter longPollingEnable of RocketMQ can configure the polling method, which is true by default
- Short polling: longPollingEnable=false, the polling time is shortPollingTimeMills, and the default is 1s
- Long polling: longPollingEnable=true, and the polling time is 5S. Suspension time of pull request: controlled by the brokerSuspendMaxTimeMillis of DefaultMQPullConsumer, the default push mode is fixed for 15s and the pull mode is fixed for 20s.
2. Outline process
[external chain picture transfer failed. The source station may have anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-b6pqzswr-1646145661686)( https://raw.githubusercontent.com/HScarb/drawio-diagrams/main/rocketmq/consume/long_polling_activity.drawio.svg )]
Let's take a look at the polling mechanism process of RocketMQ consumption according to the above activity diagram
- Consumer sends pull message request
- After receiving the request, the Broker sends it to the request processing module for processing
- Attempt to pull a message from a stored message
- If the message can be pulled, the pulled message will be returned directly
- If the message is not pulled, whether to poll and what kind of polling is determined according to whether the Broker supports suspending and whether to enable long polling.
- If suspension is supported, the pull request will be suspended
- Long polling wait 5s
- Short polling wait for 1s
- Check whether there are new messages arriving in the consumption queue. If not, continue to wait and cycle. If there is a new message, process the pending pull message request and return it to the consumer.
- If no new message arrives, it will check whether the suspension time of each pending pull request exceeds the suspension time threshold after polling. If it exceeds the suspension time threshold, it will also directly return to the consumer. Otherwise, it will continue to cycle the polling operation.
Then, according to the above process, when long polling is enabled, if no message is found in one polling, wait 5s for the next query. If there are new messages stored in these 5s, how to ensure that they can be consumed immediately?
The solution is not hard to think of, that is, after the new message is written, take the initiative to notify, so that the suspended pull request can be pulled immediately.
This is what RocketMQ does. In the doReput method after the message is stored in the CommitLog, it will judge whether it is a long polling. If so, it will send a notification to let the suspended pull request be processed immediately.
3. Detailed process
3.1 classes involved
3.1.1 PullMessageProcessor
This class is the entry class for Broker to handle Consumer pull requests. When the Broker receives the pull request sent by the Consumer, it calls the processRequest method of this class
3.1.2 PullRequestHoldService
Long polling request management thread. The suspended pull request will be saved here. Each waiting period (long polling / short polling waiting time) will check whether there is data that can be pulled in the pending request.
3.1.3 DefaultMessageStore#ReputMessageService
This thread is responsible for retransmitting the messages stored in CommitLog to generate ConsumeQueue and IndexFile indexes. After the index is generated, a reminder will be sent to the long polling thread to immediately wake up the pull request of the corresponding queue and execute the message pull.
3.2 sequence diagram
[the external chain image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-brvsznhe-1646145661687)( https://raw.githubusercontent.com/HScarb/drawio-diagrams/main/rocketmq/consume/long_polling_sequence.drawio.svg )]
It focuses on the long polling logic, and other logics are omitted
- The consumer calls pullKernelImpl() to send a pull request. When calling, the maximum time of Broker suspension is specified with brokerSuspendMaxTimeMillis. The default is 20s
- PullMessageProcess in Broker handles pull requests and queries messages from ConsumeQueue
- If no message is found, judge whether to enable long polling, and call PullRequestHoldService#suspendPullRequest() method to suspend the request
- The PullRequestHoldService thread run() method circularly waits for the polling time, and then periodically calls the checkHoldRequest() method to check whether there are messages for the pending request
- If a new message can be pulled, call notifyMessageArriving() method
- If the doReput() of the ReputMessageService is called, it indicates that a new message has arrived and needs to wake up the pending pull request. Here, a notify will also be sent, and then the notifyMessageArriving() method will be called
- The notifyMessageArriving() method will also query the maximum offset of ConsumeQueue. If there is a new message, the corresponding pull request will be awakened. The specific method is to call the executeRequestWhenWakeup() method
- The executeRequestWhenWakeup() method wakes up the pull request and calls the processRequest() method to process the request
3.3 specific logic of each class
3.3.1 PullMessageProcessor
The Broker handles the entry class of the Consumer pull request
-
RemotingCommand processRequest(ChannelHandlerContext ctx, RemotingCommand request): the entry method to process the Consumer pull request. It is called when the Consumer pull request is received. This method mainly completes the following operations
- check
- Message filtering
- Query messages from storage
- Return response to Consumer
If no message is queried from the storage, the response code will be set to responsecode PULL_ NOT_ Found and start long polling
-
Void executerequestwhenwakeup (channel, final remotingcommand request): wake up the Hold pull request and pull the message again
- This method is called when a new message is received in the long polling, immediately wakes up the pending pull requests, and then calls the processRequest method for these requests
- When do I need to remind long polling that a new message has arrived? As mentioned above, if a new message arrives during a long polling wait, a reminder will be given in the doReput method of CommitLog, and finally the executeRequestWhenWakeup method will be called
3.3.2 PullRequestHoldService
The service thread will fetch the PullRequest request from the pullRequestTable local cache variable and check whether the polling condition "whether the offset of the message to be pulled is less than the maximum offset of the consumption queue" is true. If the condition is true, it indicates that a new message has reached the Broker end, and then try to launch the RPC request of the Pull message again through the executeRequestWhenWakeup() method of PullMessageProcessor
-
pullRequestTable
private ConcurrentMap<String/* topic@queueId */, ManyPullRequest/* Pull requests accumulated in the same queue */> pullRequestTable = new ConcurrentHashMap<>(1024)
The above is the container of suspended message pull requests. It is a ConcurrentHashMap. key is the queue of pull requests, and value is all the pull requests suspended by the queue. The bottom layer of ManyPullRequest is an ArrayList whose add method is locked.
-
Suspendpullrequest (string topic, int queueid, pullrequest, pullrequest): suspend the Consumer pull request temporarily and add the request to the pullRequestTable
-
checkHoldRequest(): check all pending pull requests. If any data meets the requirements, wake up the request and execute the PullMessageProcessor#processRequest method on it
-
run(): the main loop of the thread. Every time it waits for a period of time, it calls the checkHoldRequest() method to check whether there is a request to wake up. The waiting time is determined according to the configuration of long polling / short polling. The long polling waits for 5s and the short polling waits for 1s by default
-
notifyMessageArriving(): called by checkHoldRequest() and ReputMessageService#doReput(), indicating that a new message arrives and wakes up the pending pull request of the corresponding queue
3.3.3 DefaultMessageStore#ReputMessageService
The service thread doReput() method will continuously parse data from the data storage object CommitLog on the Broker side and distribute requests, and then build two types of data: ConsumeQueue (logical consumption queue) and IndexFile (message index file).
At the same time, take out the pending pull request from the local cache variable PullRequestHoldService#pullRequestTable and execute it.
4. Source code analysis
4.1 PullMessageProcessor
4.1.1 processRequest
If no message is queried from the storage, the response code will be set to responsecode PULL_ NOT_ Found and start long polling
The following three situations will set the response code to responsecode PULL_ NOT_ FOUND:
- NO_MESSAGE_IN_QUEUE: there are no messages in the consumption queue
- OFFSET_FOUND_NULL: offset no data found
- OFFSET_OVERFLOW_ONE: the offset to be pulled is equal to the maximum offset of the queue
/**
* Handle client request entry
*
* @param channel The network channel sends the response result to the message pull client through this channel
* @param request Message pull request
* @param brokerAllowSuspend Broker Whether the client can be suspended. The default is true. True: suspend if no message is found. false: the message not found is returned directly
* @return response
* @throws RemotingCommandException When an exception occurs in the parsing request
*/
private RemotingCommand processRequest(final Channel channel, RemotingCommand request, boolean brokerAllowSuspend)
throws RemotingCommandException {
// ...
switch (response.getCode()) {
// ...
// If no new message can be pulled from the consumption queue, judge and suspend the pull request
case ResponseCode.PULL_NOT_FOUND:
// Long polling
if (brokerAllowSuspend && hasSuspendFlag) {
long pollingTimeMills = suspendTimeoutMillisLong;
if (!this.brokerController.getBrokerConfig().isLongPollingEnable()) {
pollingTimeMills = this.brokerController.getBrokerConfig().getShortPollingTimeMills();
}
String topic = requestHeader.getTopic();
long offset = requestHeader.getQueueOffset();
int queueId = requestHeader.getQueueId();
PullRequest pullRequest = new PullRequest(request, channel, pollingTimeMills,
this.brokerController.getMessageStore().now(), offset, subscriptionData, messageFilter);
this.brokerController.getPullRequestHoldService().suspendPullRequest(topic, queueId, pullRequest);
response = null;
break;
}
// ...
}
4.1.2 executeRequestWhenWakeup
In the executeRequestWhenWakeup() method of PullMessageProcessor, the request task of re pulling message is submitted asynchronously through the pullMessageExecutor of the business thread pool, that is, the processRequest() method of PullMessageProcessor business processor is re adjusted once to realize the secondary processing of Pull message request).
/**
* Wake up the pull request of Hold and pull the message again
* This method calls the thread pool, so it is not blocked
*
* @param channel passageway
* @param request Consumer Pull request
* @throws RemotingCommandException When an exception occurs in a remote call
*/
public void executeRequestWhenWakeup(final Channel channel,
final RemotingCommand request) throws RemotingCommandException {
Runnable run = new Runnable() {
@Override
public void run() {
try {
// Handle the Consumer pull request and get the return body
final RemotingCommand response = PullMessageProcessor.this.processRequest(channel, request, false);
if (response != null) {
response.setOpaque(request.getOpaque());
response.markResponseType();
try {
// Write the return body to the channel and return it to the Consumer
channel.writeAndFlush(response).addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture future) throws Exception {
if (!future.isSuccess()) {
log.error("processRequestWrapper response to {} failed",
future.channel().remoteAddress(), future.cause());
log.error(request.toString());
log.error(response.toString());
}
}
});
} catch (Throwable e) {
log.error("processRequestWrapper process request over, but response failed", e);
log.error(request.toString());
log.error(response.toString());
}
}
} catch (RemotingCommandException e1) {
log.error("excuteRequestWhenWakeup run", e1);
}
}
};
// Asynchronous execution of request processing and return
this.brokerController.getPullMessageExecutor().submit(new RequestTask(run, channel, request));
}
4.2 PullRequestHoldService
4.2.1 suspendPullRequest
/**
* Suspend (save) the client request and trigger the request when there is data
*
* @param topic theme
* @param queueId Queue number
* @param pullRequest Pull message request
*/
public void suspendPullRequest(final String topic, final int queueId, final PullRequest pullRequest) {
// Construct the key of map according to topic and queueId
String key = this.buildKey(topic, queueId);
// If the key of the map is empty, create an empty request queue and fill in the key and value
ManyPullRequest mpr = this.pullRequestTable.get(key);
if (null == mpr) {
mpr = new ManyPullRequest();
ManyPullRequest prev = this.pullRequestTable.putIfAbsent(key, mpr);
if (prev != null) {
mpr = prev;
}
}
// Save this Consumer pull request
mpr.addPullRequest(pullRequest);
}
4.2.2 checkHoldRequest
/**
* Check all long polling requests that have been suspended
* If any data meets the requirements, the request is triggered to execute again
*/
private void checkHoldRequest() {
// Traverse each queue in the pull request container
for (String key : this.pullRequestTable.keySet()) {
String[] kArray = key.split(TOPIC_QUEUEID_SEPARATOR);
if (2 == kArray.length) {
String topic = kArray[0];
int queueId = Integer.parseInt(kArray[1]);
// Get the maximum offset of the queue from the store
final long offset = this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId);
try {
// Judge whether a new message arrives according to the maximum offset obtained in the store. If so, execute the pull request operation
this.notifyMessageArriving(topic, queueId, offset);
} catch (Throwable e) {
log.error("check hold request failed. topic={}, queueId={}", topic, queueId, e);
}
}
}
}
4.2.3 run
@Override
public void run() {
log.info("{} service started", this.getServiceName());
while (!this.isStopped()) {
try {
// Wait for a certain time
if (this.brokerController.getBrokerConfig().isLongPollingEnable()) {
// Start long polling and judge whether the message arrives every 5s
this.waitForRunning(5 * 1000);
} else {
// If long polling is not started, judge whether the message arrives every 1s
this.waitForRunning(this.brokerController.getBrokerConfig().getShortPollingTimeMills());
}
long beginLockTimestamp = this.systemClock.now();
// Check whether a message arrives and wake up the pending request
this.checkHoldRequest();
long costTime = this.systemClock.now() - beginLockTimestamp;
if (costTime > 5 * 1000) {
log.info("[NOTIFYME] check hold request cost {} ms.", costTime);
}
} catch (Throwable e) {
log.warn(this.getServiceName() + " service has exception. ", e);
}
}
log.info("{} service end", this.getServiceName());
}
4.2.4 notifyMessageArriving
This method is called in two places, as shown in the following figure
This method is the core method to wake up the pull request again. Call this method to remind the PullRequestHoldService thread of the arrival of a new message
Let's see what this method does
- Obtain the list of pending pull requests according to topic and queueId
- Get the maximum offset of the queue message from the store
- Traverse all pull requests in the queue. Pull requests that meet one of the following two conditions will be processed and returned
- The maximum offset of the consumer queue is larger than the offset of the consumer pull request, which indicates that a new message can be pulled. Process the pull request
- The pull request suspension time exceeds the threshold, and the direct return message is not found
- If the above two conditions are not met, the pull request will be put back into the pullRequestTable again and wait for the next check
/**
* When a new message arrives, wake up the consumer request for long polling
*
* @param topic Message Topic
* @param queueId Message queue ID
* @param maxOffset Maximum Offset of consumption queue
*/
public void notifyMessageArriving(final String topic, final int queueId, final long maxOffset, final Long tagsCode,
long msgStoreTime, byte[] filterBitMap, Map<String, String> properties) {
// Extract the list of pending pull requests from the container according to topic and queueId
String key = this.buildKey(topic, queueId);
ManyPullRequest mpr = this.pullRequestTable.get(key);
if (mpr != null) {
// Gets the list of pending pull requests
List<PullRequest> requestList = mpr.cloneListAndClear();
if (requestList != null) {
// Predefined list of pull requests that need to continue to be suspended
List<PullRequest> replayList = new ArrayList<PullRequest>();
for (PullRequest request : requestList) {
long newestOffset = maxOffset;
// Get the maximum offset of the queue message from the store
if (newestOffset <= request.getPullFromThisOffset()) {
newestOffset = this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId);
}
// The maximum offset of the consumer queue is larger than the offset of the consumer pull request, indicating that a new message can be pulled
if (newestOffset > request.getPullFromThisOffset()) {
// Message filter matching
boolean match = request.getMessageFilter().isMatchedByConsumeQueue(tagsCode,
new ConsumeQueueExt.CqExtUnit(tagsCode, msgStoreTime, filterBitMap));
// match by bit map, need eval again when properties is not null.
if (match && properties != null) {
match = request.getMessageFilter().isMatchedByCommitLog(null, properties);
}
if (match) {
try {
// The PullMessageProcessor#processRequest method will be called to pull the message, and then the result will be returned to the consumer
this.brokerController.getPullMessageProcessor().executeRequestWhenWakeup(request.getClientChannel(),
request.getRequestCommand());
} catch (Throwable e) {
log.error("execute request when wakeup failed.", e);
}
continue;
}
}
// Check whether it has timed out. If the Consumer request has reached the timeout time, it will also trigger a response and directly return the message that it has not been found
if (System.currentTimeMillis() >= (request.getSuspendTimestamp() + request.getTimeoutMillis())) {
try {
this.brokerController.getPullMessageProcessor().executeRequestWhenWakeup(request.getClientChannel(),
request.getRequestCommand());
} catch (Throwable e) {
log.error("execute request when wakeup failed.", e);
}
continue;
}
// It does not meet the requirements at present. Put it back in the Hold list
replayList.add(request);
}
if (!replayList.isEmpty()) {
mpr.addPullRequest(replayList);
}
}
}
}
4.3 DefaultMessageStore#ReputMessageService
4.3.1 doReput
private void doReput() {
// ...
DefaultMessageStore.this.doDispatch(dispatchRequest);
// Inform the long polling thread of message consumption that a new message is dropped, and immediately wake up the pending message pull request
if (BrokerRole.SLAVE != DefaultMessageStore.this.getMessageStoreConfig().getBrokerRole()
&& DefaultMessageStore.this.brokerConfig.isLongPollingEnable()
&& DefaultMessageStore.this.messageArrivingListener != null) {
DefaultMessageStore.this.messageArrivingListener.arriving(dispatchRequest.getTopic(),
dispatchRequest.getQueueId(), dispatchRequest.getConsumeQueueOffset() + 1,
dispatchRequest.getTagsCode(), dispatchRequest.getStoreTimestamp(),
dispatchRequest.getBitMap(), dispatchRequest.getPropertiesMap());
}
Here, the NotifyMessageArrivingListener#arriving() method is called, and then pullrequestholdservice is called notifyMessageArriving().
Why not call PullRequestHoldService directly notifyMessageArriving() ？ Because the package of the class where doReput is located is store, which stores the package, while PullRequestHoldService is in the broker package
So we need a bridge, NotifyMessageArrivingListener. It is written to the DefaultMessageStore when the Broker initializes the DefaultMessageStore
4.3.2 NotifyMessageArrivingListener#arriving
public class NotifyMessageArrivingListener implements MessageArrivingListener {
@Override
public void arriving(String topic, int queueId, long logicOffset, long tagsCode,
long msgStoreTime, byte[] filterBitMap, Map<String, String> properties) {
// Remind the long polling request management container to pull the latest message immediately when a new message arrives
this.pullRequestHoldService.notifyMessageArriving(topic, queueId, logicOffset, tagsCode,
msgStoreTime, filterBitMap, properties);
}
}