Said ahead
Management request GET_EARLIEST_MSG_STORETIME for the storage time of the earliest message
Source code analysis
Enter the method org.apache.rocketmq.broker.processor.AdminBrokerProcessor#getEarliestMsgStoretime to get the storage time of the earliest messages
private RemotingCommand getEarliestMsgStoretime(ChannelHandlerContext ctx,
RemotingCommand request) throws RemotingCommandException {
final RemotingCommand response = RemotingCommand.createResponseCommand(GetEarliestMsgStoretimeResponseHeader.class);
final GetEarliestMsgStoretimeResponseHeader responseHeader = (GetEarliestMsgStoretimeResponseHeader) response.readCustomHeader();
final GetEarliestMsgStoretimeRequestHeader requestHeader =
(GetEarliestMsgStoretimeRequestHeader) request.decodeCommandCustomHeader(GetEarliestMsgStoretimeRequestHeader.class);
// Get the earliest message storage time= based on topic and queueId
long timestamp =
this.brokerController.getMessageStore().getEarliestMessageTime(requestHeader.getTopic(), requestHeader.getQueueId());
responseHeader.setTimestamp(timestamp);
response.setCode(ResponseCode.SUCCESS);
response.setRemark(null);
return response;
}
Enter the method org.apache.rocketmq.store.DefaultMessageStore#getEarliestMessageTime(java.lang.String, int) to query the storage time of the earliest messages by topic and queueId
@Override
public long getEarliestMessageTime(String topic, int queueId) {
// Query the consumer queue by topic and queueId
ConsumeQueue logicQueue = this.findConsumeQueue(topic, queueId);
if (logicQueue != null) {
// Get the smallest physical offset
long minLogicOffset = logicQueue.getMinLogicOffset();
// Get SelectMappedBufferResult = from the subscript
SelectMappedBufferResult result = logicQueue.getIndexBuffer(minLogicOffset / ConsumeQueue.CQ_STORE_UNIT_SIZE);
// Find storage time= based on SelectMappedBuffer Result
return getStoreTime(result);
}
return -1;
}
Enter the method org. apache. rocketmq. store. DefaultMessageStore # find ConsumeQueue to query the consumption queue by topic and queueId, as described earlier.
public ConsumeQueue findConsumeQueue(String topic, int queueId) {
ConcurrentMap<Integer, ConsumeQueue> map = consumeQueueTable.get(topic);
if (null == map) {
ConcurrentMap<Integer, ConsumeQueue> newMap = new ConcurrentHashMap<Integer, ConsumeQueue>(128);
ConcurrentMap<Integer, ConsumeQueue> oldMap = consumeQueueTable.putIfAbsent(topic, newMap);
if (oldMap != null) {
map = oldMap;
} else {
map = newMap;
}
}
// Find the consumer queue by queue id
ConsumeQueue logic = map.get(queueId);
if (null == logic) {
ConsumeQueue newLogic = new ConsumeQueue(
topic,
queueId,
// Consumer queue storage address user.home/store/consumequeue
StorePathConfigHelper.getStorePathConsumeQueue(this.messageStoreConfig.getStorePathRootDir()),
// Default 30W per file store
this.getMessageStoreConfig().getMapedFileSizeConsumeQueue(),
this);
ConsumeQueue oldLogic = map.putIfAbsent(queueId, newLogic);
if (oldLogic != null) {
logic = oldLogic;
} else {
logic = newLogic;
}
}
return logic;
}
Go back to the method org.apache.rocketmq.store.ConsumeQueue#getIndexBuffer to get SelectMappedBufferResult based on the smallest offset
public SelectMappedBufferResult getIndexBuffer(final long startIndex) {
int mappedFileSize = this.mappedFileSize;
long offset = startIndex * CQ_STORE_UNIT_SIZE;
if (offset >= this.getMinLogicOffset()) {
// Query the mapping file according to offset =
MappedFile mappedFile = this.mappedFileQueue.findMappedFileByOffset(offset);
if (mappedFile != null) {
SelectMappedBufferResult result = mappedFile.selectMappedBuffer((int) (offset % mappedFileSize));
return result;
}
}
return null;
}
Enter the method org.apache.rocketmq.store.MappedFileQueue#findMappedFileByOffset(long, boolean) to query the mapping file based on the minimum physical offset
public MappedFile findMappedFileByOffset(final long offset, final boolean returnFirstOnNotFound) {
try {
// Get the first mapping file in the queue
MappedFile firstMappedFile = this.getFirstMappedFile();
// Get the last mapping file in the queue
MappedFile lastMappedFile = this.getLastMappedFile();
if (firstMappedFile != null && lastMappedFile != null) {
// If the offset is not within the offset range of the index file
if (offset < firstMappedFile.getFileFromOffset() || offset >= lastMappedFile.getFileFromOffset() + this.mappedFileSize) {
LOG_ERROR.warn("Offset not matched. Request offset: {}, firstOffset: {}, lastOffset: {}, mappedFileSize: {}, mappedFiles count: {}",
offset,
firstMappedFile.getFileFromOffset(),
lastMappedFile.getFileFromOffset() + this.mappedFileSize,
this.mappedFileSize,
this.mappedFiles.size());
} else {
// Find the index location of the mapping file in the queue
int index = (int) ((offset / this.mappedFileSize) - (firstMappedFile.getFileFromOffset() / this.mappedFileSize));
MappedFile targetFile = null;
try {
// Get the index file
targetFile = this.mappedFiles.get(index);
} catch (Exception ignored) {
}
// Offset is within the range of the initial offset and the end offset of the target file
if (targetFile != null && offset >= targetFile.getFileFromOffset()
&& offset < targetFile.getFileFromOffset() + this.mappedFileSize) {
return targetFile;
}
// If the mapping file cannot be found in the queue by index, traverse the queue to find the mapping file
for (MappedFile tmpMappedFile : this.mappedFiles) {
if (offset >= tmpMappedFile.getFileFromOffset()
&& offset < tmpMappedFile.getFileFromOffset() + this.mappedFileSize) {
return tmpMappedFile;
}
}
}
// If offset=0 retrieves the first mapping file in the queue, the individual feels that it is more reasonable to put this logic in front of the judgement, or that it has a different meaning here.
if (returnFirstOnNotFound) {
return firstMappedFile;
}
}
} catch (Exception e) {
log.error("findMappedFileByOffset Exception", e);
}
return null;
}
Go up to this method, org.apache.rocketmq.store.DefaultMessageStore#getStoreTime, to get storage time from SelectMappedBufferResult
private long getStoreTime(SelectMappedBufferResult result) {
if (result != null) {
try {
final long phyOffset = result.getByteBuffer().getLong();
final int size = result.getByteBuffer().getInt();
// Find storage time= based on offset and size of SelectMappedBufferResult
long storeTime = this.getCommitLog().pickupStoreTimestamp(phyOffset, size);
return storeTime;
} catch (Exception e) {
} finally {
result.release();
}
}
return -1;
}
Enter the method org. apache. rocketmq. store. CommitLog pickupStoreTimestamp to query storage time from commitLog by minimum physical offset and file size
public long pickupStoreTimestamp(final long offset, final int size) {
if (offset >= this.getMinOffset()) {
// =>
SelectMappedBufferResult result = this.getMessage(offset, size);
if (null != result) {
try {
// Get message storage time
return result.getByteBuffer().getLong(MessageDecoder.MESSAGE_STORE_TIMESTAMP_POSTION);
} finally {
result.release();
}
}
}
return -1;
}
Enter the method org. apache. rocketmq. store. CommitLog getMessage to get SelectMappedBufferResult by minimum physical offset and file size
public SelectMappedBufferResult getMessage(final long offset, final int size) {
int mappedFileSize = this.defaultMessageStore.getMessageStoreConfig().getMapedFileSizeCommitLog();
// Find the mapping file= based on offset
MappedFile mappedFile = this.mappedFileQueue.findMappedFileByOffset(offset, offset == 0);
if (mappedFile != null) {
int pos = (int) (offset % mappedFileSize);
return mappedFile.selectMappedBuffer(pos, size);
}
return null;
}
Enter the method org.apache.rocketmq.store.MappedFileQueue#findMappedFileByOffset(long, boolean) to query the mapping file by offset, as described earlier.
public MappedFile findMappedFileByOffset(final long offset, final boolean returnFirstOnNotFound) {
try {
// Get the first mapping file in the queue
MappedFile firstMappedFile = this.getFirstMappedFile();
// Get the last mapping file in the queue
MappedFile lastMappedFile = this.getLastMappedFile();
if (firstMappedFile != null && lastMappedFile != null) {
// If the offset is not within the offset range of the index file
if (offset < firstMappedFile.getFileFromOffset() || offset >= lastMappedFile.getFileFromOffset() + this.mappedFileSize) {
LOG_ERROR.warn("Offset not matched. Request offset: {}, firstOffset: {}, lastOffset: {}, mappedFileSize: {}, mappedFiles count: {}",
offset,
firstMappedFile.getFileFromOffset(),
lastMappedFile.getFileFromOffset() + this.mappedFileSize,
this.mappedFileSize,
this.mappedFiles.size());
} else {
// Find the index location of the mapping file in the queue
int index = (int) ((offset / this.mappedFileSize) - (firstMappedFile.getFileFromOffset() / this.mappedFileSize));
MappedFile targetFile = null;
try {
// Get the index file
targetFile = this.mappedFiles.get(index);
} catch (Exception ignored) {
}
// Offset is within the range of the initial offset and the end offset of the target file
if (targetFile != null && offset >= targetFile.getFileFromOffset()
&& offset < targetFile.getFileFromOffset() + this.mappedFileSize) {
return targetFile;
}
// If the mapping file cannot be found in the queue by index, traverse the queue to find the mapping file
for (MappedFile tmpMappedFile : this.mappedFiles) {
if (offset >= tmpMappedFile.getFileFromOffset()
&& offset < tmpMappedFile.getFileFromOffset() + this.mappedFileSize) {
return tmpMappedFile;
}
}
}
// If offset=0 retrieves the first mapping file in the queue, the individual feels that it is more reasonable to put this logic in front of the judgement, or that it has a different meaning here.
if (returnFirstOnNotFound) {
return firstMappedFile;
}
}
} catch (Exception e) {
log.error("findMappedFileByOffset Exception", e);
}
return null;
}
Up to the end of the method org.apache.rocketmq.broker.processor.AdminBrokerProcessor#getEarliestMsgStoretime
Said at the end
This analysis only represents personal views, for reference only.
Joining WeChat Technology Group
Nail Technology Group