sketch
This article analyzes the open source version 4.8.0 of rocketMq; The rocketMq process is briefly described as follows:
Nameserv: as a registration center, the broker starts to register with nameserv. At the same time, it regularly sends heartbeat to nameserv to tell nameserv that it is still alive. Then, in addition to maintaining broker information, nameserv also maintains topic information, such as which brokers a topic message is sent to, and then the topic is divided into several message queue s, Now we just need to know the nameserv to maintain the broker information and topic information.
broker: an instance that stores message queues, etc
producer: there are three modes of sending messages: synchronous sending, asynchronous sending and one-way sending. One way is to ignore the results after sending, from the perspective of message production;
consumer: Message consumption and confirmation consumption. The consumption mode is through queue_offerset # go to consumeQueue to find physical_offerset, and through physical_offerset found the corresponding message content
They are connected through netty
Message store:
After receiving the message, the broker master node first stores it in the commitLog, and then returns the message storage status; Each broker node generates consumeQueue and BuildIndex through scheduled tasks; consumeQueue is used by consumers, and the stored core data is physical_offset;BuildIndex builds a keyword index for messages
Message storage is explained in three parts:
commitLog -- this article explains
commitLog Process Overview
After receiving the message, the broker gives it to the commitLog for storage. The commitLog consists of a MappedFile, which physically corresponds to a rock_ Home / commitLog / 00000000000000000000. The default size of each file is 1G. The name of the file is the starting offset plus the size of each file. For example, the name of the second file is 0000000000 1073741824. The file is read and written using the memory mapping technology (MMP) and written using the additional write mode. The offset of the message is actually the starting position of each message in the whole commitLog;
Source code analysis
When the broker is started, it will register the processor corresponding to a code on the server, and send different types of messages to different processors for processing, such as send_ The message of message code will be sent to SendMessageProcessor for processing, and the processRequest method of the corresponding processor will be called for processing. Our message producer uses send when sending messages_ Message code. Next, let's look at the processRequest method of SendMessageProcessor; The call chain is
BrokerStartup.start() -> BrokerController.start() -> NettyRemotingServer.start() -> NettyRemotingServer.prepareSharableHandlers() -> new NettyServerHandler() -> NettyRemotingAbstract.processMessageReceived() -> NettyRemotingAbstract.processRequestCommand() -> SendMessageProcessor.processRequest()
The following is an analysis of the source code of SendMessageProcessor;
Enter the red line according to the code
We will go to DefaultMessageStore#putMessage according to the source code
Next, let's look at putMessage. This time, we are talking about ordinary message storage, and we will automatically skip the logical part of the transaction code
// Get the latest mappFile
MappedFile mappedFile = this.mappedFileQueue.getLastMappedFile();
// Obtain the lock to keep the sequence of messages
putMessageLock.lock(); //spin or ReentrantLock ,depending on store config
try {
long beginLockTimestamp = this.defaultMessageStore.getSystemClock().now();
this.beginTimeInLock = beginLockTimestamp;
// Here settings are stored timestamp, in order to ensure an orderly
// global
msg.setStoreTimestamp(beginLockTimestamp);
// Recreate a file
if (null == mappedFile || mappedFile.isFull()) {
mappedFile = this.mappedFileQueue.getLastMappedFile(0); // Mark: NewFile may be cause noise
}
if (null == mappedFile) {
log.error("create mapped file1 error, topic: " + msg.getTopic() + " clientAddr: " + msg.getBornHostString());
beginTimeInLock = 0;
return new PutMessageResult(PutMessageStatus.CREATE_MAPEDFILE_FAILED, null);
}
//Append message to mappedFile
result = mappedFile.appendMessage(msg, this.appendMessageCallback);
switch (result.getStatus()) {
case PUT_OK:
break;
// When the remaining space cannot store messages, it will complete the previous messages and set them to null, return to this state, re create a new mappeFile and append messages
case END_OF_FILE:
unlockMappedFile = mappedFile;
// Create a new file, re-write the message
mappedFile = this.mappedFileQueue.getLastMappedFile(0);
if (null == mappedFile) {
// XXX: warn and notify me
log.error("create mapped file2 error, topic: " + msg.getTopic() + " clientAddr: " + msg.getBornHostString());
beginTimeInLock = 0;
return new PutMessageResult(PutMessageStatus.CREATE_MAPEDFILE_FAILED, result);
}
result = mappedFile.appendMessage(msg, this.appendMessageCallback);There are my comments in the code. The following focuses on the addition of messages
public AppendMessageResult appendMessagesInner(final MessageExt messageExt, final AppendMessageCallback cb) {
assert messageExt != null;
assert cb != null;
//Get write location
int currentPos = this.wrotePosition.get();
if (currentPos < this.fileSize) {
//Get new byte cache
ByteBuffer byteBuffer = writeBuffer != null ? writeBuffer.slice() : this.mappedByteBuffer.slice();
// Reset new location
byteBuffer.position(currentPos);
AppendMessageResult result;
if (messageExt instanceof MessageExtBrokerInner) {
// Append message
result = cb.doAppend(this.getFileFromOffset(), byteBuffer, this.fileSize - currentPos, (MessageExtBrokerInner) messageExt);
} else if (messageExt instanceof MessageExtBatch) {
result = cb.doAppend(this.getFileFromOffset(), byteBuffer, this.fileSize - currentPos, (MessageExtBatch) messageExt);
} else {
return new AppendMessageResult(AppendMessageStatus.UNKNOWN_ERROR);
}
// Update current mappedFile start location
this.wrotePosition.addAndGet(result.getWroteBytes());
this.storeTimestamp = result.getStoreTimestamp();
return result;
}There are too many contents in doAppend. We only look at the core
// Initialization of storage space
this.resetByteBuffer(msgStoreItemMemory, msgLen);
// 1 TOTALSIZE message size
this.msgStoreItemMemory.putInt(msgLen);
// 2 MAGICCODE
this.msgStoreItemMemory.putInt(CommitLog.MESSAGE_MAGIC_CODE);
// 3 BODYCRC
this.msgStoreItemMemory.putInt(msgInner.getBodyCRC());
// 4 QUEUEID
this.msgStoreItemMemory.putInt(msgInner.getQueueId());
// 5 FLAG
this.msgStoreItemMemory.putInt(msgInner.getFlag());
// 6 QUEUEOFFSET
this.msgStoreItemMemory.putLong(queueOffset);
// 7 PHYSICALOFFSET
this.msgStoreItemMemory.putLong(fileFromOffset + byteBuffer.position());
// 8 SYSFLAG
this.msgStoreItemMemory.putInt(msgInner.getSysFlag());
// 9 BORNTIMESTAMP
this.msgStoreItemMemory.putLong(msgInner.getBornTimestamp());
// 10 BORNHOST
this.resetByteBuffer(bornHostHolder, bornHostLength);
this.msgStoreItemMemory.put(msgInner.getBornHostBytes(bornHostHolder));
// 11 STORETIMESTAMP
this.msgStoreItemMemory.putLong(msgInner.getStoreTimestamp());
// 12 STOREHOSTADDRESS
this.resetByteBuffer(storeHostHolder, storeHostLength);
this.msgStoreItemMemory.put(msgInner.getStoreHostBytes(storeHostHolder));
// 13 RECONSUMETIMES
this.msgStoreItemMemory.putInt(msgInner.getReconsumeTimes());
// 14 Prepared Transaction Offset
this.msgStoreItemMemory.putLong(msgInner.getPreparedTransactionOffset());
// 15 BODY
this.msgStoreItemMemory.putInt(bodyLength);
if (bodyLength > 0)
this.msgStoreItemMemory.put(msgInner.getBody());
// 16 TOPIC
this.msgStoreItemMemory.put((byte) topicLength);
this.msgStoreItemMemory.put(topicData);
// 17 PROPERTIES
this.msgStoreItemMemory.putShort((short) propertiesLength);
if (propertiesLength > 0)
this.msgStoreItemMemory.put(propertiesData);
final long beginTimeMills = CommitLog.this.defaultMessageStore.now();
// Write messages to the queue buffer
byteBuffer.put(this.msgStoreItemMemory.array(), 0, msgLen);
AppendMessageResult result = new AppendMessageResult(AppendMessageStatus.PUT_OK, wroteOffset, msgLen, msgId,
msgInner.getStoreTimestamp(), queueOffset, CommitLog.this.defaultMessageStore.now() - beginTimeMills);
The above code is very important. The key attributes of the message are as follows
Writeoffset: This is physicsofffset. It is very important to find the key attributes of the message and locate the physical location of the message
Message length: message content
queueoffset: the offset of the queue, which can also be called logical offset. This is increased according to the id atom of the queue
Since the message content is positive, we can find the value of each attribute through physicsofffset, including the message content;
Locate the start position of the message through physicsofffset, and you can get the required data. For the change of each attribute of the message content, the size of the content will be stored before, so that you can intercept the complete message content;
The last is to brush the disk synchronously or asynchronously according to the configuration
After successful addition, unlock and brush the disk. According to the configuration, whether to brush the disk synchronously, because we are still stored in memory at this time to avoid data disappearance, we can configure synchronous disk brushing;
summary
The commitlog is a logically large file. There are many mappedfiles below. Each message has a physicsofffset (physical offset). Later, the offset of the commitlog is stored whether building an index or ConsumeQueue;
With physicsofffset, you can quickly locate the specific message start position. According to the message content setting rules and principles, you can get the attribute contents of the message body, mainly including queueid, queue offset, physicsofffset and message content,