rocketMq message store commitLog

Posted by artist-ink on Sat, 22 Jan 2022 04:03:15 +0100

sketch

This article analyzes the open source version 4.8.0 of rocketMq; The rocketMq process is briefly described as follows:

Nameserv: as a registration center, the broker starts to register with nameserv. At the same time, it regularly sends heartbeat to nameserv to tell nameserv that it is still alive. Then, in addition to maintaining broker information, nameserv also maintains topic information, such as which brokers a topic message is sent to, and then the topic is divided into several message queue s, Now we just need to know the nameserv to maintain the broker information and topic information.

broker: an instance that stores message queues, etc

producer: there are three modes of sending messages: synchronous sending, asynchronous sending and one-way sending. One way is to ignore the results after sending, from the perspective of message production;

consumer: Message consumption and confirmation consumption. The consumption mode is through queue_offerset # go to consumeQueue to find physical_offerset, and through physical_offerset found the corresponding message content

They are connected through netty

Message store:

After receiving the message, the broker master node first stores it in the commitLog, and then returns the message storage status; Each broker node generates consumeQueue and BuildIndex through scheduled tasks; consumeQueue is used by consumers, and the stored core data is physical_offset;BuildIndex builds a keyword index for messages

Message storage is explained in three parts:

commitLog -- this article explains

consumeQueue

BuildIndex

commitLog Process Overview

After receiving the message, the broker gives it to the commitLog for storage. The commitLog consists of a MappedFile, which physically corresponds to a rock_ Home / commitLog / 00000000000000000000. The default size of each file is 1G. The name of the file is the starting offset plus the size of each file. For example, the name of the second file is 0000000000 1073741824. The file is read and written using the memory mapping technology (MMP) and written using the additional write mode. The offset of the message is actually the starting position of each message in the whole commitLog;

Source code analysis

When the broker is started, it will register the processor corresponding to a code on the server, and send different types of messages to different processors for processing, such as send_ The message of message code will be sent to SendMessageProcessor for processing, and the processRequest method of the corresponding processor will be called for processing. Our message producer uses send when sending messages_ Message code. Next, let's look at the processRequest method of SendMessageProcessor; The call chain is

BrokerStartup.start() -> BrokerController.start() -> 
NettyRemotingServer.start() -> NettyRemotingServer.prepareSharableHandlers() 
-> new NettyServerHandler() -> NettyRemotingAbstract.processMessageReceived() 
-> NettyRemotingAbstract.processRequestCommand() ->
SendMessageProcessor.processRequest()

The following is an analysis of the source code of SendMessageProcessor;

Enter the red line according to the code

We will go to DefaultMessageStore#putMessage according to the source code

Next, let's look at putMessage. This time, we are talking about ordinary message storage, and we will automatically skip the logical part of the transaction code

        // Get the latest mappFile
        MappedFile mappedFile = this.mappedFileQueue.getLastMappedFile();
        // Obtain the lock to keep the sequence of messages
        putMessageLock.lock(); //spin or ReentrantLock ,depending on store config
        try {
            long beginLockTimestamp = this.defaultMessageStore.getSystemClock().now();
            this.beginTimeInLock = beginLockTimestamp;

            // Here settings are stored timestamp, in order to ensure an orderly
            // global
            msg.setStoreTimestamp(beginLockTimestamp);
            // Recreate a file
            if (null == mappedFile || mappedFile.isFull()) {
                mappedFile = this.mappedFileQueue.getLastMappedFile(0); // Mark: NewFile may be cause noise
            }
            if (null == mappedFile) {
                log.error("create mapped file1 error, topic: " + msg.getTopic() + " clientAddr: " + msg.getBornHostString());
                beginTimeInLock = 0;
                return new PutMessageResult(PutMessageStatus.CREATE_MAPEDFILE_FAILED, null);
            }
            //Append message to mappedFile
            result = mappedFile.appendMessage(msg, this.appendMessageCallback);
            switch (result.getStatus()) {
                case PUT_OK:
                    break;
                    // When the remaining space cannot store messages, it will complete the previous messages and set them to null, return to this state, re create a new mappeFile and append messages
                case END_OF_FILE:
                    unlockMappedFile = mappedFile;
                    // Create a new file, re-write the message
                    mappedFile = this.mappedFileQueue.getLastMappedFile(0);
                    if (null == mappedFile) {
                        // XXX: warn and notify me
                        log.error("create mapped file2 error, topic: " + msg.getTopic() + " clientAddr: " + msg.getBornHostString());
                        beginTimeInLock = 0;
                        return new PutMessageResult(PutMessageStatus.CREATE_MAPEDFILE_FAILED, result);
                    }
                    result = mappedFile.appendMessage(msg, this.appendMessageCallback);

There are my comments in the code. The following focuses on the addition of messages

 public AppendMessageResult appendMessagesInner(final MessageExt messageExt, final AppendMessageCallback cb) {
        assert messageExt != null;
        assert cb != null;
        //Get write location
        int currentPos = this.wrotePosition.get();

        if (currentPos < this.fileSize) {
            //Get new byte cache
            ByteBuffer byteBuffer = writeBuffer != null ? writeBuffer.slice() : this.mappedByteBuffer.slice();
            // Reset new location
            byteBuffer.position(currentPos);
            AppendMessageResult result;
            if (messageExt instanceof MessageExtBrokerInner) {
                // Append message
                result = cb.doAppend(this.getFileFromOffset(), byteBuffer, this.fileSize - currentPos, (MessageExtBrokerInner) messageExt);
            } else if (messageExt instanceof MessageExtBatch) {
                result = cb.doAppend(this.getFileFromOffset(), byteBuffer, this.fileSize - currentPos, (MessageExtBatch) messageExt);
            } else {
                return new AppendMessageResult(AppendMessageStatus.UNKNOWN_ERROR);
            }
            // Update current mappedFile start location
            this.wrotePosition.addAndGet(result.getWroteBytes());
            this.storeTimestamp = result.getStoreTimestamp();
            return result;
        }

There are too many contents in doAppend. We only look at the core

 

 // Initialization of storage space
            this.resetByteBuffer(msgStoreItemMemory, msgLen);
            // 1 TOTALSIZE message size
            this.msgStoreItemMemory.putInt(msgLen);
            // 2 MAGICCODE
            this.msgStoreItemMemory.putInt(CommitLog.MESSAGE_MAGIC_CODE);
            // 3 BODYCRC
            this.msgStoreItemMemory.putInt(msgInner.getBodyCRC());
            // 4 QUEUEID
            this.msgStoreItemMemory.putInt(msgInner.getQueueId());
            // 5 FLAG
            this.msgStoreItemMemory.putInt(msgInner.getFlag());
            // 6 QUEUEOFFSET
            this.msgStoreItemMemory.putLong(queueOffset);
            // 7 PHYSICALOFFSET
            this.msgStoreItemMemory.putLong(fileFromOffset + byteBuffer.position());
            // 8 SYSFLAG
            this.msgStoreItemMemory.putInt(msgInner.getSysFlag());
            // 9 BORNTIMESTAMP
            this.msgStoreItemMemory.putLong(msgInner.getBornTimestamp());
            // 10 BORNHOST
            this.resetByteBuffer(bornHostHolder, bornHostLength);
            this.msgStoreItemMemory.put(msgInner.getBornHostBytes(bornHostHolder));
            // 11 STORETIMESTAMP
            this.msgStoreItemMemory.putLong(msgInner.getStoreTimestamp());
            // 12 STOREHOSTADDRESS
            this.resetByteBuffer(storeHostHolder, storeHostLength);
            this.msgStoreItemMemory.put(msgInner.getStoreHostBytes(storeHostHolder));
            // 13 RECONSUMETIMES
            this.msgStoreItemMemory.putInt(msgInner.getReconsumeTimes());
            // 14 Prepared Transaction Offset
            this.msgStoreItemMemory.putLong(msgInner.getPreparedTransactionOffset());
            // 15 BODY
            this.msgStoreItemMemory.putInt(bodyLength);
            if (bodyLength > 0)
                this.msgStoreItemMemory.put(msgInner.getBody());
            // 16 TOPIC
            this.msgStoreItemMemory.put((byte) topicLength);
            this.msgStoreItemMemory.put(topicData);
            // 17 PROPERTIES
            this.msgStoreItemMemory.putShort((short) propertiesLength);
            if (propertiesLength > 0)
                this.msgStoreItemMemory.put(propertiesData);

            final long beginTimeMills = CommitLog.this.defaultMessageStore.now();
            // Write messages to the queue buffer
            byteBuffer.put(this.msgStoreItemMemory.array(), 0, msgLen);

            AppendMessageResult result = new AppendMessageResult(AppendMessageStatus.PUT_OK, wroteOffset, msgLen, msgId,
                msgInner.getStoreTimestamp(), queueOffset, CommitLog.this.defaultMessageStore.now() - beginTimeMills);

The above code is very important. The key attributes of the message are as follows

Writeoffset: This is physicsofffset. It is very important to find the key attributes of the message and locate the physical location of the message

Message length: message content

queueoffset: the offset of the queue, which can also be called logical offset. This is increased according to the id atom of the queue

Since the message content is positive, we can find the value of each attribute through physicsofffset, including the message content;

Locate the start position of the message through physicsofffset, and you can get the required data. For the change of each attribute of the message content, the size of the content will be stored before, so that you can intercept the complete message content;

The last is to brush the disk synchronously or asynchronously according to the configuration

After successful addition, unlock and brush the disk. According to the configuration, whether to brush the disk synchronously, because we are still stored in memory at this time to avoid data disappearance, we can configure synchronous disk brushing;

summary

The commitlog is a logically large file. There are many mappedfiles below. Each message has a physicsofffset (physical offset). Later, the offset of the commitlog is stored whether building an index or ConsumeQueue;

With physicsofffset, you can quickly locate the specific message start position. According to the message content setting rules and principles, you can get the attribute contents of the message body, mainly including queueid, queue offset, physicsofffset and message content,

Topics: Middleware MQ