RocketMQ message store - MappedFileQueue

Posted by kingbeastie on Wed, 09 Mar 2022 10:18:08 +0100

RocketMQ message store (III) - MappedFileQueue

The previous article explained the MappedFile class. In fact, its bottom layer is to manage the reading and writing of files by zero copy through MappedByteBuffer.

Since MappedFile is a class for managing individual files, there will be a class for managing these mappedfiles: MappedFileQueue.
We can understand the relationship between them as mappedfile and mappedfilequeue

If you want to analyze MappedFileQueue, it will feel very abstract at first. I've thought about it here. I'd better put the figure at the beginning of the article to get a general impression in my mind. When analyzing the source code later, I can refer to the figure to help understand:

1. Properties

Let's look at the code directly. The old rule is to analyze the attributes first. Here are some core attributes:

    // The directory path managed by the MappedFileQueue 
	//      1. The directory path of commitlog file is:/ store/commit/log
	//      2. The directory path of consumequeue file is:/ store/xxx_topic/x
    private final String storePath;

    // Size of each file in the directory 
	//     1. The commitlog file defaults to 1g 
	//     2. consumeQueue file (600w bytes by default)
    private final int mappedFileSize;

    //  All MappedFile collections managed under the directory 
    private final CopyOnWriteArrayList<MappedFile> mappedFiles = new CopyOnWriteArrayList<MappedFile>();

    // The service for creating MappedFile has its own thread. (MappedFile can be created asynchronously through this class)
    private final AllocateMappedFileService allocateMappedFileService;

    // Brush disk location of directory  
	//    (last MappedFile.fileName + last MappedFile.flushPosition)
    private long flushedWhere = 0;

    // Storage time of the last msg in the current directory
    private volatile long storeTimestamp = 0;

The above attributes are basically very simple. Here we need to emphasize one attribute, flushedWhere. Please understand it in combination with the above picture,

The MappedFile files in the MappedFileQueue directory are written in sequence. When the files are full, go back and create a new MappedFile. The file name of MappedFile is physical offset.

A simple example (for illustration only): assuming that the size of each file is 64bytes, the first file name is 00000. When the file is full, you need to create a second file. Then the file name of the second file is 00064. At this time, you can only write to the second file. Then, when 32bytes is written, the flushed where = 00064 + 00032 = 00096

2. Core methods


     *  broker In the startup phase, load the data used by the local disk.
     *  This method will read the files in the "storePath" directory, create mappedFile objects for the corresponding files, and add them to the List
    public boolean load() {
        // Create directory object
        File dir = new File(this.storePath);

        // Get all files in the directory
        File[] files = dir.listFiles();

        if (files != null) {
            // ascending order
            // Sort by file name
            for (File file : files) {

                if (file.length() != this.mappedFileSize) {
                    log.warn(file + "\t" + file.length()
                        + " length not matched message store config value, please check it manually");
                    return false;

                try {
                    // Create the corresponding mappedFile object for the current File
                    MappedFile mappedFile = new MappedFile(file.getPath(), mappedFileSize);

                    // Set writeposition and flushedPosition (the values given here are mappedFileSize, not exact values. The exact values need to be set in the recover phase)

                    // Add to list
          "load " + file.getPath() + " OK");
                } catch (IOException e) {
                    log.error("load file " + file + " error", e);
                    return false;

        return true;

The above code is simple and easy to understand. First, summarize the main tasks of this method, as follows:

  1. Build a File object according to the specified File directory (e.g... / store/commit/log) (Note: it is a folder).
  2. Traverse all the files in the folder and sort to get the File[] files array (Note: This is a collection of files).
  3. Traverse the sorted file collection, create a MappedFile object for each file and assign an initial value, and then store it in the MappedFiles collection

In Article 3, give MappedFile an initial value. Note: this value is only the initial value and has no effect.

After the normal Broker is started, it will first call the load() method to load all mappedfiles in the directory, and then reassign the accurate values through the relevant methods of recover.

2. getLastMappedFile

This method has three overloaded methods. Directly look at the one with the most parameters.

     * Gets the MappedFile object currently being written sequentially  
     *   (When storing messages or ConsumeQueue data, you need to obtain the MappedFile object currently being written in sequence)
     *   Note: if the MappedFile is full or there is no MappedFile found, a new MappedFile will be created
     * @param startOffset   File start offset  
     * @param needCreate    Create mappedFile when list is empty
     * @return
    public MappedFile getLastMappedFile(final long startOffset, boolean needCreate) {

        // This value controls whether a MappedFile needs to be created. When a MappedFile needs to be created, it acts as the file name
        // Two situations are created:
        //  1. There is no mappedFIle in the list
        //  2. The last mappedFile in the list (mappedFile written in the current order) is full
        long createOffset = -1;

        //  Get the last MappedFile in the list
        MappedFile mappedFileLast = getLastMappedFile();

         // Case 1: there is no mappedFile in the list
        if (mappedFileLast == null) { 
            // The createOffset value must be a multiple of mappedFileSize or 0
            createOffset = startOffset - (startOffset % this.mappedFileSize);

        // Case 2: the last mappedFile in the list (mappedFile written in the current order) is full
        if (mappedFileLast != null && mappedFileLast.isFull()) {  
            // Previous file name to Long + mappedFileSize
            createOffset = mappedFileLast.getFileFromOffset() + this.mappedFileSize;

         // Here is the logic to create a new mappedFile
        if (createOffset != -1 && needCreate) {

            // Get the absolute path of the next file to be created
            String nextFilePath = this.storePath + File.separator + UtilAll.offset2FileName(createOffset);

            // Get the absolute path of the next file to be created
            String nextNextFilePath = this.storePath + File.separator
                + UtilAll.offset2FileName(createOffset + this.mappedFileSize);
            MappedFile mappedFile = null;

            // Create MappedFile using allocateMappedFileService
            if (this.allocateMappedFileService != null) { 
                // When mappedfilesize > = 1g, the mappedFile created here will execute its preheating method
                mappedFile = this.allocateMappedFileService.putRequestAndReturnMappedFile(nextFilePath,
                    nextNextFilePath, this.mappedFileSize);
            // Create MappedFile directly (there is no preheating here)
            else {
                try {
                    mappedFile = new MappedFile(nextFilePath, this.mappedFileSize);
                } catch (IOException e) {
                    log.error("create mappedFile exception", e);

            // Add the created mappedFile to the list and return
            if (mappedFile != null) {
                if (this.mappedFiles.isEmpty()) {

            return mappedFile;
		// Come here Yes, returned when no MappedFile needs to be created.
        return mappedFileLast;

The above code is very long and may be a little difficult to understand.

The first thing to understand is what is the purpose of this method? Gets the MappedFile that is currently being written sequentially

The previously explained flushed where field in the attribute summary is similar to its example. The MappedFile currently being written in sequence must be the last file in the MappedFile set. Therefore, the code directly calls the getLastMappedFile() method to obtain the MappedFile at the end. At this time, there are three situations:

  1. The MappedFile exists and there is still writable space in the MappedFile. (this is also the best case. Just return normally)
  2. The MappedFile exists, but the MappedFile is full. (you need to create a new MappedFile)
  3. The MappedFile does not exist, which means there is no file in the directory. (you need to create a new MappedFile)

In case 2 and 3, a new MappedFile needs to be created, and there are two ways to create MappedFile:

  1. Created using other threads through allocateMappedFileService. (there is preheating operation when mappedfile > = 1g)
  2. It is created in the normal new MappedFile() method. (no preheating operation)

The preheating operation will be explained in the following articles. Here, just understand the literal meaning.

The following is a brief summary of the steps of this method:

  1. Get the last mappedFileLast in the directory
  2. Determine whether to create a new mappedlastfile
    1. If you do not need to create a new MappedFile, you can directly return to mappedFileLast
    2. Whether mappedallocateservice needs to be created according to the existing mappedallocateservice:
      1. allocateMappedFileService has the function of preheating
      2. Normal creation


     * commitLog Directory delete expired file call   
     * @param expiredTime  Expiration time
     * @param deleteFilesInterval  The time interval between deleting two files
     * @param intervalForcibly  Time interval to force resource shutdown MF Parameters passed by destroy
     * @param cleanImmediately  true Forced deletion, regardless of the expiration time
     * @return
    public int deleteExpiredFileByTime(final long expiredTime,
        final int deleteFilesInterval,
        final long intervalForcibly,
        final boolean cleanImmediately) {

        // Get mfs array (in fact, convert MappedFile collection into array) 
        Object[] mfs = this.copyMappedFiles(0);

        if (null == mfs)
            return 0;
        // Minus - 1 here is to ensure that the MappedFile currently being written in sequence will not be deleted
        int mfsLength = mfs.length - 1;

        // Records the number of files deleted
        int deleteCount = 0;

        // Deleted file collection
        List<MappedFile> files = new ArrayList<MappedFile>();
        if (null != mfs) {
            for (int i = 0; i < mfsLength; i++) {
                MappedFile mappedFile = (MappedFile) mfs[i];

                // Calculate the deadline for the survival time of the current file
                long liveMaxTimestamp = mappedFile.getLastModifiedTimestamp() + expiredTime;

                // Conditions are satisfied:
                //     Condition 1: the file survival time reaches the upper limit
                //     Condition 2: when the disk occupancy reaches the upper limit, it will be forcibly deleted
                if (System.currentTimeMillis() >= liveMaxTimestamp || cleanImmediately) {

                    // Delete file
                    if (mappedFile.destroy(intervalForcibly)) {
                        deleteCount++; // Increase delete file count

                        if (files.size() >= DELETE_FILES_BATCH_MAX) {
                        // After deleting the file, you need to sleep, and then delete the next file
                        if (deleteFilesInterval > 0 && (i + 1) < mfsLength) {
                            try {
                            } catch (InterruptedException e) {
                    } else {
                } else {
                    //avoid deleting files in the middle

        // Delete mf files that meet the deletion conditions from the list

        return deleteCount;

Although the above code is long, it is easy to understand. It is to traverse the MappedFile set under the directory, find the MappedFile that meets the deletion conditions, and then call MF Delete with the destroy () method.

Just note that this method is used to delete the CommitLog file.