Get map file schematic

• If the current file is full or the space is less than the current message length, you need to create a new file
• commitlog[1G] files need to be created through allocateMappedFileService
• Build two requests each time [create two commitlog files] and add them to the task pool
• The additional message thread is blocked through the request of the task pool [thread locking tool]
• allocateMappedFileService asynchronously and sequentially obtains task creation files from the task pool
• Complete false value filling [disk loaded into memory]
• Complete memory preheating [prevent page loss caused by memory exchange]
• After creating the file, the thread locking tool countdownlatch() Countdown() wakes up the append message thread

problem	answering question
What creates two files? (nextFilePath and nextNextFilePath)	Because the default size of a commitlog file is 1G. After creating two files, you don't need to complete the memory mapping process to obtain the files again, so as to improve the speed of adding messages
Why do I need to write a false zero?	The created memory mapping file is actually still on the hard disk. You need to fill the memory mapping file with false values to realize the page missing processing at the bottom of the operating system and complete mmap to realize memory mapping
Why do I need mlock	There is memory replacement at the bottom of the operating system. When the memory required by other processes is insufficient, swap may exchange the current memory to the hard disk, mlock locks the current memory, and the operating system replacement is not allowed to improve the performance of append messages

Source code analysis - create a new file

commitlog.asyncPutMessage message append buffer as entry to create file

• step-1: buffer full
• step-2: append the message to the memory corresponding to the commitlog file
• step-3: if the current file is insufficient, add a message to the buffer after creating a new file

public CompletableFuture<PutMessageResult> asyncPutMessage(final MessageExtBrokerInner msg) {
    
    step-1: Buffer full
    if (null == mappedFile || mappedFile.isFull()) {
        // Get a new mappedFile
        mappedFile = this.mappedFileQueue.getLastMappedFile(0); // Mark: NewFile may be cause noise
    }
    
    Append message to commitlog Memory corresponding to file
    result = mappedFile.appendMessage(msg, this.appendMessageCallback);
    switch (result.getStatus()) {
        case PUT_OK:
            step-2: Messages are normally written to the buffer
            break;
        case END_OF_FILE:
            step-3: If the current file is insufficient, create a new file
            mappedFile = this.mappedFileQueue.getLastMappedFile(0);
            ...... Delete other codes
            Message brush to memory
            result = mappedFile.appendMessage(msg, this.appendMessageCallback);
            break;
        ...... Delete other codes
        default:
            beginTimeInLock = 0;
            return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.UNKNOWN_ERROR, result));
    }
}

Source code analysis - getLastMappedFile

• step-1: asynchronous creation: build two file names
• step-2: commitlog creation method. The commitlog size is 1g, and two are created asynchronously at a time
• step-3: create consumequeue and index synchronously, and directly create a mmapedFile

public MappedFile getLastMappedFile(final long startOffset, boolean needCreate) {
    ...... Delete other codes
    if (createOffset != -1 && needCreate) {
        step-1: Asynchronous creation:Build two file names
        Wake up when the first file is created appendmessage thread 
        The second file is created asynchronously to improve the efficiency of obtaining the file next time
        String nextFilePath = this.storePath + File.separator + UtilAll.offset2FileName(createOffset);
        String nextNextFilePath = this.storePath + File.separator
            + UtilAll.offset2FileName(createOffset + this.mappedFileSize);
        MappedFile mappedFile = null;

        if (this.allocateMappedFileService != null) { 
            commitlog Creation method,commitlog Size 1 g,Create 2 asynchronously at a time
            mappedFile = this.allocateMappedFileService.putRequestAndReturnMappedFile(nextFilePath,
                nextNextFilePath, this.mappedFileSize);
        } else {
            consumequeue and index Creation method
            try {
                mappedFile = new MappedFile(nextFilePath, this.mappedFileSize);
            } catch (IOException e) {
                log.error("create mappedFile exception", e);
            }
        }
        ...... Delete other codes
        return mappedFile;
    }  
}

Source code analysis - putRequestAndReturnMappedFile

• step-1: add two requests to the task processing pool. The requestTable processing thread is blocked waiting for wake-up
• step-2: add two requests to the task processing pool requestqueue [sort by commitlog file name. If the file name is small, create the file name first and name it according to offset]
• step-3: the main thread waits for the creation of nextNextFilePath. nextNextFilePath does not need to wait

public MappedFile putRequestAndReturnMappedFile(String nextFilePath, String nextNextFilePath, int fileSize) {
    By default, two mapping file creation requests can be processed
    int canSubmitRequests = 2;
    Recalculation can submit up to several file creation requests,(Generally two)
    if (this.messageStore.getMessageStoreConfig().isTransientStorePoolEnable()) {
        if (this.messageStore.getMessageStoreConfig().isFastFailIfNoBufferInStorePool()
            && BrokerRole.SLAVE != this.messageStore.getMessageStoreConfig().getBrokerRole()) { //if broker is slave, don't fast fail even no buffer in pool
            canSubmitRequests = this.messageStore.getTransientStorePool().availableBufferNums() - this.requestQueue.size();
        }
    }
    Add to task processing pool requestTable Processing thread blocking waiting for wake-up
    AllocateRequest nextReq = new AllocateRequest(nextFilePath, fileSize);
    boolean nextPutOK = this.requestTable.putIfAbsent(nextFilePath, nextReq) == null;
    if (nextPutOK) {
        if (canSubmitRequests <= 0) {
            log.warn("[NOTIFYME]TransientStorePool is not enough, so create mapped file error, " +
                "RequestQueueSize : {}, StorePoolSize: {}", this.requestQueue.size(), this.messageStore.getTransientStorePool().availableBufferNums());
            this.requestTable.remove(nextFilePath);
            return null;
        }
        Add to task processing pool requestQueue[commitlog Sort by file name. If the file name is small, create the file name first offset Name]
        boolean offerOK = this.requestQueue.offer(nextReq);
        if (!offerOK) {
            log.warn("never expected here, add a request to preallocate queue failed");
        }
        canSubmitRequests--;
    }

    handle nextNextReq reach requestTable and requestQueue
    AllocateRequest nextNextReq = new AllocateRequest(nextNextFilePath, fileSize);
    boolean nextNextPutOK = this.requestTable.putIfAbsent(nextNextFilePath, nextNextReq) == null;
    if (nextNextPutOK) {
        if (canSubmitRequests <= 0) {
            log.warn("[NOTIFYME]TransientStorePool is not enough, so skip preallocate mapped file, " +
                "RequestQueueSize : {}, StorePoolSize: {}", this.requestQueue.size(), this.messageStore.getTransientStorePool().availableBufferNums());
            this.requestTable.remove(nextNextFilePath);
        } else {
            boolean offerOK = this.requestQueue.offer(nextNextReq);
            if (!offerOK) {
                log.warn("never expected here, add a request to preallocate queue failed");
            }
        }
    }

    if (hasException) {
        log.warn(this.getServiceName() + " service has exception. so return null");
        return null;
    }
    Main thread waiting nextFilePath Creation complete
    AllocateRequest result = this.requestTable.get(nextFilePath);
    try {
        if (result != null) {
            adopt CountDownLatch Complete information communication between threads
            boolean waitOK = result.getCountDownLatch().await(waitTimeOut, TimeUnit.MILLISECONDS);
            if (!waitOK) {
                log.warn("create mmap timeout " + result.getFilePath() + " " + result.getFileSize());
                return null;
            } else {
                this.requestTable.remove(nextFilePath);
                return result.getMappedFile();
            }
        } else {
            log.error("find preallocate mmap failed, this never happen");
        }
    } catch (InterruptedException e) {
        log.warn(this.getServiceName() + " service has exception. ", e);
    }

    return null;
}

Source code analysis - allocatemappedfileservice run

• The DefaultMessageStore constructor starts the AllocateMappedFileService thread
• Spin get request, there is no blocking, otherwise create a file

public void run() {
    log.info(this.getServiceName() + " service started");
    while (!this.isStopped() && this.mmapOperation()) {
    }
    log.info(this.getServiceName() + " service end");
}   

Source code analysis - mmapOperation

• step-1: priority blocking queue, get file creation request [no element blocking in queue]
• step-2: whether to allow out of heap memory. This mechanism will be used only when the disk is flushed asynchronously and the out of heap memory is started
• step-2.1: creation method of off heap memory
• step-2.2: if there is no out of heap memory, mmap memory mapping
• step-3: memory preheating and mlock

  private boolean mmapOperation() {
    boolean isSuccess = false;
    AllocateRequest req = null;
    try {
        step-1: Priority blocking queue,Create request[The queue has no element blocking]
        req = this.requestQueue.take();
        AllocateRequest expectedRequest = this.requestTable.get(req.getFilePath());
        ...... Delete other codes
        if (req.getMappedFile() == null) {
            long beginTime = System.currentTimeMillis();

            MappedFile mappedFile;
            step-2: Whether to allow out of heap memory,This mechanism is required only when the disk is flushed asynchronously and the off heap memory is started
            if (messageStore.getMessageStoreConfig().isTransientStorePoolEnable()) {
                step-2.1: Off heap memory creation method
                mappedFile = ServiceLoader.load(MappedFile.class).iterator().next();
                mappedFile.init(req.getFilePath(), req.getFileSize(), messageStore.getTransientStorePool());
                ...... Delete other codes
            } else {
                step-2.2: No off heap memory,be mmap Memory mapping
                mappedFile = new MappedFile(req.getFilePath(), req.getFileSize());
            }
            ...... Delete other codes
            if (mappedFile.getFileSize() >= this.messageStore.getMessageStoreConfig()
                .getMappedFileSizeCommitLog()
                &&
                this.messageStore.getMessageStoreConfig().isWarmMapedFileEnable()) {
                step-3: Memory preheating and mlock
                Write a false value of 0 to warm up, write a false value, and then the operating system finds it os page Missing pages to read physical disk data to memory
                mappedFile.warmMappedFile(this.messageStore.getMessageStoreConfig().getFlushDiskType(),
                    this.messageStore.getMessageStoreConfig().getFlushLeastPagesWhenWarmMapedFile());
            }
            ......Delete other codes
        }
    ......Delete other codes
    }finally {
        Wake up add message thread
        if (req != null && isSuccess)
            req.getCountDownLatch().countDown();
    }
    return true;
}

Source code analysis a warmMappedFile

Preheat the 1G commitlog, write a false value, give up the cpu appropriately, and then prevent swap through mlock

ps: so far, the author has not found out why to fill in the false value 0 instead of 1 or other official instructions. Readers can leave a message if they know

public void warmMappedFile(FlushDiskType type, int pages) {
    long beginTime = System.currentTimeMillis();
    ByteBuffer byteBuffer = this.mappedByteBuffer.slice();
    int flush = 0;
    long time = System.currentTimeMillis();
    Right 1 G of commitlog Warm up, write false value and give up appropriately cpu Then pass mlock prevent swap
    for (int i = 0, j = 0; i < this.fileSize; i += MappedFile.OS_PAGE_SIZE, j++) {
         Allocate memory only and call mlock This memory is not locked for the calling process, because the corresponding pages may be copied on write( copy-on-write)5. Therefore, you should write a false value in each page
         That is, only allocation, but memory mapping has not been performed

        /**
         * Copy-on-write
         * Why is 0 not 1 or another value
         * renxl: My personal understanding is that no matter it is 0, 1 or other values, it can achieve memory mapping and physical memory allocation to prevent page missing
         * However, 0 will not change its original value relatively, which is more reasonable
         * The virtual address space of the process is established, and the physical memory corresponding to the virtual memory is not allocated
         *
         *
         * When using mmap() memory allocation, only the process virtual address space is established, and the physical memory corresponding to the virtual memory is not allocated. When the process accesses these virtual memory without mapping relationship, the processor will automatically trigger a page missing exception, and then enter the kernel space to allocate physical memory, update the process cache table, and finally return to the user space to reply to the process running
         */
        byteBuffer.put(i, (byte) 0);
        // force flush when flush disk type is sync
        if (type == FlushDiskType.SYNC_FLUSH) {
            if ((i / OS_PAGE_SIZE) - (flush / OS_PAGE_SIZE) >= pages) {
                flush = i;
                mappedByteBuffer.force();
            }
        }

        Give up voluntarily cpu 
        if (j % 1000 == 0) {
            log.info("j={}, costTime={}", j, System.currentTimeMillis() - time);
            time = System.currentTimeMillis();
            try {
                Thread.sleep(0);
            } catch (InterruptedException e) {
                log.error("Interrupted", e);
            }
        }
    }

    Brush disc
    if (type == FlushDiskType.SYNC_FLUSH) {
        log.info("mapped file warm-up done, force to disk, mappedFile={}, costTime={}",
            this.getFileName(), System.currentTimeMillis() - beginTime);
        mappedByteBuffer.force();
    }
    adopt mlock Avoid memory being by the operating system swap
    this.mlock();
}

mlock

• Call the mlock function of c language to complete memory locking
• Prevent the memory from being replaced due to insufficient memory or other conditions, resulting in the next memory page shortage. The operating system needs IO from the disk

public void mlock() {
    final long beginTime = System.currentTimeMillis();
    final long address = ((DirectBuffer) (this.mappedByteBuffer)).address();
    Pointer pointer = new Pointer(address);
    {
        // Position + length
        int ret = LibC.INSTANCE.mlock(pointer, new NativeLong(this.fileSize));
        log.info("mlock {} {} {} ret = {} time consuming = {}", address, this.fileName, this.fileSize, ret, System.currentTimeMillis() - beginTime);
    }

    {
        int ret = LibC.INSTANCE.madvise(pointer, new NativeLong(this.fileSize), LibC.MADV_WILLNEED);
        log.info("madvise {} {} {} ret = {} time consuming = {}", address, this.fileName, this.fileSize, ret, System.currentTimeMillis() - beginTime);
    }
}

summary

• hj