RocketMQ message store (III) - MappedFileQueue
The previous article explained the MappedFile class. In fact, its bottom layer is to manage the reading and writing of files by zero copy through MappedByteBuffer.
Since MappedFile is a class for managing individual files, there will be a class for managing these mappedfiles: MappedFileQueue.
We can understand the relationship between them as mappedfile and mappedfilequeue
If you want to analyze MappedFileQueue, it will feel very abstract at first. I've thought about it here. I'd better put the figure at the beginning of the article to get a general impression in my mind. When analyzing the source code later, I can refer to the figure to help understand:
1. Properties
Let's look at the code directly. The old rule is to analyze the attributes first. Here are some core attributes:
// The directory path managed by the MappedFileQueue
// 1. The directory path of commitlog file is:/ store/commit/log
// 2. The directory path of consumequeue file is:/ store/xxx_topic/x
private final String storePath;
// Size of each file in the directory
// 1. The commitlog file defaults to 1g
// 2. consumeQueue file (600w bytes by default)
private final int mappedFileSize;
// All MappedFile collections managed under the directory
private final CopyOnWriteArrayList<MappedFile> mappedFiles = new CopyOnWriteArrayList<MappedFile>();
// The service for creating MappedFile has its own thread. (MappedFile can be created asynchronously through this class)
private final AllocateMappedFileService allocateMappedFileService;
// Brush disk location of directory
// (last MappedFile.fileName + last MappedFile.flushPosition)
private long flushedWhere = 0;
// Storage time of the last msg in the current directory
private volatile long storeTimestamp = 0;
The above attributes are basically very simple. Here we need to emphasize one attribute, flushedWhere. Please understand it in combination with the above picture,
The MappedFile files in the MappedFileQueue directory are written in sequence. When the files are full, go back and create a new MappedFile. The file name of MappedFile is physical offset.
A simple example (for illustration only): assuming that the size of each file is 64bytes, the first file name is 00000. When the file is full, you need to create a second file. Then the file name of the second file is 00064. At this time, you can only write to the second file. Then, when 32bytes is written, the flushed where = 00064 + 00032 = 00096
2. Core methods
1.load
/**
* broker In the startup phase, load the data used by the local disk.
* This method will read the files in the "storePath" directory, create mappedFile objects for the corresponding files, and add them to the List
*/
public boolean load() {
// Create directory object
File dir = new File(this.storePath);
// Get all files in the directory
File[] files = dir.listFiles();
if (files != null) {
// ascending order
// Sort by file name
Arrays.sort(files);
for (File file : files) {
if (file.length() != this.mappedFileSize) {
log.warn(file + "\t" + file.length()
+ " length not matched message store config value, please check it manually");
return false;
}
try {
// Create the corresponding mappedFile object for the current File
MappedFile mappedFile = new MappedFile(file.getPath(), mappedFileSize);
// Set writeposition and flushedPosition (the values given here are mappedFileSize, not exact values. The exact values need to be set in the recover phase)
mappedFile.setWrotePosition(this.mappedFileSize);
mappedFile.setFlushedPosition(this.mappedFileSize);
mappedFile.setCommittedPosition(this.mappedFileSize);
// Add to list
this.mappedFiles.add(mappedFile);
log.info("load " + file.getPath() + " OK");
} catch (IOException e) {
log.error("load file " + file + " error", e);
return false;
}
}
}
return true;
}
The above code is simple and easy to understand. First, summarize the main tasks of this method, as follows:
- Build a File object according to the specified File directory (e.g... / store/commit/log) (Note: it is a folder).
- Traverse all the files in the folder and sort to get the File[] files array (Note: This is a collection of files).
- Traverse the sorted file collection, create a MappedFile object for each file and assign an initial value, and then store it in the MappedFiles collection
In Article 3, give MappedFile an initial value. Note: this value is only the initial value and has no effect.
After the normal Broker is started, it will first call the load() method to load all mappedfiles in the directory, and then reassign the accurate values through the relevant methods of recover.
2. getLastMappedFile
This method has three overloaded methods. Directly look at the one with the most parameters.
/**
* Gets the MappedFile object currently being written sequentially
* (When storing messages or ConsumeQueue data, you need to obtain the MappedFile object currently being written in sequence)
* Note: if the MappedFile is full or there is no MappedFile found, a new MappedFile will be created
*
* @param startOffset File start offset
* @param needCreate Create mappedFile when list is empty
* @return
*/
public MappedFile getLastMappedFile(final long startOffset, boolean needCreate) {
// This value controls whether a MappedFile needs to be created. When a MappedFile needs to be created, it acts as the file name
// Two situations are created:
// 1. There is no mappedFIle in the list
// 2. The last mappedFile in the list (mappedFile written in the current order) is full
long createOffset = -1;
// Get the last MappedFile in the list
MappedFile mappedFileLast = getLastMappedFile();
// Case 1: there is no mappedFile in the list
if (mappedFileLast == null) {
// The createOffset value must be a multiple of mappedFileSize or 0
createOffset = startOffset - (startOffset % this.mappedFileSize);
}
// Case 2: the last mappedFile in the list (mappedFile written in the current order) is full
if (mappedFileLast != null && mappedFileLast.isFull()) {
// Previous file name to Long + mappedFileSize
createOffset = mappedFileLast.getFileFromOffset() + this.mappedFileSize;
}
// Here is the logic to create a new mappedFile
if (createOffset != -1 && needCreate) {
// Get the absolute path of the next file to be created
String nextFilePath = this.storePath + File.separator + UtilAll.offset2FileName(createOffset);
// Get the absolute path of the next file to be created
String nextNextFilePath = this.storePath + File.separator
+ UtilAll.offset2FileName(createOffset + this.mappedFileSize);
MappedFile mappedFile = null;
// Create MappedFile using allocateMappedFileService
if (this.allocateMappedFileService != null) {
// When mappedfilesize > = 1g, the mappedFile created here will execute its preheating method
mappedFile = this.allocateMappedFileService.putRequestAndReturnMappedFile(nextFilePath,
nextNextFilePath, this.mappedFileSize);
}
// Create MappedFile directly (there is no preheating here)
else {
try {
mappedFile = new MappedFile(nextFilePath, this.mappedFileSize);
} catch (IOException e) {
log.error("create mappedFile exception", e);
}
}
// Add the created mappedFile to the list and return
if (mappedFile != null) {
if (this.mappedFiles.isEmpty()) {
mappedFile.setFirstCreateInQueue(true);
}
this.mappedFiles.add(mappedFile);
}
return mappedFile;
}
// Come here Yes, returned when no MappedFile needs to be created.
return mappedFileLast;
}
The above code is very long and may be a little difficult to understand.
The first thing to understand is what is the purpose of this method? Gets the MappedFile that is currently being written sequentially
The previously explained flushed where field in the attribute summary is similar to its example. The MappedFile currently being written in sequence must be the last file in the MappedFile set. Therefore, the code directly calls the getLastMappedFile() method to obtain the MappedFile at the end. At this time, there are three situations:
- The MappedFile exists and there is still writable space in the MappedFile. (this is also the best case. Just return normally)
- The MappedFile exists, but the MappedFile is full. (you need to create a new MappedFile)
- The MappedFile does not exist, which means there is no file in the directory. (you need to create a new MappedFile)
In case 2 and 3, a new MappedFile needs to be created, and there are two ways to create MappedFile:
- Created using other threads through allocateMappedFileService. (there is preheating operation when mappedfile > = 1g)
- It is created in the normal new MappedFile() method. (no preheating operation)
The preheating operation will be explained in the following articles. Here, just understand the literal meaning.
The following is a brief summary of the steps of this method:
- Get the last mappedFileLast in the directory
- Determine whether to create a new mappedlastfile
- If you do not need to create a new MappedFile, you can directly return to mappedFileLast
- Whether mappedallocateservice needs to be created according to the existing mappedallocateservice:
- allocateMappedFileService has the function of preheating
- Normal creation
3.deleteExpiredFileByTime
/**
* commitLog Directory delete expired file call
* @param expiredTime Expiration time
* @param deleteFilesInterval The time interval between deleting two files
* @param intervalForcibly Time interval to force resource shutdown MF Parameters passed by destroy
* @param cleanImmediately true Forced deletion, regardless of the expiration time
* @return
*/
public int deleteExpiredFileByTime(final long expiredTime,
final int deleteFilesInterval,
final long intervalForcibly,
final boolean cleanImmediately) {
// Get mfs array (in fact, convert MappedFile collection into array)
Object[] mfs = this.copyMappedFiles(0);
if (null == mfs)
return 0;
// Minus - 1 here is to ensure that the MappedFile currently being written in sequence will not be deleted
int mfsLength = mfs.length - 1;
// Records the number of files deleted
int deleteCount = 0;
// Deleted file collection
List<MappedFile> files = new ArrayList<MappedFile>();
if (null != mfs) {
for (int i = 0; i < mfsLength; i++) {
MappedFile mappedFile = (MappedFile) mfs[i];
// Calculate the deadline for the survival time of the current file
long liveMaxTimestamp = mappedFile.getLastModifiedTimestamp() + expiredTime;
// Conditions are satisfied:
// Condition 1: the file survival time reaches the upper limit
// Condition 2: when the disk occupancy reaches the upper limit, it will be forcibly deleted
if (System.currentTimeMillis() >= liveMaxTimestamp || cleanImmediately) {
// Delete file
if (mappedFile.destroy(intervalForcibly)) {
files.add(mappedFile);
deleteCount++; // Increase delete file count
if (files.size() >= DELETE_FILES_BATCH_MAX) {
break;
}
// After deleting the file, you need to sleep, and then delete the next file
if (deleteFilesInterval > 0 && (i + 1) < mfsLength) {
try {
Thread.sleep(deleteFilesInterval);
} catch (InterruptedException e) {
}
}
} else {
break;
}
} else {
//avoid deleting files in the middle
break;
}
}
}
// Delete mf files that meet the deletion conditions from the list
deleteExpiredFile(files);
return deleteCount;
}
Although the above code is long, it is easy to understand. It is to traverse the MappedFile set under the directory, find the MappedFile that meets the deletion conditions, and then call MF Delete with the destroy () method.
Just note that this method is used to delete the CommitLog file.