1, MP4 file format analysis
1. General introduction to video files
(1) Essence of video file: record the compressed video frame and can be restored, decoded and played by the player (the key is to find the I frame and P frame of the video)
(2) The key of video files: efficient recording of information and compatibility
(3) Information of video file: index information, valid information
2. General introduction to MP4 format
(1)MP4 is upgraded from MP3 (pure audio), including video and audio
(2)MP4 is the most popular packaging format of h.264
(3) The internal * * of MP4 file adopts network byte order (big end mode)**
3. MP4 learning route
(1) Learn the organizational form and box analysis of MP4
(2) Porting and using mp4v2 open source library to package MP4
(3) Further study MP4 unpacking playback and mp4v2 source code
(4) Own programming for MP4 packaging, unpacking, segmentation, etc.
4. Officially start learning the organizational form of MP4
Reference study (required): https://www.cnblogs.com/ranson7zop/p/7889272.html
the packaging format of MP4 video file is defined based on the QuickTime container format. Therefore, referring to the format definition of QuickTime is very helpful to understand the MP4 file format. MP4 file format is a very open container, which can be used to describe almost all media structures. The media description in MP4 file is separated from media data, and the organization of media data is also very free. It does not have to be arranged in chronological order, and even media data can directly reference other files. At the same time, MP4 also supports streaming media (downloading and playing through the network). At present, MP4 is widely used to package h.264 video and AAC audio. It is the representative of HD video.
(1) The entire MP4 file is composed of several different boxes, which are packed and unpacked in boxes
(2) There is only one ftyp box (file type box) in MP4, which is located at the beginning of the whole MP4
2, MP4Info tool usage
Software and data links used:
Link: https://pan.baidu.com/s/1DFbZPI4ch_y33RUAE6whiA Extraction code: pu15 --From Baidu online disk super member V5 Sharing
Step 1: open the software
Step 2: use the software to open an MP4 video file, and the following situations will appear:
The information in the right half of the figure is obtained by analyzing the box.
The software can't open too large video files. It can only open some small video files for analysis.
3, mp4v2 porting and playing practice
mp4v2 library is an open source project dedicated to processing mp4 containers. It is written in c + + and provides c language interface.
1. Download mp4v2
(1)https://launchpad.net/ubuntu/+source/mp4v2/2.0.0~dfsg0-6
(2) Unzip and create in the directory_ install directory as the installation directory
2. Configure and compile
(1)Export environment variables so that the cross compilation chain can be found: sudo PATH=$PATH:/opt/hisi-linux/x86-arm/arm-hisiv300-linux/target/bin CC=arm-hisiv300-linux-gcc CXX=arm-hisiv300-linux-gcc Configure: ./configure --host=arm-hisiv300-linux --prefix=/home/aston/sambashare/mp4v2-2.0.0/_install --disable-option-checking --disable-debug --disable-optimize --disable-fvisibility --disable-gch --disable-largefile --disable-util --disable-dependency-tracking --disable-libtool-lock Note:_install This folder needs to be created manually, and the final compiled files will be stored in this folder (2)make (3)make install (4)Check all necessary documents
3. Deploy
(1) The generated lib is added to mpp lib
(2) Add the generated file to mpp/include in Hisilicon SDK
(3) The generated dynamic library (* so *) of lib / is added to the lib directory of the development board (the directory mounted to the host through the development board)
in case of insufficient storage space, some unused files (such as usb network card driver) can be deleted or placed in the / mnt directory mounted on the development board.
cp ~/sambashare/mp4v2-2.0.0/_install/lib/* ./ cp ~/sambashare/mp4v2-2.0.0/_install/include/* ./ -rf
4. Compile sample
(1) Replace the original sample with the provided sample folder, or give the provided folder a new name sample_mp4
Link: https://pan.baidu.com/s/1LMNPxBzWeI-94L8zKCVhpA Extraction code: gu90 --From Baidu online disk super member V5 Sharing
(2)make
6. Prepare TF Card
Reference learning (Understanding expansion): https://blog.csdn.net/li_wen01/article/details/79929730
(1)TF card is formatted as FAT32 file system. If it fails, try low-level formatting software such as SDFormat
(2) Mount the TF card after power on
mount -t vfat /dev/mmcblk0p1 /usr/mmc
if the memory card is too large, such as 16G, it may not work. The partition cannot be recognized and cannot work. I used an 8G and succeeded.
(3) First check that the TF card is available, such as creating a file on the computer and reading and writing on the development board.
7. Operation and testing
(1) Run the sample generated by make compilation in the above steps_ Venc executable file, using rtsp method to test and confirm that there are images
(2) Terminate the program, take out the TF card, connect it to the computer with the card reader, and check it through the video playback software of the computer
Note: during the function test, rtsp did not work normally, but mp4 video packaging was successful. It is analyzed and speculated that the reason may be that too much information was printed during video packaging, which affected the normal operation of rtsp.
4, Analysis of MP4 packaging source code
Source code: that is, the network disk link above. Analyze all the files in the mpp file through the sourceinsight software establishment project.
Reference learning: https://blog.csdn.net/weixin_42462202/article/details/90108485
HI_S32 SAMPLE_COMM_VENC_MP4(VENC_STREAM_S *stStream)//This function is only responsible for packaging one frame of data at a time
{
static int nRecordFlag = 0x00;//The values of these static variables are initialized only the first time the program runs
static int recording = 0x1; //For each subsequent call, the last retained value will be used
static int spsflag = 0;
static int ppsflag = 0;
static MP4TrackId video = 0;
static MP4FileHandle hMP4File = NULL;
static char recordfish = 0x1;
int j = 0;
int len = 0;
char *pData = NULL;
char isSyncSample = 0;
if(recordfish == 0x00){
return 0;
}
if(hMP4File == NULL){//The first time I entered
hMP4File = MP4CreateEx("/usr/mmc/test.mp4",0, 1, 1, 0, 0, 0, 0);//Create an mp4 file
if (hMP4File == MP4_INVALID_FILE_HANDLE) {
printf("open file fialed.\n");
return -1;
}
MP4SetTimeScale(hMP4File, 90000);
}
//Recording: indicates recording and packaging. A sequence consists of 1sps + 1PPS + SEI + 1 I frame + 29P frame
if(recording && stStream->u32Seq > 30){ //The first 30 frames can be discarded or not to ensure that they start from I frame
if(stStream->u32PackCount >= 3){//sps,pps,sei, start coding from I frame to ensure that the file can be played from the beginning
nRecordFlag = 1;
}
if(nRecordFlag){
for(j = 0;j < stStream->u32PackCount;j++){
len = stStream->pstPack[j].u32Len - stStream->pstPack[j].u32Offset;
pData = (stStream->pstPack[j].pu8Addr + stStream->pstPack[j].u32Offset);
if(stStream->pstPack[j].DataType.enH264EType == H264E_NALU_SPS){
if(spsflag == 0x00){
spsflag = 0x1;
//Write sps
printf("Write sps =================\n");
//Adding a track is equivalent to adding a flow
video = MP4AddH264VideoTrack(hMP4File, 90000, 90000 / 30, 1280, 720,
pData[4+1], //sps[1] AVCProfileIndication
pData[4+2], //sps[2] profile_compat
pData[4+3], //sps[3] AVCLevelIndication
3); // 4 bytes length before each NAL unit
MP4SetVideoProfileLevel(hMP4File, 0x7F);//Video level
MP4AddH264SequenceParameterSet(hMP4File, video, pData+4, len-4); //Add sps to mp4 file
}
continue;
}
if(stStream->pstPack[j].DataType.enH264EType == H264E_NALU_PPS){
if(ppsflag == 0x00){
ppsflag = 0x1;
//Add pps
printf("Write pps -------------------\n");
MP4AddH264PictureParameterSet(hMP4File, video, pData+4, len-4);//Add pps to mp4 file
}
continue;
}
isSyncSample = (stStream->pstPack[j].DataType.enH264EType == H264E_NALU_ISLICE) ? (1) : (0);
pData[0] = (len - 4) >> 24;
pData[1] = (len - 4) >> 16;
pData[2] = (len - 4) >> 8;
pData[3] = len - 4;
printf("Write date type = %d isSyncSample = %d\n",stStream->pstPack[j].DataType.enH264EType,isSyncSample);
MP4WriteSample(hMP4File, video, pData, len , MP4_INVALID_DURATION, 0, isSyncSample);//Write nalu
}
}
}
if((recording && stStream->u32Seq > 900)){//Control file duration
recording = 0x00;
printf("Close mp4 file\n");
MP4Close(hMP4File, 0);//Close the mp4 file. This function must be called after writing the mp4 file, otherwise the mp4 file will be damaged
hMP4File = NULL;
video = 0;
recordfish = 0x00;
}
}
5, Learning analysis of mp4v2 combined with MP4Info
1. Train of thought
(1) Modify the MP4 packaging source code and use MP4Info to view the recorded MP4 details
(2) Further depth: modify the details in the mp4v2 source code, compile and execute, and then package the MP4 video for viewing and analysis
2. Practice 1: remove sps
through the test, the video VLC player without sps and other film and television players (Tencent, etc.) cannot be played.
3. Practice 2: remove pps
through the test, the video VLC player without pps and other film and television players (Tencent, etc.) cannot be played.
From top to bottom in the figure is the complete MP4 video, remove sps and pps.
6, Add network telnet debugging
1. Why add telnet debugging
(1) The user interface of embedded linux system is commandline, which is essentially provided by busybox
(2)busybox has only one command line. Once the foreground is occupied, other operations cannot be performed
(3) There are two solutions: one is to establish multiple commandline s, and the other is to open other user interfaces.
2. Principle of telnet debugging
(1) Run telnetd in advance in the development board (this software has been brought with it in the system provided by Hisilicon)
telnet: client program, which is used by the development board to log in to other servers remotely
telnetd: server program, which is a daemon used by other clients to log in to the development board
(2) Connect to the server remotely through the client of telnet to build a user interface
(3) This is a very traditional and typical way of remote login... In fact, it has been used
3. telnet remote login debugging practice on HI3518E development board
(1) The command line executes telnetd &, and then Windows opens SecureCRT (other software with the same function can also be used, such as MobaXterm) to configure SSH to 192.168.1.10, that is, the network port of the development board
(2) Enter in CRT interface:
user name ssid: root password: Because my development board doesn't have a password,Enter directly
(3) Problem: in case of continuous restart, move the add-on in etc/profile to / etc / init In the rcS file in D. This is because opening the telnet service will load the profile many times (unable to mount again after the last mount)
profile file:
You can move these contents to / etc / init In the rcS file in D.
(4) Telnetd & can be added to rcS and loaded by default after startup
7, Hisilicon proc file system debugging interface (now mostly replaced by sys file system)
1. Principle of proc file system
Reference learning:
https://www.cnblogs.com/lidabo/p/5628020.html
https://blog.csdn.net/weixin_29202687/article/details/116963940
2. Documentation of Hisilicon proc file system debugging
Link: https://pan.baidu.com/s/1zDDRvTl3gx4GhHE4kgx-zA Extraction code: pfp8 --From Baidu online disk super member V5 Sharing
3. Additional commissioning experience documents
Link: https://pan.baidu.com/s/1Eu3LmkO3-y3YDsEARnikbw Extraction code: oifi --From Baidu online disk super member V5 Sharing
Note: This article refers to the course notes of Mr. Zhu's Internet of things lecture hall, combined with his own actual development experience and the technical articles of others on the Internet. If there is infringement, contact to delete! Limited level, welcome to communicate in the comment area.