1, Short video content production
The generation of high-quality short video content depends on the collection of short video and special effect editing, which requires the use of basic functions such as beauty, mixing, filter, speed change, picture and video mixing and clipping, subtitles and so on in the development of short video source code. On the basis of these functions, preprocessing is carried out, combined with OpenGL, AI and AR technology to produce many interesting dynamic stickers, Make short video content more creative.

The general implementation process of video recording is to collect the most original Camera picture and sound by Camera and AudioRecord, and then carry out pre-processing such as filter and noise reduction on the collected data. After processing, MediaCodec carries out hardware coding, and finally MediaMuxer is used to generate the final MP4 file.

2, Short video processing and playback
The processing and playing of video is mainly the experience of video definition and viewing fluency. In this regard, the "narrowband HD" technology can be used to save bit rate and provide a clearer viewing experience. After testing, it can save up to 20-40% bandwidth under the same video quality. In addition to bandwidth, the storage and CDN optimization of short video content are also particularly important in the development of short video source code. Generally, the content we need to upload to the cloud storage server is short video content and cover content.

What CDN optimization brings to the development of short video source code is a further experience of short video first loading and circular playback. For example, to solve the problem of slow premiere, for example, the player supports QUIC protocol, and the scheduling based on CDN can make the success rate of short video playing for the first time reach 98%. In addition, it can also be cached while playing during circular playback, so users don't need to consume stream when watching a short video repeatedly.

3, How to record video
In the Android system, if an Android device is needed to obtain a video file such as MP4, there are three mainstream methods: MediaRecorder, MediaCodec+MediaMuxer and FFmpeg.

MediaRecorder: it is a recording class directly provided to us by Android system. It is used to develop short video source code and record audio and video. It is simple and convenient. It does not need to ignore the intermediate recording process. After recording, you can directly get audio files for playback. The recorded audio files are compressed and need to set encoder, The recorded audio files can be played with the player provided by the system.

Advantages: most of them are integrated, just call relevant interfaces directly, small amount of code, simple and stable;

Disadvantages: unable to process audio in real time; The output audio format is not many.

MediaCodec+MediaMuxer: the combination of MediaCodec and MediaMuxer can also realize the function of short video source development and recording. MediaCodec is the codec class provided by Android, and MediaMuxer is the multiplexing class (generating video files). From the perspective of ease of use, it is certainly not as good as MediaRecorder, but it allows us to perform more flexible operations, such as adding watermarks to recorded videos.

Advantages: like MediaRecorder, it has low power consumption, fast speed and more flexibility

Disadvantages: limited supported formats and compatibility problems

FFmpeg:   FFmpeg (Fast forword mpeg, audio and video converter) is an open source, free and cross platform video and audio streaming solution. It provides a complete solution for recording / audio and video codec, conversion and streaming audio and video. The main function is to de protocol, de encapsulate, decode and transcode multimedia data in the development of short video source code

Advantages: very flexible and powerful format support;

Disadvantages: the audio and video codec program in C language is not very convenient to use.

Although FFmpeg is the best from the data, we have to eliminate this first because its ease of use is the worst; Secondly, MediaRecorder also needs to be excluded, so I recommend MediaCodec+MediaMuxer here.

4, Encoder parameters
Bit rate: the number of data bits transmitted per unit time during data transmission, kbps: kilobits per second. The bit rate is directly proportional to the quality and the volume of the file. If the bit rate exceeds a certain value, it has little impact on the quality of the image.

Frames: how many frames are displayed per second, fps

Key frame interval: in H.264 coding, there are many kinds of compressed image data output after coding, which can be simply divided into key frames and non key frames. The key frame can be decoded independently and regarded as the product of image compression. The non key frame contains the "difference" information with other frames, which can also be called "reference frame". Its decoding needs to refer to the key frame to decode an image. Non key frames have higher compression ratio.

5, Use of MediaCodec+MediaMuxer
Mediamuxer and mediacodec, their references http://developer.android.com/reference/android/media/MediaMuxer.html and http://developer.android.com/reference/android/media/MediaCodec.html , there is a frame used inside. This combination can realize many functions, such as editing audio and video files (combined with MediaExtractor), drawing Surface with OpenGL and generating mp4 files, screen recording and similar recording functions in Camera app (although MediaRecorder is more suitable).

One of them generates video and the other generates audio. Here, combine them to generate audio and video at the same time. The basic framework and process of short video source code development are as follows:

The first is the recording thread, mainly referring to hwencoder experiments. Receive the sampling data from the microphone through the AudioRecord class, and then throw it to the Encoder for coding:

AudioRecord audio_recorder; 
audio_recorder = new AudioRecord(MediaRecorder.AudioSource.MIC, 
 SAMPLE_RATE, CHANNEL_CONFIG, AUDIO_FORMAT, buffer_size); 
// ... 
audio_recorder.startRecording(); 
while (is_recording) { 
 byte[] this_buffer = new byte[frame_buffer_size]; 
 read_result = audio_recorder.read(this_buffer, 0, frame_buffer_size); // read audio raw data 
 // ... 
 presentationTimeStamp = System.nanoTime() / 1000; 
 audioEncoder.offerAudioEncoder(this_buffer.clone(), presentationTimeStamp); // feed to audio encoder 
 
} 

Here, you can also set the callback of AudioRecord (through setRecordPositionUpdateListener()) to trigger the reading of audio data. offerAudioEncoder() mainly sends audio sampling data into the InputBuffer of audio MediaCodec for encoding:

ByteBuffer[] inputBuffers = mAudioEncoder.getInputBuffers(); 
int inputBufferIndex = mAudioEncoder.dequeueInputBuffer(-1); 
if (inputBufferIndex >= 0) { 
 ByteBuffer inputBuffer = inputBuffers[inputBufferIndex]; 
 inputBuffer.clear(); 
 inputBuffer.put(this_buffer); 
 ... 
 mAudioEncoder.queueInputBuffer(inputBufferIndex, 0, this_buffer.length, presentationTimeStamp, 0); 
} 

Next, refer to grafika softinputsurfaceactivity and add audio processing. The main cycle is generally divided into four parts:

try { 
 // Part 1 
 prepareEncoder(outputFile); 
 ... 
 // Part 2 
 for (int i = 0; i < NUM_FRAMES; i++) { 
 generateFrame(i); 
 drainVideoEncoder(false); 
 drainAudioEncoder(false); 
 } 
 // Part 3 
 ... 
 drainVideoEncoder(true); 
 drainAudioEncoder(true); 
} catch (IOException ioe) { 
 throw new RuntimeException(ioe); 
} finally { 
 // Part 4 
 releaseEncoder(); 
} 
 

Part 1 is the preparation. In addition to the MediaCodec of video, the MediaCodec of audio is also initialized here:

MediaFormat audioFormat = new MediaFormat(); 
audioFormat.setInteger(MediaFormat.KEY_SAMPLE_RATE, 44100); 
audioFormat.setInteger(MediaFormat.KEY_CHANNEL_COUNT, 1); 
... 
mAudioEncoder = MediaCodec.createEncoderByType(AUDIO_MIME_TYPE); 
mAudioEncoder.configure(audioFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE); 
mAudioEncoder.start(); 

Part 2 enters the main loop, and the app draws directly on the Surface. Since this Surface is applied from MediaCodec with createInputSurface(), there is no need to explicitly use queueInputBuffer() to give it to the Encoder after drawing. drainVideoEncoder() and drainAudioEncoder() respectively take the encoded audio and video from the buffer (through dequeueoutputbuffer ()), and then hand them over to MediaMuxer for mixing (through writeSampleData()). Note that audio and video are synchronized through PTS (Presentation time stamp, which determines when the audio and video data of a frame is displayed or played). The time stamp of audio needs to be obtained when AudioRecord collects data from MIC and put into the corresponding bufferInfo. Since the video is drawn on the Surface, it is OK to directly use the bufferInfo from dequeueOutputBuffer(), Finally, the encoded data is sent to MediaMuxer for multi-channel mixing.

Note that Muxer will not start until audio track and video track are added. MediaCodec will return info once when calling dequeueOutputBuffer() at the beginning_ OUTPUT_ FORMAT_ Changed message. We just need to get the format of the MediaCodec here and register it in MediaMuxer. Then judge whether the current audio track and video track are ready. If so, start Muxer.

To sum up, the main logic of drainVideoEncoder() is roughly as follows. drainAudioEncoder is similar, just replace the MediaCodec of video with the MediaCodec of audio.

while(true) { 
 int encoderStatus = mVideoEncoder.dequeueOutputBuffer(mBufferInfo, TIMEOUT_USEC); 
 if (encoderStatus == MediaCodec.INFO_TRY_AGAIN_LATER) { 
 ... 
 } else if (encoderStatus == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) { 
 encoderOutputBuffers = mVideoEncoder.getOutputBuffers(); 
 } else if (encoderStatus == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) { 
 MediaFormat newFormat = mAudioEncoder.getOutputFormat(); 
 mAudioTrackIndex = mMuxer.addTrack(newFormat); 
 mNumTracksAdded++; 
 if (mNumTracksAdded == TOTAL_NUM_TRACKS) { 
 mMuxer.start(); 
 } 
 } else if (encoderStatus < 0) { 
 ... 
 } else { 
 ByteBuffer encodedData = encoderOutputBuffers[encoderStatus]; 
 ... 
 if (mBufferInfo.size != 0) { 
 mMuxer.writeSampleData(mVideoTrackIndex, encodedData, mBufferInfo); 
 } 
 mVideoEncoder.releaseOutputBuffer(encoderStatus, false); 
 if ((mBufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) { 
 break; 
 } 
 } 
 
} 

The third part is to end recording and send EOS information, so that the inner loop can be exited according to EOS in drawnvideoencoder () and drawaudioencoder. Part 4 is the cleaning work. Release the MediaCodec of audio and video, the Surface and MediaMuxer objects used by MediaCodec.

Last points:

1. At androidmanifest Add recording permission to the XML, otherwise the creation of AudioRecord object will definitely fail:

1. Audio and video shall be synchronized through PTS, and the two units shall be consistent.

2. MediaMuxer should be used in the order of constructor - > addtrack - > Start - > writesampledata - > stop. If there is both audio and video, write sampledata() before stop.

summary
The above is part of the short video source code development. According to the above process, it should work normally.