Recently, when doing mobile audio and video coding and decoding, the first thing to realize is the decoding function of mobile video. The pure FFmpeg method can also be implemented on the mobile terminal, but the efficiency is indeed slower. The 1080p video is OK, but up to 2k and 4k, the decoding speed (decoding a frame at the speed visible to the naked eye) can't stand it. Therefore, it is necessary to engage in hardware decoding at the mobile terminal to accelerate the decoding speed and release some CPU resources at the same time.
Refer to examples in FFmpeg source code
It should be the fastest idea to refer to the relevant function implementation of examples in the official source code of FFmpeg to realize your own program design. However, about video decoding, there is decode in the official source code of FFmpeg_ video. c demuxing_ decoding. c hw_ decode. c. These three decoding related files.
In fact, the schemes of the first two files are similar, except that the first one is for bare h264 streams and the second one is for encapsulated video files. It is suggested that novices can refer to the second scheme.
As for the third document, hw_decode.c. It looks like a demo of hardware decoding. Of course, it is (laughter). However, here, we can't refer to this. Referring to this file, we can implement hardware decoding on Linux or Windows platforms by using the hard decoding function implemented by cuvid, NVIDIA, Intel and other hardware manufacturers. However, the decoding of MediaCodec on Android platform has not been realized.
Trying to reference hw_decode.c in the process of realizing MediaCodec hard decoding, avcodec in 195 lines_ get_ hw_ Config failed. There is no config list to choose from.
Therefore, we still refer to demuxing_decoding.c to realize MediaCodec hard decoding on Android platform.
Then，Why？ Brief analysis of codemedic architecture
Because Android is a platform with a variety of hardware manufacturers, and MediaCodec is not a hardware manufacturer, it does not provide hardware encoding and decoding solutions. In fact, MediaCodec is more like an intermediate layer. It inherits the hardware codec capabilities of hardware manufacturers through openMAX. Finally, hardware manufacturers provide hardware codec libraries that meet the openMAX specification. Therefore, if we follow hw_encode.c, which is bound to be implemented in avcodec_ get_ hw_ The config layer cannot find a suitable configuration.
One change
Then, we can refer to demuxing_decoding.c to realize the hard decoding function of MediaCodec on Android platform. In fact, basically copy the full text into the Android native code and you can use it. You only need to change one place. That is, Dec = avcodec in line 165_ find_ decoder(st->codecpar->codec_id); Change to dec = avcodec_find_decoder_by_name("h264_mediacodec"); Just.
For convenience, we only decode the video stream in the file and simplify the whole process. Basically, the complete code is as follows:
static AVFrame *decode_frame = nullptr;
int testFFmpegMediaCodec(bool sw) {
string filename = "/sdcard/pav/hd.mp4";
AVFormatContext *fmt_ctx_ = nullptr;
int ret = 0;
if (ret = avformat_open_input(&fmt_ctx_, filename.c_str(), nullptr, nullptr) < 0) {
LOGE("Failed open file %s, ret = %d", filename.c_str(), ret);
return -1;
}
if (avformat_find_stream_info(fmt_ctx_, nullptr) < 0) {
LOGE("Failed to find stream information.");
return -1;
}
int vst_idx = av_find_best_stream(fmt_ctx_, AVMEDIA_TYPE_VIDEO, -1, -1, nullptr, false);
if (vst_idx < 0) {
LOGE("Failed to find video stream from file %s", fmt_ctx_->filename);
return -1;
}
AVCodec *pCodec = avcodec_find_decoder_by_name("h264_mediacodec");
AVCodecContext* codec_context = avcodec_alloc_context3(pCodec);
avcodec_parameters_to_context(codec_context, fmt_ctx_->streams[vst_idx]->codecpar);
if (ret = avcodec_open2(codec_context, pCodec, nullptr)) {
LOGE("Failed to open avcodec. ret = %d", ret);
return -1;
}
decode_frame = av_frame_alloc();
if (!decode_frame) {
LOGE("Failed to allocate frame.");
return -1;
}
decode_frame->format = codec_context->pix_fmt;
decode_frame->width = codec_context->width;
decode_frame->height = codec_context->height;
av_frame_get_buffer(decode_frame, 0);
ret = av_image_alloc(video_dst_data, video_dst_linesize, decode_frame->width, decode_frame->height, (AVPixelFormat)decode_frame->format, 1);
if (ret < 0) {
LOGE("Could not allocate raw video buffer.");
return -1;
} else {
LOGD("We allocate %d for raw video buffer.", ret);
}
AVPacket pkt;
av_init_packet(&pkt);
pkt.data = nullptr;
pkt.size = 0;
while (av_read_frame(fmt_ctx_, &pkt) >= 0) {
if (pkt.stream_index == vst_idx) {
ret = avcodec_send_packet(codec_context, pkt);
if (ret < 0) {
LOGE("Error submitting a packet for decoding (%s)", av_err2str(ret));
continue;
}
while (ret >= 0) {
ret = avcodec_receive_frame(codec_context, decode_frame);
if (ret < 0) {
if (ret == AVERROR_EOF) return 0;
if (ret == AVERROR(EAGAIN)) break;
}
// process with decode_frame
av_frame_unref(decode_frame);
break;
}
}
av_packet_unref(&pkt);
}
return 0;
}
A big pit
The above code is only an example and is not guaranteed to run directly. Some minor adjustments may be required. However, even if there is no problem with the above code, the hard decoding function of Android MediaCodec still cannot be realized. Among them, there is a big pit.
After I adjusted the code and ran it, I found that after decoding the first frame, the program always reported an error: error submitting a packet for coding, EAGAIN. This problem bothered me for a long time until some netizens mentioned that they need to take out the decoded frame data in the buffer first, and then send the data again for decoding.
The simplest and fastest way to change is to change avcodec_ send_ After the packet, remove the continue when RET < 0. After removing it, it can continue decoding as expected. However, many frames will be lost, because when RET < 0, the read packet is not successfully sent to the decoding queue.
Therefore, the whole decoding logic needs to be modified here. It is modified to try to get the data first, and then read the data and send it to the decoder queue. That is, it is revised as follows:
while (true) {
ret = avcodec_receive_frame(codec_context, decode_frame);
if (ret == 0) {
// process with decode_frame
av_frame_unref(decode_frame);
continue;
} else if (ret == AVERROR(EAGAIN)) {
ret = av_read_frame(fmt_ctx_, &pkt);
if (ret == AVERROR_EOF) return 0;
if (pkt.stream_index != vst_idx) {
av_packet_unref(&pkt); // Pay attention to this sentence. Missing will cause memory leakage
continue;
}
ret = avcodec_send_packet(codec_context, pkt);
if (ret < 0) {
LOGE("Error submitting a packet for decoding (%s)", av_err2str(ret));
continue;
}
}
}
Sample code only. Please handle the details yourself.
epilogue
So far, the C + + layer Android hard decoding function realized by FFmpeg + MediaCodec can be realized normally.
I did a simple test on Xiaomi MIX 2S mobile phone. The 1080 p h264 video decoding can reach 25 times the speed (that is, it takes 1 second to complete the 25 second video decoding), while the 4k h264 video can reach 3.5 times the speed.
However, 4k h264 video decoding has a high probability of flower screen phenomenon. The probability and degree of its flower screen may be related to the performance and state of the mobile phone. When the hard decoding is switched on for the first time, a video of the next 4k is decoded, and it is found that the flower screen is serious. The next day, I solved the same video at the beginning of work and found that there were basically no flower screens in the first few frames. From about 300 frames, there were small flower screens, and the flower screens in the last few frames were serious.
For the problem of flower screen, some netizens said that decoding 4k video on the mobile platform is really difficult, laborious and even impossible. If so, why can that 4k video be played smoothly on Xiaomi MIX 2S mobile phone? Are there any parameters that can be adjusted and optimized? I hope some experts can give some tips or suggestions.