FFmpeg old and new API codes

Posted by astropirate on Mon, 07 Feb 2022 13:50:08 +0100

background

FFmpeg version 2.8 was first used in the live SDK, and now the latest version of FFmpeg is 4.4. FFmpeg used by player editor is version 4.0; The internal structure of the new version of FFmpeg has also been optimized, and the efficiency and stability have been greatly improved compared with the old version. Therefore, the live SDK FFmpeg must also be upgraded.

brief introduction

FFmpeg is used in three main parts of the live SDK:

Use libavcodec to encode Audio;
Use libavcodec to encode Video;
Synthesis / streaming using libavformat;

I will first explain the process of encoding Audio and Video using the old AP Ilibavcodec. Then explain the process of coding Audio and Video with the new API. Through the comparison, you can easily know where the live SDK FFmpeg upgrade point is? Here we just compare the update points of our live SDK by analyzing the use of the old and new APIs. At the same time, we can also learn how to encode Audio, Video and Muxer through the old and new APIs of FFmpeg. Due to the limited space, the source code logic inside each API will not be explained in detail here. If you are interested, you can understand it by yourself.

libavcodec old API coded Audio

The text description feels boring. Let's first understand how to encode Audio through the old API of FFmpeg libavcodec module. Later, the functions and functions of API will be introduced in detail.

Encoder registration

av_register_all() can also use avcodec_register_all() instead. Check the source code to find AV_ register_ Avcodec was called internally by all()_ register_ all(). Its function is to register all codecs.

Find encoder

After the encoder is registered, you can use avcodec_find_encoder_by_name(),avcodec_find_encoder() gets the encoder we want. For example: through avcodec_find_encoder(AV_CODEC_ID_AAC). If we want to use libfdk AAC encoder to encode audio, we must link it during FFmpeg cross compilation. Otherwise, we will use the default AAC encoder in FFmpeg when acquiring the encoder.

Create AVCodecContext

After the encoder is created, you need to create AVCodecContext according to the encoder and initialize the coding parameters: sampling range, channel number, sampling format, etc.

AVCodecContext *avCodecContext = avcodec_alloc_context3(codec);
avCodecContext->codec_type = AVMEDIA_TYPE_AUDIO;
avCodecContext->sample_rate = 44100；
avCodecContext->bit_rate = 64000；
avCodecContext->sample_fmt = AV_SAMPLE_FMT_S16;
avCodecContext->channel_layout = AV_CH_LAYOUT_STEREO;
avCodecContext->channels = av_get_channel_layout_nb_channels(avCodecContext->channel_layout);
avCodecContext->profile = FF_PROFILE_AAC_LOW;
avCodecContext->flags |= CODEC_FLAG_GLOBAL_HEADER;
avCodecContext->codec_id = codec->id;

Turn on the encoder

After the parameters of the encoder are set, you can turn on the encoder.

if (avcodec_open2(avCodecContext, codec, NULL) < 0) {
   return -1;
}

Create AVFrame and request a PCM memory

In FFmpeg, the data before encoding and after decoding is represented by AVFrame; After encoding, the data before decoding is represented by AVPacket;

Here, we need to create an AVFrame for the data we are about to encode, and create a data storage space for it.

// Create AVFrame
AVFrame *encode_frame = av_frame_alloc();
encode_frame->nb_samples = avCodecContext->frame_size;
encode_frame->format = avCodecContext->sample_fmt;
encode_frame->channel_layout = avCodecContext->channel_layout;
encode_frame->sample_rate = avCodecContext->sample_rate;

// Request a PCM memory
int ret = av_samples_alloc_array_and_samples(&pcm_buffer, &src_samples_linesize, avCodecContext->channels, audio_nb_samples, avCodecContext->sample_fmt, 0);
if (ret < 0) {
    return -1;
}

code

The process of coding is a continuous cyclic process.

Get one frame of audio data from PCM queue to pcm_buffer
Put PCM_ The buffer is filled into the AFrame
Audio coding, get the encoded AVPacket data

// Get one frame of audio data from PCM queue to pcm_buffer
pcm_frame_callback(pcm_buffer);

int ret;
int got_packet;
AVPacket *pkt = av_packet_alloc();
pkt->duration = (int) AV_NOPTS_VALUE;
pkt->pts = pkt->dts = 0;
  
// Put PCM_ The buffer is filled into the AFrame
avcodec_fill_audio_frame(encode_frame, avCodecContext->channels, avCodecContext->sample_fmt, pcm_buffer[0], audioSamplesSize, 0);

// Audio coding, get the encoded AVPacket data
ret = avcodec_encode_audio2(avCodecContext, pkt, encode_frame, &got_packet);
if (ret < 0 || !got_packet) {
    av_packet_free(&pkt);
    return ret;
}

// write,enqueue

Destroy

if (NULL != pcm_buffer) {
    av_free(pcm_buffer);
}
if (NULL != encode_frame) {
    av_frame_free(&encode_frame);
}
if (NULL != avCodecContext) {
   avcodec_close(avCodecContext);
   av_free(avCodecContext);
}

libavcodec new API encoding Audio

Here, let's first understand how to encode Audio through the new API of FFmpeg libavcodec module. Later, the functions and functions of the core API will be introduced in detail.

As can be seen from the figure above, the idea of the new FFmpeg API remains unchanged in the total coding process, but the API called has changed. Here we focus on the use of the new API. The same parts as the old API will not be explained.

Create AVFrame and request a PCM memory

Old API creation method: first apply for a piece of memory, and then mount the memory on AVFrame after pcm data is filled.

The new API can be directly through av_frame_get_buffer() creates memory for AVFrame. But when calling AV_ frame_ get_ Before buffer(), the sampling rate, number of channels, sampling format and sampling size must be set for AVFrame.

// Create AVFrame
AVFrame *encode_frame = av_frame_alloc();
encode_frame->nb_samples = avCodecContext->frame_size;
encode_frame->format = avCodecContext->sample_fmt;
encode_frame->channel_layout = avCodecContext->channel_layout;
encode_frame->sample_rate = avCodecContext->sample_rate;

// Request a PCM memory
int ret = av_frame_get_buffer(encode_frame, 0);
if (ret < 0) {
    return -1;
}

code

FFmpeg new API coding through avcodec_send_frame(),avcodec_receive_packet() implementation. Their internal implementation principle can refer to the introduction at the bottom of the article.

    AVPacket pkt = { 0 };
    av_init_packet(&pkt);
    pkt.duration = (int) AV_NOPTS_VALUE;
    pkt.pts = pkt.dts = 0;
   
    while (true){
        do{
            ret = avcodec_receive_packet(avCodecContext, &pkt);
            // RET > = 0 get the encoded video stream
            if(ret >= 0){
                
                av_free_packet(&pkt);
                return ret;
            }
            //
            if (ret == AVERROR(EAGAIN)) {
                // Jump out of the loop.
                break;
            }
            // Coding error
            if (ret < 0) {
                av_free_packet(&pkt);
                return ret;
            }
        }while (true);

        // Get pcm data
        pcm_frame_callback(encode_frame->data);
        ret = avcodec_send_frame(avCodecContext, encode_frame);
        if(ret >= 0){
//            LOGI("avcodec_send_frame success");
        }else{
            LOGI("avcodec_send_frame error: %s\n", av_err2str(ret));
        }
        av_packet_unref(&pkt);
    }

Destroy

if (NULL != encode_frame) {
    av_frame_free(&encode_frame);
}
if (NULL != avCodecContext) {
   avcodec_free_context(avCodecContext)；
}

libavcodec old API code Video

In FFmpeg, Video is encoded through the old libavcodec API, and its process is very similar to that of Audio encoded by the old libavcodec API. I only give the API call flow chart here, and each step in the process of the flow chart will not be analyzed in detail. As long as you understand the above analysis, it is very simple here.

libavcodec new API code Video

In FFmpeg, Video is encoded through the new libavcode API, and its process is very similar to that of Audio encoded by the new libavcodec API. I only give the API call flow chart here, and each step in the process of the flow chart will not be analyzed in detail. As long as you understand the above analysis, it is very simple here.

Basic principles of audio and video codec

Avcodec in FFMPEG_ send_ Frame () and avcodec_receive_packet() is usually used at the same time. Call avcodec first_ send_ Frame () sends the audio and video frames to encoding, then calls avcodec_. receive_ Packet() gets the encoded packet. However, it should be noted that there is buffer data processing inside the encoder, so it is not guaranteed that there will be corresponding encoded data packet output for each audio and video frame. The two functions are not synchronized in timing for data processing, which needs special attention.

Usually, decoding starts through avcodec_send_frame() sends dozens of audio and video frames, corresponding to avcodec_receive_packet() has no packet output. When enough frames are sent in, avcodec_receive_packet() starts to output the data packet that was sent in for encoding at the beginning. If there are no data frames in the last dozens, avcodec should also be called_ send_ Frame() sends an empty frame to drive the encoding module to continue encoding the data in the buffer. At this time, avcodec_receive_packet() will still have packet output until averror is returned_ EOF means that all audio and video frames are encoded.

for instance:

A total of 100 video frames are sent to the encoder for encoding, and the final output video frames are also 100 data packets.

Avcodec was called 20 times before_ send_ Frame () can continuously send the 1st ~ 20th video frames, but avcodec is called for the first 20 times_ receive_ The packet() function always returns AVERROR(EAGAIN), and there is no packet output.
Call avcodec from the 21st time_ send_ Frame() starts from the 21st video frame, and then calls avcodec this time_ receive_ The packet () function can return 0, and there is a packet output, but the output packet pts is 0 (that is, the video frame corresponding to the first packet), and then avcodec_send_frame() continuously sends the 22nd, 23rd... Video frames, avcodec_receive_packet() keeps outputting the 1st, 2nd... Packets
Finally, the 100th packet passes through avcodec_ send_ The frame () feed is complete, but at this point avcodec_receive_packet() only gets the output packet of frame 82. At this time, you need to continue to call avcodec_send_frame() sends an empty frame and calls avcodec constantly at the same time_ receive_ Packet() gets the output packet until averror is returned_ EOF, you can get the last 100th output packet.

Basic principles of audio and video codec reference documents:
https://zhuanlan.zhihu.com/p/346010443

Programmer Think

FFmpeg old and new API codes

background

brief introduction

libavcodec old API coded Audio

Encoder registration

Find encoder

Create AVCodecContext

Turn on the encoder

Create AVFrame and request a PCM memory

code

Destroy

libavcodec new API encoding Audio

Create AVFrame and request a PCM memory

code

libavcodec old API code Video

libavcodec new API code Video

Basic principles of audio and video codec

Hot Topics