Recently, in the study of ffmpeg and ffplay, it is found that the source code analysis of ffmpeg unpacking on the Internet is not much and complete. Based on the ffplay source code, let's explain the ffmpeg decoding process. Here we mainly explain the key variables in the data structures of AVFormatContext, AVStream and AVIOContext, such as void*priv_data, AVStreamInternal *internal, AVFormatInternal *internal and other internal variables are the most important, because it is these internal non-public variables that are the key to storing file content, important interfaces or attributes and transferring between interfaces. To understand the true meaning of ffmpeg library to define these internal variables is the key to learning ffmpeg.
However, there are too many interfaces to write in one blog post, so students need to pay attention to other blog posts
All interfaces and data structures are written in great detail, but they have been studied for a long time. It's very troublesome and tiring to write. After reading it, give me some attention, ha ha ha ha
Key Tips:
-
Many structures in ffmpeg (AVStream, URLContext, AVFormatContext) like to use void priv_data or voidopaque variable
In fact, this variable is used to store the "substructure" of the structure. I prefer to call it external protocol or external resource (interface or data). For example, in ffplay, AVFormatContext needs to be unpacked. By detecting the file format, it is concluded that AVInputFormat is mov(MP4) format, and AVFormatContext - > priv_data = movcontext variable (mov.c file), priv in AVStream_ Data is used to store mov protocol movstramcontext structure variables, priv in URLContext_ Data is used to store the FileContext structure in the file protocol. In fact, this is to separate the protocol interface function or data from the backbone interface, so as to make the whole library scalable. So you will find that the priv of the backbone will be mentioned at the beginning of various protocol interfaces_ Data is assigned to the structure of the protocol itself. For example: mov_ read_ In header
MOVContext *mov = s->priv_data; Writing in this way is also a kind of grammatical sugar. sc will not be affected by priv_data name.
Even if the name of external variables changes, it will have little impact on the internal interface. ffmpeg interfaces mostly use this method, especially involving some external protocols
rtmp streaming media, file file, mov format, etc. -
For the naming of Context, such as URLContext, FileContext, AVFormatContext, etc., my personal understanding is the data + method (Interface) required to complete the function. For example, there is a file protocol in URLContext. In the FileContext structure, there are open, close, read and other methods and uint*data variables to store the data read from the file. This is stored level by level. In order to make the code more extensible, this kind of library is written by many people. I don't know if I can explain clearly, ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha.
-
For internal, such naming as AVStreamInternal is generally used to store data and pass it to the interface. For example, AVStreamInternal is used to store the information (location, size, pts, dts) and codec data interface of audio and video samples, and AVFormatInternal is used to store (obtained through AVStreamInternal index sample) AVPacket content. It is mainly used to store and transfer data.
Here I would like to focus on my personal understanding of AVFormatContext:
AVFormatContext structure is the structure of ffmpeg docking and unpacking, that is, directly reading (whether local file file or network streaming media) files or resources, and according to certain structure requirements (AVPacket is used to store a frame or a sample information)
To organize and store the data read out from the multimedia file. Decode and store the information in the form of AVPacket structure. Therefore, in AVFormatContext, since it is necessary to unpack, it is necessary to have the file IO interface AVIOContext and the audio and video metadata information generated when reading the multimedia file head information in AVStream. It must be noted that both AVIOContext and AVStream are lower layer protocols for docking, such as AVIOContext - > priv_ data=FileContext(file.c)
URLProtocol ff_file_protocol stores the interface for reading files, and FileContext can store file handles, avstream - > internal - > index_ Entries is used to store everyone's sample information. However, after reading a sample (such as mov(mp4)), find the information of each sample by reading the head and store it in avstream - > internal - > index_ Entries), which need to be stored in avformatinternal - > avpacketlist * packet in the form of AVPacket_ buffer_ End, and then decode.
Therefore, from this point of view, the upper structure of AVFormatContext is not directly connected with the lower protocols (FileContext, MOVContext, MOVStreamContext), which I mentioned in the above Tips and previous blog posts. The ffmpeg library needs to be written by many people and must be extensible. Changes in the lower layer will not affect the upper structure.
In fact, the naming of ffmpeg is also very reasonable. AVFormatContext is simple to read files and put them in the linked list avformatinternal - > avpacketlist * packet_ buffer_ End, wait for decoding. AVStream and AVIOContext serve this purpose.
This blog post mainly explains the overall framework of ffplay, including creating thread decoding, image display after decoding, etc
//Entry function main() function
int main(int argc, char **argv)
{
int flags;
VideoState *is;
...
//Initialize refresh frame flush_pkt, when flush_pkt packet_ queue_ put_ After the private value is assigned, the queue serial number is increased by one, that is, it is regarded as a new queue
//There is an introduction in the blog post later
av_init_packet(&flush_pkt);
flush_pkt.data = (uint8_t *)&flush_pkt;
if (!display_disable) {
int flags = SDL_WINDOW_HIDDEN;
...
//Create window (SDL library interface, which actually calls CreateWindow of window API to create window handle)
window = SDL_CreateWindow(program_name, SDL_WINDOWPOS_UNDEFINED, SDL_WINDOWPOS_UNDEFINED, default_width, default_height, flags);
SDL_SetHint(SDL_HINT_RENDER_SCALE_QUALITY, "linear");
if (window) {
//create render
renderer = SDL_CreateRenderer(window, -1, SDL_RENDERER_ACCELERATED | SDL_RENDERER_PRESENTVSYNC);
if (!renderer) {
av_log(NULL, AV_LOG_WARNING, "Failed to initialize a hardware accelerated renderer: %s\n", SDL_GetError());
renderer = SDL_CreateRenderer(window, -1, 0);
}
if (renderer) {
if (!SDL_GetRendererInfo(renderer, &renderer_info))
av_log(NULL, AV_LOG_VERBOSE, "Initialized %s renderer.\n", renderer_info.name);
}
}
if (!window || !renderer || !renderer_info.num_texture_formats) {
av_log(NULL, AV_LOG_FATAL, "Failed to create window or renderer: %s", SDL_GetError());
do_exit(NULL);
}
}
//According to input_filename file name, find out the corresponding protocol, unpack it, and create a thread for decoding
is = stream_open(input_filename, file_iformat);
if (!is) {
av_log(NULL, AV_LOG_FATAL, "Failed to initialize VideoState!\n");
do_exit(NULL);
}
//Loop playing decoded pictures
event_loop(is);
/* never returns */
return 0;
}
According to input_filename file name, find out the corresponding protocol, unpack it, and create a thread for decoding
Tips: let's say that all parameters of ffplay framework are in VideoState, which is also the top-level structure, including understanding and encapsulating AVFormatContext, decoding AVCodecInternal, etc. as mentioned above, for the convenience of coding, the upper data structure is not connected with the lower data structure. You will find that when the upper data is transferred into the interface, It will be re assigned to a new structure of the same type, so that when any variable name changes, the interface function will not be affected
//Queue structure to be decoded after unsealing
typedef struct PacketQueue {
//first_pkt is the head of the queue. Every time a packet is taken from the queue, it is taken from the first_pkt inside
//last_pkt variable is the tail pointer of MyAVPacketList when last_ When pkt is empty, it will be given to first_pkt assignment packet
MyAVPacketList *first_pkt, *last_pkt;
int nb_packets;//How many packet s are there in the queue
int size; //Queue size
int64_t duration; //Queue duration, that is, the duration of the packet
int abort_request; //Discard assignment ID
int serial; //Queue serial number
SDL_mutex *mutex; //The lock of the queue, because the assignment of the queue is in multithreading
SDL_cond *cond; //Semaphores are used for synchronization
} PacketQueue;
//According to input_filename file name, find out the corresponding protocol, unpack it, and create a thread for decoding
static VideoState *stream_open(const char *filename, AVInputFormat *iformat)
{
VideoState *is;
is = av_mallocz(sizeof(VideoState));//Open up memory space
if (!is)
return NULL;
//Assign stream serial number. Generally, there are only two streams of audio and video
is->last_video_stream = is->video_stream = -1;
is->last_audio_stream = is->audio_stream = -1;
is->last_subtitle_stream = is->subtitle_stream = -1;
is->filename = av_strdup(filename);
if (!is->filename)
goto fail;
is->iformat = iformat;
is->ytop = 0;
is->xleft = 0;
/* start video display */
//Initialize the decoded image display queue FrameQueue, including opening up the queue memory space, creating the mutex and semaphore SDL of the image display queue FrameQueue_ cond
if (frame_queue_init(&is->pictq, &is->videoq, VIDEO_PICTURE_QUEUE_SIZE, 1) < 0)
goto fail;
if (frame_queue_init(&is->subpq, &is->subtitleq, SUBPICTURE_QUEUE_SIZE, 0) < 0)
goto fail;
if (frame_queue_init(&is->sampq, &is->audioq, SAMPLE_QUEUE_SIZE, 1) < 0)
goto fail;
//Initialize the packet queue used for decoding after unpacking, including the mutex and semaphore SDL that create the PacketQueue_ cond
if (packet_queue_init(&is->videoq) < 0 ||
packet_queue_init(&is->audioq) < 0 ||
packet_queue_init(&is->subtitleq) < 0)
goto fail;
//Create read_ Semaphore of thread interface
if (!(is->continue_read_thread = SDL_CreateCond())) {
av_log(NULL, AV_LOG_FATAL, "SDL_CreateCond(): %s\n", SDL_GetError());
goto fail;
}
//Initialize the playback time of audio and video
init_clock(&is->vidclk, &is->videoq.serial);
init_clock(&is->audclk, &is->audioq.serial);
init_clock(&is->extclk, &is->extclk.serial);
is->audio_clock_serial = -1;
...
is->audio_volume = startup_volume;
is->muted = 0;
is->av_sync_type = av_sync_type;
//Create thread read_thread is used to find out the corresponding protocol, unpack it, and create thread decoding
is->read_tid = SDL_CreateThread(read_thread, "read_thread", is);
if (!is->read_tid) {
av_log(NULL, AV_LOG_FATAL, "SDL_CreateThread(): %s\n", SDL_GetError());
fail:
stream_close(is);
return NULL;
}
return is;
}
//Initialize the decoded image display queue FrameQueue, including opening up the queue memory space, creating the mutex and semaphore SDL of the image display queue FrameQueue_ cond
static int frame_queue_init(FrameQueue *f, PacketQueue *pktq, int max_size, int keep_last)
{
int i;
memset(f, 0, sizeof(FrameQueue)); //The initialization variable is 0
if (!(f->mutex = SDL_CreateMutex())) { //Create mutex
av_log(NULL, AV_LOG_FATAL, "SDL_CreateMutex(): %s\n", SDL_GetError());
return AVERROR(ENOMEM);
}
if (!(f->cond = SDL_CreateCond())) {//Create SDL_cond
av_log(NULL, AV_LOG_FATAL, "SDL_CreateCond(): %s\n", SDL_GetError());
return AVERROR(ENOMEM);
}
f->pktq = pktq;
f->max_size = FFMIN(max_size, FRAME_QUEUE_SIZE);
f->keep_last = !!keep_last;
for (i = 0; i < f->max_size; i++)//Open up queue memory
if (!(f->queue[i].frame = av_frame_alloc()))
return AVERROR(ENOMEM);
return 0;
}
//Initialize the packet queue used for decoding after unpacking, including the mutex and semaphore SDL that create the PacketQueue_ cond
static int packet_queue_init(PacketQueue *q)
{
memset(q, 0, sizeof(PacketQueue));//The initialization variable is 0
q->mutex = SDL_CreateMutex();//Create mutex
if (!q->mutex) {
av_log(NULL, AV_LOG_FATAL, "SDL_CreateMutex(): %s\n", SDL_GetError());
return AVERROR(ENOMEM);
}
q->cond = SDL_CreateCond();//Create SDL_cond
if (!q->cond) {
av_log(NULL, AV_LOG_FATAL, "SDL_CreateCond(): %s\n", SDL_GetError());
return AVERROR(ENOMEM);
}
q->abort_request = 1;
return 0;
}
Thread read_thread is used to find out the corresponding protocol, unpack it, and create thread decoding
See for details Analysis of ffplay source code avformat of ffmpeg decoding process_ find_ stream_ info,read_frame_internal,avpriv_packet_list_put interface) It is introduced in
ffplay source code analysis stream of ffmpeg decoding process_ component_ open,avcodec_receive_frame,queue_picture,packet_queue_get interface) It is introduced in
static int read_thread(void *arg)
{
//arg is assigned to a temporary variable for coding convenience
VideoState *is = arg;
AVFormatContext *ic = NULL;
int err, i, ret;
int st_index[AVMEDIA_TYPE_NB]; //stream index (type)
AVPacket pkt1, *pkt = &pkt1;
int64_t stream_start_time;
int pkt_in_play_range = 0;
AVDictionaryEntry *t;
SDL_mutex *wait_mutex = SDL_CreateMutex();
int scan_all_pmts_set = 0;
int64_t pkt_ts;
memset(st_index, -1, sizeof(st_index));
is->eof = 0;
//AVFormatContext unpacks context and opens up memory
//AVFormatInternal *internal memory
//Assignment interface io_open is used to find out the corresponding IO external protocol (file,rtmp) according to the file name
ic = avformat_alloc_context();
...
//Find out the io external protocol and assign it to AVIOContext + find out the external AVInputFormat(mov) and read the header
err = avformat_open_input(&ic, is->filename, is->iformat, &format_opts);
if (err < 0) {
print_error(is->filename, err);
ret = -1;
goto fail;
}
...
is->ic = ic;
av_format_inject_global_side_data(ic);
if (find_stream_info) {
AVDictionary **opts = setup_find_stream_info_opts(ic, codec_opts);
int orig_nb_streams = ic->nb_streams;
//Read some packet s, assign the decoder interface, and assign some parameters from AVStream to AVCodecContext*Internal for subsequent decoding
//See the link above for details
ffplay Source code analysis ffmpeg Decoding process avformat_find_stream_info,read_frame_internal,avpriv_packet_list_put Interface
err = avformat_find_stream_info(ic, opts);
....
}
/* if seeking requested, we execute it */
if (start_time != AV_NOPTS_VALUE) {
int64_t timestamp;
timestamp = start_time;
/* add the stream start time */
if (ic->start_time != AV_NOPTS_VALUE)
timestamp += ic->start_time;
ret = avformat_seek_file(ic, -1, INT64_MIN, timestamp, INT64_MAX, 0);
if (ret < 0) {
av_log(NULL, AV_LOG_WARNING, "%s: could not seek to position %0.3f\n",
is->filename, (double)timestamp / AV_TIME_BASE);
}
}
is->realtime = is_realtime(ic);
if (show_status)
av_dump_format(ic, 0, is->filename, 0);
//Generally, two streams audio and video streams are created through trak(mov format)
for (i = 0; i < ic->nb_streams; i++) {
AVStream *st = ic->streams[i];
enum AVMediaType type = st->codecpar->codec_type;
st->discard = AVDISCARD_ALL;
if (type >= 0 && wanted_stream_spec[type] && st_index[type] == -1)
if (avformat_match_stream_specifier(ic, st, wanted_stream_spec[type]) > 0)
st_index[type] = i;
}
for (i = 0; i < AVMEDIA_TYPE_NB; i++) {
if (wanted_stream_spec[i] && st_index[i] == -1) {
av_log(NULL, AV_LOG_ERROR, "Stream specifier %s does not match any %s stream\n", wanted_stream_spec[i], av_get_media_type_string(i));
st_index[i] = INT_MAX;
}
}
......
is->show_mode = show_mode;
if (st_index[AVMEDIA_TYPE_VIDEO] >= 0) {
AVStream *st = ic->streams[st_index[AVMEDIA_TYPE_VIDEO]];
AVCodecParameters *codecpar = st->codecpar;
AVRational sar = av_guess_sample_aspect_ratio(ic, st, NULL);
if (codecpar->width)
set_default_window_size(codecpar->width, codecpar->height, sar);
}
/* open the streams */
//stream_ The index parameter is the serial number of the stream. Generally, there are two stream serial numbers 0 or 1, one audio and one video
//One thread for audio and one thread for video are used to continuously obtain samples (packets) from is - > decoder * D - > packetqueue * q and decode them
//Obtain one sample at a time and decode it, and then obtain one sample and decode it
//Audio starts playing but not decoding in a thread
if (st_index[AVMEDIA_TYPE_AUDIO] >= 0) {//audio frequency
stream_component_open(is, st_index[AVMEDIA_TYPE_AUDIO]);
}
ret = -1;
if (st_index[AVMEDIA_TYPE_VIDEO] >= 0) {//video
ret = stream_component_open(is, st_index[AVMEDIA_TYPE_VIDEO]);
}
if (is->show_mode == SHOW_MODE_NONE)
is->show_mode = ret >= 0 ? SHOW_MODE_VIDEO : SHOW_MODE_RDFT;
if (st_index[AVMEDIA_TYPE_SUBTITLE] >= 0) {
stream_component_open(is, st_index[AVMEDIA_TYPE_SUBTITLE]);
}
...
//Small Tips: for (;) Because in the compilation phase, the assembler will put for (;) Optimize it and enter the cycle directly
//Loop read packet
//Audio and video samples are interspersed and encapsulated, so according to movstreamcontext - > current_ sample, keep calling AV_ read_ Just frame
//current_ There will be no duplication in the sample. This is the principle for mov format to take the packet and find the packet
for (;;) {
if (is->abort_request)
{
break;
}
if (is->paused != is->last_paused) {
is->last_paused = is->paused;
if (is->paused)
{
is->read_pause_return = av_read_pause(ic);
}
else
{
av_read_play(ic);
}
}
...
//Handle fast forward, fast backward and pause, that is, find a specific sample to decode and play according to the current playback time
if (is->seek_req) {
int64_t seek_target = is->seek_pos; //The number of seconds you need to find is converted to PTS_ Seek assignment
int64_t seek_min = is->seek_rel > 0 ? seek_target - is->seek_rel + 2: INT64_MIN;
int64_t seek_max = is->seek_rel < 0 ? seek_target - is->seek_rel - 2: INT64_MAX;
//According to the protocol in avformatcontext * IC - > avinputformat (such as mov format) and the pts value seek of the sample to be searched_ target
//Call the corresponding external protocol interface read_ seek(mov_read_seek)
//mov_read_seek function: Seek_ The value of target matches the pts of all samples, finds out the corresponding sample index, and assigns movstreamcontext - > current_ sample
//AV below_ read_ The frame will directly read the sample to be found, because av_read_frame will read the current every time_ sample
ffmpeg Unpacking mov/mp4 Analysis of format unpacking source code mov_read_header(read metadata),mov_read_packet(read sample data),mov_read_trak)It is introduced in
//ffplay takes the time of audio playback callback as pts value
ret = avformat_seek_file(is->ic, -1, seek_min, seek_target, seek_max, is->seek_flags);
if (ret < 0) {
av_log(NULL, AV_LOG_ERROR,
"%s: error while seeking\n", is->ic->url);
} else {
//Empty all packet s to be decoded in the existing queue
if (is->audio_stream >= 0) {
//Clear the variables in the PacketQueue queue and initialize to 0
packet_queue_flush(&is->audioq);
//Flush will be refreshed_ Pkt packet is assigned to the queue PacketQueue, indicating that a queue is restarted, and the queue serial number is added by one
packet_queue_put(&is->audioq, &flush_pkt);
}
if (is->subtitle_stream >= 0) {
packet_queue_flush(&is->subtitleq);
packet_queue_put(&is->subtitleq, &flush_pkt);
}
if (is->video_stream >= 0) {
//Clear the variables in the PacketQueue queue and initialize to 0
packet_queue_flush(&is->videoq);
//Flush will be refreshed_ Pkt packet is assigned to the queue PacketQueue, indicating that a queue is restarted, and the queue serial number is added by one
packet_queue_put(&is->videoq, &flush_pkt);
}
if (is->seek_flags & AVSEEK_FLAG_BYTE) {
set_clock(&is->extclk, NAN, 0);
} else {
//The pts value seek of the sample to be searched_ Target, assigned to is - > extclk as a temporary storage
set_clock(&is->extclk, seek_target / (double)AV_TIME_BASE, 0);
}
}
is->seek_req = 0;
is->queue_attachments_req = 1;
is->eof = 0;
if (is->paused)
step_to_next_frame(is);
}
...
/* if the queue are full, no need to read more */
if (infinite_buffer<1 &&
(is->audioq.size + is->videoq.size + is->subtitleq.size > MAX_QUEUE_SIZE
|| (stream_has_enough_packets(is->audio_st, is->audio_stream, &is->audioq) &&
stream_has_enough_packets(is->video_st, is->video_stream, &is->videoq) &&
stream_has_enough_packets(is->subtitle_st, is->subtitle_stream, &is->subtitleq)))) {
/* wait 10 ms */
SDL_LockMutex(wait_mutex);
SDL_CondWaitTimeout(is->continue_read_thread, wait_mutex, 10);
SDL_UnlockMutex(wait_mutex);
continue;
}
...
//Read a sample data from the file according to the protocol interface and the sample information read in the file head
ffplay Source code analysis ffmpeg Decoding process avformat_find_stream_info,read_frame_internal,avpriv_packet_list_put It is introduced in the interface
ret = av_read_frame(ic, pkt);
if (ret < 0) {
if ((ret == AVERROR_EOF || avio_feof(ic->pb)) && !is->eof) {
if (is->video_stream >= 0)
//When the stream exists, assign an empty packet to the PacketQueue queue
packet_queue_put_nullpacket(&is->videoq, is->video_stream);
if (is->audio_stream >= 0)
packet_queue_put_nullpacket(&is->audioq, is->audio_stream);
if (is->subtitle_stream >= 0)
packet_queue_put_nullpacket(&is->subtitleq, is->subtitle_stream);
is->eof = 1;
}
if (ic->pb && ic->pb->error) {
if (autoexit)
goto fail;
else
break;
}
SDL_LockMutex(wait_mutex);
SDL_CondWaitTimeout(is->continue_read_thread, wait_mutex, 10);
SDL_UnlockMutex(wait_mutex);
continue;
} else {
is->eof = 0;
}
/* check if packet is in play range specified by user, then queue, otherwise discard */
stream_start_time = ic->streams[pkt->stream_index]->start_time; //First dts
pkt_ts = pkt->pts == AV_NOPTS_VALUE ? pkt->dts : pkt->pts;
//ppkt_in_play_range is 1
pkt_in_play_range = duration == AV_NOPTS_VALUE ||
(pkt_ts - (stream_start_time != AV_NOPTS_VALUE ? stream_start_time : 0)) *
av_q2d(ic->streams[pkt->stream_index]->time_base) -
(double)(start_time != AV_NOPTS_VALUE ? start_time : 0) / 1000000
<= ((double)duration / 1000000);
//Because the audio and video are stored alternately in the file, the packet may be audio or video and stored in different queues for decoding
//In stream_ component_ open->video_ thread->get_ video_ frame->decoder_ decode_ frame->packet_ queue_ Get packet from get
if (pkt->stream_index == is->audio_stream && pkt_in_play_range) {
//Will AV_ read_ The packet taken out by frame is assigned to the PacketQueue queue through packet_queue_get get and decode
packet_queue_put(&is->audioq, pkt);
} else if (pkt->stream_index == is->video_stream && pkt_in_play_range
&& !(is->video_st->disposition & AV_DISPOSITION_ATTACHED_PIC)) {
//Will AV_ read_ The packet taken out by frame is assigned to the PacketQueue queue through packet_queue_get get and decode
packet_queue_put(&is->videoq, pkt);
} else if (pkt->stream_index == is->subtitle_stream && pkt_in_play_range) {
packet_queue_put(&is->subtitleq, pkt);
} else {
av_packet_unref(pkt);
}
}
ret = 0;
fail:
if (ic && !is->ic)
avformat_close_input(&ic);
if (ret != 0) {
SDL_Event event;
event.type = FF_QUIT_EVENT;
event.user.data1 = is;
SDL_PushEvent(&event);
}
SDL_DestroyMutex(wait_mutex);
return 0;
}
//Will AV_ read_ The packet taken out by frame is assigned to the PacketQueue queue through packet_queue_get get and decode
static int packet_queue_put(PacketQueue *q, AVPacket *pkt)
{
int ret;
SDL_LockMutex(q->mutex);//Lock
ret = packet_queue_put_private(q, pkt);
SDL_UnlockMutex(q->mutex);
if (pkt != &flush_pkt && ret < 0)
av_packet_unref(pkt);
return ret;
}
Important Tips:
As mentioned above, avformatinternal - > AVPacketList is used to store the unpacked data. The assignment of AVPacketList adopts the method of queue, and ffplay imitates this method, so let's combine the two here
ffplay queue assignment and acquisition
//ffplay queue structure
typedef struct PacketQueue {
//first_pkt is the head of the queue. Every time a packet is taken from the queue, it is taken from the first_pkt inside
//last_pkt variable is the tail pointer of MyAVPacketList. After inserting elements into the new linked list, q - > last_pkt automatically moves backward,
//When last_ When pkt is empty, it will be given to first_pkt assignment packet
MyAVPacketList *first_pkt, *last_pkt;
int nb_packets;//How many packet s are there in the queue
int size; //Queue size
int64_t duration; //Queue duration, that is, the duration of the packet
int abort_request; //Discard assignment ID
int serial; //Queue serial number
SDL_mutex *mutex; //The lock of the queue, because the assignment of the queue is in multithreading
SDL_cond *cond; //Semaphores are used for synchronization
} PacketQueue;
//To packetqueue * q - > first_ Pkt linked list assignment
//Activate semaphore for decoding
//This interface is modeled after avpriv in ffmpeg_ packet_ list_ Put interface
//Where Q - > last_ Pkt variable is the tail pointer of MyAVPacketList. After inserting elements into the new linked list, q - > last_ Pkt auto move backward
//When last_ When pkt is NULL, add packet to first_ In pkt, you need to read first every time_ Pkt content
static int packet_queue_put_private(PacketQueue *q, AVPacket *pkt)
{
MyAVPacketList *pkt1;
if (q->abort_request)//Discard assignment ID
return -1;
pkt1 = av_malloc(sizeof(MyAVPacketList));//Linked list opens up memory
if (!pkt1)
return -1;
pkt1->pkt = *pkt;
pkt1->next = NULL;
if (pkt == &flush_pkt)//If the packet is refreshed, the sequence number of the queue is increased by one, that is, it is regarded as a new queue
q->serial++;
pkt1->serial = q->serial; //Update the queue where pkt1 is located
//q->last_ Pkt variable can be regarded as an identifier
//When last_ When pkt is NULL, add packet to first_ In pkt, you need to read first every time_ Pkt content
if (!q->last_pkt)
q->first_pkt = pkt1;
else
q->last_pkt->next = pkt1;
q->last_pkt = pkt1;//Move last_pkt linked list pointer
q->nb_packets++; //The number of packs this queue already has
q->size += pkt1->pkt.size + sizeof(*pkt1);
q->duration += pkt1->pkt.duration; //Queue duration
/* XXX: should duplicate packet data in DV case */
SDL_CondSignal(q->cond); //Activate semaphore for decoding
return 0;
}
//Get packet from queue
//When first_ When pkt is null, the tail pointer of MyAVPacketList is last_pkt assignment is null
//If it's first_ No value in pkt, thread waiting, waiting for packet_ queue_ put_ First in private_ SDL after pkt assignment_ Condsignal (Q - > cond) to activate
static int packet_queue_get(PacketQueue *q, AVPacket *pkt, int block, int *serial)
{
MyAVPacketList *pkt1;
int ret;
SDL_LockMutex(q->mutex); //Lock
//Small Tips: for (;) Because in the compilation phase, the assembler will put for (;) Optimize it and enter the cycle directly
for (;;) {
if (q->abort_request) { //Whether to abandon the queue
ret = -1;
break;
}
pkt1 = q->first_pkt; //List first_pkt removal
if (pkt1) {
q->first_pkt = pkt1->next;//Move linked list pointer
if (!q->first_pkt)
//When first_ When pkt is empty, last will be identified_ Pkt assignment is null
q->last_pkt = NULL;
q->nb_packets--;//Packets in the queue minus one
q->size -= pkt1->pkt.size + sizeof(*pkt1);//Subtract the number of packet s
q->duration -= pkt1->pkt.duration;//Minus total time
*pkt = pkt1->pkt;//packet assignment
if (serial)
//Assign a value to the serial number of the retrieved packet
*serial = pkt1->serial;
av_free(pkt1);
ret = 1;
break;
} else if (!block) {
ret = 0;
break;
} else {
//If it's first_ No value in pkt, thread waiting, waiting for packet_ queue_ put_ First in private_ SDL after pkt assignment_ Condsignal (Q - > cond) to activate
SDL_CondWait(q->cond, q->mutex);
}
}
SDL_UnlockMutex(q->mutex);
return ret;
}
ffmpeg queue assignment and acquisition
//plast_pktl variable is the tail pointer of AVPacketList. After inserting elements into the new linked list, plast_pktl moves backward automatically
//When plast_ When pktl is NULL, add packet to packet_ In the buffer, you need to read the packet every time_ Buffer content
int avpriv_packet_list_put(AVPacketList **packet_buffer, AVPacketList **plast_pktl , AVPacket *pkt,
int (*copy)(AVPacket *dst, const AVPacket *src), int flags)
{
AVPacketList *pktl = av_mallocz(sizeof(AVPacketList));//Open up memory space
int ret;
if (!pktl)
return AVERROR(ENOMEM);
if (copy) {
ret = copy(&pktl->pkt, pkt);
if (ret < 0) {
av_free(pktl);
return ret;
}
} else {
//Assign the value of pkt - > data to pkt - > buf - > data
ret = av_packet_make_refcounted(pkt);
if (ret < 0) {
av_free(pktl);
return ret;
}
av_packet_move_ref(&pktl->pkt, pkt);
}
//plast_pktl variable is the tail pointer of AVPacketList. After inserting elements into the new linked list, plast_pktl moves backward automatically
//When plast_ When pktl is NULL, add packet to packet_ In the buffer, you need to read the packet every time_ Buffer content
if (*packet_buffer)
(*plast_pktl)->next = pktl;
else
*packet_buffer = pktl; //When plast_ When pktl is empty, pktl is assigned to packet_buffer
/* Add the packet in the buffered packet list. */
*plast_pktl = pktl; //Move linked list pointer
return 0;
}
//Get queue packet
//When pkt_ When buffer is null, AVPacketList tail pointer pkt_buffer_end assignment is null
int avpriv_packet_list_get(AVPacketList **pkt_buffer,
AVPacketList **pkt_buffer_end,
AVPacket *pkt)
{
AVPacketList *pktl;
if (!*pkt_buffer)
return AVERROR(EAGAIN);
pktl = *pkt_buffer;
*pkt = pktl->pkt;
*pkt_buffer = pktl->next;//Move linked list pointer
if (!pktl->next)
//When pkt_ When buffer is null, the AVPacketList tail pointer pkt_buffer_end assignment is null
*pkt_buffer_end = NULL;
av_freep(&pktl);
return 0;
}
Loop playing decoded pictures
//Loop playing decoded pictures
static void event_loop(VideoState *cur_stream)
{
SDL_Event event;
double incr, pos, frac;
//Small Tips: for (;) Because in the compilation phase, the assembler will put for (;) Optimize it and enter the cycle directly
for (;;) {
double x;
//Play pictures and audio
refresh_loop_wait_event(cur_stream, &event);
//Many messages returned by the Windows message mechanism call the SDL interface
switch (event.type) {
case SDL_KEYDOWN:
if (exit_on_keydown || event.key.keysym.sym == SDLK_ESCAPE || event.key.keysym.sym == SDLK_q) {
//Exit program
do_exit(cur_stream);
break;
}
The middle part of the students have a look. Ha, I won't repeat it here. I mainly talk about fast forward, fast backward and pause
case SDLK_p:
case SDLK_SPACE:
toggle_pause(cur_stream);
...
case SDLK_LEFT: //Left arrow rewind
incr = seek_interval ? -seek_interval : -10.0; //Default - 10.0 seconds
goto do_seek;
case SDLK_RIGHT: //Right arrow fast forward
incr = seek_interval ? seek_interval : 10.0;//Default 10.0 seconds
goto do_seek;
case SDLK_UP: //Up arrow fast forward
incr = 60.0;//Default 60.0 seconds
goto do_seek;
case SDLK_DOWN: //Down arrow rewind
incr = -60.0;//Default - 60.0 seconds
do_seek:
//Fast forward, fast backward
if (seek_by_bytes) { //Set a fast forward parameter
pos = -1;
if (pos < 0 && cur_stream->video_stream >= 0)
pos = frame_queue_last_pos(&cur_stream->pictq);
if (pos < 0 && cur_stream->audio_stream >= 0)
pos = frame_queue_last_pos(&cur_stream->sampq);
if (pos < 0)
pos = avio_tell(cur_stream->ic->pb);
if (cur_stream->ic->bit_rate)
incr *= cur_stream->ic->bit_rate / 8.0;
else
incr *= 180000.0;
pos += incr;
stream_seek(cur_stream, pos, incr, 1);
} else {
//Get the time information of the frame during playback
//If it is a pause, it is the current pts
pos = get_master_clock(cur_stream);
//Check whether pos is valid
if (isnan(pos))
pos = (double)cur_stream->seek_pos / AV_TIME_BASE;
//Add fast forward time
pos += incr;
if (cur_stream->ic->start_time != AV_NOPTS_VALUE && pos < cur_stream->ic->start_time / (double)AV_TIME_BASE)
pos = cur_stream->ic->start_time / (double)AV_TIME_BASE;
//Assign the pts of the sample to be searched to cur_stream in order to find
stream_seek(cur_stream, (int64_t)(pos * AV_TIME_BASE), (int64_t)(incr * AV_TIME_BASE), 0);
}
break;
...
}
}
}
//Play pictures and audio
static void refresh_loop_wait_event(VideoState *is, SDL_Event *event) {
double remaining_time = 0.0;
SDL_PumpEvents();
//Non blocking message loop
while (!SDL_PeepEvents(event, 1, SDL_GETEVENT, SDL_FIRSTEVENT, SDL_LASTEVENT)) {
if (!cursor_hidden && av_gettime_relative() - cursor_last_shown > CURSOR_HIDE_DELAY) {
SDL_ShowCursor(0);
cursor_hidden = 1;
}
if (remaining_time > 0.0)
av_usleep((int64_t)(remaining_time * 1000000.0));
remaining_time = REFRESH_RATE; //How often do I play a frame (FPS frame rate)
if (is->show_mode != SHOW_MODE_NONE && (!is->paused || is->force_refresh))
video_refresh(is, &remaining_time);
SDL_PumpEvents();
}
}
static void video_refresh(void *opaque, double *remaining_time) //remaining_time how long does it take to play a frame (FPS frame rate)
{
VideoState *is = opaque;
double time;
Frame *sp, *sp2;
if (!is->paused && get_master_sync_type(is) == AV_SYNC_EXTERNAL_CLOCK && is->realtime)
check_external_clock_speed(is);
if (!display_disable && is->show_mode != SHOW_MODE_VIDEO && is->audio_st) {
time = av_gettime_relative() / 1000000.0;
if (is->force_refresh || is->last_vis_time + rdftspeed < time) {
video_display(is);
is->last_vis_time = time;
}
*remaining_time = FFMIN(*remaining_time, is->last_vis_time + rdftspeed - time);
}
if (is->video_st) {
retry:
if (frame_queue_nb_remaining(&is->pictq) == 0) {
// nothing to do, no picture to display in the queue
} else {
double last_duration, duration, delay;
Frame *vp, *lastvp;
/* dequeue the picture */
lastvp = frame_queue_peek_last(&is->pictq);
vp = frame_queue_peek(&is->pictq);
if (vp->serial != is->videoq.serial) {
frame_queue_next(&is->pictq);
goto retry;
}
if (lastvp->serial != vp->serial)
is->frame_timer = av_gettime_relative() / 1000000.0;
if (is->paused)
goto display;
/* compute nominal last_duration */
last_duration = vp_duration(is, lastvp, vp);
delay = compute_target_delay(last_duration, is);
time= av_gettime_relative()/1000000.0;
if (time < is->frame_timer + delay) {
*remaining_time = FFMIN(is->frame_timer + delay - time, *remaining_time);
goto display;
}
SDL_LockMutex(is->pictq.mutex);
if (!isnan(vp->pts))
//Update and set the pts unit of the video frame as second, and pass the pts of the current frame through set_ Assign clock to is - > vidclk
update_video_pts(is, vp->pts, vp->pos, vp->serial);
SDL_UnlockMutex(is->pictq.mutex);
......
if (frame_queue_nb_remaining(&is->pictq) > 1) {
Frame *nextvp = frame_queue_peek_next(&is->pictq);
duration = vp_duration(is, vp, nextvp);
if(!is->step && (framedrop>0 || (framedrop && get_master_sync_type(is) != AV_SYNC_VIDEO_MASTER)) && time > is->frame_timer + duration){
is->frame_drops_late++;
frame_queue_next(&is->pictq);
goto retry;
}
}
...
frame_queue_next(&is->pictq);
is->force_refresh = 1;
if (is->step && !is->paused)
stream_toggle_pause(is);
}
display:
/* display picture */
if (!display_disable && is->force_refresh && is->show_mode == SHOW_MODE_VIDEO && is->pictq.rindex_shown)
//Render + display get frames in FrameQueue
video_display(is);
}
is->force_refresh = 0;
...
}
}
}
Pause processing, fast forward, fast backward
//A structure that records time during frame playback
//The variables in this are in real seconds, so it is necessary to convert them during storage, such as ST - > time_base, this is time_ Best use of base
typedef struct Clock {
double pts; //pts of frame
double pts_drift; //Difference between pts and current time
double last_updated; //Current storage last time
double speed; //The playback rate is generally 1, or fast forward playback
int serial; //Sequence number of the queue where the frame is located
int paused; //Pause flag
int *queue_serial; /* pointer to the current packet queue serial, used for obsolete clock detection */
} Clock;
//initialize variable
static void init_clock(Clock *c, int *queue_serial)
{
c->speed = 1.0;
c->paused = 0;
c->queue_serial = queue_serial;
set_clock(c, NAN, -1);
}
//Assign pts and current time to Clock
static void set_clock(Clock *c, double pts, int serial)
{
//Gets the current time converted to seconds
double time = av_gettime_relative() / 1000000.0;
set_clock_at(c, pts, serial, time);
}
//Assign current time and pts
//Each frame of audio and video will call set_clock_at recording time
//The audio will return the callback time of the hardware playback audio_ callback_ Assign, time
//The video will be updated during playback_ video_ pts update time, where pts is in stream_ component_ open->video_ It has been converted into seconds in thread
static void set_clock_at(Clock *c, double pts, int serial, double time)
{
c->pts = pts;
c->last_updated = time;
c->pts_drift = c->pts - time;
c->serial = serial;
}
//Get the time information of the frame during playback
//If it is a pause, it is the current pts
//If you fast forward or rewind, you need to get the time more accurately
//You need to multiply the difference between the time at this time (that is, the time of pressing the keyboard) and the time recorded last time by the rate and add it to pts, which is more accurate
static double get_clock(Clock *c)
{
if (*c->queue_serial != c->serial)
return NAN;
if (c->paused) {
//If it is a pause, it is the current pts
return c->pts;
} else {
//If you fast forward or rewind, you need to get the time more accurately
//You need to multiply the difference between the time at this time (that is, the time of pressing the keyboard) and the time recorded last time by the rate and add it to pts, which is more accurate
double time = av_gettime_relative() / 1000000.0;
return c->pts_drift + time - (time - c->last_updated) * (1.0 - c->speed);
}
}
//The synchronization time assigns slave to Clock *c
//The name is interesting enough. I have to use the word slave
static void sync_clock_to_slave(Clock *c, Clock *slave)
{
double clock = get_clock(c);
double slave_clock = get_clock(slave);
if (!isnan(slave_clock) && (isnan(clock) || fabs(clock - slave_clock) > AV_NOSYNC_THRESHOLD))
set_clock(c, slave_clock, slave->serial);
}
//Get clock
//master means exactly here. Hahaha, hahaha. At first, I didn't understand why it was named like this
static double get_master_clock(VideoState *is)
{
double val;
switch (get_master_sync_type(is)) {
case AV_SYNC_VIDEO_MASTER:
val = get_clock(&is->vidclk);
break;
case AV_SYNC_AUDIO_MASTER:
val = get_clock(&is->audclk);
break;
default:
val = get_clock(&is->extclk);
break;
}
return val;
}
//Get synchronization time type
//ffplay is based on AV_SYNC_AUDIO_MASTER based
static int get_master_sync_type(VideoState *is) {
if (is->av_sync_type == AV_SYNC_VIDEO_MASTER) {
if (is->video_st)
return AV_SYNC_VIDEO_MASTER;
else
return AV_SYNC_AUDIO_MASTER;
} else if (is->av_sync_type == AV_SYNC_AUDIO_MASTER) {
if (is->audio_st)
return AV_SYNC_AUDIO_MASTER;
else
return AV_SYNC_EXTERNAL_CLOCK;
} else {
return AV_SYNC_EXTERNAL_CLOCK;
}
}
//The video updates the Clock, and the pts of the current frame will be recorded for each frame playback, and assigned to is - > extclk
static void update_video_pts(VideoState *is, double pts, int64_t pos, int serial) {
/* update current video pts */
set_clock(&is->vidclk, pts, serial);
sync_clock_to_slave(&is->extclk, &is->vidclk);
}
//The audio updates the Clock, and the pts of the current frame will be recorded for each frame playback, and assigned to is - > extclk
//audio_callback_time is the time passed to the callback function when the hardware plays audio
static void sdl_audio_callback(void *opaque, Uint8 *stream, int len)
{
....
/* Let's assume the audio driver that is used by SDL has two periods. */
if (!isnan(is->audio_clock)) {
set_clock_at(&is->audclk, is->audio_clock - (double)(2 * is->audio_hw_buf_size + is->audio_write_buf_size) / is->audio_tgt.bytes_per_sec, is->audio_clock_serial, audio_callback_time / 1000000.0);
sync_clock_to_slave(&is->extclk, &is->audclk);
}
}
Suspend processing
static void toggle_pause(VideoState *is)
{
stream_toggle_pause(is);
is->step = 0;
}
static void stream_toggle_pause(VideoState *is)
{
//av_gettime_relative gets the current time, and the return value is in milliseconds
if (is->paused) {
is->frame_timer += av_gettime_relative() / 1000000.0 - is->vidclk.last_updated;
if (is->read_pause_return != AVERROR(ENOSYS)) {
is->vidclk.paused = 0;
}
//The pause is subject to the video time
//Fast forward and fast backward are subject to the audio time
set_clock(&is->vidclk, get_clock(&is->vidclk), is->vidclk.serial);
}
//Set pts or pts of temporary clock - > extclk_ Drive is assigned to is - > extclk
set_clock(&is->extclk, get_clock(&is->extclk), is->extclk.serial);
//If the previous step is pause, the pause will be canceled here. Otherwise, it will be set as pause
is->paused = is->audclk.paused = is->vidclk.paused = is->extclk.paused = !is->paused;
}