FFmpeg is a leading multimedia framework and a powerful open source library for audio and video processing. It can decode, encode, transcode, mix, separate streaming media, filter and play audio and video content in almost all formats. It can not only use API to process audio and video, but also use FFmpeg command to edit audio and video files.

It includes a plurality of decoders and encoders for audio, video and caption streams, and a plurality of bit stream filters.

FFmpeg directory and its function

libavutil: a library of functions and tools for simplifying programming, including random number generators, data structures, mathematical routines, core multimedia utilities, and so on.

libavcodec: is a library containing audio / video codecs.

libavformat: is a library containing synthesizers and separators for multimedia container formats.

libavdevice: a library containing input and output devices for capturing and rendering many common multimedia I / O software frameworks, including Video4Linux, Video4Linux2, VfW and ALSA.

Libav filter: a library containing media filters.

libswscale: a library that performs highly optimized image scaling and color space / pixel format conversion.

libswresample: a library that performs highly optimized audio resampling, rematrixing, and sample format conversion operations.

FFmpeg tool and its function

Ffmpeg: ffmpeg is a command line tool used to implement the functions of ffmpeg on the command line.

Ffplay: ffplay is a very simple and portable media player using FFmpeg library and SDL library. It is mainly used as a testing platform for various FFmpeg API s.

Ffprobe: ffprobe is a multimedia stream analysis tool. It collects information from multimedia streams and prints it in human and machine-readable forms. It can be used to detect the container type of multimedia stream, as well as the format and type of each multimedia stream. It can be used as a stand-alone application or perform more complex processing in combination with text filters.

After understanding the FFmpeg directory and its tools, the following article focuses on how to use the FFmpeg command line to process audio and video files in daily life.

Description of audio and video processing functions of FFmpeg

FFmpeg commands can be roughly divided into several modules according to their functions: recording commands, synthesis and separation of audio and video, cutting and merging of audio and video, and mutual conversion of pictures and videos. Next, we introduce how to use the FFmpeg command line to process audio and video to achieve the above functions.

Taking the Ubuntu operating system as an example, this paper installs it through the simple command sudo apt get install ffmpeg.

You can also execute the following command after installing FFmpeg on Windows, MacOS and other Linux operating systems.

1) Recording

ffmpeg -f avfoundation -list_devices true -i ""

2) Recording screen

ffmpeg -f avfoundation -i 1 -r 30 out.yuv

• f specifies to use avfoundation to collect data.
• -i specifies where to collect data, which is a file index number. On my computer, 1 represents the desktop (you can query the device index number through the above command).
• -r specifies the frame rate. According to the ffmpeg official document, - r and - framerate have the same effect, but it is found that they are different in the actual test- Framerate is used to limit input and - r is used to limit output.

3) Screen recording + sound

ffmpeg -f avfoundation -i 1:0 -r 29.97 -c:v libx264 -crf 0 -c:a libfdk_aac -profile:a aac_he_v2 -b:a 48k out.flv

• -I "1" before the 1:0 colon represents the screen index number. The "0" after the colon represents the phase number of the voice.
• -c:v, like the parameter - vcodec, represents the video encoder. C is the abbreviation of codec and v is the abbreviation of video.
• -crf is the parameter of x264. 0 table lossless compression.
• -c:a, like the parameter - acodec, represents the audio encoder.
• -profile is FDK_ Parameters of AAC. aac_he_v2 table uses AAC_HE v2 compresses data.
• -b:a specifies the audio bit rate. B is the abbreviation of bitrate and a is the abbreviation of audio.

4) Video recording

ffmpeg -framerate 30 -f avfoundation -i 0 out.mp4

• -framerate limits the capture frame rate of video. This must be set according to the prompt requirements. If it is not set, an error will be reported.
• -f specifies to use avfoundation to collect data.
• -i specifies the index number of the video device.

5) Video + Audio

ffmpeg -framerate 30 -f avfoundation -i 0:0 out.mp4

6) Recording

ffmpeg -f avfoundation -i :0 out.wav

7) Record audio raw data

ffmpeg -f avfoundation -i :0 -ar 44100 -f s16le out.pcm

8) Extract audio stream

ffmpeg -i input.mp4 -acodec copy -vn out.aac

• acodec: Specifies the audio encoder. Copy indicates to copy only without encoding and decoding.
• vn: v stands for video, n stands for no, which means no video.

9) Convert to MP3 format

ffmpeg -i input.mp4 -acodec libmp3lame  out.mp3

10) Extract video stream

ffmpeg -i input.mp4 -vcodec copy -an out.h264

• vcodec: Specifies the video encoder. Copy indicates to copy only without encoding and decoding.
• an: a stands for video, n stands for no, which means no audio.

11) Video to format

ffmpeg -i video_test.mp4 -vcodec copy -acodec copy out_convert.flv

The audio and video of the above command table are directly copied, but the packaging format of mp4 is converted to flv.

12) Remove video sound

 ffmpeg -i video_auido.mp4 -vcodec copy -an video.mp4 

13) Video compression

 1)ffmpeg -i test_ffmpeg.mp4  (The compressed file is larger and clearer, which is generally not used)
 2)ffmpeg -i out.MP4 -b:v 500k 512k_out.mp4(Reduce the video bit rate and make the compression more blurred)

14) Audio and video merging

ffmpeg -i out.h264 -i out.aac -vcodec copy -acodec copy out.mp4

15) Extract YUV data

ffmpeg -i input.mp4 -an -c:v rawvideo -pixel_format yuv420p out.yuv
 play
ffplay -s wxh out.yuv

• -c:v rawvideo specifies to convert the video to raw data
• -pixel_format yuv420p specifies that the conversion format is yuv420p

16) Video YUV to H264

ffmpeg -f rawvideo -pix_fmt yuv420p -s 640x480 -r 30 -i out.yuv -c:v libx264 -f rawvideo out.h264

17) Extract audio PCM data

ffmpeg -i out.mp4 -vn -ar 44100 -ac 2 -f s16le out.pcm
 play
ffplay -ar 44100 -ac 2 -f s16le -i out.pcm

18) PCM to WAV

ffmpeg -f s16be -ar 8000 -ac 2 -acodec pcm_s16be -i input.raw output.wav

19) Add watermark
Image watermark:

ffmpeg -i out.mp4  -vf "movie=logo.png,scale=128:72[watermask];[in][watermask] overlay=96:54 [out]" water_img.mp4

• -The movie in vf specifies the location of the logo. scale specifies the size of the logo. overlay specifies where the logo is placed.

Text watermark:

ffmpeg -i out.mp4 -vf "drawtext=fontfile=FZBaoHTJW_Xi.TTF: text='anyRTC':x=128:y=72:fontsize=24:fontcolor=red:shadowy=2" water_text.mp4

20) Video scaling

fmpeg -i out.mp4 -vf scale=iw/2:-1 scale.mp4

• -vf scale specifies to use the simple filter scale. iw in iw/2:-1 specifies to take the width of the video by integer- 1 indicates that the height varies with the width.

Reduce the resolution of the video and convert the video format

ffmpeg -i input.avi -vf scale=640:360 out.mp4

21) video clipping

ffmpeg -i VR.mov  -vf crop=in_w-300:in_h-200 -c:v libx264 -c:a copy -video_size 1280x720 vr_new.mp4

crop format: crop=out_w:out_h ❌ y

• out_w: The width of the output. You can use in_w table type in the width of the video.
• out_h: The height of the output. You can use in_h the height of the video input table.
• X: X coordinate
• Y: Y coordinate

If x and y are set to 0, crop from the upper left corner. If it is not written, it is cut from the center point.

22) video flip left and right (up and down)

ffmpeg  -i out.mp4 -filter_complex "[0:v]pad=w=2*iw[a];[0:v]hflip[b];[a][b]overlay=x=w" duicheng.mp4

• hflip flip horizontally
• If you want to change to flip vertically, you can use vflip

23) clip

ffmpeg -i out.mp4 -ss 00:00:00 -t 10 out_cut.mp4

• -ss specifies the start time of clipping, accurate to seconds
• -t the length of time after cutting.

24) audio and video merging
First, create a videolist.txt file with the following contents:
file 'test1.mp4'
file 'test2.mp4'
Then execute the following command:

ffmpeg -f concat -i videolist.txt -c copy output.mp4

25) HLS slice

ffmpeg -i out.mp4 -c:v libx264 -c:a libfdk_aac -strict -2 -f hls  out.m3u8

• -strict -2 indicates that AAC is enabled for audio
• -f hls to m3u8 format

26) video to JPEG

ffmpeg -i video_test.mp4 -r 1 -f image2 image-%3d.jpeg

27) picture format conversion

ffmpeg -i input.bmp out.jpg
ffmpeg -i input.bmp out.png

28) video screenshot by frame
It is mainly used for users to manually take screenshots or upload videos to generate thumbnails

Here: - ss position searches for the specified time [-] hh:mm:ss[.xxx]. The format of - vframes is also supported
Set how many frames of video are converted. The example command is to obtain a screenshot of the first frame in the first second.

Note: the generated screenshot is best in jpg format, which takes up less space. If other formats need to be used, it can be set according to business needs.

ffmpeg -ss 00:00:01 -y -i video_test.mp4 -vframes 1 snap.jpg

29) video to GIF

ffmpeg -i video_test.mp4 -r 1 -f image2 image-%3d.jpeg

30) picture to video

ffmpeg -f image2 -i image-%3d.jpeg out_img_video.mp4

31) video caption srt

ffmpeg -i video_test.mp4 -vf subtitles=subtitle.srt out_subtitle.mp4

32) add music + subtitles to the video

ffmpeg -i video_test.mp4 -i audio_bg.mp3 -vf subtitles=all_mp3_srt.srt out_mp3_subtitle.mp4

FFmpeg is a powerful library specialized in audio and video processing. Many players are developed based on FFmpeg. The functions mentioned in this article are only part of the usable functions of FFmpeg listed in the small series. More FFmpeg command lines can realize the function of processing audio and video. You can go to the FFmpeg official website to consult relevant documents.