Although this information is already outdated, the original material is still a popular source of inspiration for various useful content on the topic of FFmpeg. However, there is still no complete translation of the original into Russian. We correct the annoying omission, for it is better late than never.And although we tried, difficulties of translation are inevitable in such a voluminous text . Report bugs (preferably in private messages) - together we will do better.Table of contents
EDISON.
, C C++.
! ;-)
UPD: This guide has been updated as of February 2015.FFmpeg is a great library for creating video applications as well as general-purpose utilities. FFmpeg takes care of the entire video processing routine, performing all decoding, encoding, multiplexing and demultiplexing. Which greatly simplifies the creation of media applications. Everything is quite simple and quick, written in C, you can decode almost any codec available today, as well as encode to some other formats.The only catch is that the documentation is mostly missing. There is one tutorial ( in the original, here is a link to an already non-existent web page - note translator), which covers the basics of FFmpeg and the automatic generation of doxygen docks. And nothing more. Therefore, I decided to independently figure out how to use FFmpeg to create working digital video and audio applications, and at the same time document the process and present it in the form of a textbook.There is an FFplay program that comes with FFmpeg. It is simple, written in C, implements a full-fledged video player using FFmpeg. My first lesson was an updated version of the original lesson by Martin Boehme ( in the original, a link to an already defunct web page - a translator’s note ) - I dragged some pieces from there. And also in a series of my lessons I will show the process of creating a working video player based on ffplay.cFabrice Bellard. Each lesson will present a new idea (or even two) with an explanation of its implementation. Each chapter comes with a C listing, which you can compile and run on your own. The source files will show how this program works, how its individual parts work, and also demonstrate minor technical details that are not covered in this guide. When we are done, we will have a working video player written in less than 1000 lines of code!When creating the player, we will use SDL to output audio and video media file. SDL is an excellent cross-platform multimedia library used in MPEG playback programs, emulators and many video games. You will need to download and install the SDL libraries on your system in order to compile the programs from this guide.This tutorial is for people with good programming experience. At the very least, you need to know C, and also have an understanding of concepts such as queues, mutexes, etc. There should be some understanding of multimedia; for example, things like waveforms and the like. However, being a guru in these matters is not necessary, as many concepts will be explained in the course of the lessons.Please feel free to send me error messages, questions, comments, ideas, features, whatever, at Dranger Doggy Gmail dot Com.
Read also on the blog of the
company EDISON:
FFmpeg libav manual
Lesson 1: Creating Screencaps ← ⇑ →
Full listing: tutorial01.c
#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libswscale/swscale.h>
#include <stdio.h>
#if LIBAVCODEC_VERSION_INT < AV_VERSION_INT(55,28,1)
#define av_frame_alloc avcodec_alloc_frame
#define av_frame_free avcodec_free_frame
#endif
void SaveFrame(AVFrame *pFrame, int width, int height, int iFrame) {
FILE *pFile;
char szFilename[32];
int y;
sprintf(szFilename, "frame%d.ppm", iFrame);
pFile=fopen(szFilename, "wb");
if(pFile==NULL)
return;
fprintf(pFile, "P6\n%d %d\n255\n", width, height);
for(y=0; y<height; y++)
fwrite(pFrame->data[0]+y*pFrame->linesize[0], 1, width*3, pFile);
fclose(pFile);
}
int main(int argc, char *argv[]) {
AVFormatContext *pFormatCtx = NULL;
int i, videoStream;
AVCodecContext *pCodecCtxOrig = NULL;
AVCodecContext *pCodecCtx = NULL;
AVCodec *pCodec = NULL;
AVFrame *pFrame = NULL;
AVFrame *pFrameRGB = NULL;
AVPacket packet;
int frameFinished;
int numBytes;
uint8_t *buffer = NULL;
struct SwsContext *sws_ctx = NULL;
if(argc < 2) {
printf("Please provide a movie file\n");
return -1;
}
av_register_all();
if(avformat_open_input(&pFormatCtx, argv[1], NULL, NULL)!=0)
return -1;
if(avformat_find_stream_info(pFormatCtx, NULL)<0)
return -1;
av_dump_format(pFormatCtx, 0, argv[1], 0);
videoStream=-1;
for(i=0; i<pFormatCtx->nb_streams; i++)
if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_VIDEO) {
videoStream=i;
break;
}
if(videoStream==-1)
return -1;
pCodecCtxOrig=pFormatCtx->streams[videoStream]->codec;
pCodec=avcodec_find_decoder(pCodecCtxOrig->codec_id);
if(pCodec==NULL) {
fprintf(stderr, "Unsupported codec!\n");
return -1;
}
pCodecCtx = avcodec_alloc_context3(pCodec);
if(avcodec_copy_context(pCodecCtx, pCodecCtxOrig) != 0) {
fprintf(stderr, "Couldn't copy codec context");
return -1;
}
if(avcodec_open2(pCodecCtx, pCodec, NULL)<0)
return -1;
pFrame=av_frame_alloc();
pFrameRGB=av_frame_alloc();
if(pFrameRGB==NULL)
return -1;
numBytes=avpicture_get_size(PIX_FMT_RGB24, pCodecCtx->width,
pCodecCtx->height);
buffer=(uint8_t *)av_malloc(numBytes*sizeof(uint8_t));
avpicture_fill((AVPicture *)pFrameRGB, buffer, PIX_FMT_RGB24,
pCodecCtx->width, pCodecCtx->height);
sws_ctx = sws_getContext(pCodecCtx->width,
pCodecCtx->height,
pCodecCtx->pix_fmt,
pCodecCtx->width,
pCodecCtx->height,
PIX_FMT_RGB24,
SWS_BILINEAR,
NULL,
NULL,
NULL
);
i=0;
while(av_read_frame(pFormatCtx, &packet)>=0) {
if(packet.stream_index==videoStream) {
avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished, &packet);
if(frameFinished) {
sws_scale(sws_ctx, (uint8_t const * const *)pFrame->data,
pFrame->linesize, 0, pCodecCtx->height,
pFrameRGB->data, pFrameRGB->linesize);
if(++i<=5)
SaveFrame(pFrameRGB, pCodecCtx->width, pCodecCtx->height,
i);
}
}
av_free_packet(&packet);
}
av_free(buffer);
av_frame_free(&pFrameRGB);
av_frame_free(&pFrame);
avcodec_close(pCodecCtx);
avcodec_close(pCodecCtxOrig);
avformat_close_input(&pFormatCtx);
return 0;
}
Overview
Movie files have several main components. First, the file itself is called a container , and the type of container determines how data is represented in the file. Examples of containers are AVI and Quicktime . Further, there are several threads in the file; in particular, there is usually an audio stream and a video stream . (“Stream” is a funny word for “a sequence of data items available according to the timeline.”) Data items in a stream are called frames . Each stream is encoded by one or another type of codec . The codec determines how the actual data to diruyutsya and DecemberAudited - hence the name of the codec. Examples of codecs are DivX and MP3. Packets are then read from the stream. Packets are pieces of data that can contain bits of data that are decoded into raw frames, which we can finally manipulate in our application. For our purposes, each packet contains full frames (or several frames if it is audio).Working with video and audio streams is very simple even at the most basic level:10 OPEN video_stream FROM video.avi
20 READ packet FROM video_stream INTO frame
30 IF frame NOT COMPLETE GOTO 20
40 DO SOMETHING WITH frame
50 GOTO 20
Working with multimedia using FFmpeg is almost as simple as in this program, although in some programs the “MAKE [...]” step can be very difficult. In this tutorial, we will open the file, count the video stream inside it, and our “MAKE [...]” will write the frame to the PPM file.Open file
First things first, let's see what happens first when you open the file. Using FFmpeg, we first initialize the desired library:#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <ffmpeg/swscale.h>
...
int main(int argc, charg *argv[]) {
av_register_all();
This registers all available file formats and codecs in the library, so they will be used automatically when opening a file with the appropriate format / codec. Note that you need to call av_register_all () only once, so we do it here in main (). If you wish, you can register only selective file formats and codecs, but usually there is no particular reason to do so.Now open the file:AVFormatContext *pFormatCtx = NULL;
if(avformat_open_input(&pFormatCtx, argv[1], NULL, 0, NULL)!=0)
return -1;
Get the file name from the first argument. This function reads the file header and stores the file format information in the AVFormatContext structure that we passed. The last three arguments are used to specify the file format, buffer size, and format parameters. By setting them to NULL or 0, libavformat will detect everything automatically.This function only looks at the header, so now we need to check the stream information in the file:
if(avformat_find_stream_info(pFormatCtx, NULL)<0)
return -1;
This function passes valid data to pFormatCtx -> streams . We get acquainted with a convenient debugging function, showing us what's inside:
av_dump_format(pFormatCtx, 0, argv[1], 0);
Now pFormatCtx -> streams is just an array of pointers of size pFormatCtx -> nb_streams . We will go through it until we find the video stream:int i;
AVCodecContext *pCodecCtxOrig = NULL;
AVCodecContext *pCodecCtx = NULL;
videoStream=-1;
for(i=0; i<pFormatCtx->nb_streams; i++)
if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_VIDEO) {
videoStream=i;
break;
}
if(videoStream==-1)
return -1;
pCodecCtx=pFormatCtx->streams[videoStream]->codec;
Information about the codec in the stream is located in a place called the " codec context ". It contains all the information about the codec that the stream uses, and now we have a pointer to it. But we still have to find the real codec and open it:AVCodec *pCodec = NULL;
pCodec=avcodec_find_decoder(pCodecCtx->codec_id);
if(pCodec==NULL) {
fprintf(stderr, "Unsupported codec!\n");
return -1;
}
pCodecCtx = avcodec_alloc_context3(pCodec);
if(avcodec_copy_context(pCodecCtx, pCodecCtxOrig) != 0) {
fprintf(stderr, "Couldn't copy codec context");
return -1;
}
if(avcodec_open2(pCodecCtx, pCodec)<0)
return -1;
Please note that you can not directly use AVCodecContext from the video stream! Therefore, you have to use a vcodec_copy_context () to copy the context to a new location (of course, after memory is allocated for it).Data storage
Now we need a place to store the frame:AVFrame *pFrame = NULL;
pFrame=av_frame_alloc();
Since we plan to output PPM files that are stored in 24-bit RGB, we will need to convert our frame from its own format to RGB. FFmpeg will do it for us. For most projects (including this one), you need to convert the start frame to a specific format. Select a frame for the converted frame:
pFrameRGB=av_frame_alloc();
if(pFrameRGB==NULL)
return -1;
Despite the fact that we selected the frame, we still need a place to accommodate the raw data when converting it. We use avpicture_get_size to get the right sizes and allocate the necessary space manually:uint8_t *buffer = NULL;
int numBytes;
numBytes=avpicture_get_size(PIX_FMT_RGB24, pCodecCtx->width,
pCodecCtx->height);
buffer=(uint8_t *)av_malloc(numBytes*sizeof(uint8_t));
av_malloc is an analogue of the C-function malloc from FFmpeg, which is a simple wrapper around malloc that provides alignment of memory addresses, etc. By the way, this does not protect against memory leaks, double freeing or other problems that occur with malloc .Now we use avpicture_fill to associate the frame with our newly allocated buffer. Regarding AVPicture : the AVPicture structure is a subset of the AVFrame structure - the beginning of the AVFrame structure is identical to the AVPicture structure .
avpicture_fill((AVPicture *)pFrameRGB, buffer, PIX_FMT_RGB24,
pCodecCtx->width, pCodecCtx->height);
We are already at the finish line! Now we are ready to read from the stream!Reading data
Now, to read the entire video stream, we read the next package, decrypt it in our frame, and as soon as the decryption is complete, convert the frame and save it:struct SwsContext *sws_ctx = NULL;
int frameFinished;
AVPacket packet;
sws_ctx = sws_getContext(pCodecCtx->width,
pCodecCtx->height,
pCodecCtx->pix_fmt,
pCodecCtx->width,
pCodecCtx->height,
PIX_FMT_RGB24,
SWS_BILINEAR,
NULL,
NULL,
NULL
);
i=0;
while(av_read_frame(pFormatCtx, &packet)>=0) {
if(packet.stream_index==videoStream) {
avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished, &packet);
if(frameFinished) {
sws_scale(sws_ctx, (uint8_t const * const *)pFrame->data,
pFrame->linesize, 0, pCodecCtx->height,
pFrameRGB->data, pFrameRGB->linesize);
if(++i<=5)
SaveFrame(pFrameRGB, pCodecCtx->width,
pCodecCtx->height, i);
}
}
av_free_packet(&packet);
}
Nothing complicated: av_read_frame () reads the package and saves it in the AVPacket structure . Please note that we only distribute the structure of the package - FFmpeg provides us with the internal data that packet.data points to . This frees av_free_packet () a bit later . avcodec_decode_video () converts the packet to frame. However, we may not have all the information that we need for the frame after decoding the packet, therefore avcodec_decode_video () sets frameFinished when we have the next frame. Finally, we use sws_scale () to convert from our own format ( pCodecCtx ->pix_fmt ) in RGB. Remember that you can cast an AVFrame pointer to an AVPicture pointer. Finally, we pass the information about the frame, height and width of our SaveFrame function.Speaking of packages. Technically, a packet can contain only part of a frame, as well as other data bits. However, the FFmpeg parser guarantees that the packets we receive contain either a full frame or even several frames.Now all that remains to be done is to use the SaveFrame functionto write the RGB information to a PPM file. Although we are superficially dealing with the PPM format itself; believe me, everything works here:void SaveFrame(AVFrame *pFrame, int width, int height, int iFrame) {
FILE *pFile;
char szFilename[32];
int y;
sprintf(szFilename, "frame%d.ppm", iFrame);
pFile=fopen(szFilename, "wb");
if(pFile==NULL)
return;
fprintf(pFile, "P6\n%d %d\n255\n", width, height);
for(y=0; y<height; y++)
fwrite(pFrame->data[0]+y*pFrame->linesize[0], 1, width*3, pFile);
fclose(pFile);
}
We perform a standard file open, etc., and then record the RGB data. The file is written line by line. A PPM file is simply a file in which RGB information is presented as a long line. If you know the colors of HTML, it will be like marking the colors of each pixel from the first end to the last, something like # ff0000 # ff0000 .... , like for a red screen. (In fact, it is stored in binary format and without a separator, but I hope you catch the idea.) The title indicates how wide and high the image is, as well as the maximum size of the RGB values.Now back to our main () function . As soon as we finish reading from the video stream, we just need to clear everything:
av_free(buffer);
av_free(pFrameRGB);
av_free(pFrame);
avcodec_close(pCodecCtx);
avcodec_close(pCodecCtxOrig);
avformat_close_input(&pFormatCtx);
return 0;
As you can see, we use av_free for the memory allocated using avcode_alloc_frame and av_malloc .That's all the code! Now, if you are using Linux or a similar platform, then run:gcc -o tutorial01 tutorial01.c -lavutil -lavformat -lavcodec -lz -lavutil -lm
If you have an older version of FFmpeg, you may need to remove -lavutil :gcc -o tutorial01 tutorial01.c -lavformat -lavcodec -lz -lm
Most graphics programs must open the PPM format. Check it out on some movie files whose screencaps were made using our program.
Lesson 2: Displaying the screen ← ⇑ →
Full listing: tutorial02.c
#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libswscale/swscale.h>
#include <SDL.h>
#include <SDL_thread.h>
#ifdef __MINGW32__
#undef main
#endif
#include <stdio.h>
#if LIBAVCODEC_VERSION_INT < AV_VERSION_INT(55,28,1)
#define av_frame_alloc avcodec_alloc_frame
#define av_frame_free avcodec_free_frame
#endif
int main(int argc, char *argv[]) {
AVFormatContext *pFormatCtx = NULL;
int i, videoStream;
AVCodecContext *pCodecCtxOrig = NULL;
AVCodecContext *pCodecCtx = NULL;
AVCodec *pCodec = NULL;
AVFrame *pFrame = NULL;
AVPacket packet;
int frameFinished;
float aspect_ratio;
struct SwsContext *sws_ctx = NULL;
SDL_Overlay *bmp;
SDL_Surface *screen;
SDL_Rect rect;
SDL_Event event;
if(argc < 2) {
fprintf(stderr, "Usage: test <file>\n");
exit(1);
}
av_register_all();
if(SDL_Init(SDL_INIT_VIDEO | SDL_INIT_AUDIO | SDL_INIT_TIMER)) {
fprintf(stderr, "Could not initialize SDL - %s\n", SDL_GetError());
exit(1);
}
if(avformat_open_input(&pFormatCtx, argv[1], NULL, NULL)!=0)
return -1;
if(avformat_find_stream_info(pFormatCtx, NULL)<0)
return -1;
av_dump_format(pFormatCtx, 0, argv[1], 0);
videoStream=-1;
for(i=0; i<pFormatCtx->nb_streams; i++)
if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_VIDEO) {
videoStream=i;
break;
}
if(videoStream==-1)
return -1;
pCodecCtxOrig=pFormatCtx->streams[videoStream]->codec;
pCodec=avcodec_find_decoder(pCodecCtxOrig->codec_id);
if(pCodec==NULL) {
fprintf(stderr, "Unsupported codec!\n");
return -1;
}
pCodecCtx = avcodec_alloc_context3(pCodec);
if(avcodec_copy_context(pCodecCtx, pCodecCtxOrig) != 0) {
fprintf(stderr, "Couldn't copy codec context");
return -1;
}
if(avcodec_open2(pCodecCtx, pCodec, NULL)<0)
return -1;
pFrame=av_frame_alloc();
#ifndef __DARWIN__
screen = SDL_SetVideoMode(pCodecCtx->width, pCodecCtx->height, 0, 0);
#else
screen = SDL_SetVideoMode(pCodecCtx->width, pCodecCtx->height, 24, 0);
#endif
if(!screen) {
fprintf(stderr, "SDL: could not set video mode - exiting\n");
exit(1);
}
bmp = SDL_CreateYUVOverlay(pCodecCtx->width,
pCodecCtx->height,
SDL_YV12_OVERLAY,
screen);
sws_ctx = sws_getContext(pCodecCtx->width,
pCodecCtx->height,
pCodecCtx->pix_fmt,
pCodecCtx->width,
pCodecCtx->height,
PIX_FMT_YUV420P,
SWS_BILINEAR,
NULL,
NULL,
NULL
);
i=0;
while(av_read_frame(pFormatCtx, &packet)>=0) {
if(packet.stream_index==videoStream) {
avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished, &packet);
if(frameFinished) {
SDL_LockYUVOverlay(bmp);
AVPicture pict;
pict.data[0] = bmp->pixels[0];
pict.data[1] = bmp->pixels[2];
pict.data[2] = bmp->pixels[1];
pict.linesize[0] = bmp->pitches[0];
pict.linesize[1] = bmp->pitches[2];
pict.linesize[2] = bmp->pitches[1];
sws_scale(sws_ctx, (uint8_t const * const *)pFrame->data,
pFrame->linesize, 0, pCodecCtx->height,
pict.data, pict.linesize);
SDL_UnlockYUVOverlay(bmp);
rect.x = 0;
rect.y = 0;
rect.w = pCodecCtx->width;
rect.h = pCodecCtx->height;
SDL_DisplayYUVOverlay(bmp, &rect);
}
}
av_free_packet(&packet);
SDL_PollEvent(&event);
switch(event.type) {
case SDL_QUIT:
SDL_Quit();
exit(0);
break;
default:
break;
}
}
av_frame_free(&pFrame);
avcodec_close(pCodecCtx);
avcodec_close(pCodecCtxOrig);
avformat_close_input(&pFormatCtx);
return 0;
}
SDL and video
For drawing on the screen we will use SDL. SDL stands for Simple Direct Layer . It is an excellent cross-platform multimedia library used in many projects. You can get the library on the official website or download the developer package for your operating system, if any. You will need libraries to compile the code from this lesson (all the other lessons, by the way, this also applies).SDL has many methods for drawing on the screen. One way to display movies is what is called YUV overlay .Formally, not even YUV, but YCbCr. Some people, by the way, get very burned when “YCbCr” is called as “YUV”. Generally speaking, YUV is an analog format, and YCbCr is a digital format. FFmpeg and SDL in their code and in macros designate YCbCr as YUV, but that is.YUV is a method of storing raw image data such as RGB. Roughly speaking, Y is a component of brightness , and U and V are components of color . (This is more complicated than RGB because part of the color information is discarded, and you can only have 1 measurement of U and V for every 2 measurements of Y ). YUV overlayin SDL accepts a raw YUV dataset and displays it. It accepts 4 different kinds of YUV formats, but YV12 is the fastest of them. There is another YUV format called YUV420P that matches YV12, except that the arrays of U and V are swapped. 420 means that it is sampled at a ratio of 4: 2: 0, that is, for every 4 measurements of brightness there is 1 color measurement, so the color information is distributed in quarters. This is a good way to save bandwidth because the human eye still does not notice these changes. The Latin letter “P” in the name indicates that the format is “planar”, which simply means that the components are Y ,U and V are in separate arrays. FFmpeg can convert images to YUV420P , which is very helpful, because many video streams are already stored in this format or are easily converted to it.Thus, our current plan is to replace the SaveFrame () function from the previous lesson and display our frame instead. But first you need to get acquainted with the basic features of the SDL library. To get started, connect the libraries and initialize the SDL:#include <SDL.h>
#include <SDL_thread.h>
if(SDL_Init(SDL_INIT_VIDEO | SDL_INIT_AUDIO | SDL_INIT_TIMER)) {
fprintf(stderr, "Could not initialize SDL - %s\n", SDL_GetError());
exit(1);
}
SDL_Init () essentially tells the library which functions we will use. SDL_GetError (), of course, this is our convenient function for debugging.Display creation
Now we need a place on the screen to arrange the elements. The main area for displaying images with SDL is called the surface :SDL_Surface *screen;
screen = SDL_SetVideoMode(pCodecCtx->width, pCodecCtx->height, 0, 0);
if(!screen) {
fprintf(stderr, "SDL: could not set video mode - exiting\n");
exit(1);
}
So we set up a screen with a given width and height. The next option is the bit depth of the screen - 0 - this is a special value that means "the same as the current display."Now we create a YUV overlay on this screen so that we can output video to it, and configure our SWSContext to convert image data to YUV420 :SDL_Overlay *bmp = NULL;
struct SWSContext *sws_ctx = NULL;
bmp = SDL_CreateYUVOverlay(pCodecCtx->width, pCodecCtx->height,
SDL_YV12_OVERLAY, screen);
sws_ctx = sws_getContext(pCodecCtx->width,
pCodecCtx->height,
pCodecCtx->pix_fmt,
pCodecCtx->width,
pCodecCtx->height,
PIX_FMT_YUV420P,
SWS_BILINEAR,
NULL,
NULL,
NULL
);
As mentioned, we use YV12 to display the image and get the YUV420 data from FFmpeg.Image display
Well, that was easy enough! Now we just need to show the image. Let's go all the way to the place where we had the finished shot. We can get rid of everything that we had for the RGB frame and we are going to replace SaveFrame () with our display code. To display the image, we are going to create an AVPicture structure and set the data pointers and line size for it for our YUV overlay : if(frameFinished) {
SDL_LockYUVOverlay(bmp);
AVPicture pict;
pict.data[0] = bmp->pixels[0];
pict.data[1] = bmp->pixels[2];
pict.data[2] = bmp->pixels[1];
pict.linesize[0] = bmp->pitches[0];
pict.linesize[1] = bmp->pitches[2];
pict.linesize[2] = bmp->pitches[1];
sws_scale(sws_ctx, (uint8_t const * const *)pFrame->data,
pFrame->linesize, 0, pCodecCtx->height,
pict.data, pict.linesize);
SDL_UnlockYUVOverlay(bmp);
At first, we block the overlay, because we plan to write to it. This is a good habit so that later there are no problems. The AVPicture structure , as shown above, has a data pointer, which is an array of 4 pointers. Since here we are dealing with the YUV420P , we have only 3 channels and therefore only 3 data sets. Other formats may have a fourth pointer for the alpha channel or something else. Line size is what it looks like. Similar structures in our YUV overlay are variables for pixels and heights. (Pitches, pitches - if expressed in terms of SDL to indicate the width of a given data line.) So, we indicate three pict.data arrays on our overlay, so when we write inpict , we are actually recording in our overlay, which, of course, already has the necessary space allocated specifically for it. In the same way, we get line size information directly from our overlay. We change the conversion format to PIX_FMT_YUV420P and use sws_scale as before.Image drawing
But we still need to specify for the SDL so that it really shows the data that we provided to it. We also pass a rectangle to this function, which indicates where the movie should go, to what width and height it should be scaled. Thus, the SDL scales for us, and this can help your GPU to scale faster:SDL_Rect rect;
if(frameFinished) {
sws_scale(sws_ctx, (uint8_t const * const *)pFrame->data,
pFrame->linesize, 0, pCodecCtx->height,
pict.data, pict.linesize);
SDL_UnlockYUVOverlay(bmp);
rect.x = 0;
rect.y = 0;
rect.w = pCodecCtx->width;
rect.h = pCodecCtx->height;
SDL_DisplayYUVOverlay(bmp, &rect);
}
Now our video is displayed!Let's show one more feature of SDL: system of events . SDL is configured in such a way that when you enter or move the mouse in the SDL application or send a signal to it, an event is generated. Your program then checks for these events if it is intended to process user input. Your program can also create events to send SDL events to the system. This is especially useful for multi-threaded programming with SDL, which we will see in lesson number 4. In our program, we are going to check events immediately after the processing of the package. At the moment, we are going to handle the SDL_QUIT event so that we can exit:SDL_Event event;
av_free_packet(&packet);
SDL_PollEvent(&event);
switch(event.type) {
case SDL_QUIT:
SDL_Quit();
exit(0);
break;
default:
break;
}
And so we live! We get rid of all the old garbage and we are ready to compile. If you use Linux or something like Linux, the best way to compile using the SDL libraries is:gcc -o tutorial02 tutorial02.c -lavformat -lavcodec -lswscale -lz -lm \
`sdl-config --cflags --libs`
sdl-config simply displays the necessary flags for gcc to correctly enable the SDL libraries. You may have to do something else to make this compile on your system; please check the SDL documentation for your system for any firefighter. Once compiled, continue and run.What happens when you run this program? The video seems to be going crazy! In fact, we simply display all the video frames as quickly as we can extract them from the movie file. We don’t have the code right now to find out when we need to show the video. In the end (in lesson number 5) we will begin to synchronize the video. But at the moment we are missing something equally important: the sound!
Lesson 3: Play Sound ← ⇑ →
Full listing: tutorial03.c
#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libswscale/swscale.h>
#include <SDL.h>
#include <SDL_thread.h>
#ifdef __MINGW32__
#undef main
#endif
#include <stdio.h>
#include <assert.h>
#if LIBAVCODEC_VERSION_INT < AV_VERSION_INT(55,28,1)
#define av_frame_alloc avcodec_alloc_frame
#define av_frame_free avcodec_free_frame
#endif
#define SDL_AUDIO_BUFFER_SIZE 1024
#define MAX_AUDIO_FRAME_SIZE 192000
typedef struct PacketQueue {
AVPacketList *first_pkt, *last_pkt;
int nb_packets;
int size;
SDL_mutex *mutex;
SDL_cond *cond;
} PacketQueue;
PacketQueue audioq;
int quit = 0;
void packet_queue_init(PacketQueue *q) {
memset(q, 0, sizeof(PacketQueue));
q->mutex = SDL_CreateMutex();
q->cond = SDL_CreateCond();
}
int packet_queue_put(PacketQueue *q, AVPacket *pkt) {
AVPacketList *pkt1;
if(av_dup_packet(pkt) < 0) {
return -1;
}
pkt1 = av_malloc(sizeof(AVPacketList));
if (!pkt1)
return -1;
pkt1->pkt = *pkt;
pkt1->next = NULL;
SDL_LockMutex(q->mutex);
if (!q->last_pkt)
q->first_pkt = pkt1;
else
q->last_pkt->next = pkt1;
q->last_pkt = pkt1;
q->nb_packets++;
q->size += pkt1->pkt.size;
SDL_CondSignal(q->cond);
SDL_UnlockMutex(q->mutex);
return 0;
}
static int packet_queue_get(PacketQueue *q, AVPacket *pkt, int block)
{
AVPacketList *pkt1;
int ret;
SDL_LockMutex(q->mutex);
for(;;) {
if(quit) {
ret = -1;
break;
}
pkt1 = q->first_pkt;
if (pkt1) {
q->first_pkt = pkt1->next;
if (!q->first_pkt)
q->last_pkt = NULL;
q->nb_packets--;
q->size -= pkt1->pkt.size;
*pkt = pkt1->pkt;
av_free(pkt1);
ret = 1;
break;
} else if (!block) {
ret = 0;
break;
} else {
SDL_CondWait(q->cond, q->mutex);
}
}
SDL_UnlockMutex(q->mutex);
return ret;
}
int audio_decode_frame(AVCodecContext *aCodecCtx, uint8_t *audio_buf, int buf_size) {
static AVPacket pkt;
static uint8_t *audio_pkt_data = NULL;
static int audio_pkt_size = 0;
static AVFrame frame;
int len1, data_size = 0;
for(;;) {
while(audio_pkt_size > 0) {
int got_frame = 0;
len1 = avcodec_decode_audio4(aCodecCtx, &frame, &got_frame, &pkt);
if(len1 < 0) {
audio_pkt_size = 0;
break;
}
audio_pkt_data += len1;
audio_pkt_size -= len1;
data_size = 0;
if(got_frame) {
data_size = av_samples_get_buffer_size(NULL,
aCodecCtx->channels,
frame.nb_samples,
aCodecCtx->sample_fmt,
1);
assert(data_size <= buf_size);
memcpy(audio_buf, frame.data[0], data_size);
}
if(data_size <= 0) {
continue;
}
return data_size;
}
if(pkt.data)
av_free_packet(&pkt);
if(quit) {
return -1;
}
if(packet_queue_get(&audioq, &pkt, 1) < 0) {
return -1;
}
audio_pkt_data = pkt.data;
audio_pkt_size = pkt.size;
}
}
void audio_callback(void *userdata, Uint8 *stream, int len) {
AVCodecContext *aCodecCtx = (AVCodecContext *)userdata;
int len1, audio_size;
static uint8_t audio_buf[(MAX_AUDIO_FRAME_SIZE * 3) / 2];
static unsigned int audio_buf_size = 0;
static unsigned int audio_buf_index = 0;
while(len > 0) {
if(audio_buf_index >= audio_buf_size) {
audio_size = audio_decode_frame(aCodecCtx, audio_buf, sizeof(audio_buf));
if(audio_size < 0) {
audio_buf_size = 1024;
memset(audio_buf, 0, audio_buf_size);
} else {
audio_buf_size = audio_size;
}
audio_buf_index = 0;
}
len1 = audio_buf_size - audio_buf_index;
if(len1 > len)
len1 = len;
memcpy(stream, (uint8_t *)audio_buf + audio_buf_index, len1);
len -= len1;
stream += len1;
audio_buf_index += len1;
}
}
int main(int argc, char *argv[]) {
AVFormatContext *pFormatCtx = NULL;
int i, videoStream, audioStream;
AVCodecContext *pCodecCtxOrig = NULL;
AVCodecContext *pCodecCtx = NULL;
AVCodec *pCodec = NULL;
AVFrame *pFrame = NULL;
AVPacket packet;
int frameFinished;
struct SwsContext *sws_ctx = NULL;
AVCodecContext *aCodecCtxOrig = NULL;
AVCodecContext *aCodecCtx = NULL;
AVCodec *aCodec = NULL;
SDL_Overlay *bmp;
SDL_Surface *screen;
SDL_Rect rect;
SDL_Event event;
SDL_AudioSpec wanted_spec, spec;
if(argc < 2) {
fprintf(stderr, "Usage: test <file>\n");
exit(1);
}
av_register_all();
if(SDL_Init(SDL_INIT_VIDEO | SDL_INIT_AUDIO | SDL_INIT_TIMER)) {
fprintf(stderr, "Could not initialize SDL - %s\n", SDL_GetError());
exit(1);
}
if(avformat_open_input(&pFormatCtx, argv[1], NULL, NULL)!=0)
return -1;
if(avformat_find_stream_info(pFormatCtx, NULL)<0)
return -1;
av_dump_format(pFormatCtx, 0, argv[1], 0);
videoStream=-1;
audioStream=-1;
for(i=0; i<pFormatCtx->nb_streams; i++) {
if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_VIDEO &&
videoStream < 0) {
videoStream=i;
}
if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_AUDIO &&
audioStream < 0) {
audioStream=i;
}
}
if(videoStream==-1)
return -1;
if(audioStream==-1)
return -1;
aCodecCtxOrig=pFormatCtx->streams[audioStream]->codec;
aCodec = avcodec_find_decoder(aCodecCtxOrig->codec_id);
if(!aCodec) {
fprintf(stderr, "Unsupported codec!\n");
return -1;
}
aCodecCtx = avcodec_alloc_context3(aCodec);
if(avcodec_copy_context(aCodecCtx, aCodecCtxOrig) != 0) {
fprintf(stderr, "Couldn't copy codec context");
return -1;
}
wanted_spec.freq = aCodecCtx->sample_rate;
wanted_spec.format = AUDIO_S16SYS;
wanted_spec.channels = aCodecCtx->channels;
wanted_spec.silence = 0;
wanted_spec.samples = SDL_AUDIO_BUFFER_SIZE;
wanted_spec.callback = audio_callback;
wanted_spec.userdata = aCodecCtx;
if(SDL_OpenAudio(&wanted_spec, &spec) < 0) {
fprintf(stderr, "SDL_OpenAudio: %s\n", SDL_GetError());
return -1;
}
avcodec_open2(aCodecCtx, aCodec, NULL);
packet_queue_init(&audioq);
SDL_PauseAudio(0);
pCodecCtxOrig=pFormatCtx->streams[videoStream]->codec;
pCodec=avcodec_find_decoder(pCodecCtxOrig->codec_id);
if(pCodec==NULL) {
fprintf(stderr, "Unsupported codec!\n");
return -1;
}
pCodecCtx = avcodec_alloc_context3(pCodec);
if(avcodec_copy_context(pCodecCtx, pCodecCtxOrig) != 0) {
fprintf(stderr, "Couldn't copy codec context");
return -1;
}
if(avcodec_open2(pCodecCtx, pCodec, NULL)<0)
return -1;
pFrame=av_frame_alloc();
#ifndef __DARWIN__
screen = SDL_SetVideoMode(pCodecCtx->width, pCodecCtx->height, 0, 0);
#else
screen = SDL_SetVideoMode(pCodecCtx->width, pCodecCtx->height, 24, 0);
#endif
if(!screen) {
fprintf(stderr, "SDL: could not set video mode - exiting\n");
exit(1);
}
bmp = SDL_CreateYUVOverlay(pCodecCtx->width,
pCodecCtx->height,
SDL_YV12_OVERLAY,
screen);
sws_ctx = sws_getContext(pCodecCtx->width,
pCodecCtx->height,
pCodecCtx->pix_fmt,
pCodecCtx->width,
pCodecCtx->height,
PIX_FMT_YUV420P,
SWS_BILINEAR,
NULL,
NULL,
NULL
);
i=0;
while(av_read_frame(pFormatCtx, &packet)>=0) {
if(packet.stream_index==videoStream) {
avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished, &packet);
if(frameFinished) {
SDL_LockYUVOverlay(bmp);
AVPicture pict;
pict.data[0] = bmp->pixels[0];
pict.data[1] = bmp->pixels[2];
pict.data[2] = bmp->pixels[1];
pict.linesize[0] = bmp->pitches[0];
pict.linesize[1] = bmp->pitches[2];
pict.linesize[2] = bmp->pitches[1];
sws_scale(sws_ctx, (uint8_t const * const *)pFrame->data,
pFrame->linesize, 0, pCodecCtx->height,
pict.data, pict.linesize);
SDL_UnlockYUVOverlay(bmp);
rect.x = 0;
rect.y = 0;
rect.w = pCodecCtx->width;
rect.h = pCodecCtx->height;
SDL_DisplayYUVOverlay(bmp, &rect);
av_free_packet(&packet);
}
} else if(packet.stream_index==audioStream) {
packet_queue_put(&audioq, &packet);
} else {
av_free_packet(&packet);
}
SDL_PollEvent(&event);
switch(event.type) {
case SDL_QUIT:
quit = 1;
SDL_Quit();
exit(0);
break;
default:
break;
}
}
av_frame_free(&pFrame);
avcodec_close(pCodecCtxOrig);
avcodec_close(pCodecCtx);
avcodec_close(aCodecCtxOrig);
avcodec_close(aCodecCtx);
avformat_close_input(&pFormatCtx);
return 0;
}
Audio
Now we would like the sound to play in the application. SDL also provides us with methods for playing sound. The SDL_OpenAudio () function is used to open the audio device itself. It takes as arguments the SDL_AudioSpec structure , which contains all the information about the audio that we are going to play.Before we show how to configure this, we first explain how the computer processes audio in general. Digital audio consists of a long stream of samples, each of which represents a specific meaning of a sound wave. Sounds are recorded at a specific sampling rate, which simply indicates how quickly each sample is played, and measured by the number of samples per second. The approximate sampling frequencies are 22,050 and 44,100 samples per second, which are the speeds used for radio and CD, respectively. In addition, most audio can have more than one channel for stereo or surround sound, so, for example, if the sample is in stereo, the samples will come two at a time. When we get the data from the movie file, we don’t know how many samples we will get, but FFmpeg doesn’t produce broken samples — this also means that it will not separate the stereo sample.The method for playing audio in SDL is as follows. Sound parameters are configured: sampling frequency, number of channels, etc. And also set the callback function and user data. When we start playing sound, the SDL will constantly call this callback function and ask it to fill the audio buffer with a certain number of bytes. After we put this information into the SDL_AudioSpec structure , we call SDL_OpenAudio (), which will open the audio device and return another AudioSpec structure to us . These are the characteristics that we will actually use - there is no guarantee that we will get exactly what we asked for!Audio setting
Just keep it in mind for now, because we don’t have any information about audio streams yet! Let's go back to the place in our code where we found the video stream and find out which stream is the audio stream:
videoStream=-1;
audioStream=-1;
for(i=0; i < pFormatCtx->nb_streams; i++) {
if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_VIDEO
&&
videoStream < 0) {
videoStream=i;
}
if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_AUDIO &&
audioStream < 0) {
audioStream=i;
}
}
if(videoStream==-1)
return -1;
if(audioStream==-1)
return -1;
Here we can get all the information we want from the AVCodecContext from the stream, just like we did with the video stream:AVCodecContext *aCodecCtxOrig;
AVCodecContext *aCodecCtx;
aCodecCtxOrig=pFormatCtx->streams[audioStream]->codec;
If you remember, in previous lessons, we still have to open the audio codec itself. It's simple:AVCodec *aCodec;
aCodec = avcodec_find_decoder(aCodecCtx->codec_id);
if(!aCodec) {
fprintf(stderr, "Unsupported codec!\n");
return -1;
}
aCodecCtx = avcodec_alloc_context3(aCodec);
if(avcodec_copy_context(aCodecCtx, aCodecCtxOrig) != 0) {
fprintf(stderr, "Couldn't copy codec context");
return -1;
}
avcodec_open2(aCodecCtx, aCodec, NULL);
In the context of the codec contains all the information necessary to configure our audio:wanted_spec.freq = aCodecCtx->sample_rate;
wanted_spec.format = AUDIO_S16SYS;
wanted_spec.channels = aCodecCtx->channels;
wanted_spec.silence = 0;
wanted_spec.samples = SDL_AUDIO_BUFFER_SIZE;
wanted_spec.callback = audio_callback;
wanted_spec.userdata = aCodecCtx;
if(SDL_OpenAudio(&wanted_spec, &spec) < 0) {
fprintf(stderr, "SDL_OpenAudio: %s\n", SDL_GetError());
return -1;
}
Let's go over each item:- freq (frequency): sampling rate, as explained earlier.
- format (): SDL , . «S» «S16SYS» «», 16 , 16 , «SYS» , , . , avcodec_decode_audio2 .
- channels (): .
- silence (): , . 0.
- samples (): , , SDL , . - 512 8192; FFplay, , 1024.
- callback (callback): here we pass the real callback function. We'll talk more about the callback function later.
- userdata : The SDL will give our callback a null pointer to any user data we want. We want to let him know about our codec context; a little lower it will be clear why.
Finally, open the audio with SDL_OpenAudio .Queues
And it is necessary! Now we are ready to extract audio information from the stream. But what to do with this information? We will continuously receive packets from the movie file, but at the same time, the SDL will call the callback function! The solution is to create some kind of global structure into which we can insert audio packets so that our audio_callback has something to receive audio data! So, here is what we will do to create the packet queue. FFmpeg even has a structure to help with this: AVPacketList , which is just a linked list for packages. Here is our queue structure:typedef struct PacketQueue {
AVPacketList *first_pkt, *last_pkt;
int nb_packets;
int size;
SDL_mutex *mutex;
SDL_cond *cond;
} PacketQueue;
First, we must indicate that nb_packets is different in size - the size refers to the size of the byte we get from packet-> size . Notice we have a mutex and a condition variable. This is because the SDL performs the audio process as a separate stream. If we do not block the queue properly, we can really ruin our data. Let's see how the queue is implemented. Every self-respecting programmer should know how to create queues, but we will also show how to do this, so that it is easier for you to learn the SDL functions.First, we create a function to initialize the queue:void packet_queue_init(PacketQueue *q) {
memset(q, 0, sizeof(PacketQueue));
q->mutex = SDL_CreateMutex();
q->cond = SDL_CreateCond();
}
Then create a function to place objects in our queue:int packet_queue_put(PacketQueue *q, AVPacket *pkt) {
AVPacketList *pkt1;
if(av_dup_packet(pkt) < 0) {
return -1;
}
pkt1 = av_malloc(sizeof(AVPacketList));
if (!pkt1)
return -1;
pkt1->pkt = *pkt;
pkt1->next = NULL;
SDL_LockMutex(q->mutex);
if (!q->last_pkt)
q->first_pkt = pkt1;
else
q->last_pkt->next = pkt1;
q->last_pkt = pkt1;
q->nb_packets++;
q->size += pkt1->pkt.size;
SDL_CondSignal(q->cond);
SDL_UnlockMutex(q->mutex);
return 0;
}
SDL_LockMutex () blocks the mutex in the queue so that we can add something, and then SDL_CondSignal () sends a signal to our get function (if it expects it) through our conditional variable to tell it that there is data and can be continued, for further unlock mutex.Here is the corresponding get function . Notice how SDL_CondWait () creates the function block (i.e. pauses until we get the data) if we tell it to do this:int quit = 0;
static int packet_queue_get(PacketQueue *q, AVPacket *pkt, int block) {
AVPacketList *pkt1;
int ret;
SDL_LockMutex(q->mutex);
for(;;) {
if(quit) {
ret = -1;
break;
}
pkt1 = q->first_pkt;
if (pkt1) {
q->first_pkt = pkt1->next;
if (!q->first_pkt)
q->last_pkt = NULL;
q->nb_packets--;
q->size -= pkt1->pkt.size;
*pkt = pkt1->pkt;
av_free(pkt1);
ret = 1;
break;
} else if (!block) {
ret = 0;
break;
} else {
SDL_CondWait(q->cond, q->mutex);
}
}
SDL_UnlockMutex(q->mutex);
return ret;
}
As you can see, we wrapped the function in an eternal cycle, so we will definitely get some data if we want to block it. We avoid looping forever using the SDL_CondWait () function . Essentially, all CondWait does is wait for a signal from SDL_CondSignal () (or SDL_CondBroadcast ()) and then continue. However, it looks like we caught it in a mutex - if we hold the lock, our put function cannot queue anything! However, what SDL_CondWait () also does for us is to unblock the mutex that we give it, and then try again to lock it as soon as we receive the signal.For every fireman
You also see that we have a global quit variable that we check to make sure that we did not set an output signal in the program (SDL automatically processes TERM signals , etc.). Otherwise, the thread will continue forever, and we will have to kill the program with kill -9 : SDL_PollEvent(&event);
switch(event.type) {
case SDL_QUIT:
quit = 1;
We will set the exit flag to 1.We feed packages
It remains only to configure our queue:PacketQueue audioq;
main() {
...
avcodec_open2(aCodecCtx, aCodec, NULL);
packet_queue_init(&audioq);
SDL_PauseAudio(0);
SDL_PauseAudio () finally starts the audio unit. It reproduces silence if it does not receive data; but this does not happen immediately.So, we have a queue configured, now we are ready to send packets to her. We move on to our package reading cycle:while(av_read_frame(pFormatCtx, &packet)>=0) {
if(packet.stream_index==videoStream) {
....
}
} else if(packet.stream_index==audioStream) {
packet_queue_put(&audioq, &packet);
} else {
av_free_packet(&packet);
}
Please note that we do not release the package after queuing it. We will release it later when we decrypt.Retrieving packages
Now let's finally make our audio_callback function to fetch packets from the queue. The callback should look like:void callback(void *userdata, Uint8 *stream, int len)
userdata , is the pointer that we gave the SDL, stream is the buffer into which we will write audio data, and len is the size of this buffer. Here is the code:void audio_callback(void *userdata, Uint8 *stream, int len) {
AVCodecContext *aCodecCtx = (AVCodecContext *)userdata;
int len1, audio_size;
static uint8_t audio_buf[(AVCODEC_MAX_AUDIO_FRAME_SIZE * 3) / 2];
static unsigned int audio_buf_size = 0;
static unsigned int audio_buf_index = 0;
while(len > 0) {
if(audio_buf_index >= audio_buf_size) {
audio_size = audio_decode_frame(aCodecCtx, audio_buf,
sizeof(audio_buf));
if(audio_size < 0) {
audio_buf_size = 1024;
memset(audio_buf, 0, audio_buf_size);
} else {
audio_buf_size = audio_size;
}
audio_buf_index = 0;
}
len1 = audio_buf_size - audio_buf_index;
if(len1 > len)
len1 = len;
memcpy(stream, (uint8_t *)audio_buf + audio_buf_index, len1);
len -= len1;
stream += len1;
audio_buf_index += len1;
}
}
In fact, this is a simple loop that extracts data from another function that we wrote, audio_decode_frame (), saves the result in an intermediate buffer, tries to write len bytes to the stream and receives more data if we still do not have enough or save it for later, if we have something left. The audio_buf is 1.5 times the size of the largest audio frame that FFmpeg will give us, which gives us a good margin.Final Audio Decryption
Let's look at the insides of the audio_decode_frame decoder :int audio_decode_frame(AVCodecContext *aCodecCtx, uint8_t *audio_buf,
int buf_size) {
static AVPacket pkt;
static uint8_t *audio_pkt_data = NULL;
static int audio_pkt_size = 0;
static AVFrame frame;
int len1, data_size = 0;
for(;;) {
while(audio_pkt_size > 0) {
int got_frame = 0;
len1 = avcodec_decode_audio4(aCodecCtx, &frame, &got_frame, &pkt);
if(len1 < 0) {
audio_pkt_size = 0;
break;
}
audio_pkt_data += len1;
audio_pkt_size -= len1;
data_size = 0;
if(got_frame) {
data_size = av_samples_get_buffer_size(NULL,
aCodecCtx->channels,
frame.nb_samples,
aCodecCtx->sample_fmt,
1);
assert(data_size <= buf_size);
memcpy(audio_buf, frame.data[0], data_size);
}
if(data_size <= 0) {
continue;
}
return data_size;
}
if(pkt.data)
av_free_packet(&pkt);
if(quit) {
return -1;
}
if(packet_queue_get(&audioq, &pkt, 1) < 0) {
return -1;
}
audio_pkt_data = pkt.data;
audio_pkt_size = pkt.size;
}
}
The whole process actually starts near the end of the function, where we call packet_queue_get (). We take the packet from the queue and save the information from it. Then, when we have the package to work, we call avcodec_decode_audio4 (), which is very similar to its sister function avcodec_decode_video (), except that in this case the package can have more than one frame. Therefore, you may need to call it several times to get all the data from the packet. Having received the frame, we simply copy it to our audio buffer, making sure that data_size is smaller than our audio buffer. Also, remember about casting audio_bufto the correct type, because SDL gives an 8-bit int buffer, and FFmpeg gives us data in a 16-bit int buffer. You should also consider the difference between len1 and data_size . len1 is the size of the package we used, and data_size is the amount of raw data returned.When we have some data, we immediately return to find out if we need to get more data from the queue or we are done. If we still need to process the package, then stick to it. If you have completed the package, then finally release it.And it's all! We have audio transferred from the main read loop to the queue, which is then read by the audio_callback function, which transfers this data to the SDL, and the SDL transfers to your sound card. Go ahead and compile:gcc -o tutorial03 tutorial03.c -lavutil -lavformat -lavcodec -lswscale -lz -lm \
`sdl-config --cflags --libs`
Gip-gip-hooray! The video is still carried at maximum speed, but the sound is already playing as it should. Why is that? Yes, because the audio information has a sampling frequency - we pump out the audio information as fast as it turns out, but the audio is simply played in this stream in accordance with its sampling frequency.We are almost ripe for video and audio synchronization, but first we need to carry out a small reorganization of the program. The method of queuing sound and playing it using a separate stream worked very well: it made the code more manageable and more modular. Before we start synchronizing video with audio, we need to simplify the code. In the next series we will produce control flows!
Lesson 4: Multiple Threads ← ⇑ →
Full listing tutorial04.c
#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libswscale/swscale.h>
#include <SDL.h>
#include <SDL_thread.h>
#ifdef __MINGW32__
#undef main
#endif
#include <stdio.h>
#include <assert.h>
#include <math.h>
#if LIBAVCODEC_VERSION_INT < AV_VERSION_INT(55,28,1)
#define av_frame_alloc avcodec_alloc_frame
#define av_frame_free avcodec_free_frame
#endif
#define SDL_AUDIO_BUFFER_SIZE 1024
#define MAX_AUDIO_FRAME_SIZE 192000
#define MAX_AUDIOQ_SIZE (5 * 16 * 1024)
#define MAX_VIDEOQ_SIZE (5 * 256 * 1024)
#define FF_REFRESH_EVENT (SDL_USEREVENT)
#define FF_QUIT_EVENT (SDL_USEREVENT + 1)
#define VIDEO_PICTURE_QUEUE_SIZE 1
typedef struct PacketQueue {
AVPacketList *first_pkt, *last_pkt;
int nb_packets;
int size;
SDL_mutex *mutex;
SDL_cond *cond;
} PacketQueue;
typedef struct VideoPicture {
SDL_Overlay *bmp;
int width, height;
int allocated;
} VideoPicture;
typedef struct VideoState {
AVFormatContext *pFormatCtx;
int videoStream, audioStream;
AVStream *audio_st;
AVCodecContext *audio_ctx;
PacketQueue audioq;
uint8_t audio_buf[(AVCODEC_MAX_AUDIO_FRAME_SIZE * 3) / 2];
unsigned int audio_buf_size;
unsigned int audio_buf_index;
AVFrame audio_frame;
AVPacket audio_pkt;
uint8_t *audio_pkt_data;
int audio_pkt_size;
AVStream *video_st;
AVCodecContext *video_ctx;
PacketQueue videoq;
struct SwsContext *sws_ctx;
VideoPicture pictq[VIDEO_PICTURE_QUEUE_SIZE];
int pictq_size, pictq_rindex, pictq_windex;
SDL_mutex *pictq_mutex;
SDL_cond *pictq_cond;
SDL_Thread *parse_tid;
SDL_Thread *video_tid;
char filename[1024];
int quit;
} VideoState;
SDL_Surface *screen;
SDL_mutex *screen_mutex;
VideoState *global_video_state;
void packet_queue_init(PacketQueue *q) {
memset(q, 0, sizeof(PacketQueue));
q->mutex = SDL_CreateMutex();
q->cond = SDL_CreateCond();
}
int packet_queue_put(PacketQueue *q, AVPacket *pkt) {
AVPacketList *pkt1;
if(av_dup_packet(pkt) < 0) {
return -1;
}
pkt1 = av_malloc(sizeof(AVPacketList));
if (!pkt1)
return -1;
pkt1->pkt = *pkt;
pkt1->next = NULL;
SDL_LockMutex(q->mutex);
if (!q->last_pkt)
q->first_pkt = pkt1;
else
q->last_pkt->next = pkt1;
q->last_pkt = pkt1;
q->nb_packets++;
q->size += pkt1->pkt.size;
SDL_CondSignal(q->cond);
SDL_UnlockMutex(q->mutex);
return 0;
}
static int packet_queue_get(PacketQueue *q, AVPacket *pkt, int block)
{
AVPacketList *pkt1;
int ret;
SDL_LockMutex(q->mutex);
for(;;) {
if(global_video_state->quit) {
ret = -1;
break;
}
pkt1 = q->first_pkt;
if (pkt1) {
q->first_pkt = pkt1->next;
if (!q->first_pkt)
q->last_pkt = NULL;
q->nb_packets--;
q->size -= pkt1->pkt.size;
*pkt = pkt1->pkt;
av_free(pkt1);
ret = 1;
break;
} else if (!block) {
ret = 0;
break;
} else {
SDL_CondWait(q->cond, q->mutex);
}
}
SDL_UnlockMutex(q->mutex);
return ret;
}
int audio_decode_frame(VideoState *is, uint8_t *audio_buf, int buf_size) {
int len1, data_size = 0;
AVPacket *pkt = &is->audio_pkt;
for(;;) {
while(is->audio_pkt_size > 0) {
int got_frame = 0;
len1 = avcodec_decode_audio4(is->audio_ctx, &is->audio_frame, &got_frame, pkt);
if(len1 < 0) {
is->audio_pkt_size = 0;
break;
}
data_size = 0;
if(got_frame) {
data_size = av_samples_get_buffer_size(NULL,
is->audio_ctx->channels,
is->audio_frame.nb_samples,
is->audio_ctx->sample_fmt,
1);
assert(data_size <= buf_size);
memcpy(audio_buf, is->audio_frame.data[0], data_size);
}
is->audio_pkt_data += len1;
is->audio_pkt_size -= len1;
if(data_size <= 0) {
continue;
}
return data_size;
}
if(pkt->data)
av_free_packet(pkt);
if(is->quit) {
return -1;
}
if(packet_queue_get(&is->audioq, pkt, 1) < 0) {
return -1;
}
is->audio_pkt_data = pkt->data;
is->audio_pkt_size = pkt->size;
}
}
void audio_callback(void *userdata, Uint8 *stream, int len) {
VideoState *is = (VideoState *)userdata;
int len1, audio_size;
while(len > 0) {
if(is->audio_buf_index >= is->audio_buf_size) {
audio_size = audio_decode_frame(is, is->audio_buf, sizeof(is->audio_buf));
if(audio_size < 0) {
is->audio_buf_size = 1024;
memset(is->audio_buf, 0, is->audio_buf_size);
} else {
is->audio_buf_size = audio_size;
}
is->audio_buf_index = 0;
}
len1 = is->audio_buf_size - is->audio_buf_index;
if(len1 > len)
len1 = len;
memcpy(stream, (uint8_t *)is->audio_buf + is->audio_buf_index, len1);
len -= len1;
stream += len1;
is->audio_buf_index += len1;
}
}
static Uint32 sdl_refresh_timer_cb(Uint32 interval, void *opaque) {
SDL_Event event;
event.type = FF_REFRESH_EVENT;
event.user.data1 = opaque;
SDL_PushEvent(&event);
return 0;
}
static void schedule_refresh(VideoState *is, int delay) {
SDL_AddTimer(delay, sdl_refresh_timer_cb, is);
}
void video_display(VideoState *is) {
SDL_Rect rect;
VideoPicture *vp;
float aspect_ratio;
int w, h, x, y;
int i;
vp = &is->pictq[is->pictq_rindex];
if(vp->bmp) {
if(is->video_ctx->sample_aspect_ratio.num == 0) {
aspect_ratio = 0;
} else {
aspect_ratio = av_q2d(is->video_ctx->sample_aspect_ratio) *
is->video_ctx->width / is->video_ctx->height;
}
if(aspect_ratio <= 0.0) {
aspect_ratio = (float)is->video_ctx->width /
(float)is->video_ctx->height;
}
h = screen->h;
w = ((int)rint(h * aspect_ratio)) & -3;
if(w > screen->w) {
w = screen->w;
h = ((int)rint(w / aspect_ratio)) & -3;
}
x = (screen->w - w) / 2;
y = (screen->h - h) / 2;
rect.x = x;
rect.y = y;
rect.w = w;
rect.h = h;
SDL_LockMutex(screen_mutex);
SDL_DisplayYUVOverlay(vp->bmp, &rect);
SDL_UnlockMutex(screen_mutex);
}
}
void video_refresh_timer(void *userdata) {
VideoState *is = (VideoState *)userdata;
VideoPicture *vp;
if(is->video_st) {
if(is->pictq_size == 0) {
schedule_refresh(is, 1);
} else {
vp = &is->pictq[is->pictq_rindex];
schedule_refresh(is, 40);
video_display(is);
if(++is->pictq_rindex == VIDEO_PICTURE_QUEUE_SIZE) {
is->pictq_rindex = 0;
}
SDL_LockMutex(is->pictq_mutex);
is->pictq_size--;
SDL_CondSignal(is->pictq_cond);
SDL_UnlockMutex(is->pictq_mutex);
}
} else {
schedule_refresh(is, 100);
}
}
void alloc_picture(void *userdata) {
VideoState *is = (VideoState *)userdata;
VideoPicture *vp;
vp = &is->pictq[is->pictq_windex];
if(vp->bmp) {
SDL_FreeYUVOverlay(vp->bmp);
}
SDL_LockMutex(screen_mutex);
vp->bmp = SDL_CreateYUVOverlay(is->video_ctx->width,
is->video_ctx->height,
SDL_YV12_OVERLAY,
screen);
SDL_UnlockMutex(screen_mutex);
vp->width = is->video_ctx->width;
vp->height = is->video_ctx->height;
vp->allocated = 1;
}
int queue_picture(VideoState *is, AVFrame *pFrame) {
VideoPicture *vp;
int dst_pix_fmt;
AVPicture pict;
SDL_LockMutex(is->pictq_mutex);
while(is->pictq_size >= VIDEO_PICTURE_QUEUE_SIZE &&
!is->quit) {
SDL_CondWait(is->pictq_cond, is->pictq_mutex);
}
SDL_UnlockMutex(is->pictq_mutex);
if(is->quit)
return -1;
vp = &is->pictq[is->pictq_windex];
if(!vp->bmp ||
vp->width != is->video_ctx->width ||
vp->height != is->video_ctx->height) {
SDL_Event event;
vp->allocated = 0;
alloc_picture(is);
if(is->quit) {
return -1;
}
}
if(vp->bmp) {
SDL_LockYUVOverlay(vp->bmp);
dst_pix_fmt = PIX_FMT_YUV420P;
pict.data[0] = vp->bmp->pixels[0];
pict.data[1] = vp->bmp->pixels[2];
pict.data[2] = vp->bmp->pixels[1];
pict.linesize[0] = vp->bmp->pitches[0];
pict.linesize[1] = vp->bmp->pitches[2];
pict.linesize[2] = vp->bmp->pitches[1];
sws_scale(is->sws_ctx, (uint8_t const * const *)pFrame->data,
pFrame->linesize, 0, is->video_ctx->height,
pict.data, pict.linesize);
SDL_UnlockYUVOverlay(vp->bmp);
if(++is->pictq_windex == VIDEO_PICTURE_QUEUE_SIZE) {
is->pictq_windex = 0;
}
SDL_LockMutex(is->pictq_mutex);
is->pictq_size++;
SDL_UnlockMutex(is->pictq_mutex);
}
return 0;
}
int video_thread(void *arg) {
VideoState *is = (VideoState *)arg;
AVPacket pkt1, *packet = &pkt1;
int frameFinished;
AVFrame *pFrame;
pFrame = av_frame_alloc();
for(;;) {
if(packet_queue_get(&is->videoq, packet, 1) < 0) {
break;
}
avcodec_decode_video2(is->video_ctx, pFrame, &frameFinished, packet);
if(frameFinished) {
if(queue_picture(is, pFrame) < 0) {
break;
}
}
av_free_packet(packet);
}
av_frame_free(&pFrame);
return 0;
}
int stream_component_open(VideoState *is, int stream_index) {
AVFormatContext *pFormatCtx = is->pFormatCtx;
AVCodecContext *codecCtx = NULL;
AVCodec *codec = NULL;
SDL_AudioSpec wanted_spec, spec;
if(stream_index < 0 || stream_index >= pFormatCtx->nb_streams) {
return -1;
}
codec = avcodec_find_decoder(pFormatCtx->streams[stream_index]->codec->codec_id);
if(!codec) {
fprintf(stderr, "Unsupported codec!\n");
return -1;
}
codecCtx = avcodec_alloc_context3(codec);
if(avcodec_copy_context(codecCtx, pFormatCtx->streams[stream_index]->codec) != 0) {
fprintf(stderr, "Couldn't copy codec context");
return -1;
}
if(codecCtx->codec_type == AVMEDIA_TYPE_AUDIO) {
wanted_spec.freq = codecCtx->sample_rate;
wanted_spec.format = AUDIO_S16SYS;
wanted_spec.channels = codecCtx->channels;
wanted_spec.silence = 0;
wanted_spec.samples = SDL_AUDIO_BUFFER_SIZE;
wanted_spec.callback = audio_callback;
wanted_spec.userdata = is;
if(SDL_OpenAudio(&wanted_spec, &spec) < 0) {
fprintf(stderr, "SDL_OpenAudio: %s\n", SDL_GetError());
return -1;
}
}
if(avcodec_open2(codecCtx, codec, NULL) < 0) {
fprintf(stderr, "Unsupported codec!\n");
return -1;
}
switch(codecCtx->codec_type) {
case AVMEDIA_TYPE_AUDIO:
is->audioStream = stream_index;
is->audio_st = pFormatCtx->streams[stream_index];
is->audio_ctx = codecCtx;
is->audio_buf_size = 0;
is->audio_buf_index = 0;
memset(&is->audio_pkt, 0, sizeof(is->audio_pkt));
packet_queue_init(&is->audioq);
SDL_PauseAudio(0);
break;
case AVMEDIA_TYPE_VIDEO:
is->videoStream = stream_index;
is->video_st = pFormatCtx->streams[stream_index];
is->video_ctx = codecCtx;
packet_queue_init(&is->videoq);
is->video_tid = SDL_CreateThread(video_thread, is);
is->sws_ctx = sws_getContext(is->video_ctx->width, is->video_ctx->height,
is->video_ctx->pix_fmt, is->video_ctx->width,
is->video_ctx->height, PIX_FMT_YUV420P,
SWS_BILINEAR, NULL, NULL, NULL
);
break;
default:
break;
}
}
int decode_thread(void *arg) {
VideoState *is = (VideoState *)arg;
AVFormatContext *pFormatCtx;
AVPacket pkt1, *packet = &pkt1;
int video_index = -1;
int audio_index = -1;
int i;
is->videoStream=-1;
is->audioStream=-1;
global_video_state = is;
if(avformat_open_input(&pFormatCtx, is->filename, NULL, NULL)!=0)
return -1;
is->pFormatCtx = pFormatCtx;
if(avformat_find_stream_info(pFormatCtx, NULL)<0)
return -1;
av_dump_format(pFormatCtx, 0, is->filename, 0);
for(i=0; i<pFormatCtx->nb_streams; i++) {
if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_VIDEO &&
video_index < 0) {
video_index=i;
}
if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_AUDIO &&
audio_index < 0) {
audio_index=i;
}
}
if(audio_index >= 0) {
stream_component_open(is, audio_index);
}
if(video_index >= 0) {
stream_component_open(is, video_index);
}
if(is->videoStream < 0 || is->audioStream < 0) {
fprintf(stderr, "%s: could not open codecs\n", is->filename);
goto fail;
}
for(;;) {
if(is->quit) {
break;
}
if(is->audioq.size > MAX_AUDIOQ_SIZE ||
is->videoq.size > MAX_VIDEOQ_SIZE) {
SDL_Delay(10);
continue;
}
if(av_read_frame(is->pFormatCtx, packet) < 0) {
if(is->pFormatCtx->pb->error == 0) {
SDL_Delay(100);
continue;
} else {
break;
}
}
if(packet->stream_index == is->videoStream) {
packet_queue_put(&is->videoq, packet);
} else if(packet->stream_index == is->audioStream) {
packet_queue_put(&is->audioq, packet);
} else {
av_free_packet(packet);
}
}
while(!is->quit) {
SDL_Delay(100);
}
fail:
if(1){
SDL_Event event;
event.type = FF_QUIT_EVENT;
event.user.data1 = is;
SDL_PushEvent(&event);
}
return 0;
}
int main(int argc, char *argv[]) {
SDL_Event event;
VideoState *is;
is = av_mallocz(sizeof(VideoState));
if(argc < 2) {
fprintf(stderr, "Usage: test <file>\n");
exit(1);
}
av_register_all();
if(SDL_Init(SDL_INIT_VIDEO | SDL_INIT_AUDIO | SDL_INIT_TIMER)) {
fprintf(stderr, "Could not initialize SDL - %s\n", SDL_GetError());
exit(1);
}
#ifndef __DARWIN__
screen = SDL_SetVideoMode(640, 480, 0, 0);
#else
screen = SDL_SetVideoMode(640, 480, 24, 0);
#endif
if(!screen) {
fprintf(stderr, "SDL: could not set video mode - exiting\n");
exit(1);
}
screen_mutex = SDL_CreateMutex();
av_strlcpy(is->filename, argv[1], sizeof(is->filename));
is->pictq_mutex = SDL_CreateMutex();
is->pictq_cond = SDL_CreateCond();
schedule_refresh(is, 40);
is->parse_tid = SDL_CreateThread(decode_thread, is);
if(!is->parse_tid) {
av_free(is);
return -1;
}
for(;;) {
SDL_WaitEvent(&event);
switch(event.type) {
case FF_QUIT_EVENT:
case SDL_QUIT:
is->quit = 1;
SDL_Quit();
return 0;
break;
case FF_REFRESH_EVENT:
video_refresh_timer(event.user.data1);
break;
default:
break;
}
}
return 0;
}
Overview
Last time, we added audio support using the SDL audio features. SDL launched a thread making callbacks for the function that we defined each time a sound was needed. Now we are going to do the same with the video display. This makes the code more modular and easier to work with - especially if you want to add synchronization. So where do we start?Note that our main function handles a lot: it goes through the event loop, reads packets and decodes the video. What we are going to do is to divide everything into parts: we will have a stream responsible for decoding packets; then these packets are added to the queue and read by the corresponding audio and video streams. We have already tuned the audio stream as required; with a video stream it will be somewhat more difficult, since we will have to ensure that the video is shown on our own. We will add the actual display code to the main loop. But instead of showing the video every time we execute the loop, we integrate the video display into the event loop. The idea is to decode the video, save the received frame in another queue, then create your own event ( FF_REFRESH_EVENT), which we add to the event system, then when our event loop sees this event, it will display the next frame in the queue. Here is a convenient ASCII illustration of what is happening:The main reason for moving video display control through the event loop is that with the SDL_Delay stream we can precisely control when the next video frame appears on the screen. When we finally synchronize the video in the next lesson, just add a code that will schedule the next video update so that the correct image appears on the screen at the right time.Simplify the code
Let's clear the code a bit. We have all this information about audio and video codecs, and we're going to add queues, buffers, and God knows what else. All these things are for a certain logical unit, namely - for the film. So, we intend to create a large structure containing all this information called VideoState .typedef struct VideoState {
AVFormatContext *pFormatCtx;
int videoStream, audioStream;
AVStream *audio_st;
AVCodecContext *audio_ctx;
PacketQueue audioq;
uint8_t audio_buf[(AVCODEC_MAX_AUDIO_FRAME_SIZE * 3) / 2];
unsigned int audio_buf_size;
unsigned int audio_buf_index;
AVPacket audio_pkt;
uint8_t *audio_pkt_data;
int audio_pkt_size;
AVStream *video_st;
AVCodecContext *video_ctx;
PacketQueue videoq;
VideoPicture pictq[VIDEO_PICTURE_QUEUE_SIZE];
int pictq_size, pictq_rindex, pictq_windex;
SDL_mutex *pictq_mutex;
SDL_cond *pictq_cond;
SDL_Thread *parse_tid;
SDL_Thread *video_tid;
char filename[1024];
int quit;
} VideoState;
Here we see hints of what we are going to get in the end. First, we see the basic information - the format context and the audio and video stream indices, as well as the corresponding AVStream objects . Then we see that some of these audio buffers are moved to this structure. They ( audio_buf , audio_buf_size , etc.) were intended for information about audio that was still there (or was missing). We added another queue for video and a buffer (which will be used as a queue; for this we do not need any extravagant queues) for decoded frames (saved as an overlay). VideoPicture Structure- this is our own creation (we will see what will be in it when we come to it). You can also notice that we have allocated pointers for two additional streams that we will create, as well as an exit flag and a movie file name.So, now we return to the main function in order to see how this changes our program. Let's set up our VideoState structure :int main(int argc, char *argv[]) {
SDL_Event event;
VideoState *is;
is = av_mallocz(sizeof(VideoState));
av_mallocz () is a good function that will allocate memory for us and zero it.Then we initialize our locks for the display buffer ( pictq ), because since the event loop calls our display function - remember, the display function will retrieve pre-decoded frames from pictq . At the same time, our video decoder will put information into it - we don’t know who gets there first. I hope you understand that this is a classic race condition. Therefore, we distribute it now before starting any topics. Let's also copy the name of our movie into VideoState :av_strlcpy(is->filename, argv[1], sizeof(is->filename));
is->pictq_mutex = SDL_CreateMutex();
is->pictq_cond = SDL_CreateCond();
av_strlcpy is a function from FFmpeg that performs some additional border checks besides strncpy .Our first thread
Let's run our threads and do something real:schedule_refresh(is, 40);
is->parse_tid = SDL_CreateThread(decode_thread, is);
if(!is->parse_tid) {
av_free(is);
return -1;
}
schedule_refresh is a function that we will define later. What she does is tell the system to produce FF_REFRESH_EVENT after the specified number of milliseconds. This, in turn, will call the video update function when we see it in the event queue. But now let's look at SDL_CreateThread ().SDL_CreateThread () does just that - it spawns a new thread that has full access to all the memory of the original process, and starts the thread executed by the function we give it. This function will also transmit user-defined data. In this case, we call decode_thread () and attach our VideoState structure. There is nothing new in the first half of the function; it just does the job of opening the file and finding the index of audio and video streams. The only thing we do differently is to keep the format context in our large structure. After we found our stream indexes, we call another function that we define, stream_component_open (). This is a fairly natural way to separate, and since we do a lot of similar things to set up the video and audio codec, we reuse some code, making it a function. Stream_component_openfunction() Is the place where we discover our codec decoder, configure the sound parameters, save important information in our large structure and launch our audio and video streams. Here we also insert other parameters, such as forced use of the codec instead of its automatic detection, etc. Like this:int stream_component_open(VideoState *is, int stream_index) {
AVFormatContext *pFormatCtx = is->pFormatCtx;
AVCodecContext *codecCtx;
AVCodec *codec;
SDL_AudioSpec wanted_spec, spec;
if(stream_index < 0 || stream_index >= pFormatCtx->nb_streams) {
return -1;
}
codec = avcodec_find_decoder(pFormatCtx->streams[stream_index]->codec->codec_id);
if(!codec) {
fprintf(stderr, "Unsupported codec!\n");
return -1;
}
codecCtx = avcodec_alloc_context3(codec);
if(avcodec_copy_context(codecCtx, pFormatCtx->streams[stream_index]->codec) != 0) {
fprintf(stderr, "Couldn't copy codec context");
return -1;
}
if(codecCtx->codec_type == AVMEDIA_TYPE_AUDIO) {
wanted_spec.freq = codecCtx->sample_rate;
wanted_spec.callback = audio_callback;
wanted_spec.userdata = is;
if(SDL_OpenAudio(&wanted_spec, &spec) < 0) {
fprintf(stderr, "SDL_OpenAudio: %s\n", SDL_GetError());
return -1;
}
}
if(avcodec_open2(codecCtx, codec, NULL) < 0) {
fprintf(stderr, "Unsupported codec!\n");
return -1;
}
switch(codecCtx->codec_type) {
case AVMEDIA_TYPE_AUDIO:
is->audioStream = stream_index;
is->audio_st = pFormatCtx->streams[stream_index];
is->audio_ctx = codecCtx;
is->audio_buf_size = 0;
is->audio_buf_index = 0;
memset(&is->audio_pkt, 0, sizeof(is->audio_pkt));
packet_queue_init(&is->audioq);
SDL_PauseAudio(0);
break;
case AVMEDIA_TYPE_VIDEO:
is->videoStream = stream_index;
is->video_st = pFormatCtx->streams[stream_index];
is->video_ctx = codecCtx;
packet_queue_init(&is->videoq);
is->video_tid = SDL_CreateThread(video_thread, is);
is->sws_ctx = sws_getContext(is->video_st->codec->width, is->video_st->codec->height,
is->video_st->codec->pix_fmt, is->video_st->codec->width,
is->video_st->codec->height, PIX_FMT_YUV420P,
SWS_BILINEAR, NULL, NULL, NULL
);
break;
default:
break;
}
}
This is almost the same as the code that we had before, except that now it is generalized for audio and video. Note that instead of aCodecCtx, we have configured our large structure as user data for our audio callback. We also saved the streams themselves as audio_st and video_st . We also added our video queue and set it up just like our audio queue. The bottom line is to run video and audio streams. These bits do this: SDL_PauseAudio(0);
break;
is->video_tid = SDL_CreateThread(video_thread, is);
Remember SDL_PauseAudio () from the last lesson. SDL_CreateThread () is used the same way. Back to our video_thread () function .Before that, let's get back to the second half of our decode_thread () function . Essentially, it's just a for loop that reads a package and puts it in the right queue: for(;;) {
if(is->quit) {
break;
}
if(is->audioq.size > MAX_AUDIOQ_SIZE ||
is->videoq.size > MAX_VIDEOQ_SIZE) {
SDL_Delay(10);
continue;
}
if(av_read_frame(is->pFormatCtx, packet) < 0) {
if((is->pFormatCtx->pb->error) == 0) {
SDL_Delay(100);
continue;
} else {
break;
}
}
if(packet->stream_index == is->videoStream) {
packet_queue_put(&is->videoq, packet);
} else if(packet->stream_index == is->audioStream) {
packet_queue_put(&is->audioq, packet);
} else {
av_free_packet(packet);
}
}
There is nothing really new here, except that we now have a maximum size for our audio and video queue, and we have added read error checking. The format context has a ByteIOContext structure inside called pb . ByteIOContext is a structure that basically stores all the information about low-level files.After our for loop, we have all the code to wait for the rest of the program to complete or inform about it. This code is instructive because it shows how we push events - something that we will need later to display the video: while(!is->quit) {
SDL_Delay(100);
}
fail:
if(1){
SDL_Event event;
event.type = FF_QUIT_EVENT;
event.user.data1 = is;
SDL_PushEvent(&event);
}
return 0;
We get values for custom events using the SDL constant SDL_USEREVENT . The first user event must be set to SDL_USEREVENT , the next SDL_USEREVENT + 1 , etc. FF_QUIT_EVENT is defined in our program as SDL_USEREVENT + 1 . We can also pass user data if necessary, and here we pass our pointer to a large structure. Finally, we call SDL_PushEvent (). In our event loop switch, we just put this in the SDL_QUIT_EVENT sectionthat we had before. We will see our cycle of events in more detail; for now, just be sure that when we press FF_QUIT_EVENT we catch it later and switch the exit flag.Receive frame: video_thread
After you have prepared the codec, you can start the video stream. This stream reads packets from the video queue, decodes the video into frames, and then calls the queue_picture function to place the processed frame in the image queue:int video_thread(void *arg) {
VideoState *is = (VideoState *)arg;
AVPacket pkt1, *packet = &pkt1;
int frameFinished;
AVFrame *pFrame;
pFrame = av_frame_alloc();
for(;;) {
if(packet_queue_get(&is->videoq, packet, 1) < 0) {
break;
}
avcodec_decode_video2(is->video_st->codec, pFrame, &frameFinished, packet);
if(frameFinished) {
if(queue_picture(is, pFrame) < 0) {
break;
}
}
av_free_packet(packet);
}
av_free(pFrame);
return 0;
}
Most of this function should be understood by now. We just copied the avcodec_decode_video2 function here , simply replacing some arguments; for example, we have an AVStream stored in our large structure, so we get our codec from there. We just continue to receive packets from our video queue until someone tells us to exit or we find an error.Queue frame
Let's take a look at the function that stores our decoded pFrame in our image queue. Since our image queue is an overlay of SDL (presumably to allow the video display function to perform as few calculations as possible), we need to convert our frame into it. The data that we store in the image queue is the structure that we created:typedef struct VideoPicture {
SDL_Overlay *bmp;
int width, height;
int allocated;
} VideoPicture;
Our large structure contains a buffer of these files, where we can store them. However, we need to distribute SDL_Overlay ourselves (pay attention to the assigned flag, which shows whether we did it or not).To use this queue, we have two pointers - the write index and the read index. We also track how many actual images are in the buffer. To write to the queue, we first wait until our buffer is cleared, so that we have a place to store our VideoPicture . Then we check to see if we set the overlay in our record index? If not, you need to allocate memory. We must also reallocate the buffer if the window size has changed!int queue_picture(VideoState *is, AVFrame *pFrame) {
VideoPicture *vp;
int dst_pix_fmt;
AVPicture pict;
SDL_LockMutex(is->pictq_mutex);
while(is->pictq_size >= VIDEO_PICTURE_QUEUE_SIZE &&
!is->quit) {
SDL_CondWait(is->pictq_cond, is->pictq_mutex);
}
SDL_UnlockMutex(is->pictq_mutex);
if(is->quit)
return -1;
vp = &is->pictq[is->pictq_windex];
if(!vp->bmp ||
vp->width != is->video_st->codec->width ||
vp->height != is->video_st->codec->height) {
SDL_Event event;
vp->allocated = 0;
alloc_picture(is);
if(is->quit) {
return -1;
}
}
Let's take a look at the alloc_picture () function :void alloc_picture(void *userdata) {
VideoState *is = (VideoState *)userdata;
VideoPicture *vp;
vp = &is->pictq[is->pictq_windex];
if(vp->bmp) {
SDL_FreeYUVOverlay(vp->bmp);
}
SDL_LockMutex(screen_mutex);
vp->bmp = SDL_CreateYUVOverlay(is->video_st->codec->width,
is->video_st->codec->height,
SDL_YV12_OVERLAY,
screen);
SDL_UnlockMutex(screen_mutex);
vp->width = is->video_st->codec->width;
vp->height = is->video_st->codec->height;
vp->allocated = 1;
}
You should recognize the function SDL_CreateYUVOverlay , which we have moved from our main loop to this section. This code should be reasonably clear by now. However, now we have a mutex lock, because two threads cannot simultaneously write information to the screen! This will not allow our alloc_picture function to interfere with another function that will display the picture. (We created this lock as a global variable and initialized it in main (); see the code.) Remember that we keep the width and height in the VideoPicture structure , because we need to make sure that the size of our video does not change for some reason.Ok, we've settled it, and we have our overlay YUV, dedicated and ready to receive the image. Let's go back to queue_picture and look at the code to copy the frame to the overlay. This part should be familiar to you:int queue_picture(VideoState *is, AVFrame *pFrame) {
if(vp->bmp) {
SDL_LockYUVOverlay(vp->bmp);
dst_pix_fmt = PIX_FMT_YUV420P;
pict.data[0] = vp->bmp->pixels[0];
pict.data[1] = vp->bmp->pixels[2];
pict.data[2] = vp->bmp->pixels[1];
pict.linesize[0] = vp->bmp->pitches[0];
pict.linesize[1] = vp->bmp->pitches[2];
pict.linesize[2] = vp->bmp->pitches[1];
sws_scale(is->sws_ctx, (uint8_t const * const *)pFrame->data,
pFrame->linesize, 0, is->video_st->codec->height,
pict.data, pict.linesize);
SDL_UnlockYUVOverlay(vp->bmp);
if(++is->pictq_windex == VIDEO_PICTURE_QUEUE_SIZE) {
is->pictq_windex = 0;
}
SDL_LockMutex(is->pictq_mutex);
is->pictq_size++;
SDL_UnlockMutex(is->pictq_mutex);
}
return 0;
}
Here, most of it is just the code that we used earlier to populate the YUV overlay with our frame. The last bit simply “adds” our value to the queue. The queue works, values are added to it until it is full, and reading from it occurs while there is at least something in it. Therefore, it all depends on the value of is -> pictq_size , which requires us to block it. So, what are we doing here: increase the record pointer (and if necessary, start over), then block the queue and increase its size. Now our reader will know that there is more information about the queue, and if this makes our queue full, and our recorder will know about it.Video display
That's all for our video thread! Now we have completed all the free threads, except for one - remember how we called the schedule_refresh () function a long time ago ? Take a look at what actually happened:
static void schedule_refresh(VideoState *is, int delay) {
SDL_AddTimer(delay, sdl_refresh_timer_cb, is);
}
SDL_AddTimer () is an SDL function that simply performs a callback to a user-defined function after a certain number of milliseconds (and, if necessary, transfers some user-defined data). We will use this function to schedule video updates - each time we call it, it sets a timer that will trigger an event, which, in turn, will cause our main () function to call a function that extracts a frame from our queue picture and displays her! Phew! Three “which / which / which” in one sentence! So, let’s do the first thing to do - fire this event. This sends us to:static Uint32 sdl_refresh_timer_cb(Uint32 interval, void *opaque) {
SDL_Event event;
event.type = FF_REFRESH_EVENT;
event.user.data1 = opaque;
SDL_PushEvent(&event);
return 0;
}
The event is launched by our old friend. FF_REFRESH_EVENT is defined here as SDL_USEREVENT + 1 . It should be noted that when we return 0, the SDL stops the timer, so the callback does not execute again.Now that we have called FF_REFRESH_EVENT again , we need to process it in our event loop:for(;;) {
SDL_WaitEvent(&event);
switch(event.type) {
case FF_REFRESH_EVENT:
video_refresh_timer(event.user.data1);
break;
what sends us here to this function, which actually extracts data from our image queue:void video_refresh_timer(void *userdata) {
VideoState *is = (VideoState *)userdata;
VideoPicture *vp;
if(is->video_st) {
if(is->pictq_size == 0) {
schedule_refresh(is, 1);
} else {
vp = &is->pictq[is->pictq_rindex];
schedule_refresh(is, 80);
video_display(is);
if(++is->pictq_rindex == VIDEO_PICTURE_QUEUE_SIZE) {
is->pictq_rindex = 0;
}
SDL_LockMutex(is->pictq_mutex);
is->pictq_size--;
SDL_CondSignal(is->pictq_cond);
SDL_UnlockMutex(is->pictq_mutex);
}
} else {
schedule_refresh(is, 100);
}
}
At the moment, this function is pretty simple: it processes the queue while we have something, sets a timer to display the next video frame, calls video_display to actually show the video on the screen, then increases the counter in the queue, while reducing its size. You may notice that we are not really doing anything with vp in this function, and here's why: this is ahead. But a little later. We are going to use it to access time information when we begin to synchronize video with audio. Here, take a look at the place in the code where the comment “Timing code goes here” is written. In this section we are going to find out how soon we should show the next video frame, and then enter this value in the schedule_refresh function(). At the moment, we just enter a fictitious value of 80. Technically, you can guess and check this value and recompile it for each movie, but: 1) it will start to slow down after a while and 2) it's pretty stupid. Although, in the future we will return to this point.We are almost done. There is only one thing left to do: show the video! Here is the video_display function :void video_display(VideoState *is) {
SDL_Rect rect;
VideoPicture *vp;
float aspect_ratio;
int w, h, x, y;
int i;
vp = &is->pictq[is->pictq_rindex];
if(vp->bmp) {
if(is->video_st->codec->sample_aspect_ratio.num == 0) {
aspect_ratio = 0;
} else {
aspect_ratio = av_q2d(is->video_st->codec->sample_aspect_ratio) *
is->video_st->codec->width / is->video_st->codec->height;
}
if(aspect_ratio <= 0.0) {
aspect_ratio = (float)is->video_st->codec->width /
(float)is->video_st->codec->height;
}
h = screen->h;
w = ((int)rint(h * aspect_ratio)) & -3;
if(w > screen->w) {
w = screen->w;
h = ((int)rint(w / aspect_ratio)) & -3;
}
x = (screen->w - w) / 2;
y = (screen->h - h) / 2;
rect.x = x;
rect.y = y;
rect.w = w;
rect.h = h;
SDL_LockMutex(screen_mutex);
SDL_DisplayYUVOverlay(vp->bmp, &rect);
SDL_UnlockMutex(screen_mutex);
}
}
Since the screen can be of any size (we installed 640x480, and there are ways to configure it so that the user resizes), you need to dynamically determine how large the rectangular area should be for our film. So, first you need to find out the aspect ratio of our film, just the width divided by height. Some codecs will have an odd aspect ratio of the sample, which is simply the width / height of one pixel or sample. Since the height and width values in our codec context are measured in pixels, the actual aspect ratio is equal to the aspect ratio multiplied by the aspect ratio for the sample. Some codecs will show an aspect ratio of 0, which means that each pixel simply has a size of 1x1. Then we scale the film in such a wayso that it fits on the screen as much as possible. Bit reversal& -3 simply rounds the value to the nearest multiple of four. Then center the movie and call SDL_DisplayYUVOverlay () to make sure that the screen mutex is used to access it.And it's all? Are we done? You still need to rewrite the audio code to use the new VideoStruct , but these are trivial changes that can be seen in the sample code. The last thing we need to do is change our callback for the internal exit callback function in FFmpeg:VideoState *global_video_state;
int decode_interrupt_cb(void) {
return (global_video_state && global_video_state->quit);
}
Set global_video_state to a large structure in main ().So that's it! We compile:gcc -o tutorial04 tutorial04.c -lavutil -lavformat -lavcodec -lswscale -lz -lm \
`sdl-config --cflags --libs`
and enjoy the movie without syncing! In the next step, we will finally create a really working video player!
Lesson 5: Video Sync ← ⇑ →
Full listing tutorial05.c
#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libswscale/swscale.h>
#include <SDL.h>
#include <SDL_thread.h>
#ifdef __MINGW32__
#undef main
#endif
#include <stdio.h>
#include <assert.h>
#include <math.h>
#if LIBAVCODEC_VERSION_INT < AV_VERSION_INT(55,28,1)
#define av_frame_alloc avcodec_alloc_frame
#define av_frame_free avcodec_free_frame
#endif
#define SDL_AUDIO_BUFFER_SIZE 1024
#define MAX_AUDIO_FRAME_SIZE 192000
#define MAX_AUDIOQ_SIZE (5 * 16 * 1024)
#define MAX_VIDEOQ_SIZE (5 * 256 * 1024)
#define AV_SYNC_THRESHOLD 0.01
#define AV_NOSYNC_THRESHOLD 10.0
#define FF_REFRESH_EVENT (SDL_USEREVENT)
#define FF_QUIT_EVENT (SDL_USEREVENT + 1)
#define VIDEO_PICTURE_QUEUE_SIZE 1
typedef struct PacketQueue {
AVPacketList *first_pkt, *last_pkt;
int nb_packets;
int size;
SDL_mutex *mutex;
SDL_cond *cond;
} PacketQueue;
typedef struct VideoPicture {
SDL_Overlay *bmp;
int width, height;
int allocated;
double pts;
} VideoPicture;
typedef struct VideoState {
AVFormatContext *pFormatCtx;
int videoStream, audioStream;
double audio_clock;
AVStream *audio_st;
AVCodecContext *audio_ctx;
PacketQueue audioq;
uint8_t audio_buf[(AVCODEC_MAX_AUDIO_FRAME_SIZE * 3) / 2];
unsigned int audio_buf_size;
unsigned int audio_buf_index;
AVFrame audio_frame;
AVPacket audio_pkt;
uint8_t *audio_pkt_data;
int audio_pkt_size;
int audio_hw_buf_size;
double frame_timer;
double frame_last_pts;
double frame_last_delay;
double video_clock;
AVStream *video_st;
AVCodecContext *video_ctx;
PacketQueue videoq;
struct SwsContext *sws_ctx;
VideoPicture pictq[VIDEO_PICTURE_QUEUE_SIZE];
int pictq_size, pictq_rindex, pictq_windex;
SDL_mutex *pictq_mutex;
SDL_cond *pictq_cond;
SDL_Thread *parse_tid;
SDL_Thread *video_tid;
char filename[1024];
int quit;
} VideoState;
SDL_Surface *screen;
SDL_mutex *screen_mutex;
VideoState *global_video_state;
void packet_queue_init(PacketQueue *q) {
memset(q, 0, sizeof(PacketQueue));
q->mutex = SDL_CreateMutex();
q->cond = SDL_CreateCond();
}
int packet_queue_put(PacketQueue *q, AVPacket *pkt) {
AVPacketList *pkt1;
if(av_dup_packet(pkt) < 0) {
return -1;
}
pkt1 = av_malloc(sizeof(AVPacketList));
if (!pkt1)
return -1;
pkt1->pkt = *pkt;
pkt1->next = NULL;
SDL_LockMutex(q->mutex);
if (!q->last_pkt)
q->first_pkt = pkt1;
else
q->last_pkt->next = pkt1;
q->last_pkt = pkt1;
q->nb_packets++;
q->size += pkt1->pkt.size;
SDL_CondSignal(q->cond);
SDL_UnlockMutex(q->mutex);
return 0;
}
static int packet_queue_get(PacketQueue *q, AVPacket *pkt, int block)
{
AVPacketList *pkt1;
int ret;
SDL_LockMutex(q->mutex);
for(;;) {
if(global_video_state->quit) {
ret = -1;
break;
}
pkt1 = q->first_pkt;
if (pkt1) {
q->first_pkt = pkt1->next;
if (!q->first_pkt)
q->last_pkt = NULL;
q->nb_packets--;
q->size -= pkt1->pkt.size;
*pkt = pkt1->pkt;
av_free(pkt1);
ret = 1;
break;
} else if (!block) {
ret = 0;
break;
} else {
SDL_CondWait(q->cond, q->mutex);
}
}
SDL_UnlockMutex(q->mutex);
return ret;
}
double get_audio_clock(VideoState *is) {
double pts;
int hw_buf_size, bytes_per_sec, n;
pts = is->audio_clock;
hw_buf_size = is->audio_buf_size - is->audio_buf_index;
bytes_per_sec = 0;
n = is->audio_ctx->channels * 2;
if(is->audio_st) {
bytes_per_sec = is->audio_ctx->sample_rate * n;
}
if(bytes_per_sec) {
pts -= (double)hw_buf_size / bytes_per_sec;
}
return pts;
}
int audio_decode_frame(VideoState *is, uint8_t *audio_buf, int buf_size, double *pts_ptr) {
int len1, data_size = 0;
AVPacket *pkt = &is->audio_pkt;
double pts;
int n;
for(;;) {
while(is->audio_pkt_size > 0) {
int got_frame = 0;
len1 = avcodec_decode_audio4(is->audio_ctx, &is->audio_frame, &got_frame, pkt);
if(len1 < 0) {
is->audio_pkt_size = 0;
break;
}
data_size = 0;
if(got_frame) {
data_size = av_samples_get_buffer_size(NULL,
is->audio_ctx->channels,
is->audio_frame.nb_samples,
is->audio_ctx->sample_fmt,
1);
assert(data_size <= buf_size);
memcpy(audio_buf, is->audio_frame.data[0], data_size);
}
is->audio_pkt_data += len1;
is->audio_pkt_size -= len1;
if(data_size <= 0) {
continue;
}
pts = is->audio_clock;
*pts_ptr = pts;
n = 2 * is->audio_ctx->channels;
is->audio_clock += (double)data_size /
(double)(n * is->audio_ctx->sample_rate);
return data_size;
}
if(pkt->data)
av_free_packet(pkt);
if(is->quit) {
return -1;
}
if(packet_queue_get(&is->audioq, pkt, 1) < 0) {
return -1;
}
is->audio_pkt_data = pkt->data;
is->audio_pkt_size = pkt->size;
if(pkt->pts != AV_NOPTS_VALUE) {
is->audio_clock = av_q2d(is->audio_st->time_base)*pkt->pts;
}
}
}
void audio_callback(void *userdata, Uint8 *stream, int len) {
VideoState *is = (VideoState *)userdata;
int len1, audio_size;
double pts;
while(len > 0) {
if(is->audio_buf_index >= is->audio_buf_size) {
audio_size = audio_decode_frame(is, is->audio_buf, sizeof(is->audio_buf), &pts);
if(audio_size < 0) {
is->audio_buf_size = 1024;
memset(is->audio_buf, 0, is->audio_buf_size);
} else {
is->audio_buf_size = audio_size;
}
is->audio_buf_index = 0;
}
len1 = is->audio_buf_size - is->audio_buf_index;
if(len1 > len)
len1 = len;
memcpy(stream, (uint8_t *)is->audio_buf + is->audio_buf_index, len1);
len -= len1;
stream += len1;
is->audio_buf_index += len1;
}
}
static Uint32 sdl_refresh_timer_cb(Uint32 interval, void *opaque) {
SDL_Event event;
event.type = FF_REFRESH_EVENT;
event.user.data1 = opaque;
SDL_PushEvent(&event);
return 0;
}
static void schedule_refresh(VideoState *is, int delay) {
SDL_AddTimer(delay, sdl_refresh_timer_cb, is);
}
void video_display(VideoState *is) {
SDL_Rect rect;
VideoPicture *vp;
float aspect_ratio;
int w, h, x, y;
int i;
vp = &is->pictq[is->pictq_rindex];
if(vp->bmp) {
if(is->video_ctx->sample_aspect_ratio.num == 0) {
aspect_ratio = 0;
} else {
aspect_ratio = av_q2d(is->video_ctx->sample_aspect_ratio) *
is->video_ctx->width / is->video_ctx->height;
}
if(aspect_ratio <= 0.0) {
aspect_ratio = (float)is->video_ctx->width /
(float)is->video_ctx->height;
}
h = screen->h;
w = ((int)rint(h * aspect_ratio)) & -3;
if(w > screen->w) {
w = screen->w;
h = ((int)rint(w / aspect_ratio)) & -3;
}
x = (screen->w - w) / 2;
y = (screen->h - h) / 2;
rect.x = x;
rect.y = y;
rect.w = w;
rect.h = h;
SDL_LockMutex(screen_mutex);
SDL_DisplayYUVOverlay(vp->bmp, &rect);
SDL_UnlockMutex(screen_mutex);
}
}
void video_refresh_timer(void *userdata) {
VideoState *is = (VideoState *)userdata;
VideoPicture *vp;
double actual_delay, delay, sync_threshold, ref_clock, diff;
if(is->video_st) {
if(is->pictq_size == 0) {
schedule_refresh(is, 1);
} else {
vp = &is->pictq[is->pictq_rindex];
delay = vp->pts - is->frame_last_pts;
if(delay <= 0 || delay >= 1.0) {
delay = is->frame_last_delay;
}
is->frame_last_delay = delay;
is->frame_last_pts = vp->pts;
ref_clock = get_audio_clock(is);
diff = vp->pts - ref_clock;
sync_threshold = (delay > AV_SYNC_THRESHOLD) ? delay : AV_SYNC_THRESHOLD;
if(fabs(diff) < AV_NOSYNC_THRESHOLD) {
if(diff <= -sync_threshold) {
delay = 0;
} else if(diff >= sync_threshold) {
delay = 2 * delay;
}
}
is->frame_timer += delay;
actual_delay = is->frame_timer - (av_gettime() / 1000000.0);
if(actual_delay < 0.010) {
actual_delay = 0.010;
}
schedule_refresh(is, (int)(actual_delay * 1000 + 0.5));
video_display(is);
if(++is->pictq_rindex == VIDEO_PICTURE_QUEUE_SIZE) {
is->pictq_rindex = 0;
}
SDL_LockMutex(is->pictq_mutex);
is->pictq_size--;
SDL_CondSignal(is->pictq_cond);
SDL_UnlockMutex(is->pictq_mutex);
}
} else {
schedule_refresh(is, 100);
}
}
void alloc_picture(void *userdata) {
VideoState *is = (VideoState *)userdata;
VideoPicture *vp;
vp = &is->pictq[is->pictq_windex];
if(vp->bmp) {
SDL_FreeYUVOverlay(vp->bmp);
}
SDL_LockMutex(screen_mutex);
vp->bmp = SDL_CreateYUVOverlay(is->video_ctx->width,
is->video_ctx->height,
SDL_YV12_OVERLAY,
screen);
SDL_UnlockMutex(screen_mutex);
vp->width = is->video_ctx->width;
vp->height = is->video_ctx->height;
vp->allocated = 1;
}
int queue_picture(VideoState *is, AVFrame *pFrame, double pts) {
VideoPicture *vp;
int dst_pix_fmt;
AVPicture pict;
SDL_LockMutex(is->pictq_mutex);
while(is->pictq_size >= VIDEO_PICTURE_QUEUE_SIZE &&
!is->quit) {
SDL_CondWait(is->pictq_cond, is->pictq_mutex);
}
SDL_UnlockMutex(is->pictq_mutex);
if(is->quit)
return -1;
vp = &is->pictq[is->pictq_windex];
if(!vp->bmp ||
vp->width != is->video_ctx->width ||
vp->height != is->video_ctx->height) {
SDL_Event event;
vp->allocated = 0;
alloc_picture(is);
if(is->quit) {
return -1;
}
}
if(vp->bmp) {
SDL_LockYUVOverlay(vp->bmp);
vp->pts = pts;
dst_pix_fmt = PIX_FMT_YUV420P;
pict.data[0] = vp->bmp->pixels[0];
pict.data[1] = vp->bmp->pixels[2];
pict.data[2] = vp->bmp->pixels[1];
pict.linesize[0] = vp->bmp->pitches[0];
pict.linesize[1] = vp->bmp->pitches[2];
pict.linesize[2] = vp->bmp->pitches[1];
sws_scale(is->sws_ctx, (uint8_t const * const *)pFrame->data,
pFrame->linesize, 0, is->video_ctx->height,
pict.data, pict.linesize);
SDL_UnlockYUVOverlay(vp->bmp);
if(++is->pictq_windex == VIDEO_PICTURE_QUEUE_SIZE) {
is->pictq_windex = 0;
}
SDL_LockMutex(is->pictq_mutex);
is->pictq_size++;
SDL_UnlockMutex(is->pictq_mutex);
}
return 0;
}
double synchronize_video(VideoState *is, AVFrame *src_frame, double pts) {
double frame_delay;
if(pts != 0) {
is->video_clock = pts;
} else {
pts = is->video_clock;
}
frame_delay = av_q2d(is->video_ctx->time_base);
frame_delay += src_frame->repeat_pict * (frame_delay * 0.5);
is->video_clock += frame_delay;
return pts;
}
int video_thread(void *arg) {
VideoState *is = (VideoState *)arg;
AVPacket pkt1, *packet = &pkt1;
int frameFinished;
AVFrame *pFrame;
double pts;
pFrame = av_frame_alloc();
for(;;) {
if(packet_queue_get(&is->videoq, packet, 1) < 0) {
break;
}
if(packet_queue_get(&is->videoq, packet, 1) < 0) {
break;
}
pts = 0;
avcodec_decode_video2(is->video_ctx, pFrame, &frameFinished, packet);
if((pts = av_frame_get_best_effort_timestamp(pFrame)) == AV_NOPTS_VALUE) {
pts = 0;
}
pts *= av_q2d(is->video_st->time_base);
if(frameFinished) {
pts = synchronize_video(is, pFrame, pts);
if(queue_picture(is, pFrame, pts) < 0) {
break;
}
}
av_free_packet(packet);
}
av_frame_free(&pFrame);
return 0;
}
int stream_component_open(VideoState *is, int stream_index) {
AVFormatContext *pFormatCtx = is->pFormatCtx;
AVCodecContext *codecCtx = NULL;
AVCodec *codec = NULL;
SDL_AudioSpec wanted_spec, spec;
if(stream_index < 0 || stream_index >= pFormatCtx->nb_streams) {
return -1;
}
codec = avcodec_find_decoder(pFormatCtx->streams[stream_index]->codec->codec_id);
if(!codec) {
fprintf(stderr, "Unsupported codec!\n");
return -1;
}
codecCtx = avcodec_alloc_context3(codec);
if(avcodec_copy_context(codecCtx, pFormatCtx->streams[stream_index]->codec) != 0) {
fprintf(stderr, "Couldn't copy codec context");
return -1;
}
if(codecCtx->codec_type == AVMEDIA_TYPE_AUDIO) {
wanted_spec.freq = codecCtx->sample_rate;
wanted_spec.format = AUDIO_S16SYS;
wanted_spec.channels = codecCtx->channels;
wanted_spec.silence = 0;
wanted_spec.samples = SDL_AUDIO_BUFFER_SIZE;
wanted_spec.callback = audio_callback;
wanted_spec.userdata = is;
if(SDL_OpenAudio(&wanted_spec, &spec) < 0) {
fprintf(stderr, "SDL_OpenAudio: %s\n", SDL_GetError());
return -1;
}
is->audio_hw_buf_size = spec.size;
}
if(avcodec_open2(codecCtx, codec, NULL) < 0) {
fprintf(stderr, "Unsupported codec!\n");
return -1;
}
switch(codecCtx->codec_type) {
case AVMEDIA_TYPE_AUDIO:
is->audioStream = stream_index;
is->audio_st = pFormatCtx->streams[stream_index];
is->audio_ctx = codecCtx;
is->audio_buf_size = 0;
is->audio_buf_index = 0;
memset(&is->audio_pkt, 0, sizeof(is->audio_pkt));
packet_queue_init(&is->audioq);
SDL_PauseAudio(0);
break;
case AVMEDIA_TYPE_VIDEO:
is->videoStream = stream_index;
is->video_st = pFormatCtx->streams[stream_index];
is->video_ctx = codecCtx;
is->frame_timer = (double)av_gettime() / 1000000.0;
is->frame_last_delay = 40e-3;
packet_queue_init(&is->videoq);
is->video_tid = SDL_CreateThread(video_thread, is);
is->sws_ctx = sws_getContext(is->video_ctx->width, is->video_ctx->height,
is->video_ctx->pix_fmt, is->video_ctx->width,
is->video_ctx->height, PIX_FMT_YUV420P,
SWS_BILINEAR, NULL, NULL, NULL
);
break;
default:
break;
}
}
int decode_thread(void *arg) {
VideoState *is = (VideoState *)arg;
AVFormatContext *pFormatCtx;
AVPacket pkt1, *packet = &pkt1;
int video_index = -1;
int audio_index = -1;
int i;
is->videoStream=-1;
is->audioStream=-1;
global_video_state = is;
if(avformat_open_input(&pFormatCtx, is->filename, NULL, NULL)!=0)
return -1;
is->pFormatCtx = pFormatCtx;
if(avformat_find_stream_info(pFormatCtx, NULL)<0)
return -1;
av_dump_format(pFormatCtx, 0, is->filename, 0);
for(i=0; i<pFormatCtx->nb_streams; i++) {
if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_VIDEO &&
video_index < 0) {
video_index=i;
}
if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_AUDIO &&
audio_index < 0) {
audio_index=i;
}
}
if(audio_index >= 0) {
stream_component_open(is, audio_index);
}
if(video_index >= 0) {
stream_component_open(is, video_index);
}
if(is->videoStream < 0 || is->audioStream < 0) {
fprintf(stderr, "%s: could not open codecs\n", is->filename);
goto fail;
}
for(;;) {
if(is->quit) {
break;
}
if(is->audioq.size > MAX_AUDIOQ_SIZE ||
is->videoq.size > MAX_VIDEOQ_SIZE) {
SDL_Delay(10);
continue;
}
if(av_read_frame(is->pFormatCtx, packet) < 0) {
if(is->pFormatCtx->pb->error == 0) {
SDL_Delay(100);
continue;
} else {
break;
}
}
if(packet->stream_index == is->videoStream) {
packet_queue_put(&is->videoq, packet);
} else if(packet->stream_index == is->audioStream) {
packet_queue_put(&is->audioq, packet);
} else {
av_free_packet(packet);
}
}
while(!is->quit) {
SDL_Delay(100);
}
fail:
if(1){
SDL_Event event;
event.type = FF_QUIT_EVENT;
event.user.data1 = is;
SDL_PushEvent(&event);
}
return 0;
}
int main(int argc, char *argv[]) {
SDL_Event event;
VideoState *is;
is = av_mallocz(sizeof(VideoState));
if(argc < 2) {
fprintf(stderr, "Usage: test <file>\n");
exit(1);
}
av_register_all();
if(SDL_Init(SDL_INIT_VIDEO | SDL_INIT_AUDIO | SDL_INIT_TIMER)) {
fprintf(stderr, "Could not initialize SDL - %s\n", SDL_GetError());
exit(1);
}
#ifndef __DARWIN__
screen = SDL_SetVideoMode(640, 480, 0, 0);
#else
screen = SDL_SetVideoMode(640, 480, 24, 0);
#endif
if(!screen) {
fprintf(stderr, "SDL: could not set video mode - exiting\n");
exit(1);
}
screen_mutex = SDL_CreateMutex();
av_strlcpy(is->filename, argv[1], sizeof(is->filename));
is->pictq_mutex = SDL_CreateMutex();
is->pictq_cond = SDL_CreateCond();
schedule_refresh(is, 40);
is->parse_tid = SDL_CreateThread(decode_thread, is);
if(!is->parse_tid) {
av_free(is);
return -1;
}
for(;;) {
SDL_WaitEvent(&event);
switch(event.type) {
case FF_QUIT_EVENT:
case SDL_QUIT:
is->quit = 1;
SDL_Quit();
return 0;
break;
case FF_REFRESH_EVENT:
video_refresh_timer(event.user.data1);
break;
default:
break;
}
}
return 0;
}
WARNING
When I just wrote this guide, all my synchronization code was taken from the then version of ffplay.c . Today it is a completely different program, and updates in the FFmpeg libraries (and in ffplay.c itself) have led to fundamental changes. Although this code still works, it is already outdated, and there are many other improvements that could be used in this guide.How video syncs
Until now, we had an almost useless movie player. Yes, it plays video, and yes, it plays audio, but that’s not quite what we would call a movie. So what do we do then?PTS and DTS
Fortunately, audio and video streams contain information about how fast and at what points in time they should be played. Audio streams have a sampling rate, and video streams have frames per second. However, if we just synchronize the video by counting the number of frames and multiplying by the frame rate, there is a good chance that it does not synchronize with the sound. Therefore, we will go the other way. Packets from the stream can have the so-called decoding time stamp ( DTS - from d ecoding t ime s tamp ) and presentation time stamp ( PTS - from p resentation t ime stamp ). To understand these two meanings, you need to know how movies are stored. Some formats, such as MPEG, use what they call B-frames ( Bed and is bi-directional, England. Bidirectional ). Two other types of frames are called I-frames and P-frames ( I is internal , i nner , and P means predicted , p redicted ). I-frames contain the full image. P framesdepend on previous I- and P-frames and are different from previous frames, or you can also name - deltas. B-frames are similar to P-frames, but depend on the information contained in both previous and subsequent frames! The fact that a frame may not contain the image itself, but differences with other frames - explains why we may not have a finished frame after calling avcodec_decode_video2 .Let's say we have a movie in which 4 frames in this sequence: IBBP . Then we need to find out the information from the last P-frame before we can display any of the two previous B-frames. Because of this, frames can be stored in a sequence that does not match the actual display order: IPBB. That's what the decoding time stamp and presentation time stamp for each frame are for. The decoding time stamp tells us when we need to decode something, and the presentation time stamp tells us when we need to display something. So, in this case, our stream may look like this: PTS: 1 4 2 3 DTS: 1 2 3 4Stream: IPBBAs a rule, PTS and DTS are different only when the stream being played contains B-frames.When we receive a package from av_read_frame (), it contains the PTS and DTS values for the information that is inside the package. But what we really need is the PTS of our newly decoded raw frame, in which case we know when it needs to be displayed.Fortunately, FFmpeg provides us with the “best possible timestamp” we can get using the av_frame_get_best_effort_timestamp () function .Synchronization
In order for frames to be displayed in turn, it would be nice to know when to display a specific video frame. But how exactly do we do it? The idea is this: after we show the frame, we figure out when the next frame should be shown. Then just pause, after which we update the video after this period of time. As expected, we check the PTS value of the next frame on the system clock to see how long our wait time should be. This approach works, but there are two problems that need to be addressed.First, the question is, when will the next PTS be? You will say that you can simply add the video frequency to the current PTS - and you will, in principle, be right. However, some varieties of video will require repeating frames. This means that you have to repeat the current frame a certain number of times. This may cause the program to display the next frame too soon. This must be taken into account.The second problem is that, in the program that we have written at the moment, video and audio are joyfully rushing forward until they bother to synchronize at all. We would not have to worry about it if everything by itself worked perfectly. But your computer is not perfect, as are many video files. Thus, we have three options: synchronize audio with video, synchronize video with audio or synchronize both audio and video with an external clock (for example, with your computer). Now we are going to synchronize the video with audio.Coding: receiving a PTS frame
Now let's write something directly. We need to add a few more parts to our large structure, and we will do it the way we need. First, let's take a look at our video thread. Remember that here we collect packets that have been queued by our decoding stream? In this part of the code, we need to get the PTS for the frame that avcodec_decode_video2 gave us . The first way we talked about is getting the DTS of the last processed packet, which is pretty simple: double pts;
for(;;) {
if(packet_queue_get(&is->videoq, packet, 1) < 0) {
break;
}
pts = 0;
len1 = avcodec_decode_video2(is->video_st->codec,
pFrame, &frameFinished, packet);
if(packet->dts != AV_NOPTS_VALUE) {
pts = av_frame_get_best_effort_timestamp(pFrame);
} else {
pts = 0;
}
pts *= av_q2d(is->video_st->time_base);
We set the PTS to zero if we cannot determine its value.Well, that was easy. Technical note: as you can see, we use int64 for PTS. This is because the PTS is stored as an integer. This value is a timestamp that corresponds to the time dimension in timebb . For example, if the stream has 24 frames per second, PTS from 42 will indicate that the frame should be used where the 42nd frame should be, provided that we have frames replaced every 1/24 second (of course, this will not necessarily be so in fact).We can convert this value to seconds by dividing by the frame rate. Time_base valuethe stream will be equal to 1 divided by the frame rate (for content with a fixed frame rate), therefore, to get the PTS in seconds, we multiply by time_base .Code further: synchronization and use of PTS
So now we have all the ready-made PTS. Now we will take care of those two synchronization problems, which were discussed a little higher. We are going to define a synchronize_video function that will update the PTS to synchronize with everything. This function, finally, will also deal with cases where we do not get the PTS value for our frame. At the same time, we need to track when the next frame is expected so that we can correctly set the refresh rate. We can do this using the internal video_clock value , which tracks how much time has passed for the video. We add this value to our large structure:typedef struct VideoState {
double video_clock;
Here is the synchronize_video function , which is pretty clear:double synchronize_video(VideoState *is, AVFrame *src_frame, double pts) {
double frame_delay;
if(pts != 0) {
is->video_clock = pts;
} else {
pts = is->video_clock;
}
frame_delay = av_q2d(is->video_st->codec->time_base);
frame_delay += src_frame->repeat_pict * (frame_delay * 0.5);
is->video_clock += frame_delay;
return pts;
}
As you can see, we take into account repeated frames in this function.Now, let's get our correct PTS and queue the frame using queue_picture by adding a new pts argument :
if(frameFinished) {
pts = synchronize_video(is, pFrame, pts);
if(queue_picture(is, pFrame, pts) < 0) {
break;
}
}
The only thing that changes in queue_picture is that we store this pts value in the VideoPicture structure that we queue. Thus, we must add the pts variable to the structure and add these lines of code:typedef struct VideoPicture {
...
double pts;
}
int queue_picture(VideoState *is, AVFrame *pFrame, double pts) {
... stuff ...
if(vp->bmp) {
... convert picture ...
vp->pts = pts;
... alert queue ...
}
So now we have the images queued up with the correct PTS values, so let's take a look at our video update feature. You can recall from the last lesson that we simply faked it and installed an update of 80 ms. Well, now we are going to find out what should really be there.Our strategy is to predict the time of the next PTS by simply measuring the time between the current pts and the previous one. At the same time, we need to synchronize the video with audio. We are going to make an audio clock.: An internal value that keeps track of the position of the audio we are playing. It is like a digital readout on any mp3 player. Since we synchronize the video with the sound, the video stream uses this value to find out if it is too far ahead or too far behind.We will return to implementation later; Now let's assume that we have the get_audio_clock functionwhich will give us time on the audio clock. As soon as we get this value, what needs to be done if the video and audio are not synchronized? It would be foolish to just try to jump to the right package through a search or something else. Instead, we simply adjust the value that we calculated for the next update: if the PTS is too far behind the audio time, we double our estimated delay. If the PTS is too far ahead of the playing time, we just update as quickly as possible. Now that we have the configured update or delay time, we are going to compare it with the clock of our computer, leaving frame_timer running . This frame timer summarizes all our estimated delays during movie playback. In other words, this frame_timer- This is the time indicating when to display the next frame. We simply add a new delay to the frame timer, compare it with the time on the clock of our computer, and use this value to plan the next update. This can be a little confusing, so read the code carefully:void video_refresh_timer(void *userdata) {
VideoState *is = (VideoState *)userdata;
VideoPicture *vp;
double actual_delay, delay, sync_threshold, ref_clock, diff;
if(is->video_st) {
if(is->pictq_size == 0) {
schedule_refresh(is, 1);
} else {
vp = &is->pictq[is->pictq_rindex];
delay = vp->pts - is->frame_last_pts;
if(delay <= 0 || delay >= 1.0) {
delay = is->frame_last_delay;
}
is->frame_last_delay = delay;
is->frame_last_pts = vp->pts;
ref_clock = get_audio_clock(is);
diff = vp->pts - ref_clock;
sync_threshold = (delay > AV_SYNC_THRESHOLD) ? delay : AV_SYNC_THRESHOLD;
if(fabs(diff) < AV_NOSYNC_THRESHOLD) {
if(diff <= -sync_threshold) {
delay = 0;
} else if(diff >= sync_threshold) {
delay = 2 * delay;
}
}
is->frame_timer += delay;
actual_delay = is->frame_timer - (av_gettime() / 1000000.0);
if(actual_delay < 0.010) {
actual_delay = 0.010;
}
schedule_refresh(is, (int)(actual_delay * 1000 + 0.5));
video_display(is);
if(++is->pictq_rindex == VIDEO_PICTURE_QUEUE_SIZE) {
is->pictq_rindex = 0;
}
SDL_LockMutex(is->pictq_mutex);
is->pictq_size--;
SDL_CondSignal(is->pictq_cond);
SDL_UnlockMutex(is->pictq_mutex);
}
} else {
schedule_refresh(is, 100);
}
}
We make several checks: firstly, we make sure that the delay between the current PTS and the previous PTS makes sense. If there is no need for a delay, then the audio and video just coincided at this point and just use the last delay. Then we make sure that the synchronization threshold is fulfilled, because perfect synchronization never happens. FFplay uses a value of 0.01 for the threshold. We also make sure that the synchronization threshold is never less than the intervals between the PTS values. Finally, set the minimum update value to 10 milliseconds (indeed, it seems like they should skip the frame here, but let's not worry about that).We added a bunch of variables to the big structure, so don't forget to check the code. Also, do not forget to initialize the frame timer and the initial delay of the previous frame in stream_component_open : is->frame_timer = (double)av_gettime() / 1000000.0;
is->frame_last_delay = 40e-3;
Sync: audio clock
The time has come to realize the audio clock. We can update the time in our audio_decode_frame function , where we decode the audio. Now remember that we do not always process a new package every time we call this function, so there are two areas where you need to update the clock. The first place is where we get the new package: just install the sound clock on the PTS package. Then, if the packet has several frames, we save the audio playback time by counting the number of samples and multiplying them by a given sampling frequency per second. So, when we have the package:
if(pkt->pts != AV_NOPTS_VALUE) {
is->audio_clock = av_q2d(is->audio_st->time_base)*pkt->pts;
}
And as soon as we process the package:
pts = is->audio_clock;
*pts_ptr = pts;
n = 2 * is->audio_st->codec->channels;
is->audio_clock += (double)data_size /
(double)(n * is->audio_st->codec->sample_rate);
A few minor nuances: the function template has been changed and now includes pts_ptr , so be sure to change it. pts_ptr is the pointer that we use to tell audio_callback the audio packet pts . This will be used next time to synchronize audio with video.Now we can finally implement our get_audio_clock function . It is not as simple as getting the value is -> audio_clock , if you think about it. Please note that we set PTS audio every time we process it, but if you look at the audio_callback function, it will take time to move all the data from our audio packet to our output buffer. This means that the value in our audio clock may be too far ahead. Therefore, we need to check how much we have to write. Here is the complete code:double get_audio_clock(VideoState *is) {
double pts;
int hw_buf_size, bytes_per_sec, n;
pts = is->audio_clock;
hw_buf_size = is->audio_buf_size - is->audio_buf_index;
bytes_per_sec = 0;
n = is->audio_st->codec->channels * 2;
if(is->audio_st) {
bytes_per_sec = is->audio_st->codec->sample_rate * n;
}
if(bytes_per_sec) {
pts -= (double)hw_buf_size / bytes_per_sec;
}
return pts;
}
You should now understand why this function works;)So, that's it! We compile:gcc -o tutorial05 tutorial05.c -lavutil -lavformat -lavcodec -lswscale -lz -lm \
`sdl-config --cflags --libs`
It happened! You can watch the movie on a self-made player. In the next lesson, we’ll look at audio synchronization, and then learn how to search.Translations on the Edison Blog: