With the advent of heavy video formats such as 4K (Ultra HD), the problem of video stream decoding efficiency has become quite relevant. On an average computer, special measures must be taken in order to be able to process such a video stream in real time. The article describes the possible ways to increase the decoding speed of video streams in FFmpeg-based solutions, and presents the results of experiments on measuring the decoding speed for 4K video streams encoded in H264 and HEVC (H265).

Table of contents

  1. Three ways to increase the decoding speed of the video stream
    1.1. Connection of additional work streams in standard decoders
    1.2. Connecting hardware acceleration in standard decoders
    1.3. Using special decoders that implement decoding on GPUs
  2. Measuring decoding speed
  3. Notes on QSV decoders
  Resources

1. Three ways to increase the decoding speed of a video stream

We will consider three ways to increase the decoding speed of a video stream.

1. Connection of additional worker threads (threads) in standard decoders.
2. Connection of hardware acceleration (HW Acceleration) in standard decoders.
3. Using special decoders that implement decoding on GPUs.

CPU, .

, FFmpeg. - : — Windows 10, — Intel i5 8400 2.80 (6 hyper-threading), — Intel UHD Graphics 630, — 16 , FFmpeg 4.2.1, zeranoe.

FFmpeg , .

1.1.

(, , ) , . avcodec_open2() thread_count AVCodecContext . — threads , avcodec_open2().

, , (h264, hevc, vp9) , theora .

-threads.

1.2.

FFmpeg . FFmpeg API libavutil/hwcontext.h. enum AVHWDeviceType, . FFmpeg :

void print_hwtypes_all()
{
    AVHWDeviceType hwtype = AV_HWDEVICE_TYPE_NONE;
    while ((hwtype = av_hwdevice_iterate_types(hwtype)) !=
                            AV_HWDEVICE_TYPE_NONE)
    {
        printf("%s\n", av_hwdevice_get_type_name(hwtype));
    }
}

- :

    cuda
    dxva2
    qsv
    d3d11va

, cuda Nvidia , qsv Intel Quick Sync Video (QSV), Intel (. [1]), dxva2 d3d11va DirectX Video Acceleration (. [2]), Windows, (Intel, Nvidia, AMD).

( ). , , :

void print_hwtypes(const char* dec_name)
{
    const AVCodec* decoder = avcodec_find_decoder_by_name(dec_name);
    for (int i = 0; ; ++i) { 
        const AVCodecHWConfig *config =
               avcodec_get_hw_config(decoder, i);
        if (config) {
            if (config->methods &
                    AV_CODEC_HW_CONFIG_METHOD_HW_DEVICE_CTX) {
                printf("%s\n", 
                  av_hwdevice_get_type_name(config->device_type));
            }
        }
        else {
            break;
        }
    }
}

- h264, hevc, vp9, vc1 :

    dxva2
    d3d11va
    cuda

theora .

, hw_decode.c. [3], 2expres.

void init_hwdevice(AVHWDeviceType hwtype, AVCodecContext *codec_ctx)
{
    AVBufferRef *dev_ctx = NULL;
    int ret = av_hwdevice_ctx_create(&dev_ctx, hwtype, NULL, NULL, 0);
    if (ret >= 0) {
        codec_ctx->get_format = get_hw_format; // . hw_decode.c
        codec_ctx->hw_device_ctx = av_buffer_ref(dev_ctx);
//   dev_ctx,    
//   av_buffer_unref()
    }
}

, , av_hwframe_transfer_data(). dxva2 d3d11va NV12.

-hwaccel.

1.3. ,

FFmpeg , .

Intel Quick Sync Video (QSV), , Intel i3, i5, i7, i9. . [1]. _qsv. FFmpeg : h264_qsv, hevc_qsv, vp8_qsv, mpeg2_qsv, vc1_qsv.

NVDEC, NVENC Nvidia. _cuvid. FFmpeg : h264_cuvid, hevc_cuvid, mpeg2_cuvid, vc1_cuvid, vp8_cuvid, vp9_cuvid, mjpeg_cuvid, mpeg4_cuvid.

:

AVStream *strm;
// ...
AVCodecID cid = strm->codecpar->codec_id;
const AVCodec *decoder = avcodec_find_decoder(cid);

. :

const AVCodec *decoder = (cid == AV_CODEC_ID_H264)
    ? avcodec_find_decoder_by_name("h264_qsv")
    : avcodec_find_decoder(cid);

-c:v -i

ffmpeg -c:v h264_qsv -i INPUT ...

2.

, H264, HEVC(H265). — 38402160 (Ultra HD), — 30 /. — h264, hevc QSV — h264_qsv, hevc_qsv. 4 : , , , dxva2. dxva2 , d3d11va, . , . : , 32- BGRA libswscale. ( 12- YUV420P NV12.) , , ( ). , 100%, , , . . 64- FFmpeg.

, , . — BGRA. , , .

32- FFmpeg. , : hevc 2-3 . .

. -benchmark . :

ffmpeg -benchmark -i INPUT -an -f null -
ffmpeg -benchmark -threads N -i INPUT -an -f null -
ffmpeg -benchmark -c:v h264_qsv -i INPUT -an -f null -
ffmpeg -benchmark -hwaccel dxva2 -i INPUT -an -f null -
ffmpeg -benchmark -i INPUT -an -pix_fmt bgra -f null -

fps, speed . -threads N auto, , 100%.

3. QSV

FFmpeg QSV : h264_qsv, hevc_qsv, vp8_qsv, mpeg2_qsv, vc1_qsv. . mpeg2_qsv , vc1_qsv . , , , -, , .

. , — avcodec_flush_buffers(). , .

[1] Intel Quick Sync Video.
[2] DirectX Video Acceleration.
[3] FFmpeg DXVA2.