Get best of the week

Simple zero-copy rendering of hardware accelerated video in QML

Introduction


The purpose of this article is to demonstrate how you can make friends with third-party video buffers and QML. The main idea is to use the standard QML component of VideoOutput. It allows you to palm off third-party sources, it is well documented and has a backend supporting GL_OES_EGL_image_external.


The idea that this might suddenly be useful arose after I tried to run examples of working with the camera in Qt, and on the embedded platform they worked at a speed of 3-5 frames per second. It became clear that out of the box there was no question of any zero-copy, although the platform supports all this very well. In fairness, on the desktop, VideoOutput and Camera work, as expected, quickly and without unnecessary copying. But in my task, alas, it was impossible to do with the existing classes for capturing video, and I wanted to get video from a third-party source, which could be an arbitrary GStreamer pipeline for decoding video, for example, from a file or RTSP stream, or a third-party API that can be integrated into the base Qt's classes are somewhat dubious. You can, of course, once again reinvent the wheel and write your component with drawing through OpenGL,but it immediately seemed a deliberately dead end and complicated way.


Everything led to the fact that you need to figure out how it really works, and write a small application confirming the theory.


Theory


VideoOutput supports custom source, provided that


  1. the passed object can accept QAbstractVideoSurface directly through the videoSurface property
  2. or through mediaObject with QVideoRendererControl [link] .

, QtMultimedia QAbstractVideoBuffer , QPixmap GLTexture EGLImage. videonode_egl, samplerExternalOES. , QAbstractVideoBuffer EGLImage, videnode_egl.
EGLImage , , .



Video Overview.


Qt OpenGL ES , Qt . , .


, GStreamer :


v4l2src ! appsink

V4L2Source, QAbstractVideoSurface.


class V4L2Source : public QQuickItem
{
    Q_OBJECT
    Q_PROPERTY(QAbstractVideoSurface* videoSurface READ videoSurface WRITE
                   setVideoSurface)
    Q_PROPERTY(QString device MEMBER m_device READ device WRITE setDevice)
    Q_PROPERTY(QString caps MEMBER m_caps)
public:
    V4L2Source(QQuickItem* parent = nullptr);
    virtual ~V4L2Source();

    void setVideoSurface(QAbstractVideoSurface* surface);
    void setDevice(QString device);

public slots:
    void start();
    void stop();

private slots:
    void setWindow(QQuickWindow* win);
    void sync();

signals:
    void frameReady();
...
}

, setWinow() — QQuickItem::windowChanged() callback QQuickWindow::beforeSynchronizing().


VideoOutput EGLImage, QAbstractVideoSurface QAbstractVideoBuffer::HandleType :


void V4L2Source::setVideoSurface(QAbstractVideoSurface* surface)
{
    if (m_surface != surface && m_surface && m_surface->isActive()) {
        m_surface->stop();
    }
    m_surface = surface;
    if (surface
            ->supportedPixelFormats(
                QAbstractVideoBuffer::HandleType::EGLImageHandle)
            .size() > 0) {
        EGLImageSupported = true;
    } else {
        EGLImageSupported = false;
    }

    if (m_surface && m_device.length() > 0) {
        start();
    }
}

, callback':


GstAppSinkCallbacks V4L2Source::callbacks = {.eos = nullptr,
                                             .new_preroll = nullptr,
                                             .new_sample =
                                                 &V4L2Source::on_new_sample};

V4L2Source::V4L2Source(QQuickItem* parent) : QQuickItem(parent)
{
    m_surface = nullptr;
    connect(this, &QQuickItem::windowChanged, this, &V4L2Source::setWindow);

    pipeline = gst_pipeline_new("V4L2Source::pipeline");
    v4l2src = gst_element_factory_make("v4l2src", nullptr);
    appsink = gst_element_factory_make("appsink", nullptr);

    GstPad* pad = gst_element_get_static_pad(appsink, "sink");
    gst_pad_add_probe(pad, GST_PAD_PROBE_TYPE_QUERY_BOTH, appsink_pad_probe,
                      nullptr, nullptr);
    gst_object_unref(pad);

    gst_app_sink_set_callbacks(GST_APP_SINK(appsink), &callbacks, this,
                               nullptr);

    gst_bin_add_many(GST_BIN(pipeline), v4l2src, appsink, nullptr);
    gst_element_link(v4l2src, appsink);

    context = g_main_context_new();
    loop = g_main_loop_new(context, FALSE);
}

void V4L2Source::setWindow(QQuickWindow* win)
{
    if (win) {
        connect(win, &QQuickWindow::beforeSynchronizing, this,
                &V4L2Source::sync, Qt::DirectConnection);
    }
}

GstFlowReturn V4L2Source::on_new_sample(GstAppSink* sink, gpointer data)
{
    Q_UNUSED(sink)
    V4L2Source* self = (V4L2Source*)data;
    QMutexLocker locker(&self->mutex);
    self->ready = true;
    self->frameReady();
    return GST_FLOW_OK;
}

// Request v4l2src allocator to add GstVideoMeta to buffers
static GstPadProbeReturn
appsink_pad_probe(GstPad* pad, GstPadProbeInfo* info, gpointer user_data)
{
    if (info->type & GST_PAD_PROBE_TYPE_QUERY_BOTH) {
        GstQuery* query = gst_pad_probe_info_get_query(info);
        if (GST_QUERY_TYPE(query) == GST_QUERY_ALLOCATION) {
            gst_query_add_allocation_meta(query, GST_VIDEO_META_API_TYPE, NULL);
        }
    }
    return GST_PAD_PROBE_OK;
}

, GMainContext GMainLoop .


Qt::DirectConnection setWindow() — callback , OpenGL .


V4L2Source::on_new_sample() v4l2src appsink , VideoOutput .


sink appsink , v4l2src . , / .


VideoOutput sync():


// Make sure this callback is invoked from rendering thread
void V4L2Source::sync()
{
    {
        QMutexLocker locker(&mutex);
        if (!ready) {
            return;
        }
        // reset ready flag
        ready = false;
    }
    // pull available sample and convert GstBuffer into a QAbstractVideoBuffer
    GstSample* sample = gst_app_sink_pull_sample(GST_APP_SINK(appsink));
    GstBuffer* buffer = gst_sample_get_buffer(sample);
    GstVideoMeta* videoMeta = gst_buffer_get_video_meta(buffer);

    // if memory is DMABUF and EGLImage is supported by the backend,
    // create video buffer with EGLImage handle
    videoFrame.reset();
    if (EGLImageSupported && buffer_is_dmabuf(buffer)) {
        videoBuffer.reset(new GstDmaVideoBuffer(buffer, videoMeta));
    } else {
        // TODO: support other memory types, probably GL textures?
        // just map memory
        videoBuffer.reset(new GstVideoBuffer(buffer, videoMeta));
    }

    QSize size = QSize(videoMeta->width, videoMeta->height);
    QVideoFrame::PixelFormat format =
        gst_video_format_to_qvideoformat(videoMeta->format);

    videoFrame.reset(new QVideoFrame(
        static_cast<QAbstractVideoBuffer*>(videoBuffer.get()), size, format));

    if (!m_surface->isActive()) {
        m_format = QVideoSurfaceFormat(size, format);
        Q_ASSERT(m_surface->start(m_format) == true);
    }
    m_surface->present(*videoFrame);
    gst_sample_unref(sample);
}

appsink, GstVideoMeta ( , , fallback , - , ) QAbstractVideoBuffer : EGLImage (GstDmaVideoBuffer) None (GstVideoBuffer). QVideoFrame .


GstDmaVideoBuffer GstVideoBuffer :


#define GST_BUFFER_GET_DMAFD(buffer, plane)                                    \
    (((plane) < gst_buffer_n_memory((buffer))) ?                               \
         gst_dmabuf_memory_get_fd(gst_buffer_peek_memory((buffer), (plane))) : \
         gst_dmabuf_memory_get_fd(gst_buffer_peek_memory((buffer), 0)))

class GstDmaVideoBuffer : public QAbstractVideoBuffer
{
public:
    // This  should be called from renderer thread
    GstDmaVideoBuffer(GstBuffer* buffer, GstVideoMeta* videoMeta) :
        QAbstractVideoBuffer(HandleType::EGLImageHandle),
        buffer(gst_buffer_ref(buffer)), m_videoMeta(videoMeta)

    {
        static PFNEGLCREATEIMAGEKHRPROC eglCreateImageKHR =
            reinterpret_cast<PFNEGLCREATEIMAGEKHRPROC>(
                eglGetProcAddress("eglCreateImageKHR"));
        int idx = 0;
        EGLint attribs[MAX_ATTRIBUTES_COUNT];

        attribs[idx++] = EGL_WIDTH;
        attribs[idx++] = m_videoMeta->width;
        attribs[idx++] = EGL_HEIGHT;
        attribs[idx++] = m_videoMeta->height;
        attribs[idx++] = EGL_LINUX_DRM_FOURCC_EXT;
        attribs[idx++] = gst_video_format_to_drm_code(m_videoMeta->format);
        attribs[idx++] = EGL_DMA_BUF_PLANE0_FD_EXT;
        attribs[idx++] = GST_BUFFER_GET_DMAFD(buffer, 0);
        attribs[idx++] = EGL_DMA_BUF_PLANE0_OFFSET_EXT;
        attribs[idx++] = m_videoMeta->offset[0];
        attribs[idx++] = EGL_DMA_BUF_PLANE0_PITCH_EXT;
        attribs[idx++] = m_videoMeta->stride[0];
        if (m_videoMeta->n_planes > 1) {
            attribs[idx++] = EGL_DMA_BUF_PLANE1_FD_EXT;
            attribs[idx++] = GST_BUFFER_GET_DMAFD(buffer, 1);
            attribs[idx++] = EGL_DMA_BUF_PLANE1_OFFSET_EXT;
            attribs[idx++] = m_videoMeta->offset[1];
            attribs[idx++] = EGL_DMA_BUF_PLANE1_PITCH_EXT;
            attribs[idx++] = m_videoMeta->stride[1];
        }
        if (m_videoMeta->n_planes > 2) {
            attribs[idx++] = EGL_DMA_BUF_PLANE2_FD_EXT;
            attribs[idx++] = GST_BUFFER_GET_DMAFD(buffer, 2);
            attribs[idx++] = EGL_DMA_BUF_PLANE2_OFFSET_EXT;
            attribs[idx++] = m_videoMeta->offset[2];
            attribs[idx++] = EGL_DMA_BUF_PLANE2_PITCH_EXT;
            attribs[idx++] = m_videoMeta->stride[2];
        }
        attribs[idx++] = EGL_NONE;

        auto m_qOpenGLContext = QOpenGLContext::currentContext();
        QEGLNativeContext qEglContext =
            qvariant_cast<QEGLNativeContext>(m_qOpenGLContext->nativeHandle());

        EGLDisplay dpy = qEglContext.display();
        Q_ASSERT(dpy != EGL_NO_DISPLAY);

        image = eglCreateImageKHR(dpy, EGL_NO_CONTEXT, EGL_LINUX_DMA_BUF_EXT,
                                  (EGLClientBuffer) nullptr, attribs);
        Q_ASSERT(image != EGL_NO_IMAGE_KHR);
    }

...

    // This should be called from renderer thread
    ~GstDmaVideoBuffer() override
    {
        static PFNEGLDESTROYIMAGEKHRPROC eglDestroyImageKHR =
            reinterpret_cast<PFNEGLDESTROYIMAGEKHRPROC>(
                eglGetProcAddress("eglDestroyImageKHR"));

        auto m_qOpenGLContext = QOpenGLContext::currentContext();
        QEGLNativeContext qEglContext =
            qvariant_cast<QEGLNativeContext>(m_qOpenGLContext->nativeHandle());

        EGLDisplay dpy = qEglContext.display();
        Q_ASSERT(dpy != EGL_NO_DISPLAY);
        eglDestroyImageKHR(dpy, image);
        gst_buffer_unref(buffer);
    }

private:
    EGLImage image;
    GstBuffer* buffer;
    GstVideoMeta* m_videoMeta;
};

class GstVideoBuffer : public QAbstractPlanarVideoBuffer
{
public:
    GstVideoBuffer(GstBuffer* buffer, GstVideoMeta* videoMeta) :
        QAbstractPlanarVideoBuffer(HandleType::NoHandle),
        m_buffer(gst_buffer_ref(buffer)), m_videoMeta(videoMeta),
        m_mode(QAbstractVideoBuffer::MapMode::NotMapped)
    {
    }

    QVariant handle() const override
    {
        return QVariant();
    }

    void release() override
    {
    }

    int map(MapMode mode,
            int* numBytes,
            int bytesPerLine[4],
            uchar* data[4]) override
    {
        int size = 0;
        const GstMapFlags flags =
            GstMapFlags(((mode & ReadOnly) ? GST_MAP_READ : 0) |
                        ((mode & WriteOnly) ? GST_MAP_WRITE : 0));
        if (mode == NotMapped || m_mode != NotMapped) {
            return 0;
        } else {
            for (int i = 0; i < m_videoMeta->n_planes; i++) {
                gst_video_meta_map(m_videoMeta, i, &m_mapInfo[i],
                                   (gpointer*)&data[i], &bytesPerLine[i],
                                   flags);
                size += m_mapInfo[i].size;
            }
        }
        m_mode = mode;
        *numBytes = size;
        return m_videoMeta->n_planes;
    }

    MapMode mapMode() const override
    {
        return m_mode;
    }

    void unmap() override
    {
        if (m_mode != NotMapped) {
            for (int i = 0; i < m_videoMeta->n_planes; i++) {
                gst_video_meta_unmap(m_videoMeta, i, &m_mapInfo[i]);
            }
        }
        m_mode = NotMapped;
    }

    ~GstVideoBuffer() override
    {
        unmap();
        gst_buffer_unref(m_buffer);
    }

private:
    GstBuffer* m_buffer;
    MapMode m_mode;
    GstVideoMeta* m_videoMeta;
    GstMapInfo m_mapInfo[4];
};

QML :


import QtQuick 2.10
import QtQuick.Window 2.10
import QtQuick.Layouts 1.10
import QtQuick.Controls 2.0
import QtMultimedia 5.10
import v4l2source 1.0

Window {
    visible: true
    width: 640
    height: 480
    title: qsTr("qml zero copy rendering")
    color: "black"

    CameraSource {
        id: camera
        device: "/dev/video0"
        onFrameReady: videoOutput.update()
    }

    VideoOutput {
        id: videoOutput
        source: camera
        anchors.fill: parent
    }

    onClosing: camera.stop()
}


The purpose of this article was to show how to integrate an existing API that can produce hardware-accelerated video with QML and use existing components for rendering without copying (well, or in the worst case, with one, but without expensive software conversion to RGB).


Code Link


References


Source: https://habr.com/ru/post/undefined/

More articles:


All Articles