Record video UI autotests working in headless Chrome

Hello everyone!

In this article I want to talk about how the task of recording video auto-tests working in headless Chrome was solved (there should be no problems with recording in non-headless mode). Several different approaches will be considered, and difficulties and ways to solve them will be told.

Formulation of the problem

  1. Tests run under Windows
  2. Tests use Selenium Web Driver + Headless Chrome
  3. Tests run in multiple threads

For fallen tests, you need to save the video, while

  1. Runtime should grow by no more than 10%.
  2. Do with a minimum of changes in the current implementation

If you are interested in a solution, welcome to cat.

The naive approach. Screenshots


Our test framework has a low-level wrapper over Selenium. Therefore, the first implementation was very simple and extremely naive: code was added to all places that change the page (Click, Set textbox, Navigate, etc.), which saves a screenshot through the Selenium Web Driver

Driver.TakeScreenshot().SaveAsFile(screenshotPath);

Test execution time with this approach has grown significantly. Reason: the operation of saving the screenshot does not work at all quickly - from 0.5 seconds to several seconds.

Screenshots in a separate stream


Instead of code that saves screenshots in all places that change the page (Click, Set textbox, Navigate), a code was added that constantly saves screenshots in a separate stream

Hidden text
...
var timeTillNextFrame = TimeSpan.Zero;
while (!_stopThread.WaitOne(timeTillNextFrame))
{
    var screenShotDriver = Driver as ITakesScreenshot;
    if (screenShotDriver == null)
    {
        continue;
    }

    var screenShot = screenShotDriver.GetScreenshot();
    ...
}


The test execution time was still very long. I did not understand the reason for the delay. Most likely, Selenium refuses to do something while the screenshot is being saved. Perhaps another instance of Selenium would help, ending in the same session.

Screenshots in a separate thread through Puppeteer


It wasn’t very interesting to do two instances of Selenium, since I had long wanted to try puppeteer-sharp in practice - and here I found a suitable reason. On the side of Selenium, Puppeteer was created, which simply connected to Chrome, already created through Selenium

Hidden text
var options = new ConnectOptions()
{
    BrowserURL = $"http://127.0.0.1:{debugPort}"
};

_puppeteerBrowser = Puppeteer.ConnectAsync(options).GetAwaiter().GetResult();


The test went its own way through Selenium, and Puppeteer took screenshots in a separate thread

Hidden text
...
var timeTillNextFrame = TimeSpan.Zero;
while (!_stopThread.WaitOne(timeTillNextFrame))
{
    var pages = _puppeteerBrowser.PagesAsync().GetAwaiter().GetResult();
    if (pages.Length <= 0)
    {
        continue;
    }
    
    var page = pages[0];
    
    page.SetViewportAsync(new ViewPortOptions
    {
        Width = screenWidth,
        Height = screenHeight
    }).GetAwaiter().GetResult();
    
    var screen = page.ScreenshotStreamAsync().GetAwaiter().GetResult();
    ...
}


This implementation gave encouraging results, the execution time increased by 10%.

Minuses

  1. The time to save screenshots through Puppeteer is not instant, some of the frames will be lost, and they may turn out to be something interesting to parse.
  2. If Selenium switches tabs, you need to notify Puppeteer, otherwise it will only screenshot the first tab in the code above (maybe there is a way to find the active tab - you need to look).

The first minus turned out to be blocking for me, so we move on to the next solution.

Screencast


Chrome has an interesting feature - Page.startScreencast . According to the description - she just does what is needed - she casts the changed frames so that those who wish can intercept them and do something interesting with them.

In both Selenium and Puppeteer, you can start Page.startScreencast, but you cannot add handlers in either one or the other. The hoteller is already voiced - we are waiting for the implementation.

I tried making friends with the ChromeDevTools library . Unfortunately, I could not quickly establish good relations with her. After further searches, a solution was found for ScreenCast in mafredri / cdp. The unnecessary navigation was removed from the original example and the necessary input parameters were added:

Hidden text
package main

import (
    "os"
    "context"
    "fmt"
    "io/ioutil"
    "log"
    "time"
    "flag"

    "github.com/mafredri/cdp"
    "github.com/mafredri/cdp/devtool"
    "github.com/mafredri/cdp/protocol/page"
    "github.com/mafredri/cdp/rpcc"
)

func main() {

    folderPtr := flag.String("folder", "", "folder path for screenshots: example c:\\temp\\screens\\")
    chromePtr := flag.String("chrome", "http://localhost:9222", "chrome connection - example: http://localhost:9222")
    
    widthPtr := flag.Int("width", 1280, "screencast width")
    heightPtr := flag.Int("height", 720, "screencast height")
    qualityPtr := flag.Int("quality", 100, "screencast quality")
    
    flag.Parse()

    if err := run(*folderPtr, *chromePtr, *widthPtr, *heightPtr, *qualityPtr); err != nil {
        panic(err)
    }
}

func run(folder string, chromeConnection string, width int, height int, quality int) error {
    ctx, cancel := context.WithCancel(context.TODO())
    defer cancel()
    
    chromePath := chromeConnection
    folderPath := folder

    devt := devtool.New(chromePath)

    pageTarget, err := devt.Get(ctx, devtool.Page)
    if err != nil {
        return err
    }

    conn, err := rpcc.DialContext(ctx, pageTarget.WebSocketDebuggerURL)
    if err != nil {
        return err
    }
    defer conn.Close()

    c := cdp.NewClient(conn)

    err = c.Page.Enable(ctx)
    if err != nil {
        return err
    }

    // Start listening to ScreencastFrame events.
    screencastFrame, err := c.Page.ScreencastFrame(ctx)
    if err != nil {
        return err
    }

    go func() {
        defer screencastFrame.Close()

        for {
            ev, err := screencastFrame.Recv()
            if err != nil {
                log.Printf("Failed to receive ScreencastFrame: %v", err)
                os.Exit(0)
            }
            log.Printf("Got frame with sessionID: %d: %+v", ev.SessionID, ev.Metadata)

            err = c.Page.ScreencastFrameAck(ctx, page.NewScreencastFrameAckArgs(ev.SessionID))
            if err != nil {
                log.Printf("Failed to ack ScreencastFrame: %v", err)
                os.Exit(0)
            }

            // Write to screencast_frame-[timestamp].png.
            name := fmt.Sprintf("screencast_frame-%d.png", ev.Metadata.Timestamp.Time().Unix())
            
            filePath := folderPath + name

            // Write the frame to file (without blocking).
            go func() {
                err = ioutil.WriteFile(filePath, ev.Data, 0644)
                if err != nil {
                    log.Printf("Failed to write ScreencastFrame to %q: %v", name, err)
                }
            }()
        }
    }()

    screencastArgs := page.NewStartScreencastArgs().
        SetQuality(quality).
        SetMaxWidth(width).
        SetMaxHeight(height).
        SetEveryNthFrame(1).
        SetFormat("png")
    err = c.Page.StartScreencast(ctx, screencastArgs)
    if err != nil {
        return err
    }

    // Random delay for our screencast.
    time.Sleep(600 * time.Second)

    err = c.Page.StopScreencast(ctx)
    if err != nil {
        return err
    }

    return nil
}


Further, this file was compiled by the command:

go build -o screencast.exe main.go

And I was able to use it in a C # solution with tests:

Hidden text
var startInfo = new ProcessStartInfo(screenCastPath)
{
    WindowStyle = ProcessWindowStyle.Minimized,

    Arguments = $"-folder={_framesFolderPath} " +
                $"-chrome=http://localhost:{_debugPort} " +
                "-width=1024 " +
                "-height=576 " +
                "-quality=0"
};

Process.Start(startInfo);


A separate stream for recording screenshots was thrown out as unnecessary. The work algorithm was as follows:

  1. Starting Chrome via Selenium
  2. We start Screencast through the assembled binary - it connects to Chrome and starts saving the stream of frames to the folder we specify
  3. At the end of the test, close Chrome - the binary closes automatically
  4. If the test fails - create a video
  5. We clean the folder with frames

This approach gave the best result in terms of execution time (there are practically no delays). Plus, he provided maximum information on the test (there are practically no lost frames).

Cons

1. Low resolution for screencast. If you run the tests in a couple of streams and set the resolution to 2560 * 1440 for Chrome, then the buffer allocated for data transfer will overflow .

2. With an increase in resolution, the load on the CPU increases.

As a result, under screencast, I chose the resolution of 1024 * 576 - on this resolution the tests worked fine in 6 threads, the processor worked in comfortable mode (6 core i7-5820).

We collect video


It remains to collect frames in the video. For this, I used the SharpAvi library

Hidden text
private void GenerateVideoFromScreens(string videoPath)
{
    try
    {
        var videoWriter = new AviWriter(videoPath) { FramesPerSecond = 1, EmitIndex1 = true };

        var videoStream = videoWriter.AddMotionJpegVideoStream(1024, 576);

        var screens = new DirectoryInfo(_framesFolderPath).GetFiles().OrderBy(f => f.CreationTimeUtc.Ticks).ToList();
        foreach (var screen in screens)
        {
            try
            {
                using (var bmp = new Bitmap(screen.FullName))
                {
                    var bits = bmp.LockBits(new Rectangle(0, 0, videoStream.Width, videoStream.Height), ImageLockMode.ReadOnly, PixelFormat.Format32bppRgb);
                    var videoFrame = new byte[videoStream.Width * videoStream.Height * 4];
                    Marshal.Copy(bits.Scan0, videoFrame, 0, videoFrame.Length);
                    bmp.UnlockBits(bits);

                    videoStream.WriteFrameAsync(
                        true,
                        videoFrame,
                        0,
                        videoFrame.Length).GetAwaiter().GetResult();
                }
            }
            catch(Exception ex)
            {
                // ignore all video related errors per frame
            }
        }

        videoWriter.Close();
    }
    catch
    {
        // ignore all video related errors per streams
    }
}


Upscale Pictures


Since the resolution of screencast is very small 1024 * 576, you need to set a small resolution for Chrome itself, otherwise there will be problems with small text.

Chrome 2560 * 1440 -> screencast at 1024 * 576 = small text is practically unreadable
Chrome 1920 * 1080 -> screencast at 1024 * 576 = small text is read with difficulty
Chrome 1408 * 792 -> screencast at 1024 * 576 = small text is read without problems

The resulting video 1024 * 576 can be improved - if the frames are scaled down to 1920 * 1080 using the PhotoSauce library

Hidden text
public Bitmap ResizeImage(Bitmap bitmap, int width)
{
	using (var inStream = new MemoryStream())
	{
		bitmap.Save(inStream, ImageFormat.Png);
		inStream.Position = 0;
		using (MemoryStream outStream = new MemoryStream())
		{
			var settings = new ProcessImageSettings { Width = width };
			MagicImageProcessor.ProcessImage(inStream, outStream, settings);
			return new Bitmap(outStream);
		}
	}
}


As a result, the following parameters were obtained: Chrome works in 1408 * 792, ScreenCast in 1024 * 576, the final video for viewing upscales to 1920 * 1080. Click here to see an example of the final result.

thank


Thank you to everyone who has read - if there is a simpler solution to the original problem, please write in the comment. Any criticism is also accepted, including malicious criticism as described above.

All health and speedy end to coveted restrictions!

All Articles