Get best of the week

WebRTC on Android: how to enable hardware encoding on multiple devices

For video calls in Badoo, we use the WebRTC standard and the H.264 codec. If you believe the documentation, this codec should work without problems on any Android device since Android 5.0. But in practice, everything turned out to be not quite so. In this article, I will talk about the features of implementing hardware encoding for the H.264 codec in WebRTC and how to make it work on more devices.



Why H.264?


When connected via WebRTC, all devices participating in the session transmit various communication parameters, including video and audio codecs. If devices support multiple codecs (for example, VP8 and H.264), priority codecs for the platform are listed first. This data is used at the reconciliation stage in WebRTC, after which only codecs that are supported by all devices remain. An example of such data with decryption can be seen in this document .

In the case of video calls, if one of the devices does not support the H.264 codec, both devices can switch, for example, to the VP8 codec, which does not depend on the hardware implementation on the device. But our application is available on a variety of gadgets, including smartphones from previous generations. Therefore, for video communications, we wanted to use hardware encoding whenever possible: it reduces the load on the processor and does not eat the battery so much, which is critical for outdated gadgets. H.264 hardware encoding support is implemented on a large number of devices, unlike the same VP8 .

H.264 support on Android


If you believe the description of support for multimedia formats , decoding H.264 Baseline Profile should work on all Android devices, and encoding - starting with Android 3.0. In Badoo, we support devices starting with Android 5.0, so we shouldn't have any problems. But it was not so simple: even in gadgets with the fifth version, we found a large number of features.

With what it can be connected?

As you know, when developing a new device on Android, any manufacturer needs to pass the Compatibility Test Suite. It runs on a PC connected to the device, and its results must be sent to Google to confirm that the device meets the requirements of the Android OS of the specified version. Only after that the gadget can be released to the market.

We are interested in multimedia tests in this test suite, and more specifically, video encoding and decoding tests. I decided to stop on tests EncodeDecodeTest , MediaCodecTest , DecoderTest and EncoderTest , as they are present on all versions of Android since 4.3. The graph of the number of lines of code in these tests looks like this:



Prior to version 4.3, most of these tests simply did not exist, and their significant increase fell on versions 5 and 7. Therefore, we can say that before version Android 4.3 Google did not check the compliance of devices with its specifications for encoding and decoding video, but in version 5.0 greatly improved this check.

It would seem that this indicates that starting from version 5.0 with encoding everything should be in order. But, given my previous experience with decoding streaming video on Android, I was sure that it was not. It was enough to look at the number of topics about coding in the Google group discuss-webrtc .

WebRTC source files, which are in the public domain, helped us search for pitfalls. Let's consider them in more detail.

H.264 support in WebRTC


Let's start with the HardwareVideoEncoderFactory .

There is a method with the talking name isHardwareSupportedInCurrentSdkH264:

private boolean isHardwareSupportedInCurrentSdkH264(MediaCodecInfo info) {
  // First, H264 hardware might perform poorly on this model.
  if (H264_HW_EXCEPTION_MODELS.contains(Build.MODEL)) {
    return false;
  }
  String name = info.getName();
  // QCOM H264 encoder is supported in KITKAT or later.
  return (name.startsWith(QCOM_PREFIX) && Build.VERSION.SDK_INT >= Build.VERSION_CODES.KITKAT)
      // Exynos H264 encoder is supported in LOLLIPOP or later.
      || (name.startsWith(EXYNOS_PREFIX)
             && Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP);
}

As we can see, support for hardware encoding on Android is implemented only for Qualcomm and Exynos chipsets. Why is there no support for other chipsets in the standard WebRTC implementation? Most likely, this is due to the peculiarities of the implementation of hardware codecs manufacturers. And it’s often possible to identify these features only at production, since it is not always possible to find particular devices.

All codec descriptions on the device are stored in the media_codecs.xml file. Here, for example, is this file for the Pixel XL and for the HUAWEI P8 lite . When you get a list of codecs using the getCodecInfos () method of the MediaCodecList object, this file is parsed - and the codecs stored in it are returned. This operation and the correct filling of this file by the manufacturer are covered in the CTS test.MediaCodecListTest , which also increased from 160 lines of code in Android 4.3 to 740 lines in Android 10.

In Badoo, we changed the isHardwareSupportedInCurrentSdkH264 method code, rejecting the “white” codec list and replacing it with a “black” list of program codec prefixes that are listed in WebRTC:

static final String[] SOFTWARE_IMPLEMENTATION_PREFIXES = {"OMX.google.", "OMX.SEC."};

But you can’t just take and implement support for all codecs without paying any attention to manufacturers' features. From the names of topics devoted to hardware coding on Android in the discuss-webrtc group , we can understand that in this case we will definitely encounter errors. Basically, they appear at the codec configuration stage.

Codec configuration options


The initialization of the codec for encoding is as follows:

MediaCodec mediaCodec = createByCodecName(codecName);
MediaFormat format = MediaFormat.createVideoFormat(mime, width, height);
format.setInteger(MediaFormat.KEY_BIT_RATE, targetBitrateBps);
format.setInteger(MediaFormat.KEY_BITRATE_MODE, VIDEO_ControlRateConstant);
format.setInteger(MediaFormat.KEY_COLOR_FORMAT, colorFormat);
format.setInteger(MediaFormat.KEY_FRAME_RATE, targetFps);
format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, keyFrameIntervalSec);
mediaCodec.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);

It is easy to make a mistake in some of these parameters, which will throw an exception when configuring the codec and disrupt the application. Also, when working with a codec, you may need to adjust its bitrate depending on various factors, since the codec itself does it wrong. For this, in WebRTC, the BaseBitrateAdjuster class is responsible , which has two descendants:


Accordingly, for each codec, you must choose your own bitrate adjustment mechanism. Let us consider in more detail the features of setting the initialization parameters for hardware codecs.

Stream resolution


After receiving the MediaCodecInfo object for the codec, you can examine the codec in more detail by getting its capabilities in the CodecCapabilities class . From them you can find out if the codec supports the selected resolution and frame rate. If it supports these parameters, they can be set safely.

However, sometimes this rule does not work. We are faced with the fact that codecs with the prefix “OMX.MARVELL.” encoded incorrectly, showing green stripes at the edges of the screen if the resolution of the stream was different from 4: 3. At the same time, the codec itself claimed that the selected resolution and frame rate are supported.

Bit rate mode


The standard mode for all video codecs is a constant bitrate. However, once we had to use a variable bitrate:

format.setInteger(MediaFormat.KEY_BITRATE_MODE, VIDEO_ControlRateVariable);

This happened on a Lenovo A1000 device with the Spreadtrum chipset (now Unisoc) starting with the prefix “OMX.sprd.”. A search on the Internet led us to a six-year-old post on Firefox OS that describes this problem and how to solve it.

Color format


When using the codec in byte-buffer mode, you must select the correct format. This is usually done using a function of the following form:

@Nullable
static Integer selectColorFormat(int[] supportedColorFormats, CodecCapabilities capabilities) {
  for (int supportedColorFormat : supportedColorFormats) {
    for (int codecColorFormat : capabilities.colorFormats) {
      if (codecColorFormat == supportedColorFormat) {
        return codecColorFormat;
      }
    }
  }
  return null;
}

Roughly speaking, the first of the supported color formats is always selected.

However, in the case of HUAWEI codecs starting with the prefixes "OMX.IMG.TOPAZ.", "OMX.hisi." and “OMX.k3.”, it didn’t work, and after a long search we found a solution: no matter what format these codecs return, you need to use the COLOR_FormatYUV420SemiPlanar format . The thread helped us to figure this out in one Chinese forum.

Bitrate Adjustment


The standard WebRTC code contains the following :

private BitrateAdjuster createBitrateAdjuster(VideoCodecMimeType type, String codecName) {
  if (codecName.startsWith(EXYNOS_PREFIX)) {
    if (type == VideoCodecMimeType.VP8) {
        // Exynos VP8 encoders need dynamic bitrate adjustment.
        return new DynamicBitrateAdjuster();
      } else {
        // Exynos VP9 and H264 encoders need framerate-based bitrate adjustment.
        return new FramerateBitrateAdjuster();
    }
  }
  // Other codecs don't need bitrate adjustment.
  return new BaseBitrateAdjuster();
}

As you can see from this code, for all chipsets except Exynos, bitrate adjustment is turned off. But this applies only to Qualcomm, since only Exynos and Qualcomm are supported in the standard code. Experimenting with various values ​​of this setting, as well as searching the Internet, we found out that for codecs with prefixes “OMX.MTK.” it also needs to be turned on. It is also necessary to do this for HUAWEI codecs starting with the prefix "OMX.IMG.TOPAZ.", "OMX.hisi." or "OMX.k3.". This is due to the fact that these codecs do not use timestamps of frames to adjust the bitrate, assuming that all frames come with the same frequency set when configuring the codec.

In conclusion, I will give a list of codecs that we received for devices on Android 5.0 and 5.1. They were of interest to us primarily because on newer versions of Android the situation is improving and non-standard codecs are becoming smaller.

This can be seen in the graph below. The logarithmic scale to better show rare cases.


As we can see, most devices had Spreadtrum, MediaTek, HUAWEI and MARVELL chipsets - therefore, our changes helped enable hardware encoding on these gadgets.

Result


Although we assumed that some devices would have problems working with H.264, Android was again able to surprise us. As we can see from Badoo user statistics, there are still quite a few devices in the hands of users in 2014-2016, which they do not want or cannot update. And although the situation with the release of Android updates for new devices is already much better than a few years ago, the proportion of previous generation gadgets is declining quite slowly and will have to be maintained for quite some time.

Now WebRTC is being actively developed by Google due to its use in the Stadia project (here's the videowith details on this topic), so it will become better and better and, most likely, will become the standard for implementing video communications. I hope that this article will help you understand the features of working with H.264 in WebRTC and use it in your projects.

More articles:


All Articles