Advances supporting the next-generation AV1 codec have intensified over the past year, adding new urgency to the question of when the technology might become useful in WebRTC-based real-time interactive streaming applications.
There’s no question live video encoding supported by AV1 would benefit real-time streaming every bit as much as it stands to benefit traditional HTTP streaming. The codec achieves bitrate reductions in the range of 30%-40% compared to HEVC/H.265 and 60%-70% relative to AVC/H.264. Moreover, AV1 is available royalty-free with major tech sector backing through the Alliance for Open Media (AOMedia), whose members include Amazon, AMD, Apple, Cisco, Google, Hulu, IBM, LG, Meta, Microsoft, Netflix, Nvidia, Intel, IBM, Samsung, and many other leading players.
But the amount of computing power it takes to run the encoding and decoding processes has been a major impediment to AV1 adoption, especially when it comes to encoding at live video frame rates. As demonstrated in countless tests, there’s been no practical way to use AV1 to encode video in real time except at very low levels of resolution. For example, a series of recent tests conducted by cloud CDN and compute platform operator Cloudflare found that even if all 48 cores of the top-line AMD EPYC 7642 CPU processor were used to encode a 4K clip commonly used in such testing, it would take 43 seconds to encode two seconds.
But the perspective on the possibilities is shifting rapidly. And we at Red5 Pro are paying close attention to developments.
1. AV1 Live Encoding Is Closer than Ever to Commercial Viability
In 2021, Google released its desktop Chrome 90 browser with an AV1 encoder optimized for video conferencing with WebRTC. The encoder is targeted to two- to four-core CPUs for 2 Mbps transmission of 720p video at 30fps. This points to the fact that encoding AV1 for low bit-rate transmissions at sub-1080p levels is now doable on most PCs, which means the codec can be widely used to get decent-quality video over low-bandwidth mobile links that are the sole means of internet access in many parts of the world.
Cisco took another step in this direction in 2021 on the fringes of its Webex video conferencing service. The company now gives users the option to turn on live AV1 encoding to improve the quality of shared displays of video clips during conference calls.
Anyone on such a call with a device capable of decoding a low-bitrate AV1 stream will see the clip with much greater clarity. Cisco said it expects to expand the use of AV1 on Webex over time but had not announced any further steps as of YE 2022.
A much more dramatic sign of progress on the AV1 encoding front came in late 2022 with the introduction of powerful new AV1-enabled graphics engines by Intel, AMD, and Nvidia. As of this writing, for the first time, live AV1 encoding in fast-action gameplay can be executed by some versions of Intel’s Arc Alchemist and Nvidia’s RTX 4000 series running on gaming cards, which became commercially available in October. AMD released its far less expensive RX 7000 platform in early 2023 with no lofty claims for live AV1 encoding. Even so, early comparisons of the three tech companies’ latest advances, like this one reported by PC World, found them more or less on par with one another when it came to AV1 encoding of content for recording and archival usage.
It’s clear the AV1 live streaming barrier has been broken for gamers who want to shell out $1,500 or so to purchase the Intel or Nvidia gaming cards now on sale from multiple vendors. For example, testing of live video encoding with Intel Arc’s AV1 encoder against leading H.264 GPU encoders, as reported by PC World, found that when set to encode HD 1080p video at 3.5 Mbps, the quality of the video is superior to the output of H.264 encoders delivering the content at nearly twice that bitrate.
Of course, GPU cards significantly outperform encoders running on CPU cores or CPU cores enhanced with GPU acceleration, which remain the primary encoding engines in OTT streaming services. But these latest GPU platform advances point to the likelihood that the densification and architectural advances achieved with gaming encoders will be adopted in succeeding generations of hybrid CPU/GPU processors.
Another hardware advance signaling a role for live AV1 encoding occurred at the 2022 IBC trade show in Amsterdam, where RealSprint, the Swedish company operating the Vindral CDN service, revealed it would be introducing AV1 live encoding on its platform using ASIC (application-specific integrated circuit) hardware technology from NETINT Technologies. NETINT’s low-latency 4K AV1 encoder harbingers the emergence of ASICs as a potential game changer in advanced codec operations, according to NETINT COO Alex Liu, as quoted by Streaming Media. “[These applications] can only be met with ASIC hardware encoding,” Liu said.
2. Browser, Player Support for AV1 Decoding Is Gaining Wider Traction
Meanwhile, on the decoding side, there’s been a surge in browser and app support for AV1 software decoding on high-power devices. For now, this means that entities streaming on-demand content can use AV1 encoding with high confidence that a growing share of users will be able to take advantage of the bandwidth savings, which are beneficial to distributors as well as receivers.
The Google-backed Chrome browser and apps based on Android 10 or later can support software-based AV1 decoding on any device with enough processing power to accommodate the codec. Firefox, too, supports AV1, as does the Edge browser when used on PCs running Windows 10 or later.
Apple’s Safari browser and iOS operating system are holdouts, but it’s generally assumed that Apple, as a member of AOMedia, will fall in line at some point. Notably, the AVPlayer API incorporated with iOS and tvOS now offers optional support for AV1, meaning any app providers who choose to use the player can deliver AV1-encoded content to most iPhones and all TVs connected to the Apple TV module.
It’s important to note that browser and OS support for AV1 isn’t essential in the case of apps that incorporate support for the codec in their encoding and player software decoding mechanisms as long as targeted devices have the CPU or CPU with hardware acceleration resources to support decoding. Meta, for example, which has bypassed the inclusion of HEVC with its Facebook and Instagram Reels video apps in favor of using AV1, is finding more iPhones than Android mobile devices have enough hardware processing power to handle the codec, as reported in a post on NETINT’s site.
Citing a talk given by Meta software engineer Ryan Lei at the company’s Video @Scale Fall 2022 conference, the NETINT blog post reported Meta uses H.264, VP9, and AV1 to encode Reels and that it streams content using the most efficient encodes supported by targeted devices. Meta has found VP9 produces a 48% bitrate savings compared to H.264, while AV1 cuts the H.264 bitrate by 65%. Meta chose AV1 over HEVC based on tests that showed AV1 delivered a 30% improvement over HEVC.
Netflix, too, has found it makes sense to take advantage of AV1. It began using the codec for streaming its content to Android mobile devices in 2020 and, a year later, said it would use AV1 to stream content to any smart TVs that could support the decoding. At the time, Netflix said these included a wide range of Samsung TVs dating from 2020, as well as any TV connected to PlayStation 4 Pro, Amazon Fire TV running Fire OS 7 or above, and Android TV devices running Android OS 10 or above.
Netflix recently issued stats attesting to the benefits to be gained even at this early stage through more advanced alternatives to H.264 encoding. The streaming giant reported analysis of its output during one month in early 2022 showed that its contribution to global traffic volume would have been 24% higher without its reliance on per-title optimized HEVC and AV1 encoding. The company said these advances, along with its CDN-related Open Connect Program, had contributed to over $1 billion in savings for ISPs in 2021.
3. Device Hardware Support for AV1 Decoding Is Picking Up Steam
These developments are moving in tandem with ever more support for AV1 among device OEMs. Some of this support is in response to pressure from Google, which in 2021 began requiring AV1 decoding support in all new Android TV and Google TV smart TVs, media dongles, and set-top boxes. In addition, AV1 hardware decoding is built into the latest Chromecast TV streaming adaptor and the Tensor chipsets powering the latest Google Pixel smartphones.
Beyond the Google orbit, the push from Netflix, AOMedia member Hulu, Meta, and other major players looking to exploit AV1 is persuading chipmakers and OEMs to incorporate direct support for AV1 decoding in their hardware processing, thereby eliminating the need for software decoders in app-based players.
Hardware-accelerated AV1 decoding is fast becoming a ubiquitous feature of new smartphones regardless of the relationship to Android. AV1 decoding is already built into the Samsung Exynos 2200 and MediaTek Dimensity 1000 SoC smartphone chipsets. And Qualcomm, by far the dominant microprocessor supplier for Android smartphones, has broken with its avoidance of hardware decoding support for AV1 with an announcement that its forthcoming Snapdragon 8 Gen 2 chipset will support AV1 playback for HD and UHD formats extending to HDR (high dynamic range)-enhanced 8K.
As for smart TVs, beyond the Samsung models cited by Netflix and those like Sony’s Google TV model that have built-in hardware decoding support for AV1, new models from LG, Sony, Philips, TLC, and probably others are running on chipsets that can accommodate AV1 processing. But none of these brands have spelled out what, if any, steps they’ve taken to create the algorithmic paths in their firmware that would enable decoding without the aid of player software.
This may be the next shoe to drop as AV1 momentum builds. For example, LG, acknowledging its latest models have the horsepower to support AV1 and many other “smarter features,” recently announced it had joined AOMedia “to work collaboratively to further the development and adoption of AV1-enabled innovative online video streaming solutions.”
Getting this right at the core hardware level isn’t easy. While LG had previously issued documentation for developers claiming support for AV1 on UHD TV sets running WebOS 5.0 or above, testing performed by Bitmovin in conjunction with its October AV1 report found that some of those LG models couldn’t handle AV1 content. With this new commitment to the codec, that’s likely to change.
OEMs’ reluctance to expend more energy on AV1 support likely has to do with the fact that, until recently, the codec profiles hadn’t clearly spelled out support for HDR processing. This has been rectified with documentation clarifying how to use HDR 10 and 10+ with AV1, with work on Dolby Vision still in progress, according to Netflix senior research scientist Andrey Norkin. As reported in September by Streaming Media, Norkin said Netflix, which uses HDR with all its original content, has begun testing how AV1-encoded HDR video performs in playback with players that have been tweaked to support HDR.
4. WebRTC & AV1 Are More Compatible than Ever
Clearly, AV1 has a strong toehold in the marketplace with the potential to burst into the mainstream once live video encoding becomes practical. Fortunately, whenever that happens, putting AV1 to use with interactive real-time streaming over WebRTC will be a fairly straightforward process, thanks to preparations for that eventuality that are already or close to being baked into both standards.
The two protocols are intrinsically compatible, as noted two years ago by Bernard Aboba, chief architect at Microsoft and co-chair of W3C WebRTC and IETF Web Transport working groups. In an interview posted on the webrtcHacks site, Aboba said, “AV1 doesn’t require any changes to WebRTC PeerConnections per se….I think of it as the next in the lineage of VP8 [and] VP9….It has some H.264 kind of MAL (Multimedia Adaptation Layer) unit semantics, so it’s a little bit like a cross between H.264 and VP9.”
But he noted there were some important steps that needed to be taken in both development arenas to achieve smooth interoperability between AV1 and WebRTC. Since then, there’s been significant progress in this direction.
For example, it’s important that WebRTC be able to exploit Scalable Video Coding (SVC), which enables the generation of multiple content resolutions from a single encoder as a cost-effective alternative to server-side transcoding used with adaptive bitrate (ABR) streaming. AV1 is the first codec to incorporate SVC by default, and now completion of the draft Real-Time Protocol (RTP) Payload Format for AV1 specification within AOMedia has enabled AV1 SVC to be executed with WebRTC, which operates on RTP. In August 2022, WC3 issued SVC Extension for WebRTC as a draft specification spelling out the use of SVC with any type of codec used with WebRTC.
Another step toward compatibility involved aligning AV1 and WebRTC approaches to end-to-end security, which has been accomplished. Without going into technical details, key steps in this process have been the release of the RTP Payload Format for AV1 and the adoption of the IETF’s SFrame specification with the security intrinsic to WebRTC.
5. Red5 Pro Is Poised to Deliver Superior Results when Live AV1 Becomes Viable
So, now, the use of AV1 with WebRTC-based real-time interactive streaming (RTIS) platforms, including Red5 Pro’s XDN, primarily depends on when encoding support at high video profiles becomes practical. While, as noted above, standard-issue CPUs can support AV1 encoding with low-resolution video, which is helpful in the video conferencing business, XDN architecture is meant to provide support for scaling RTIS to an unlimited number of end users with sub-400ms latencies over any distance at levels of video quality common to live-streamed content.
Red5 Pro will be moving ahead with support for AV1 whenever it makes sense to put the codec to use on our platform. Judging by all the progress we’ve seen to date, those final steps are likely to be taken sooner than later.
Here it’s important to note that when that happens, XDN users will have the option to exploit much greater encoding efficiencies with AV1 than they’ll find in other real-time streaming scenarios. This results from the integration of XDN with the performance-enhancing capabilities of a software codec processing layer defined by the new MPEG-5 Low Complexity Enhancement Video Coding (LCEVC) standard. In late 2021 XDN became the first RTIS platform to include support for LCEVC in its SDKs.
As validated in recent tests conducted by Meta and Intel, LCEVC enhancements used with AV1 encoding cuts computations by about 40% with no loss in quality. And LCEVC was shown to broaden the set of mobile devices capable of playing back full HD AV1 content while extending battery life by 50% compared to non-enhanced streams using state-of-the-art AV1 software decoders.
Critically, there’s no disruption to connectivity with devices that don’t support LCEVC. Any devices not equipped with the lightweight LCEVC player will interact with the LCEVC-enhanced AV1 encoding at the native AV1 level.
As described in this blog, LCEVC has been proven to offer similar benefits with all codecs, including the ability to significantly reduce bitrates at a given level of quality, if that is the goal. In such cases, compression gains of about 40% are achieved with no increase in computational complexity, which, for example, is tantamount to reaching HEVC levels of compression with H.264 codecs.
As we await further developments in accelerated CPU support for live-stream AV1 encoding, we are also mindful that HEVC, with its substantial gains over H.264, has now reached mass market penetration at the hardware level in smartphones, smart TVs, and other devices. Be sure to stay tuned for further developments on this front, as well.
With video streaming moving into a new era of multi-codec options extending compression efficiencies far beyond the H.264 baseline, we’re just at the beginning stages of the advancements that will be essential to meeting the challenges ahead. Graphic realism in multiplayer gaming, fully immersive social engagement in virtual reality, 8K video entertainment, and the need for superior quality video in every use case where RTIS is in play will require aggressive but cost-effective use of the best compression technology available at every step of the way.
Be sure to stay in touch as we build on the XDN foundation to make that journey as smooth and productive as it can be. Contact us at email@example.com or schedule a call.