The next stage in video streaming revolves around increasing access to 4K–Ultra High Definition–video. Right now, the major obstacles to implementing 4k on a wide scale is the large amount of data it consumes and the delay it takes to send it. Currently, the codec in widest use is H.264–also known as AVC. Given H.264’s high compression ratios, it is not well suited for the high bandwidth demands of 4K streaming.

In order to meet this challenge, a variety of next-gen codecs have been released. Though some of these next-gen codecs are stable, many of them are still actively in development to make them more practical, increase their availability, and make them more useful in general.

However, it looks like there may be a way to solve the bandwidth and latency issues without having to swamp through the video codec morass. This post will examine the different options currently available and present 9 reasons for why the winning combo right now is LCEVC and WebRTC.

So what are the current options and who are the major players?

Earlier this year, the Moving Picture Experts Group (MPEG) finalized H.266, also known as Versatile Video Coding (VVC), as well as taking major steps toward final approval of Parts 1 and 2 to MPEG-5, known –respectively– as Essential Video Coding (EVC) and Low Complexity Enhancement Video Coding (LCEVC). Though EVC and LCEVC are both on track for final approval before year’s end, there could be some delays owing to uncertainties related to the pandemic and the upheaval at MPEG, which recently underwent reorganization under administration of ISO/IEC JTC 1 (Joint Technical Committee of the International Standards Organization and the International Electrotechnical Commission).

Designed to outperform the royalty-free AV1 codec, H.266 (VVC) is the only new standard of the group that can really be considered next-generation due to the high degree of compression it can achieve. However, that increased compression also causes a large increase in processing time making it untenable for widespread use today. It is worth noting that AV1 is backed by the Alliance for Open Media (AOMedia) whose members include Amazon, Apple, Facebook, Google, Microsoft, Cisco, Intel, IBM, Netflix, Samsung, and others. AV1 also has quite high processing times currently making it impractical for real-time video streaming use cases. Perhaps the designers of VVC and AV1 were really only looking at high latency use cases, or ones where latency isn’t a factor like video on demand.

It’s also worth pointing out one of the best features of the older most universally supported and codec, h.264: it’s fast! The h.264 compression algorithm is much simpler than the newer codecs, and this means that encode and decode times are very fast. Because of this h.264 is really well suited for real-time video streaming despite it creating much higher bandwidth video.

Now without further ado, let’s look at the reasons LCEVC and WebRTC are such a powerful combination for real-time video streaming.

1) Increased Compression

Simply put, LCEVC offers increased compression (around 40%) without the downside of high processing times that plague the new cocecs EVC, VVC, and AV1.

EVC is designed as a superior, lower-cost alternative to AVC (H.264) and HEVC (H.265) using elements of each, where 25-30% compression gains are achieved for AVC with a baseline royalty-free profile and for HEVC with a second profile that incurs royalty costs.

EVC is designed as a superior, lower-cost alternative to AVC (H.264) and HEVC (H.265) using elements of each. There are two profiles involved for EVC: a “Baseline” royalty-free profile for AVC and a “Main” royalty incurring profile for HEVC.  According to Streaming Media Contributing Editor Jan Ozer’s research, Baseline offers a compression improvement of 31% over AVC, while Main provides a compression improvement of 26% over HEVC. Of course as we already mentioned, it comes with high processing times making it impractical for ultra low latency live streaming use cases.

From a practical perspective, this is why LCEVC is so attractive. While we are comparing it to the other modern codecs, LCEVC isn’t exactly a new codec per say. Rather, LCEVC adds an extra processing layer to increase compression ratios in the 40% range, and it wraps existing codecs such as h.264, VP9 and AV1.

LCEVC is able to achieve higher compression by creating a hybrid stream where two codecs essentially work together. The first component of an LCEVC stream is a lower resolution segment containing the base codec, which can be any codec such as AVC, HEVC, AV1, etc. That low fidelity version is paired with enhancement information so that the low-resolution base can be upscaled by the corresponding decoder to a higher resolution. LCEVC is more suitable for software processor implementation since the process lowers the complexity and resulting CPU consumption. Importantly, this adds flexibility around the hardware options currently available. It stands up well against AVC providing at least 36% improvement and improving upon HEVC bit rates with a much needed reduction in encoder computation.

When LCEVC is used to enhance a codec like AVC (h.264), MPEG says, “the compression efficiency for the aggregate stream is appreciably higher than that of the n-th generation MPEG codec used at full resolution and as close as possible to that of the (n+1)-th generation MPEG codec (e.g., HEVC) used at full resolution, at bandwidths and operating conditions relevant to mass-market distribution.” In other words, you get HEVC-level compression using AVC encoding with no reduction in quality.

A test conducted by the Streaming Learning Center–operated by Jan Ozer–confirms MPEG’s statement. In his write-up Ozer reports “impressive” results, “Overall, the analysis shows that if you substitute LCEVC with a base layer of x264 [the open-source software used for encoding H.264] for x.264 at 1080p resolution in a live use case, you should achieve about 40% bandwidth savings without noticeable quality loss, and you won’t need dedicated decoding hardware on the playback side.”

Ozer’s reference to a “base layer of x264” has to do with the fact that the basic AVC encode is set for a resolution, in this case 960×540, that is well below the resolution to be achieved with the LCEVC enhancement. The LCEVC processing closes the gap between whatever resolution is chosen for the base layer and the targeted resolution.

For another example let’s consider live 4K UHD for events such as a live sports broadcast. Typically such high quality requires 10-bit encoding and a 60-fps frame rate. Using plain H.264 this would result in streaming with an untenable 50-60 Mbps bitrate. However, using LCEVC with an H.264 base codec, those same parameters could be supported at 30-36 Mbps. Given the ongoing bandwidth expansion in seasoned broadband markets, that is a reasonable target.

Sidenote: It should be stated that a 4K sports stream delivered to native H.264 decoders would be transmitted at a much lower bitrate, resulting in a lower-quality viewing experience. However, this is likely to be of little consequence in situations where the content is streamed to authorized users in high-speed broadband environments.

While the real winning combination for live real-time video streaming is LCEVC + h.264 due to the low processing time, it’s still possible to use newer codecs. In cases where HEVC or a next-gen codec like AV1 or the new MPEG VVC is brought into play, the bitrates required for live streaming in 4K would be reduced by margins similar to those achieved with H.264 (about 40%). From a bandwidth perspective, this would make it easier to support more data-intensive applications like VR. For example, with HEVC in the case of 4K, live sports video that would normally require transmissions in the 25-30 Mbps with 10-bit encoding and 60-fps could be streamed at 15-18 Mbps.

It’s also important to note that all these benefits apply when it comes to encoding for HD and SD.

For example, an H.264-encoded 1080p high frame rate live sports stream that normally requires a top bitrate of around 8 Mbps can be delivered at equivalent quality at 4.8 Mbps when the LCEVC is used, or a similar rate that would be used with HEVC encoding alone. In the near future when faster encoders for these new codecs emerge, LCEVC will be ready to take advantage of them. Fast encoding can really only be done practically when the processing work to do it is very low which leads us to our next point, processing power.

2) Reduction in Processing Power

While it’s fairly obvious why you would want to reduce processing power and CPU consumption (better performance/battery life, less drain on resources, etc.) it’s rather less obvious how LCEVC performs that reduction.

Traditionally, MPEG uses a linear macro-block approach to segmenting and encoding each frame in a video sequence. LCEVC uses a different approach with a hierarchical technique that uses the parallel processing and scaling capabilities of today’s microprocessors to apply compression frame by frame. Interestingly, parallel processing is more analogous to how the human visual system brings a scene into focus as attention shifts or the scene itself changes within the broader field of vision.

Happening in real time, the processing begins by specifying the gist of the information associated with any pictorial element such as a tree or a person’s face. It then adds additional information to achieve the required quality level for that element relative to the quality reached in the uncompressed frame. In this way, LCEVC determines how elements in the frame need to be reconstructed in strict adherence to variations across points of greater or lesser focus as captured in the original signal.

By removing inter-block prediction, LCEVC allows for parallel processing of small 2×2 or 4×4 independent transform kernels. This is key to the dramatic improvement in encoding time as well as a reduction in use of processing power. David Ronca, director of video encoding at Facebook, notes that eliminating barriers that have made HEVC, AV1 and other advanced codecs ill-suited for live streaming is vital to the future evolution of video services.

“The longer-term answer to video encoding cannot be to simply add more CPU capacity,” aid Ronca in a recent LCEVC Broadcast Bridge article. “This is an unsustainable model, both financially and environmentally. Codec research must emphasize both compression efficiency and computational efficiency.”

When LCEVC is used with HEVC, tests have shown that LCEVC produces a net reduction of about 64% in total processing time compared to the time consumed by HEVC alone. It does so by minimizing processing power devoted to the core codec and shifting the rest of the processing to the enhancement layer. With H.264, the encoding time, which in native mode is considerably less than that of HEVC, is still reduced by about 58%. More on the encoding time below.

Due to a lack of specific testing, there currently is no formal data on how well LCEVC performs with more processing intensive codecs like AV1 and VVC. However, it is logical to presume that similar or even greater gains are achievable.

3) Reduced Encoding Time

While it’s a good thing that codecs such as HEVC and VCC reduce the stream size, they come with an unfortunate side effect. All that additional compression in order to make the stream smaller means that it needs to perform additional processing. That additional processing consumes more CPU and takes more time. Like breaking down boxes for the recycling bin, flattening all that cardboard means you can fit more, but it takes more time than just tossing the empty box straight in the bin.

This is what makes LCEVC so useful. As discussed above, LCEVC is able to increase compression without the subsequent increase in compression time. It reduces total encoding time by a significant amount compared to the time consumed by any given baseline codec to achieve a targeted level of resolution without LCEVC. This is a potential game-changer, especially for live streaming.

But why is a reduced encoding time so important in the first place?

4) Low Latency

Lower encoding time means the video data can be packaged for delivery over an internet stream sooner which ultimately results in lower latency for the entire streaming process. Lower encoding time equals low latency.

Latency is very important to live streaming since anything with a long delay isn’t really live anymore. Even a delay of one or two seconds between the broadcaster and subscriber substantially degrades an interactive live streaming experience. Any sort of live stream application needs low latency in order to have a natural flow.

There are a great variety of use cases that absolutely need real-time streaming with a latency of under 500ms. Sports broadcasts need to prevent spoilers from Twitter and texts while ensuring bets are placed in a timely fashion. Live event broadcasts should allow for crowd response in real-time, live auctions need to make sure bids are registered to the right item at the right time, and social platforms need to ensure smooth back and forth conversations.

For live video streaming latency measured in seconds will always be too long.

As detailed previously, the WebRTC protocol is able to provide that low latency. Built on web-standards, WebRTC supports live streams with under 500 milliseconds of latency.

Furthermore, initial testing indicates that LCEVC-enabled software or hardware-based encoders can be effectively combined with the Red5 Pro cross-cloud platform to facilitate real-time streaming with a new generation of processing-intensive video formats. Depending on which core codec is used, this applies not only to 4K and–eventually–8K UHD, but also to formats devised for virtual reality, 360 degree viewing, and other innovations.

The fast encode times of h.264 with the addition of low latency LCEVC processing mean that it’s the perfect combination for WebRTC based delivery.

5) Backward Compatibility

Another positive aspect of LCEVC, is how the hybrid model of using two codecs (a base codec and an upscale codec) allows for backwards compatibility. Any device equipped to decode in the core format that isn’t running the LCEVC client software will be able to display the video at the level of resolution supported by the baseline encode without the enhancement. Thus running LCEVC with a widely used codec–such as H.264–will ensure widespread compatibility.

Again, h.264 + LCEVC + WebRTC allows for a huge win. Even if you don’t have an LCEVC based player, a person will still be able to play the stream.

6) Cost Savings

In respect to current conditions where the AVC codec dominates usage across the world, reducing the size of a stream will result in substantial cost savings. Smaller streams means paying less for sending data over transport links and through CDN tollgates. Thus, with the widespread popularity of OTT applications, a distributor who can implement a low-cost means of improving compression efficiency stands to make a very large profit.

Due to the reduction in bandwidth, huge costs savings can also be had with LCEVC compressed video when used with cloud based deployments like Red5 Pro where cloud providers often charge significant data transfer rates. For example AWS charges 5¢ per GB even in their most cost effective regions like US East Ohio, and that’s after transferring 150 TB of data. Reducing your bandwidth of your streams by 40% can have a big impact on your total cost of operation.

Additionally, having the flexibility to deploy on a cloud platform with a hosting agnostic solution like Red5 Pro, means you can choose the most cost effective cloud provider for you. You can even port it over to a different solution in the cloud provider changes their prices.

The enhancement processing technology that powers LCEVC was developed by U.K. startup V-Nova and gained commercial traction as a proprietary solution with deployments in areas with weaker bandwidth infrastructure such as Africa and India. While V-Nova has yet to announce the licensing terms for making their technology available for MPEG standardization, they have made it clear that costs associated with LCEVC will be minimal as compared to fees incurred with HEVC.

Finally, when it comes to introducing UHD or other advanced video formats, there are major cost savings to be realized. In addition to avoiding HEVC royalty fees, LCEVC eliminates the encoder and other expenses incurred with introducing a next-gen codec. Even without any royalties encoders can be expensive as is. In the case of LCEVC, the additional processing is accommodated via software running on commodity appliances with no need to replace existing encoders.

7) HTML5 Implementation

Another point for LCEVC comes from the fact that it’s fairly easy to implement it in HTML5. Virtually any device can support a thin LCEVC client embedded in the service provider’s app player or downloaded independently to viewers’ devices. Eventually, there should be virtually no barriers to scaling LCEVC’s reach, due to the MPEG requirement that web browsers should be able to support LCEVC in HTML5 JavaScript without plug-ins or browser upgrades.

This means that a future version of the Red5 Pro HTML5 SDK with LCEVC wouldn’t be that hard to build, and such a thing might just be in the works.

8) Best Quality in Restricted Networks

LCEVC also stands to create new opportunities for live and on-demand video distribution in low-bandwidth markets where people largely rely on 2G or 3G mobile connections for Internet access. An H.264-encoded 720p/30 fps stream at 3.5 Mbps would be reduced to 2.1 Mbps using H.264 with LCEVC, while, at 480p, a good quality experience could be delivered to cell phones at 1.2 Mbps.

9) Hardware Support

While it remains to be seen how encoding vendors will react to the availability of LCEVC, prospects for adoption in hardware-based products have been bolstered by commitments from two chipmakers. Xilinx, U.S.-based inventor of the FPGA (field programmable gate array) has already incorporated V-Nova’s technology into its chipsets, and Canadian SoC (system-on-chip) supplier NETINT Technologies, plans to introduce support for LCEVC in 2021.

MPEG surprised many industry observers and vendors by actually meeting the daunting list of requirements they had for LCEVC. As evidenced by the progress of the standard toward finalization and by recently reported third-party testing results such as the Streaming Learning Center, all benchmarks have been achieved; a remarkable accomplishment.

Red5 Pro can be used with any hardware or software encoding platform. For distributors who rely on Red5 Pro to achieve industry-leading real-time streaming performance, there’s much to gain by requesting support for LCEVC from their encoding suppliers.

To learn more about what we are working in in this space, contact info@red5.net or schedule a call.