The process of sending and receiving live video over the internet is complicated. Encoding images and audio, transporting it, then decoding it for display involves a web of different methods with a multitude of available options to perform those methods. One of those important components is the codec used to encode and decode the media along with defining the types of tools used to perform the streaming.

Basically, in order to stream a video over the internet, it must first use a microphone to capture the audio and a camera to capture the series of images that will make up the video. Then that raw data must be compressed (encoded) into a codec, broadcast over an internet connection (using a transport protocol), sent to some kind of server-side solution (typically a CDN or a cloud-based cluster like Red5 Pro), and subsequently decompressed (decoded) for the subscriber to finally watch the video.

There are quite a few codecs currently in use today including, VP8/9, h.264 (AVC), h.265 (HEVC), and AV1. In previous posts, we covered AV1, and VP8/9 so this post will focus mainly on h.265. Of course, the most important thing is what codec works best. Thus we will present the case for how h.264 is currently a more effective choice for low latency live streaming.

What is h.265?

The h.265 codec, or High Efficiency Video Coding (HEVC), was developed through a joint effort by the Video Coding Experts Group (VCEG) and the Moving Picture Experts Group (MPEG).  Approved in April 2013, it was designated the official successor to h.264, also known as Advanced Video Coding (AVC). It improves upon the compression efficiency of h.264 reducing the size of the video by around 50%.

What is h.264?

H.264, or AVC is currently the most widely adopted video codec. As of September of 2019, it is used by 91% of video industry developers. Like h.265, h.264 was also developed by the Moving Picture Experts Group (MPEG) as an improvement over previous standards with an aim to deliver efficiently compressed high-quality video over the internet.

Licensed by the MPEG-LA organization, H.264 is protected by many patents. However, in 2013 Cisco essentially paid for the patent license to open it up for free general use. Cisco did this by open-sourcing an encoder and decoder called openH264. This opened the door for wide adoption of the h.264 codec, and implementations of openH264 showed up in all the web browsers.

AVC vs. HEVC – What Is the Difference?

Logically, one might assume that since HEVC (h.265) is the most recently developed codec it would be the most effective. In reality, it is more complex. As intended, HEVC offers better compression efficiency than AVC (h.264). An h.264 will take up about twice the space as an equivalent HEVC file. A smaller file size means it requires less storage space which, more importantly, corresponds to less bandwidth consumption when streamed. HEVC’s advanced compression efficiency is due to how it processes macroblocks. HEVC also features improved deblocking and motion compensation filters, better variable-block-size segmentation, sample adaptive offset filtering, and advanced motion vector prediction and precision. This page from the x265 group has a great explanation of these terms and how it can improve HEVC’s efficiency.

With all these advantages, why isn’t HEVC the better codec for live streaming? We will cover 6 reasons.

1) Higher Encoding Speeds

Among the reasons that make h.264 better than h.265, one the biggest is latency. Latency is connected to encoding speed since the longer it takes to encode the data, the longer the stream will be delayed.

As stated earlier, HEVC produces a smaller video than AVC. The smaller file size makes it easier and faster to transport it over the internet since there is less data to stream. However, the savings from sending a smaller file is outweighed by the length of time it takes to encode an h.265 file. Compared to AVC, HEVC needs to perform much more compression resulting in a much longer length of time to prepare the file for streaming. Thus, it will make it longer to send the video. Despite taking less bandwidth, the increased encoding times ultimately surpasses the time saved in stream delivery.

Those less familiar with the live streaming industry may not see why latency is so important.  A long delay between the broadcaster and subscriber substantially degrades an interactive live streaming experience. Any sort of live stream application needs low latency in order to have a natural flow. In that regard, long is only a matter of seconds.

Latency affects a variety of use cases that all need real-time live streaming. Sports broadcasts need to prevent spoilers and ensure bets are placed in a timely fashion. Live events should allow for crowd response in real-time, live auctions need to ensure bids are getting registered at the right time, and social platforms need to ensure smooth back and forth conversations.

Essentially the extra overhead from HEVC means that it will have more latency when compared to AVC encoding.

2) Limited Adoption

In order to work with the codec, hardware, or software encoders need to support it. The concern regarding patent licensing is a major reason behind HEVC suffering from low adoption since its release. While AVC has only 1, HEVC has four patent pools related to it: MPEG LA, Technicolor, HEVC Advance, and Velos Media. This makes it more expensive which has discouraged more widespread adoption thus limiting it to specific hardware encoders and mobile chipsets. As mentioned earlier, Cisco open-sourced its h.264 implementation and released it as a free binary download. That boosted AVC implementation to the widespread use it currently enjoys.

While there are HEVC software encoders available (which by the way also carry the same license restrictions), the use of software encoders will create performance issues as it will consume more CPU. Since live streaming is already a CPU intensive process, anything that further increases the CPU load will be doubly problematic as battery life will decrease and the latency potentially increases.

3) Not supported in Browsers

As opposed to AVC, HEVC is not supported by the majority of internet browsers. Only Internet Explorer (now being phased out), Edge, and Safari support HEVC. Unfortunately, even when a browser supports HEVC there is no guarantee that it will actually work since the device running the browser will still need to support HEVC hardware encoding. Furthermore, even if everything is fully supported WebRTC (used for real-time transport) tends to not work correctly.

The widespread use of laptops and mobile devices for internet browsing means that not being able to watch the video in internet browsers severely limits how users can access live streaming. Supporting a wide variety of devices is the only way to ensure that a variety of users can have access to it.

4) Reducing Bandwidth Is More Important for High Resolutions

Part of the advantage of higher compression rates and smaller file sizes is that it consumes less bandwidth meaning that users can still stream high-quality videos at lower internet speeds. The reality is that with a global average of 42.63 Mbps upload speed for fixed broadband connections, most places have internet speeds high enough to support even 4K streaming over AVC. Mobile devices have significantly lower average speeds at 10.93 Mbps, but even that is more than fast enough for 1080p.

This diagram from Boxcast shows that the average worldwide connection speeds are definitely able to handle the upload speed requirements at all tiers of resolutions.

There are methods to assist users in countries with slower internet speeds such as adaptive bitrate (ABR) and transcoding. ABR will adjust the bitrate to deliver the smoothest possible experience and transcoding will split broadcasts into multiple qualities for the highest possible experience depending upon the available bandwidth. Some mobile devices may be stuck on 2 or 3G connections, but devices the size of your hand don’t really need 1080p resolutions to look good. 720 or even 480 will still display with good quality.

However, the reality is that smaller video will save companies money in regards to bandwidth costs. More compressed video means less data to stream which results in not having to pay as much for data sent over CDN or cloud networks. H.264 live streaming may cost a little more but it delivers better results that create better-performing applications that will attract more users. While that is certainly nice, it is only at really high-resolution settings such as 4K that halving the data consumption makes a substantial difference.

Of course, saving money is certainly an important thing no matter what the scale. That brings us to our next point that provides the best of both worlds; better compression with the same real-time performance.

5) LCEVC Circumvents the Whole Argument

LCEVC (Low Complexity Enhancement Video Coding) adds an extra processing layer to increase compression ratios in the 40% range with existing and future versions of MPEG or other codecs such as VP9 and AV1 (more on this below). As we covered in a previous article, LCEVC is a very promising piece of technology. Essentially, LCEVC enables the current batch of protocols that have already been adopted in one form or another to be transformed into more efficient protocols in and of themselves.

Currently, it appears that content providers will be able to combine LCEVC-enabled software or hardware-based encoders with the Red5 Pro cross-cloud platform to facilitate real-time streaming with the newer, processing-intensive video formats. Depending on which core codec is used, this will apply to both 4K and the eventual adoption of 8K UHD, in addition to formats devised for 360-degree viewing, virtual reality, and other innovations.

That’s because virtually any device can support a thin LCEVC client embedded in the service provider’s app player or downloaded independently to viewers’ devices. Furthermore, the accompanying HTML5 JavaScript implementation means plugin-in free browsers support with LCEVC. Given these considerations, widespread implementation should be fairly straightforward.

6) AV1 is the Next Step

AV1 is poised to be the next widely used codec. It achieves an even greater amount of compression over h.265, and h.265 enjoyed more compression over h.264. Thus it will consume even less bandwidth than h.265.

Furthermore, the consortium behind AV1 has all of the major players involved and it’s royalty-free. This solves two of the biggest issues that plagued all the earlier codecs AVC, HEVC, VP8 and VP9: patents and compatibility.

The Alliance of Open Media (the organization behind AV1) consists of 48 members, including all browser vendors and most chipset vendors. That means that AV1 will eventually benefit from widespread adoption on browsers and smartphones unlike earlier issues with other codecs stemming from the fact that Google, Apple and Microsoft usually don’t share their toys with each other.

All that is holding AV1 back right now is the lack of real-time encoders. Once those become widely available, AV1 (especially when paired with LCEVC) will be the way forward.

Bottom Line: h.264 / AVC Works Better Today

Though streaming with h.265 improves upon elements of h.264 live streaming, those improvements end-up causing other issues that work against the practical use of HEVC. Longer encoding times create higher latency, concerns about patents limit adoption, browser support is lacking, and the real-world viewing resolution of most live streams means that HEVC will likely not enjoy any sort of widespread use. AV1 without it’s patent restrictions and even better compression are poised to replace both h.264 and h.265. Even more important, LCEVC is poised to sidestep the entire codec issue making it all but null and void.

Think we missed something in our analysis? Let us know by sending an email to info@red5.net or schedule a call with us to find out more about Red5 Pro.