The coronavirus has greatly disrupted many of the things we previously enjoyed doing. Especially attending concerts, sports events, and theatre productions. Gathering large crowds together in performance halls or athletic fields is unsafe in the midst of a pandemic. However, even before the virus there were problems with attending live events in-person. For some fans they may not have been able to afford a ticket, travel to a venue or the tickets were sold out before they got a chance to purchase them. Accordingly, it is imperative that we figure out a better way to translate the excitement of a live event over a virtual platform. Not only is this important for current conditions, but once employed, these virtual live event platforms will continue to be valuable once life goes back to normal.
This is why it’s even more important than ever to think hard about the design of your streaming infrastructure. Live video is complicated which means that a single developer will not possess all the knowledge and skills to build an application from scratch. To achieve your goals, you and your team will need to pull together various technologies: pure open source, proprietary, or a more common combination of all of the above.
Let’s take a look at some of the key features you should consider in choosing a platform to build an entertainment streaming app:
The excitement of a live event comes directly from the crowd as they cheer their favorite artist or sports team. It’s the combined enthusiasm of those fans that drives the experience. As the crowd reacts to events the athletes on the field or the performers on stage feed off that energy. In other words, it’s the interaction between the crowd itself and those they are watching that makes live events so much fun.
Another exciting aspect is fans betting on the action. This adds another layer of emotional attachment to all the fans simply cheering for their team. With the laws around sports gambling opening up, it will play a large part in the future of live event broadcasts.
A few different elements are involved in realistically simulating this interactivity. To replicate the social aspect of attending a live event, you can create in-app chat rooms so fans can share their reactions to everything happening in real time. These rooms could be created by using a flexible API to develop a bespoke chat system or integrate with a third party provider. Further extending that API could enable push notifications targeted to match a users’ interests, or an automatic signaling system that lets people know when someone replies to their comments. By mimicking the natural pace of a conversation, this will help them to engage with content and brings the experience closer to how it would be in-person.
There are methods for manipulating the video to increase visual interactivity as well. The creation of video calls between a group of friends would allow them to watch the event together in a virtual setting. The goal is to create a virtual experience as close to reality as possible, in this case everyone sitting together in the stands. Constructing a series of screens in the stands could foster a virtual crowd with a video feed coming from everyone watching the event, such as what the NBA has done with Microsoft Teams. Since this would essentially be a large collection of two way connections, the athletes or performers could hear the crowd and build off of their feedback. Perhaps with advancements in VR, friends will be able to gather virtually in the same room emulating the experience of sitting together in the concert hall and listening to the music.
Features such as the ability to switch camera angles would further enhance the streaming experience. This multi-view capability could include wide angle shots of the stage or field, to close-up shots of backup vocalists and player reactions. By setting up cameras in different sides of the venue, the action would be experienced in a way that could only be provided with a live streaming application.
Interactivity depends heavily upon the next two categories; latency and synchronization.
Live events need to be shown in real-time so viewers can experience everything as it is happening. How can a live event be truly live if it takes seconds to deliver the “live” event broadcast? You need real-time latency.
This year, America’s largest sporting event, The Superbowl, experienced broadcast latencies of at least 28 seconds, and that was the fastest!? Even those texting with an old Nokia phone could send out a spoiler well under 28 seconds. Whether it’s a text or a tweet, that spoiler would ruin the excitement.
Solving for the real-time latency issue, requires a departure from traditional CDN architecture to switch to modern protocols developed around low latency. Unfortunately, most streaming platforms and CDNs are built on older protocols which are too slow due to the fact that they are based on HTTP. Attempts to improve those protocols such as Low Latency HLS, CMAF, Apple Low Latency HLS and even non-HTTP approaches like WebSockets and Media Source Extensions have all fallen short of delivering actual real-time latency. Considering that they deliver a latency measured in seconds, they will never be fast enough for interactive experiences that consumers want.
WebRTC is currently the only realistic choice for browser based applications to deliver sub 500ms of real-time latency. In addition to WebRTC, RTSP can be used in Native mobile apps to achieve the same performance. The delivery of actual real time latency helps with synchronization as well.
As previously mentioned, the prevention of spoilers is very important. While synchronization will help with this, that’s not the only thing it addresses.
The proper synchronization of data with the video stream is an important consideration as well. For example, game statistics shown in a real-time graphic display should correlate with what is currently happening on the screen. A delay could lead to an inaccurate score causing confusion for fans who might mistakenly think that a goal didn’t count due to a rule violation.
Synchronization is also very useful for sports betting. Ensuring that accurate bets are placed at the correct time is important. Furthermore, there shouldn’t be a delay between what is currently happening at the game or race and when the action is seen by those watching it. This is due to the fact that there would be an unfair advantage if someone who is at the event places a bet or texts someone outside the event before everyone else can see what happened.
Besides improving the fan experience, collecting user data on viewer watching habits is a useful asset. Tracking which camera angles fans were watching or if they clicked off the feed at any point is valuable information for the event broadcaster. Of course, that data is only useful if the synchronization is accurate.
Synchronization is also key in making sure that users’ comments align correctly with what they are seeing. If the comments come staggered out of order, it will disrupt the flow and make the experience less enjoyable.
Usually, the main objective for a live broadcast is to pull in as many viewers as possible. Successfully streaming to a large audience obviously requires scalability. Without the ability to support more users they consequently won’t be able to watch. That’s fundamentally a bad user experience.
Even though scaling is a pretty easy concept to understand, configuring a fully scalable architecture is a little more involved. Enabling a large number of fans (especially for chart-topping performers) is only part of the equation. The other part is ensuring that all the fans subscribing to the stream get the same experience. Everyone watching the broadcast should view it with the best quality possible and see it in real time.
The HTTP based CDN architecture provides scalability, but with a major tradeoff in latency. As previously discussed, this is due to the need for CDNs to cache data in different physical data centers causing a slow down in the live streaming process.
The use of cloud-based infrastructure circumvents CDN limitations. The configuration of cloud-based autoscaling enables servers to dynamically spin up or down as needed. Autoscaling server clusters set up in different geographic regions, ensure that fans streaming from across the globe can still access the video stream. To get around the issue of different hosting providers in different parts of the world, a cross-cloud solution allows you to access the best regional data center regardless of your provider.
Scalability should not come at the expense of profitability. Flexible scaling is also an important factor to consider when building an entertainment streaming app. The cost of running large servers could be wasted on too much extra capacity if you end up with a smaller than expected crowd. Even worse if the event is cancelled. As the name suggests, autoscaling spins up servers on-demand and in real-time to meet the current load.
Scaling up is great, but scaling down is important as well. By spinning down unneeded servers this reduces the chance of running with too much capacity if fans drop off the stream.
There is certainly still a need for efficient software as within a cluster, the number of connections per server instance should be maximized to make the most out of the available CPU and RAM. While dynamic scaling is useful, you shouldn’t have to constantly be spinning up new servers every single time a few subscribers join the broadcast.
Another important consideration is client side optimizations. A streaming solution that supports codecs featuring hardware acceleration will be much more efficient at encoding and decoding. This is different from software encoding as software takes much longer to encode and decode the stream. A good example of this is the more widely supported h.264 over H.65 and VP9.
6) Good Experience
Until now, we haven’t covered one of the most important factors: the perspective of the fans. How does everything look? Can the stream be easily accessed?
A key element for building an entertainment live streaming app, is supporting a full range of resolution sizes from 4K down to 240p (and eventually up to 8K). Ideally, all the fans connecting to the event broadcast would have the best connection speed possible. That is unfortunately not realistic so you will have to take steps to make sure that your users will get the smoothest possible experience regardless of connectivity.
Three features work together to provide the best possible experience: NACK, ABR, and transcoding.
Given that WebRTC is a UDP based protocol, one of its most critical features is NACK, which is a method of resending critical packets. A bad network connection will likely result in the client dropping packets. Rather than trying to resend each and every one of the packets, NACK identifies the ones that are most important and resends those. This prevents the network from getting further clogged with redundant requests. This will help keep the stream flowing and looking good even under poor network conditions, and doesn’t have the drawbacks of packet backups in TCP based systems.
Adaptive Bitrate (ABR) allows the client to request a lower bitrate that is more appropriate to the connectivity they are experiencing at that moment. That will ensure a smooth connection despite poor connectivity. Transcoding splits the stream into a variety of quality ladders (For example: high, medium, and low) so that users who can support the highest quality can subscribe to it, while users with poorer connections can still watch.
WebRTC (already recommended for real-time latency) provides quality assurances as well. Unlike slower protocols, WebRTC responds to stream conditions in real-time. As such, REMB messages sent over RTCP (an element of WebRTC used for sending flow control messages) allow the edge node to deliver just the right size stream for every network situation.
7) Forward Thinking Platform
Future requirements are an important consideration when you build an entertainment streaming app. The platform should be built to evolve as live streaming technology changes.
A good example of this shift in technology is Flash. Once a widely used piece of technology, Flash will no longer be officially supported by the end of 2020. In regards to live streaming, HLS is next on the list for discontinuation.
Fortunately, those following the live streaming industry had already seen that Flash was eventually going to go away. They recognized that a new web-standard WebRTC codec was the best choice to replace the ultra low latency video streaming provided by Flash. This is due to the fact that it enjoys the support of Mozilla, Opera, Apple and Google, which will ensure that it will remain up to date and functional for the foreseeable future.
Ensuring that your application continues to function well into the future is just as important as making sure it works well now. Since technology is prone to change, the software used to build your application must continue to evolve. As demonstrated by Flash, fundamental changes can occur that require far in advance planning to make sure that your service remains uninterrupted.
Make sure that what ever stream8ng platform you choose is looking into or now supporting cutting-edge protocols, codecs, and general streaming technologies like SRT, LCEVC, and QICK.
8) Multi-Platform Support
Reaching your users where they are physically is just one part of the equation. The other side is supporting them how they want to watch your broadcast. With different smartphones, laptops with different browsers, as well as tablets, fans will want to watch the event numerous ways. Part of attracting a broad audience comes from fully supporting them.
The ability to run directly in all the major browsers without a plugin means that WebRTC enjoys a wide range of support. WebRTC can be run in mobile browsers as well. However, the most effective solution to ensure consistent performance in mobile devices is a mobile SDK. As VR becomes more and more viable, support for IoT devices such as headsets remains an important consideration as well.
Those with older devices that don’t support modern protocols will benefit from failover support to RTMP and HLS.
Broadcasting the streams from different devices is another consideration. Drones for example, could capture video from different angles so that they could be incorporated into the stream as discussed earlier.
When you build an entertainment live streaming app you must use customizable software. Multi-view cameras, security considerations, collecting information on viewing habits and controlling your own back-end infrastructure (more on that below) all depend upon a flexible software platform.
Also, custom features will help your application to stand out. For example, fans might access on-demand data such as integrated heartbeat monitors on players and important stats from the game. This will further enhance the live experience and increase engagement.
There is also the consideration of adding VR integrations with third party software and hardware support. Since VR is still an evolving technology, customization is very important so it can be flexible enough to adapt to future requirements. The live event streaming software must be able to handle the large amount of data that VR applications require.
Of course, depending upon your business model, there is also the consideration of ad insertions.
Being able to build your own server side logic can also be a major feature in choosing a video streaming platform. Make sure that you are able to build in those custom webhooks and tie into your own authentication. Oftentimes a provider’s API can come up short on key features you need for your application. Being able to build your own custom API can be the difference between success and failure.
Fans don’t always have the time to sit down and enjoy watching a live event. Building a recording feature lets them save a game or concert so they can watch it when they have the time. Expanding this functionality further you could add more personal recording options or in-game replays. Of course, making sure that everything lines up correctly in sequence depends upon synchronization as discussed earlier.
Recordings created by the platform should easily integrate with VOD services and CDNs and be configured to be stored on cloud storage as needed.
11) Flexible Off-Site Production
Already an ongoing trend, the shift to move workflows to a partially or fully of-site production space was further accelerated by covid. Effectively communicating with coworkers outside of a shared office depends upon two things: cloud-based hosting and latency.
Cloud-based hosting enables employees to connect together regardless of where they are physically located. Additionally, home-based production facilities save you money by eliminating the need to send out production vans or pay for costly, latency-burdened satellite uplinks.
Latency is key for ensuring natural conversation flows, being able to react to events happening in real-time and to connect with other people on the production team. By using elements such as SharedObjects, you can responsively edit a feed. Real time latency allows you to all speak at the same time as if you were in the same office so you can all be on the same page when looking at a particular frame.
Remote collaboration enables the development of interactive applications, new types of content, and extra features, most of which are already covered in this post. This is due to the fact that effective off-site production will capture the raw A/V feeds, including on-site announcers’ commentary.
Hackers and illegal streaming are a big problem for live event broadcasting. A good way to track the source of a pirated stream is forensic watermarking. Considering how important it is to respond quickly, latency plays a prominent role in the effectiveness of watermarking.
Encrypting streams is an important part of preventing hackers from gaining access. Encryption is the process by which information is encoded into a format that makes it unrecognizable until it is decoded with a key. In this way, information can be securely sent over the open internet from a client to a server. Once it reaches the server it must be decrypted in order to be rerouted to other pipelines so it can eventually be read and understood.
Stream encryption is handled by WebRTC since it automatically encrypts all streams. What about unauthorized users? Red5 Pro’s round trip authentication system enables stream authentication before gaining access to the streams. This enables fully secure access with the ability to customize how access is given to users.
Another important choice is where and how you will host your entertainment streaming platform. While choosing a convenient Platform as a Service (PaaS) solution can be an easy way to go, it can have really consequences if you end up in a service trap. As Platform as a Service (PaaS) providers make decisions, you may need to move your application onto one that better supports your needs.
Business decisions that are almost entirely outside of your control, can greatly affect your business. PaaS providers are subject to the same issues many other businesses face. Internal decisions shaped by evolving market trends or a more dramatic company buyout can result in big changes. They could drop support for a region where many fans live, or change their pricing. Apprehension over having to rebuild your entire application from scratch can lock you into working with a company that could slow down your growth.
The best option is to use a solution with a flexible API that is hosting agnostic and allows you to build your own server side application logic. This allows you the freedom to port your application over to another hosting provider. It also means that if you want to add features that you didn’t foresee early in your development, you can do so. Multi-platform support removes the risk of being permanently bound to a single provider. The aforementioned mobile SDKs should be portable as well as you don’t want to tear everything down and have to build it all up again.
14) Technical Support
Especially when using highly customizable software, even the best build platforms might need the occasional technical support. Advanced support contracts, chat channels and online ticketing systems can all be leveraged to make sure those needs are met. Without proper support, you may not be taking advantage of all the features you could add to your live entertainment app. While it may be tempting to opt for pure open source streaming solutions, make sure that these projects have the option for paid support and consulting from the creators before using them.
Covid has greatly affected our daily lives and accelerated efforts already in progress regarding live streaming. This is especially true for sports, theatre and music venues struggling to safely open up and increase their viewership. The above features will greatly aid in building a live streaming application that could effectively bring us “back” to these events and in some ways even accentuate them.
Interested in building your own live event broadcasting platform? Let us show you what Red5 Pro can do by sending an email to firstname.lastname@example.org or schedule a call.