How to Perfect 5G In-Venue Experiences with Real-Time Precision

in-venue experiences
SHARE

As 5G service providers announce headline-generating commitments to transform what people experience when they attend events at stadiums and concert halls an unresolved challenge to those efforts has gone unnoticed. Simply put, 5G alone can’t reliably deliver in-venue experiences with perfectly synchronized live and on-screen content. And while some mobile network operators (MNOs) may not… Continue reading How to Perfect 5G In-Venue Experiences with Real-Time Precision

As 5G service providers announce headline-generating commitments to transform what people experience when they attend events at stadiums and concert halls an unresolved challenge to those efforts has gone unnoticed. Simply put, 5G alone can’t reliably deliver in-venue experiences with perfectly synchronized live and on-screen content. And while some mobile network operators (MNOs) may not have reached the point in service development where this becomes a concern, others have.

A lot is at stake. MNOs are spending billions to saturate venues with 5G coverage. Wherever they are on the development curve, most are committed to offering benefits like multiscreen zoom-in viewing, individually controlled instant replay, action-relevant data feeds, venue-wide views of bathroom and food lines, and surveillance camera feeds showing traffic status and parking availability.

If it turns out those experiences are less than satisfactory, anticipated returns on those investments could be out of reach. That’s why eliminating barriers to real-time precision is essential to generating the levels of enthusiasm among consumers that will ensure long-term success in the competition for venue and producer licensing deals.

Much is made of the fact that this new era in live venue experiences would be impossible without 5G’s much lower latency profile compared to previous generations of mobile. And that’s true. After all, in terms of the time consumed by the radio access network (RAN) as it executes conversions between transport payloads and radio signals, 5G can cut latency from the multi-second delays experienced with LTE and earlier generations to as little as 1ms.

But that doesn’t address the latencies incurred over IP network links between the 5G picocells dotting a venue and the servers transcoding and streaming the live productions. Nor does it address the fact that orchestrating and transcoding the output of those productions, often involving multiple camera feeds and graphic streams, can add untenable levels of latency to the distribution process.

Delivering 5G Experiences at the Speed of Sound

To be clear, this is primarily a speed-of-sound as opposed to a speed-of-light-related issue. That’s because the speed of sound is the key to a user’s perception of whether the onscreen experience is perfectly synchronized with the venue experience.

While there’s no way to deliver the 5G streams visually at the speed of light, the human brain can’t perceive minor visual latency. As long as there’s no more than roughly 200ms latency between onscreen reception and what’s seen from the stage or field, it feels seamless. But the echo effects with out-of-synch sound are discernable and can be irritating at a much lower latency threshold.

Given that MNOs want to turn heads with their commitments to in-venue 5G services, you won’t find much in the way of public discussion about their need for the ultimate in synchronized off- and on-screen experience. But rest assured, MNOs who aspire to be at the pinnacle of best-of-class services are on the hunt for that ideal experience and some are discovering they can meet the challenge. The solution lies with real-time synchronicity of onsite and onscreen experience via a real-time interactive streaming (RTIS) infrastructure that leverages the speeds enabled by a WebRTC-based platform.

As the provider of a widely-deployed platform that is demonstrating these goals can be met, we at Red5 Pro are well aware of the demand. And because it’s in reach, it won’t be long before at least some MNOs are normalizing a level of in-venue experience that will forever transform ticket buyers’ expectations. And, eventually, they are likely to go even further with pervasive area-wide uses of RTIS infrastructure with their in-venue and other 5G services.

The Worldwide MNO Pursuit of In-Venue 5G Service Opportunities

Before exploring how close we are to seeing these capabilities come to life on deployments of Red5 Pro’s Experience Delivery Network (XDN) platform, it’s well worth considering what’s already in play as carriers put 5G to in-venue use across the globe. These efforts offer a pretty good idea of what we can expect from carriers who leverage RTIS to deliver the ultimate in live event user experiences.

In the U.S., the three major carriers are seizing the 5G in-venue opportunity with varying degrees of commitment to advanced features. So far, Verizon has made the biggest splash publicly with announcements of long-term in-stadium 5G service deals with the NFL and NHL as well as deployments related to recent Indianapolis 500 races, the 2022 Formula 1 Miami Grand Prix, and concerts at venues managed by Live Nation.

According to the Groupe Speciale Mobile Association (GSMA), Verizon has deployed 5G millimeter wave networks in 25 NFL stadiums and 35 other sports and concert venues nationwide. The latest display of Verizon’s NFL-related offerings occurred at the recent Super Bowl in Phoenix, where the seven-stream Multi-View service now offered at most of its 5G venues enabled users to shift among seven camera angles with both live action and personally activated replays.

There was an augmented-reality (AR) element as well, enabling users to create personalized displays of player statistics and selfie images merged with live stadium footage. For example, in the latter case, people could capture a selfie appearing virtually on the jumbotron and post the picture to social media. They could also participate in a game during the second quarter that involved moving their phones to virtually toss a football into a virtual pickup truck parked in the middle of the field.

Verizon reports positive results from such efforts as people grow more accustomed to making use of the new capabilities. The carrier noted that while 60% of attendees at both the 2023 and 2022 Super Bowls were Verizon customers, game-day data usage spiked by 57% this year compared to last.

AT&T and T-Mobile, too, have capitalized on much denser 5G deployments at stadiums and other entertainment venues across the U.S. over the past year. But, with occasional exceptions, they’ve chosen to rely on better coverage as the driver of audience engagement as opposed to offering venue-related feature enhancements. AT&T, for example, reported that a major upgrade to its 5G presence at the 2023 Super Bowl venue in Phoenix led to an 81% surge in data usage compared to usage at the 2022 Super Bowl in Los Angeles.

But both carriers make clear they have development initiatives underway that are likely to spawn more in-venue experiences and special features over time. So far, AT&T has limited its in-venue 5G enhancements to features that don’t involve fan engagement with live action at the main event.  Many of these, like the options available during the days leading up to the 2023 College Football Playoff, involve access to special concerts or other side shows to the main event. And some are AR-related, as in the case of the 2023 NBA Allstar game in Salt Lake City, where fans could participate as avatars in an interactive AR game featuring two NBA stars.

T-Mobile has populated many venues with its 5G small cells but has seldom publicized venue-specific apps. Instead, like AT&T, it publicizes the fact that its in-venue deployments provide people with market-high levels of throughput, which mitigates congestion resulting from the massive amount of content users typically generate while attending live events.

One example of an occasional venue-specific set of features offered by T-Mobile was the 3D panoramic rendering of MLB’s pre-Allstar game Home Run Derby, which allowed users to track a ball’s path and view stats relaying its velocity, distance, and hang time. T-Mobile is seeking to inspire greater developer engagement in app creation, which may lead to more in-venue apps. For example, one developer has created an audio app T-Mobile subscribers can use with their earbuds and headphones to tune into a venue’s soundboard to get the pure sound musicians hear.

Outside the U.S. there are many examples of what’s still a nascent movement toward in-venue 5G experiences that will require tight synchronization with live action. In Canada, for example, Rogers Communications has embarked on an initiative leveraging new 5G spectrum capacity in partnership with AWS, Ericsson, software developer Immersiv.io, and venue operator Maple Leaf Sports & Entertainment Ltd. to implement AR-based immersive viewing at hockey games.

One of the early moves to commercial offerings of enhanced in-venue 5G experiences in Europe can be found in Germany, where Vodafone reached an agreement with Bundesliga, Germany’s largest soccer league, to establish advanced 5G infrastructure in all the stadiums used by the league. This has allowed Vodafone to offer a multi-view and replay app developed for Sky Sports customers wherever those installations are complete.

France-based Orange is another European telecom with a transnational footprint pursuing the development of advanced 5G in-venue user experiences. The company is using Marseille’s Orange Vélodrome soccer stadium as a testbed for the development of apps on the advanced 5G millimeter-wave infrastructure it has installed there. One of several publicized apps in trial there employs AR technology in conjunction with AWS edge computing and other partners’ technologies to superimpose game and player data on AR users’ fields of vision.

Similarly, U.K. MNO Three has contracted with the Premier League’s Chelsea club and Stamford Bridge stadium in London to implement 5G infrastructure that will serve as a testbed for app development. Here, too, a mix of multi-viewing, replay, and real-time displays of stats with AR support is said to be in the offing.

One of the most advanced uses of 5G in live events, albeit not tied to a specific venue, occurred at the 2022 Tour de France cycling competition. This was another Orange project billed as an experiment but offered to all 5G subscribers in villages and other designated locations along portions of the race route. The app supported AR projections of a 3D map of the race with data on competition in each of the 21 race stages and used standalone holographic technology to present a volumetric hologram of race director Christian Prudhomme speaking directly to users.

As all of these commercial and experimental initiatives attest, there’s broad consensus on the audience-drawing power of 5G-delivered in-venue experiences. Going forward. the challenge will be to take whatever steps are necessary to ensure the real-time synchronization of onscreen experiences with what’s happening on stages and playing fields.

The Potent Combination of 5G NG and XDN Technologies

On the 5G side of the ledger capabilities built into the 5G NG standard that are essential to the full realization of the potential include:

  • Network slicing, which logically defines a private network with the allocation of bandwidth, resilience, and latency that operates independently of the wider public network.
  • Distributed RAN infrastructure with high cell density, which divides the functions of next-generation 5G node base stations (gNBs) between central baseband units (CUs) and what can amount to 200 or more distributed units (DUs) or small cells at a large venue.
  • Ultra-Reliable and Low Latency Communications (URLLC), which is the mechanism that cuts RAN processing latency to 1 ms.

In-venue service providers are also well advised to take advantage of multi-access edge computing (MEC) solutions like AWS’s Wavelength and Microsoft’s Azure public MEC. These are small-footprint extensions of the core compute platforms that help to reduce latency by localizing compute and storage services within a carrier’s network edge facilities.

Carriers utilizing Wavelength Zones include Verizon, KDDI, SK Telecom, Vodafone, and Bell Canada. Azure names AT&T as its mobile partner on its MEC. Verizon has the largest MEC footprint with Wavelength Zones in 13 U.S. cities.

But, as mentioned at the outset, even with the implementation of all these measures, carriers seeking to deliver the ultimate in application synchronization with live events require a means of reducing latencies incurred on links between production processes and the 5G facilities. This is the role that can be played by Red5 Pro’s XDN platform in deployments that are optimized to eliminate any perceptual latency no matter how complex the applications running on attendees’ smartphones might be.

The ability to satisfy this requirement begins with the basic performance parameters of XDN architecture, which, like many RTIS platforms leverages WebRTC but at far greater performance levels. In use cases ranging across consumer, enterprise, institutional, and government markets, XDN infrastructure is supporting frame-accurate synchronization of all video feeds with no limits on the scalability of interactive video communications that can traverse any distance at sub-400ms latencies over one or more cloud platforms.  

XDN also leverages the Real-Time Transport Protocol (RTP) underlying WebRTC to enable the delivery of content packaged in other distribution profiles, including Real-Time Streaming Protocol (RTSP), Real-Time Messaging Protocol (RTMP), Secure Reliable Transport (SRT), and MPEG-TS. But WebRTC is the most commonly used format by virtue of its support in all the major browsers, which obviates the need for plug-in client software.

XDN supports transcoding and delivery of content streams in multiple bitrate profiles common to HTTP-based adaptive bitrate technology by intelligently distributing stream segments to each user at bitrates best suited to local conditions. And the platform automatically switches to one-way higher latency HTTP Live Streaming (HLS) in the rare instance where a receiving device is incompatible with any of the other supported formats.

XDN infrastructure is built on a hierarchically deployed server software stack consisting of one or more Origin Nodes, where content is ingested and can be transcoded if necessary; intelligent Edge Nodes that deliver live unicast streams to assigned areas of end users, and, in large-scale deployments, Relay Nodes that serve to connect Origin Nodes with arrays of Edge Nodes. The real-time streams from core sources and end users are orchestrated for distribution across the node clusters with fully redundant fail-safe performance.

XDN nodes can be deployed in public cloud or private cloud facilities anywhere in the world. By leveraging both containerized and virtual machine-based iterations of data center virtualization, the XDN platform enables the flexibility and speed of resource utilization that is essential to seamless cross-cloud operations and scalability.

All of these capabilities are brought to bear with XDN support for 5G in-venue streaming operations in conjunction with additional measures that leverage the localized distribution environment to cut latency to the levels essential to ideal performance. Here the end-to-end latency must be no greater than 40ms-60ms, which roughly matches the range of speed-of-sound lag times between a speaker and an end user within anywhere from 60 to 120 feet of the nearest venue speaker.

One of the factors enabling Red5 Pro to meet these stringent requirements has to do with the approach to node clustering employed with XDN architecture. This enables configurations of multiple interconnected Origin and Edge Nodes that provide more than one endpoint for ingesting publisher- and user-generated content and ensure the shortest path to distribution to end users is always taken.

In a crowded stadium, latency-reducing cluster orchestration may involve dozens of XDN nodes. No matter how many streams comprise the live 5G in-venue experience, they all reach every end user at the same time.

Another major ingredient to the latency-reduction process has to do with our ability to instantiate XDN Nodes in MEC coverage areas. In the case of the Wavelength MEC environment, this can be done immediately insofar as the XDN platform is the only RTIS infrastructure that AWS has authorized for pre-integration with Wavelength. Pre-integrations with other MECs are likely, but, for now, integration must be performed manually.

Finally, to ensure even further reductions in latency, Red5 Pro is enabling the use of hardware acceleration with XDN encoding resources that can be leveraged in the live production playout process. These capabilities, which have been demonstrated with the use of the AMD Supernova chipset at recent trade shows, reduce latency by about 8ms to achieve 2ms latency in the encoding process.

———————————————————————————————-

The net result of all these measures is to achieve a reduction of in-venue 5G application latencies to the sub-40ms level. This meets the speed of sound criterion for eliminating any gap between what an end user positioned as close as 60 feet from a venue speaker hears from that source and the audio generated on the smartphone.

The stage is being set for an explosion of next-generation 5G in-venue services that will transform consumers’ expectations of what they’re getting with their ticket purchases. To learn more about what’s in store contact info@red5.net or schedule a call.