How to Unleash GStreamer for Live Streaming in Real Time

The emergence of real-time interactive streaming has created a field of dreams for developers who can innovate with the same alacrity they’ve come to expect in the traditional streaming environment.

For many if not most developers, whether the fast track to commercialization will remain open in the real-time streaming era hinges on the answer to a simple question. Will they be able to continue exploiting the open-source GStreamer platform with the flexibility, speed, and scalability that has made it the go-to multimedia framework for innovation over the past few years?

GStreamer’s open-source framework provides vendors a common approach to offering commercial solutions as plugins that developers can bring into their service pipelines with a few mouse clicks. Plugin elements relating to operating systems, codecs, streaming modes, editors, filters, feature functionalities, descriptive metadata, and much else can be mixed and matched to support just about any multimedia application at any point in the distribution chain, from video capture to device playout.

But there’s a serious challenge when it comes to live-streamed content. The elements guiding encoding, packaging, categorization, rendering, and other aspects of content operations on a per-stream basis must perform their tasks in precise frame-by-frame synchronization with the primary payload no matter how many users are involved.

Efforts over the years that resulted in tight synchronization of elements used with payloads live-streamed over HTTP-based content delivery networks (CDNs) have been instrumental to expanding GStreamer adoption rates. The question now is whether the same will be true with real-time streaming platforms, which don’t rely on HTTP and where there’s no room for adding latency on streams traversing the GStreamer pipelines.

The XDN Path to Using GStreamer for Live Streaming in Real Time

There’s no doubt this can be done when it comes to activating GStreamer plugins with real-time streaming over Red5 Pro’s Experience Delivery Network (XDN) platform. Red5 Pro has taken multiple steps to ensure GStreamer pipelines designed for real-time use cases can operate on XDN infrastructure at any scale with tight synchronization across all elements.

This is vital not only to content flows targeted to end users. It’s also crucial to enabling plugin and applications developers to collaborate among themselves and with suppliers of core processors in order to fashion rapid responses to soaring demand for real-time solutions.

In all cases, GStreamer plugin operations benefit from the same mechanisms that ensure real-time synchronization of everything else that’s delivered on the XDN platform. Any number of users from a handful to millions can be simultaneously engaged with complex component-synchronized streams registering end-to-end latencies no greater than 200ms-400ms over any distance in any direction.

In some situations, latencies are reduced even farther, as when XDN infrastructure is instantiated in conjunction with AWS Wavelength facilities positioned with mobile carriers’ 5G onramps. See this blog for a description of the unique use of real-time streaming facilities with AWS Wavelength resulting from a new partnership between AWS and Red5 Pro.

Beyond these baseline contributions to effective use of GStreamer in live-streaming scenarios, Red5 Pro is working with others in the GStreamer space to perform integrations and customize XDN processes for specific types of collaborative development projects. Presently, for example, there’s a spate of activity in the accelerated hardware domain where CPU/GPU suppliers are working with developers to formulate the best approaches to utilizing hardware acceleration, often in conjunction with hardware prototypes that are still undergoing refinements based on market needs.

Remote collaboration in GStreamer development is also driven by the same requirements impacting all industries in the wake of the Covid 19 outbreak. Developers who were accustomed to working together in the same locations need to be able to work at the same level of productivity from afar.

Whatever type of collaboration might be in play, XDN connectivity ensures everyone is participating in real time. We’ll take a deeper look at how the XDN platform facilitates the use of GStreamer in real-time streaming scenarios farther down. But, first, it might be useful to consider what the use of GStreamer in the real-time streaming environment means to the development community.

A Versatile Environment for Real-Time Application Development

As in traditional streaming, developers focused on real-time streaming can benefit from the fact that GStreamer makes it much easier to evolve use cases by transitioning from simple to complex pipeline workflows through a consistent, standardized approach to tapping plugins for elements essential to any given task. By one count, there are 230 plugins cataloged in over 30 GStreamer libraries offering in excess of 1,600 elements supporting a vast range of functionalities.

The framework works with all major operating systems, including Linux, Windows, Android, Max OS X, iOS, and those based on Unix-derived platforms like Solaris and the Berkeley Software Distribution systems (BSDs). And it runs on 32- and 64-bit iterations of the leading hardware architectures, including x86, ARM, MIPS, SPARC, and PowerPC.

These attributes have provided developers with a powerful environment for keeping pace with evolving needs in the one-way streaming market. The benefits will be even more important to developers as the nuances bearing on the formats, range of devices, feature enhancements, user data analysis and much else grow more complex in the real-time interactive streaming era.

An Exploding Field of Opportunities for Developers

As described in this white paper and multiple blogs, the emerging generation of real-time video use cases extends across all segments of society.

In the consumer domain, they include the synchronization of shared real-time experiences with live-streamed sports, esports, and other events; operations of e-commerce in all its permutations, and the vast realm of experiences tied to social networking, multiplayer gaming, and virtual and other versions of extended reality. An even greater variety of opportunities for developers can be found in the myriad ways enterprises, municipalities, medical professionals, educational institutions, the military, and other sectors are exploiting interactive video streamed in real time.

The ability to leverage GStreamer in this environment is fundamental to everyone’s success.

A good example of both the nuanced variability and the precise synchronicity developers must accommodate can be found in applications that rely on the use of time stamping with GPS readings. An interactive real-time video streaming infrastructure with the scalability and performance parameters of the XDN platform can support frame-by-frame GPS accuracy in any use case that benefits from associating user location with interactive video engagement.

Another real-time usage scenario for GStreamer has to do with the aforementioned collaboration between hardware suppliers and developers in the creation and validation of approaches to exploiting next-generation microprocessor architectures. One such endeavor involves GStreamer innovator Fluendo’s work with Intel in efforts to facilitate developers’ use of the chipmaker’s Deep Link hardware-acceleration platform.

As described by Intel, Deep Link technology uses dynamic power sharing and AI to harness CPU and GPU resources in devices to accelerate content creation, gaming, and productivity. Fluendo is developing GStreamer components to take advantage of Deep Link to enhance decoding performance in set-top boxes and other devices.

AI is a major force behind many other GStreamer development activities in the real-time streaming domain. Indeed, considering the pervasive presence of AI across all realms of consumer, business, government, medical and scientific activity, this is where the developer field of dreams approaches infinity.

A good illustration of how GStreamer is facilitating AI usage can be found in NVIDIA’s DeepStream SDK. Using an extensible standard structure for metadata, DeepStream enables the application of AI in GStreamer pipelines through batch-level and user-specific metadata to generate business insights, improve processing efficiencies, optimize edge-to-cloud connectivity, and execute the many other Intelligent Video Analytics (IVA) tasks in the NVIDIA plugin portfolio.

Through real-time connectivity, the SDK can bring multiple streams from different sources into play, NVIDIA notes. “Whether it’s at a traffic intersection to reduce vehicle congestion, health and safety monitoring at hospitals, surveying retail aisles for better customer satisfaction, sports analytics or at a manufacturing facility to detect component defects, every application demands reliable, real-time IVA,” the company says.

Addressing the Multi-Protocol Real-Time Streaming Imperative

In such scenarios, the streams delivered from video cameras need to reach all targeted endpoints, be they machines or people, with no increase in latency. A large share of live-streamed video is delivered in real time from streamers embedded in or attached to cameras via either the Real-Time Streaming Protocol (RTSP) or Secure Reliable Transport (SRT), neither of which is supported in the major web browsers. Consequently, use cases are better served when the streams can be delivered to their targets over WebRTC, the real-time streaming platform that achieves broader reach by virtue of its support in all the major browsers.

Automatic, latency-free transition of RTSP and SRT streams to transport over WebRTC is one of the important roles played by XDN architecture in opening the world of real-time video streaming applications to GStreamer developers. Developers can utilize APIs in the GStreamer framework to manually set up the conversion of RTSP streams to WebRTC streams, but, as evidenced in this complicated how-to set of instructions provided by Google, this is a difficult, time-consuming process.

Moreover, developers pursuing this course would not benefit from the additional aspects of multi-protocol flexibility that come with reliance on XDN architecture. This flexibility stems from the fact that the platform uses the Real-Time Transport Protocol (RTP), which is the transport core for both WebRTC and RTSP, as the foundation for all streams delivered over XDN infrastructure.

By retaining the RTSP and SRT formats, the XDN architecture ensures that the content is received by targeted clients that support the stream-originating protocol but not WebRTC. The same principle applies to video ingested from sources that employ Real-Time Messaging Protocol (RTMP) and MPEG-TS (Transport Protocol).

In all cases, the automatic transition to WebRTC ensures the broadest possible reach through receptivity by all targeted devices that are compatible with at least one of the major browsers as well as devices without that browser support that are attuned to a given protocol used by the live-stream source. Moreover, in the rare instances when a targeted device doesn’t work with WebRTC or the originating protocol, the XDN platform streams the content over HTTP Live Streaming (HLS), which negates the real-time latency and multi-directional benefits but at least ensures no device is left out of the flow.

Beyond this multi-protocol flexibility, multi-cloud scalability and persistent performance are hallmarks of XDN architecture that are essential to any real-time streaming applications executed on GStreamer pipelines. Hierarchies of automatically orchestrated XDN Origin, Relay, and Edge Nodes can be implemented in cross-cloud configurations by leveraging pre-integrations with major suppliers like AWS, Google Cloud, Microsoft Azure, and DigitalOcean, or through integrations with many other cloud platforms enabled by Red5 Pro’s use of the Terraform multi-cloud toolset

The XDN Stream Manager works in real time to orchestrate set-up configurations, load balancing, and traffic routing with fail-safe redundancy across all cloud instantiations. Automated scaling mechanisms enable real-time responses to fluctuations in traffic demand or the need to add new broadcasters and end users.

XDN architecture also provides full support for the multi-profile transcodes used with ABR streaming without imposing the latencies and other impediments incurred with HTTP-based streaming. It does this by utilizing intelligent Edge Node interactions with client devices to deliver content in the profiles appropriate to each user. In addition, as described in this blog, Red5 Pro is leveraging Edge Node intelligence to facilitate next-generation gaming, XR, internet of things (IoT), and other applications that depend on CPU/GPU processing at the deepest edge locations to achieve maximum reductions in latency.

Customized Approaches to Optimizing XDN Support for GStreamer Development

Adding to the benefits to GStreamer development stemming from these intrinsic capabilities, XDN architecture lends itself to customizations that can provide additional support for specific GStreamer projects. Notably, plugins associated with a given GStreamer pipeline can include elements that direct the use of XDN intelligence, servers, and other components to help execute applications.

By leveraging GStreamer APIs designed to support the use of JavaScript-based elements available in GStreamer Java libraries, Red5 Pro and its partners can formulate XDN functions utilizing a Red5 Pro service that employs WebSockets to receive and act on data tags within live streams. With WebSocket support for connecting client- and server-side Java code in XDN operations, partners can create Java-based applications in GStreamer that provide business logic for real-time communications, data gathering, edge-based analytics, and much else. The XDN utilizes the WebRTC datachannel to transmit all GStreamer-related communications.

Such capabilities have important implications for CPU/GPU suppliers who want to work with large communities of GStreamer developers to optimize the use of hardware for specific applications such as multiplayer gaming. Red5 Pro will be bringing some of these efforts to light in the months ahead.

——————————————————

Clearly, GStreamer will be playing an ever-larger role in bringing the new era of real-time video streaming to life. The fact that XDN architecture supports tightly synchronized real-time streaming over multiple seamlessly interconnected protocols and clouds opens a realm of virtually unlimited possibilities in the use of GStreamer to embellish real-time video services and applications.

To learn more about the use of XDN infrastructure in the development and delivery of applications over GStreamer pipelines, contact info@red5.net or schedule a call.