Breakthrough Fueling Virtual Reality Medical Applications With Video Communications Has Far-Reaching Implications

A ground-breaking approach to socializing virtual reality experiences has taken shape with development of a healthcare platform supporting video communications during VR-based therapy sessions. redacted, a leading supplier of therapeutic products, is employing the solution in use cases involving treatment delivered through patients’ VR headsets. In some cases, friends and family as well as the caregiver can share a 2D 3600 view of the patient’s VR experience while communicating with each other and the patient in real time via video. In others, the video communication is between the patient and caregiver.

The ability to engage a large number of people in live video communications with someone undergoing such treatment is an important enhancement to the growing role of VR in patient care. More broadly, it has significant ramifications for enabling social interactions between VR users and people without headsets in many scenarios, including shared engagements with live-streamed sports and other events, as well as non-entertainment use cases beyond healthcare.

Embellishing VR Therapy

The Redacted VR communications platform was developed with reliance on real-time interactive video connectivity, multi-video stream mixing, authentication processes, and other capabilities provided by Red5 Pro. Leveraging the unlimited scalability of Red5 Pro’s Experience Delivery Network (XDN) platform, the solution is designed to support as many as 18,000 concurrent 60-minute VR sessions with up to ten or more participants in each.

The VR therapy sessions are conducted by healthcare professionals, who remotely transmit commands to headset controls that manage the VR experience. The sessions employ a custom-designed version of HTC’s VIVE Focus head-mounted device (HMD), which persistently streams a 2D view of the patient’s immersive experience over the XDN infrastructure to the connected therapist’s and any other participants’ smartphones or computers.

At the same time, the XDN supports the patient’s and other parties’ ability to communicate with each other via real-time audio/video streams. The remote participants appear in virtual screens stacked together within the patient’s virtual environment and in gallery arrays running adjacent to the streamed VR view on each of their devices.

Initial use cases forRedacted’s new platform involve VR-therapy offered through two solution sets. One is designed for physical rehabilitation sessions, during which the caregiver guides the patient through exercises, game playing and other interactions with the VR content. The other is a collection of immersive content offered with VR-enabled tools used in cognitive therapy, where the focus is on providing patients an uplifting escape from the confines of day-to-day living space.

Real-time connectivity supporting video communications enters the VR space at a key moment when the healthcare industry is finding ever more ways to put virtual experience to use, not only in patient therapy but also in surgery, diagnostics, basic research, and training. According to a recent study of global VR adoption performed by Grandview Research, healthcare is the fastest growing usage segment, outstripping consumer entertainment as well as all other industrial sectors.

Beyond healthcare, the significance of any platform that might be devised to link VR users and non-users through video communications is clear. Such capabilities will put an end to the social separation that has limited market acceptance of VR.

Exploiting XDN Architecture for Video Communications in VR

The multi-cloud XDN architecture makes this possible with support for streaming video-rich payloads over any distance in any direction to and from any number of users. This can be done with end-to-end latencies no greater than 200-400ms and as low as 50ms or less under certain conditions. For example, as described in this blog, latencies approach the vanishing point when 5G onramps to the cloud like Amazon’s AWS Wavelength are used with XDN infrastructure.

XDN performance parameters are achieved through automated orchestration of a software stack running on Origin, Relay, and Edge Node clusters instantiated in private or public clouds. Cross-cloud configurations supporting fully redundant fail-safe performance are enabled through pre-integrations with AWS, Microsoft Azure, Google Cloud Platform and DigitalOcean and in easily executed implementations with a dozen other cloud providers utilizing the open-source Terraform multi-cloud toolset.

XDN architecture relies on a transport foundation provided by the Real-Time Transport Protocol (RTP) to support streaming via WebRTC and the Real-Time Streaming Protocol (RTSP). XDN infrastructure can also ingest content generated via Real-Time Messaging Protocol (RTMP), Secure Reliable Transport (SRT) or MPEG-Transport Protocol (TS) for transport over RTP to reach devices tuned to those protocols that don’t work with the major browsers supporting WebRTC.

XDN technology also provides full support for the multi-profile transcodes used with ABR streaming by utilizing intelligent Edge Node interactions with client devices to deliver content in the profiles appropriate to each user. Redacted has made use of this capability for all streaming on its platform. As demonstrated in these and other aspects of the engagement with Redacted, the flexibility intrinsic to XDN architecture and Red5 Pro’s developmental expertise bodes well for other efforts in the VR Communications space. Additional instances of how the flexibility of XDN architecture and SDK tools enables capabilities essential to video communications as reflected in the Redacted platform include:

• Development of messaging delivered through the XDN – These purpose-built messages support:
  • session setup and delivery of virtual room codes guiding participants to the shared VR experience,
  • HMD control commands generated remotely by caregivers and participants,
  • distribution of shared HMD data related to motion sensors and controls.
• Session security restricting access to authorized participants – Red5 Pro’s Round Trip Authentication process accepts or rejects clients without knowing the details of the security policies implemented in the authentication server.
• Bandwidth-saving consolidation of multiple participant streams into a single stream
  • XDN architecture enables the mixing of up to hundreds of source streams into a single customized composite stream.
• Use of Red5 Pro’s Mixer technology to create “mix-minus audio” streams specific to each participant – This eliminates the audio loopback effect that occurs when each user’s audio input is picked up by their mics and fed back into the conference stream.

Much has been written of late about the role VR will play in what many are referring to as the Metaverse era, when remotely dispersed people in virtual and real-world environments are able to seamlessly interact as a routine aspect of activities at work, at home and on the go. As with much else pertaining to the emerging Metaverse, XDN infrastructure is demonstrating that the integration of video communications connecting those within and outside the VR experience is now a reality.

For a look at other ways interactive real-time video connectivity is bringing the Metaverse to life with support for VR and other extended reality (XR) modes, see this blog. To learn more about XDN technology, contact info@red5.net or schedule a call.

* The provider has requested anonymity to protect the competitive advantage it has gained through use of XDN technology to enable video communications with administration of VR-enhanced therapy.