Target, Capture, and Act Faster with Live-Streaming Video Surveillance

INTRODUCTION

Video surveillance is undergoing a major transformation in multiple market segments worldwide as system operators discover they now have the ability to leverage real-time visibility into unfolding events across multiple points of observation.

Faster, smarter reactions to developments become the norm when it’s possible to analyze, often with the help of AI-assisted intelligence, tightly synchronized high-resolution camera feeds aggregated in real time from all locations relevant to momentary dynamics. This can be done whether cameras are positioned terrestrially, off shore, under water, on drones, or in smartphones carried by field personnel. In fact, real-time streaming is even coming into play with use of video surveillance in space missions.

Demand for these capabilities “is growing really rapidly,” according to David St. Claire, project manager at FLIR Systems. St. Claire has a broad view of the global surveillance market stemming from FLIR’s role as a leading supplier of advanced thermal (infrared) and other imaging techniques used with AI-assisted analytics to support aerial and terrestrial surveillance applications in public safety, transportation, smart city operations, military, industrial, and consumer-facing scenarios.

“Managing widespread environmental catastrophes as well as routine traffic problems, all in real time is critical,” St. Claire noted during the virtual XDN Summit in early 2021. “First responders on location can stream in real time to command centers. We never could do that before.”

Indeed, such capabilities are out of reach when traditionally streamed camera feeds are delivered out of sync with seconds of lag time behind what’s happening in real time. In those situations, even with the assistance of advanced analytics, operators must await extraction and pooling of critical tidbits of information from individual video streams to formulate what amounts to an after-the-fact view of developments.

These delays can be a matter of life and death during fires, floods, energy outages, hurricanes, tornadoes, robberies, mass shootings, military operations and all the other crises first responders have to deal with. And delays in full-perspective visual assessments of malfunctions in factory production, oil drilling, space flight, and any other situation where complex industrial operations are in play add significant costs to damage control.

There’s no longer any reason to view these impediments as intrinsic to use of video surveillance. In growing numbers, developers of intelligent surveillance platforms and their customers are making crisis management far more effective through reliance on real-time streaming capabilities that meet the performance standards set by the Experience Delivery Network (XDN) technology developed by Red5 Pro.

XDN instantiations work as well in many-to-one or many-to-many surveillance streaming scenarios as they do in the one-to-many and interactive real-time streaming applications now taking hold across the internet. As described in the white paper “The World Needs an Interactive Real-Time Streaming Infrastructure,” those uses of XDN technology are becoming ever more common in entertainment, social networking, game playing, gambling, ecommerce, telemedicine, business collaboration, worker training, and much else.

When it comes to video surveillance, “one of the interesting things about XDN technology is its many-to-one capabilities,” said Sam Recine, vice president of Americas and Asia Pacific sales for the AV/IT Group at Matrox Video. As a supplier of microprocessors, hardware appliances, and software in multiple fields, including surveillance, Matrox is witnessing “heavy investment in what you can do in real-time situations,” Recine said.

One dramatic case in point involves a military application where Matrox equipment encodes multiple real-time streamed video and data feeds from drones into a single stream, which is then delivered in real time to distant monitoring locations. “We’re really pushing the envelope with ultra-low latency delivery of high-fidelity video transparently combined with data,” he noted.

Whatever the ratio of sources-to-receivers might be at any moment and regardless of the number of end points or where they are, XDN infrastructure streams video, audio, and ancillary metadata and other content with end-to-end latencies registering at 200 to 400 milliseconds over global distances. Latencies are even lower in localized distribution scenarios common to surveillance applications.

The fact that this is done with open-source technology is especially meaningful, St. Claire noted. “We’re finally in a position with XDN technology that allows us to focus on open, not proprietary protocols, which is really big for our industry,” he said.

XDN infrastructure can be used to stream the content end to end via the open-source Real-Time Streaming Protocol (RTSP) employed with high-resolution digital surveillance cameras and smartphones. Or, in instances where end points aren’t equipped with RTSP-compatible players, the XDN converts the streams for transport via WebRTC, which is supported for playback by all the major browsers, obviating the need for plug-ins. The platform also seamlessly interoperates with video streamed by cameras that use the Secure Reliable Transport (SRT) protocol.

In all cases, another great benefit that comes with the use of XDN technology in surveillance operations is built-in support for robust protection of streamed content. RTSP, SRT and WebRTC mandate DRM-caliber security through AES encryption and key management as defined by the Secure Real-time Transport Protocol (SRTP).

As St. Claire observed, this is key to expediting operational set-ups. “With XDN technology we are able to transmit highly encrypted video in real time, which gives us an opportunity to use our technology securely in emergency operations that require instant management,” he said.

The ensuing discussion begins with a review of the trends that underlie the surging use of video surveillance worldwide (exclusive of nefarious applications employed by autocratic governments), including the trends in multiple market sectors that call for the real-time capabilities enabled by an XDN infrastructure. This is followed by a deeper look at XDN technology and a concluding segment describing commercial implementations in video surveillance.

As shall be seen, these applications of XDN technology involve everything from public safety and emergency management to operations in business, the military, border protection, and space transport.

TRENDS DRIVING EMERGENCE OF REAL-TIME VIDEO SURVEILLANCE

Recent research leaves no doubt that use of video surveillance is spreading globally at an unprecedented rate. By the calculations of Emergen Research, $86.33 billion will be spent globally on video surveillance technology and services in 2027, representing a compound annual growth rate (CAGR) of 9.6% over the $41.26 billion registered in 2019.

In a five-year forecast, BIS Research forecasts spending will grow at a 10.06% CAGR, from $31.8 billion in 2020 to $51.36 billion in 2025. Allied Market Research puts the growth rate even higher, predicting a 14.6% CAGR will lead to a $144.85 billion spending total in 2027.

The Role of AI-Assisted Analytics

There’s also general agreement among researchers that AI-assisted analytics will play an ever larger role in video surveillance. For example, Memoori, a specialist in smart infrastructure research, predicts spending on AI in surveillance intelligence, starting near zero in 2017, could reach $3 billion by 2023.

The transition to intelligent video surveillance rests on a massive market shift from reliance on analog closed-circuit television (CCTV) cameras to IP-based digital cameras and digitized networking. High-resolution digital cameras equipped to stream live video to monitoring stations near and far provide the image clarity with support for digital zooming and frame-rate adjustments that have made it possible to execute the advances in AI computer vision essential to automated analysis.

There’s been a lot of publicity—and controversy—around ever improving attempts at AI-assisted face recognition, but the truth is, AI is contributing many other types of information as well for automated analysis at speeds and depth that would be impossible manually. Applications like object recognition, event detection, motion estimation, identification of behavioral and audio anomalies, and image restoration are feeding analytics engines with data that can be put to use in real time.

Such processes have become a normalized feature of mainstream surveillance systems, many of which run the analytics components of their products in the cloud. “The traditional video surveillance companies naturally see Deep Learning and Artificial Intelligence as a feature to run off their existing products, offering a way to both differentiate themselves and add further value to their products,” Memoori reports.

Early successes with intelligent video surveillance have had a major impact on market expectations, according to Allied Research. The previously referenced report cites dramatic increases in demand for sophisticated systems that can monitor public places, borders, ports, transportation infrastructure, buildings, factories, agricultural operations, and much else.

Use Cases Calling for Better Approaches to Surveillance

Yet, for all that’s been accomplished, the use of intelligent video surveillance during live events has only scratched the surface of what can be achieved when human monitors and their analytics platforms can react to multiple video feeds streamed in real time. As discussed in the introduction and explained at greater length below, such capabilities are now possible when Red5 Pro’s experience delivery network technology is used to ingest an aggregation of high-resolution video streams for simultaneous delivery to internet-connected monitoring posts wherever they may be in less than half a second.

This can’t be done when traditional HTTP-based streaming technology is employed. While many CDN operators have adopted the latest advances in HTTP latency reduction, they can’t avoid the multi-second delays that are inimical to real-time surveillance operations. Nor can CDNs scale ingest points to accommodate the dynamic coverage requirements that often arise in video surveillance operations.

The need for the type of support provided by XDN technology is evident across multiple market segments, including:

● Law Enforcement
Police need to be able to respond as fast and as knowledgeably as possible to active shooter and other emergencies. Guidance from surveillance administrators tracking wrongdoers in real time can save lives, especially when they can rely on advanced analytics to derive useful information faster from all the relevant points of video coverage.

In an advisory addressing the need for faster response times in active school shootings, the Department of Homeland Security notes that “the average duration of active shooter incidents in institutions of higher education within the United States is 12.5 minutes. In contrast, the average response time of campus and local law enforcement from the beginning of the incident to the scene is 18 minutes. This stark contrast between response requirements and response capability produces a considerable delta of dead, injured or potential victims.”

In a real-time surveillance environment, 911 dispatchers and action coordinators can immediately match the caller’s location with the nearest camera feeds to provide a comprehensive perspective on the perpetrator’s whereabouts. In cases where video content analytics (VCA) is in play, guidance can be enhanced through the automated perusal of multiple live video streams to provide more information than officials can amass manually.

For example, in the event of vehicular flight from a crime scene, the video feeds along multiple routes and from hovering drones can be instantly searched and analyzed to pinpoint a license plate or a vehicle of a certain make or color. When drones are in play, once the fleeing vehicle is found, they can be assigned to track it wherever it goes.

● Public Safety and Traffic Management
Real-time surveillance will contribute greatly to municipalities’ ability to track accidents, crowd behavior, traffic patterns, and other activities in public spaces. Immediate responses to violations of social distancing and other preventive measures in the COVID-19 era have become an especially significant facet of live video surveillance.

With the application of VCA across multiple live feeds, the smart city vision comes to life. For example, leveraging actionable video intelligence, cities can dynamically update traffic light schedules in response to traffic flows, manage use of high-occupancy vehicle (HOV) lanes, and deliver more timely messaging through digital highway signs. VCA identification of anomalous events as they crop up across a city’s perusal of live camera feeds can trigger preventive action against incipient fires, rowdy behavior, robberies, and other emergencies.

● Emergency Response, Border Patrol, and Military Operations
Real-time intelligent video surveillance becomes more difficult with the volatility and size of target areas in these scenarios. Drones are a big part of the solution.

They are playing major roles in efforts to combat fires, floods, and other emergencies, providing coverage outside the range of fixed cameras and, in the case of smoke-choked air space, outside the range of helicopters and airplanes. And reliance on real-time responses to drone-delivered video has become intrinsic to military and border patrol operations worldwide.

But drones raise the bar in terms of the precision that must be applied with their video output. Synchronous aggregation of those feeds is essential to avoid errors in situational analysis that can occur even with minute distance-related time disparities in ingestion by distribution servers.

● Business Operations
Video surveillance in business has moved beyond security to uses in myriad other applications where real-time visibility will play an ever-greater role. Where security is concerned, real-time visibility with heavy video coverage indoors and out will not only help catch bad actors but also contribute to deterrence, especially if workers and others understand activities are being viewed in real time.

Video-aided insight complementing data readouts from sensors, meters, and machine components have become essential to tracking and analyzing operational performance in large-scale industrial scenarios such as factories, construction, mining, drilling, nuclear plants, and agriculture. The same is true when it comes to responding to supply chain disruptions at loading docks, storage facilities, and other points of vulnerability to fire, accidents, and other events.

Commercial space operations, too, call for access to feeds from video cameras as well as sensors and other sources on board spacecraft. Large flows of video and data from myriad monitoring points need to be captured by terrestrial systems and distributed in sync at ultra-low latency to all observation centers.

● Residential Applications
The same principles regarding security maintenance in government and commercial operations apply with use of real-time video surveillance in and around households. Whole neighborhoods served by a given surveillance provider will generate high volumes of live video feeds that can be ingested for real-time distribution to providers’ monitoring stations.

Real-time surveillance is also a key component to growing reliance on remote monitoring of the elderly and others in need of regular medical scrutiny. Live video will be an essential complement to sensor data when it comes to keeping tabs on people’s well-being 24/7.

HOW XDN TECHNOLOGY ENABLES EVENT-WIDE VISIBILITY IN REAL TIME

Red5 Pro’s multicloud XDN platform is providing support for multidirectional, highly scalable real-time streaming across a vast range of consumer, business, government, and institutional use cases where old approaches to current needs are inadequate, including most of the scenarios described above. It doesn’t matter whether the situation calls for one-to-many, many-to-many, or many-to-one connectivity at any moment in time, the technological basics are the same.

Video Streaming with Open-Source Protocols

The XDN uses the Real-time Transport Protocol (RTP), which underlies IP-based voice communications, as the foundation for enabling video delivered from any XDN-connected source to be streamed over any distance to any number of receivers at end-to-end latencies in the 200 to 400 millisecond range or even lower when transmissions are confined to a local area. RTP is the foundation for both WebRTC (Real-Time Communications), originally developed for peer-to-peer video communications, and Real-Time Streaming Protocol (RTSP), a one-to-many video streaming alternative to HTTP that became an IETF standard in 1998.

In the case of digital video surveillance cameras streaming over RTSP, a Red5 Pro server positioned at the multi-stream aggregation point can ingest and package any number of camera feeds for synchronized distribution to one or more monitoring posts, including any cloud locations where analytics engines are in play. Real-time performance can be achieved using the RTSP protocol end to end, provided that receiving devices are running a client player that supports the protocol.

But if clients are not natively equipped to support RTSP, the XDN repackages content ingested from RTSP streams for distribution via WebRTC without adding latency. WebRTC needs no plug-ins because it is supported by all the major browsers.

XDN infrastructure is also ideally suited to stream video from surveillance cameras that use Secure Reliable Transport (SRT), which has gained traction as an open-source real-time alternative to RTSP for streaming high-resolution video. This is an outgrowth of SRT’s role as a secure ultra-low latency successor to satellite distribution for premium video playout in the broadcast television industry.

The XDN platform is seamlessly interoperable with SRT (as is also the case with the Real Time Messaging Protocol (RTMP) and MPEG-TS modes used in other sectors). Multiple SRT-based surveillance streams, like those delivered via RTSP, are aggregated in real-time for ingestion onto the XDN for real-time streaming to surveillance command centers.

Another benefit resulting from XDN support for SRT is automatic implementation of end-to-end security. Both SRT and the foundational RTP protocol used with the XDN platform rely on the encryption protection afforded by the Secure Real-Time Transport Protocol (SRTP), as discussed below.

A Highly Scalable Multicloud Architecture

The scalability and other aspects to XDN versatility are anchored in a hierarchical architecture of server software stacks positioned in origin, relay, and edge nodes that are holistically orchestrated by the platform’s operations system (OS). The OS stream manager, processing performance data in real time, applies automated scaling mechanisms to add or remove nodes in response to fluctuations in traffic demand or the need to add new points of ingestion or reception.

The OS also accommodates the fail-safe redundancy essential to persistent surveillance operations. To create a high degree of stability, XDNs built with Red5 Pro use the stream manager’s autoscaling mechanism to create cluster-wide redundancy. When a node goes offline or malfunctions, all the processing can be moved to another node.

These capabilities also apply to load balancing on a stream manager. This means that in the event of anticipated heavy traffic, more than one stream manager can be set up behind the cloud platform’s load balancer service. This ensures that traffic requests such as (broadcast / subscribe) are evenly distributed among multiple stream manager instances to prevent flooding of requests on a single instance.

The multicloud capabilities of an XDN platform enable surveillance operators to choose the cloud environments they want to operate, including the ability to orchestrate real-time streaming across multiple clouds. Several global cloud service providers (including AWS, Microsoft Azure, Google Cloud Platform, and DigitalOcean) are pre-integrated with the platform.

Moreover, a large number of additional cloud providers can be tied into XDN operations via the widely used Terraform open-source multicloud toolset, which facilitates cross-cloud instantiations by enabling infrastructure-as-a-service (IaaS) APIs to be abstracted for access through a Terraform Cloud API specific to each cloud operator. The XDN stream manager can also be manually integrated to work with the APIs of any cloud provider that isn’t integrated with Terraform.

Ironclad Payload Protection

All content streamed on the Red5 Pro XDN platform is secured using SRTP, which protects users from the unauthorized access to content that many surveillance operations require. Moreover, the Red5 Pro XDN SDKs have been designed to hide the complexities of signaling and other steps related to setting up SRTP protection, making it easy to set up a secure streaming infrastructure.

Red5 Pro also provides the means by which all end points in the surveillance operation are authenticated as clients when requesting a stream action on the platform. In short, content streamed on Red5 Pro’s XDN provides the same high level of security commonly achieved through the use of costly digital rights management (DRM) systems, but at a fraction of the cost.

REAL-TIME VIDEO SURVEILLANCE IN ACTION

XDN technology has been deployed to enable better results from video surveillance in a rapidly expanding number of scenarios worldwide. The following descriptions highlight some examples of how suppliers are increasing the appeal of their surveillance solutions through the use of an XDN infrastructure.

Novetta
Customers of Novetta, a provider of actionable intelligence solutions, are using the XDN platform with systems employed in defense, federal law enforcement, and government intelligence. For example, a major U.S. military branch and the U.S. Border Patrol units are among the high-profile entities that use Novetta’s real-time multi-intelligence solution, Ageon ISR, to deliver actionable insights derived from sensors and radar as well as video feeds from drones, Humvees, underwater vehicles, body cameras, and even canine patrols to remotely dispersed command centers.

Simultaneous delivery of video streams is essential to these operations, according to Christopher Regan, Novetta’s director of engineering for C4ISR (the Defense Department’s acronym for Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance). “If we can’t provide that video solution, low-latency, high-quality live streaming to a browser, overall we’re kind of dead in the water,” Regan said.

Aiforsite Oy
Another realm of video surveillance activity where XDN technology is employed can be found in the real-time construction site management support offered by the Finnish company Aiforsite Oy. The firm’s AI for Construction (AIC) intelligent software-as-a-service solution enables data-driven site management and remote work, relying in part on video streams from fixed cameras and drones flying over construction sites.

Clients obtain real-time situational awareness through 360-degree visibility, which enables them to analyze overall project progress in comparison to 3D images developed from construction industry Building Information Modeling (BIM) tools. Streaming over an XDN infrastructure also helps them monitor and manage productivity of workers, tools, and materials in real time.

Matrox Video
As discussed in the introduction, Matrox Video vice president Sam Racine said he is witnessing demand for XDN support in multiple surveillance applications, including the aforementioned drone-based military application. Matrox provides a wide array of components for such endeavors, including IP cores, application-specific integrated circuits (ASICs), circuit boards, as well as a variety of imaging-related products such as video controllers, frame grabbers, and 3D sensors. The 3D sensors enhance surveillance performance by applying a dual-camera, single-laser design with embedded vision algorithms in laser triangulations to represent objects and scenes through the generation of individual profiles and depth maps.

Racine also noted the company is facilitating customers’ use of XDN technology by building “highly dense capture points.” He said these solutions are able to “spit out many different resolutions, protocols, [and] timing formats,” resulting in a significant reduction in transcoding latency.

FLIR Systems
FLIR, another company highlighted in the introduction, is a leader in surveillance technology that relies on thermal (infrared) as well as visible light captures to provide around-the-clock visibility into the monitored situations. When these components are combined with AI-assisted analytics in real-time surveillance (enabled by an XDN infrastructure), FLIR customers in public safety, transportation, industrial operations, and other use cases gain the situational awareness essential to effective responses to events, noted FLIRs project manager David St. Claire.

Whether surveillance managers are engaged in monitoring and reacting to the “natural flow of traffic and fender benders or major emergencies like fires, hurricanes, and earthquakes, we have a growing need for this kind of [analytics-generated] metadata combined with video to enable real-time reactions to the overarching situation,” St. Claire said. Recognition of such needs is new in some areas like traffic management, he noted.

“The transportation sector has adopted video technology predominantly disseminating video for situations where latency doesn’t matter,” he said, in reference to the widely deployed analog CCTV devices that dot the world’s streets and highways. But conversions to digital cameras are gaining momentum, driven in part by the “opportunity to employ instant management in emergency operations,” he added.

A case in point is the work underway at the California Department of Transportation (Caltrans), which is using FLIR technology with XDN support for real-time traffic surveillance in the highly congested District 7 area of operations serving Los Angeles and Ventura Counties. The benefits to the state are amplified by FLIR’s Cameleon Intelligent Transportation System (ITS) software solution, which supports control over all ITS devices, including dynamic message signs (DMSs), gates, detector stations, and other elements.

The management software also allows Caltrans operators to easily share information with other agencies while maintaining control over their systems. Caltrans has recommended broader conversion to such capabilities across the state.

Video Surveillance in Space Transport

Aerospace Corp. is taking government video surveillance operations into space. As the operator of the only federally funded research and development center committed exclusively to the space enterprise, the company has been contracted by NASA to provide agency-wide specialized engineering, evaluation, and test services, including expertise in support of mission success.

In order to monitor all video and data streams from rockets on launchpads and in transit, Aerospace captures raw camera and sensor feeds from various locations on the spacecraft for ingestion onto the XDN platform. These inputs are streamed in real time for simultaneous reception at monitoring sites, where personnel employ advanced analytics to deliver immediate assessments of onboard developments.

Another space-related application of XDN technology implemented by a private supplier of commercial transport vehicles involves transmission of onboard camera outputs from ground-based receivers to multiple company locations. This allows thousands of staff members to view the live feeds simultaneously at launch and during flight.

An executive, who doesn’t want the company to be named, described the set-up by stressing that reliability, simultaneity, and sub–half-second latency are vital to operations. “You no longer feel like you’re part of a shared experience when you’re watching something out of sync with the person sitting right next to you,” she said.
“More than anything,” she added, “we need a stable platform. With thousands of people watching hundreds of feeds, you can end up very quickly with ten, twenty thousand concurrent views. I need a platform that can scale and grow with that, that won’t topple over as you expand the network and increase the load on it.”

CONCLUSION

Transformative approaches to video surveillance are underway in multiple market segments worldwide with execution of faster, smarter responses based on real-time visibility into unfolding events across multiple points of observation.

As live video surveillance becomes ever more consequential to maintaining public safety, preventing disruptions in commerce, managing military operations, securing property, and much else, the need for real-time visibility across all fields of view has become a top priority. Major advances on two fronts have set the stage for this to happen:

● High-resolution digital cameras equipped to stream live video to monitoring stations have moved far beyond the limitations of analog cameras with their grainy records of bad behavior.

● New analytics engines employing AI are able to instantaneously process immense amounts of data from monitoring posts to provide highly detailed insights into events based on the identification of details that can take hours to draw out manually.

But the digital advantage is highly constrained when human monitors and their analytics engines can’t see what’s happening in real time. Rather than processing camera feeds that are streamed out of sync with seconds of lag time behind live events, managers of surveillance operations must be able to shape reactions based on insights drawn from synchronized aggregations of video streamed at sub–half-second latencies from multiple locations.

This new approach to video surveillance is unfolding in a growing number of scenarios where smart-city operations, emergency responders, businesses, the military, and space transporters are relying on Red5 Pro’s multicloud XDN platform to enable comprehensive real-time insight into live events. These surveillance system operators are able to coordinate scrutiny and analysis of one or more occurrences to whatever extent of geographic coverage is necessary.

No matter how many cameras are involved and where they are deployed, their output is fed to XDN origin nodes for simultaneous real-time distribution to all points involved in the formulation of event responses, including cloud-based analytics engines as well as manned posts. Surveillance managers are assured the scalability, redundancy, and content protection they need to ensure their operations are adequate to the tasks at hand.

To learn more about the Red5 Pro XDN platform and the role it can play in real-time intelligent video surveillance contact info@red5.net or schedule a call.