Red5 Pro Clustering

With a typical high capacity web site which serves static files and scripts, load balancing is used and every client facing server is a mirror image of a master with a shared database.

Live streams are almost always unicast over UDP or TCP to a distribution server. The subscribers are also TCP or UDP unicast receivers. When it is time to add more capacity for live content, it is not practical to have the broadcaster publish to additional servers. As such, other servers must be able to acquire and repeat the streams in real time.

The Clustering Plugin available from Red5 Pro allows for:


Clustering is available for Pro Accounts Only

To enable clustering, you will need to have a Pro License associated with your Red5 Pro Account.

To purchase a Pro License, visit the Overview Section of your account and fill in the required information under Upgrade to Pro License.


Terminology

The follow terms will be used throughout this document.

Cluster

One or more active servers which are expected to make real-time data streams available for consumption. These are usually audio and video streams but can also be other forms of data.

Server Node

A server instance that is part of an active cluster.

Broadcaster

The source of a live data stream, usually a webcam and microphone or smart phone. It can encompass the user, network session, or function of stream generation.

Origin

An Origin accepts broadcasters. Origins can also be configured to accept subscribers. The server can be in a hybrid mode where it repeats streams from another Origin. A stand-alone server is an Origin and an Edge.

Edge

An Edge accepts subscribers. An Edge is configured to have at least one origin to repeat content from. It can be configured to communicate with more than one Origin.

Clustering

Clustering is how we configure multiple servers to be Edges and Origins interconnected in order to distribute real time data to all receiving clients.


Enabling Clustering

To enable Clustering between Red5Pro servers, you need to uncomment the ClusterServiceResolver in the list of serviceResolvers in webapps/red5-default.xml on all Origin and Edge servers.

Remove the lines around the bean id="clusterServiceResolver" node from the file:

webapps/red5-default.xml

 <bean id="global.serviceInvoker" class="org.red5.server.service.ServiceInvoker">
    <property name="serviceResolvers">
      <list>
        <bean id="scopeServiceResolver" class="org.red5.server.service.ScopeServiceResolver"/>
        <bean id="handlerServiceResolver" class="org.red5.server.service.HandlerServiceResolver"/>
        <bean id="contextServiceResolver" class="org.red5.server.service.ContextServiceResolver"/>
        <!-- Uncomment as part of the clustering configuration
        <bean id="clusterServiceResolver" class="com.red5pro.cluster.plugin.ClusterServiceResolver"/>
         -->
      </list>
    </property>
  </bean>


Configuration

Included in the conf/ directory of your Red5 Pro Server install is a file named cluster.xml. The cluster.xml file is a Spring Bean resource definition used to configure the cluster.

The properties defined on the resource will differ based on whether the deployed server is an Origin or an Edge. Thye are:

Configuration of Origin

Most of the properties defined in the cluster.xml file will have no effect on the inner-workings of a Red5 Pro Server designated as an Origin. However, it is a requirement that the password attribute is defined set as the same value for each Edge and Origin assocaited within a cluster.

Any modifications to cluster.xml will necessitate restarting the Red5 Pro server.

To configure an Origin,

  1. Modify the changeme value assigned to the password attribute in the cluster.xml file.
  2. Modify THIS SERVER's public ip from 0.0.0.0
  3. If you do not want the Origin to be part of the round-robin subscribers, the change the "privateInstance" value to "true"

conf/cluster.xml

<bean name="clusterConfig" class="com.red5pro.cluster.ClusterConfiguration">
  <property name="origins">
    <list>
      <!-- Add origin ips and optionally the port if its not default 1935. Uncomment origin entries-->
      <!-- <value>0.0.0.1</value> -->
      <!-- <value>0.0.0.2:1935</value> -->
    </list>
  </property>

  <!-- edge/origin link cluster password -->
  <property name="password" value="changeme" />

  <!-- THIS SERVER'S public ip used for round-robin at the origin server. -->
  <property name="publicIp" value="0.0.0.0" />

  <!-- THIS SERVER'S public RTMP port.-->
  <property name="publicPort" value="1935" />

  <!-- edges and origin can be excluded from round robin queries. -->
  <!-- set PrivateInstance to true to exclude this instance in round robin subscriber queries. -->
  <property name="privateInstance" value="false" />

  <!-- For an edge, it is the interval in seconds to retry to connect to origin if the origin becomes unavailable. -->
  <property name="retryDuration" value="30" />
</bean>

Additionally, there is a web.xml file included in the _webapps/root/WEB-INF/__ directory of the Red5 Pro Server install that defines the Round Robin servlet (note: this file does not need to be modified). This is the service from which subscribers will request playback IP of the clustered stream:

webapps/root/WEB-INF/web.xml

<servlet>
  <servlet-name>cluster</servlet-name>
  <servlet-class>
    com.red5pro.cluster.plugin.agent.ClusterWebService
  </servlet-class>
  <load-on-startup>-1</load-on-startup>
</servlet>

<servlet-mapping>
  <servlet-name>cluster</servlet-name>
  <url-pattern>/cluster</url-pattern>
</servlet-mapping>

This will return the IP that subscribers will use to consume the stream with the following structure. The GET request will be made on a URI with the following structure:

Configuration of Edge

When setting up an Edge server to be part of the Cluster, you will need to:

  1. Modify the changeme value assigned to the password attribute in the cluster.xml file to be the same as the one set for the Origin.
  2. Modify THIS SERVER's public ip from 0.0.0.0
  3. Define a list of Origins in the origin attribute (one entry for each origin that the edge will be connecting to)
  4. If you do not want the Edge to be part of the round-robin subscribers, the change the "privateInstance" value to "true"

Any modifications to cluster.xml will necessitate restarting the Red5 Pro server. You can add Edges to an active cluster without restarting the Origin.

conf/cluster.xml

<bean name="clusterConfig" class="com.red5pro.cluster.ClusterConfiguration">
  <property name="origins">
    <list>
      <!-- Add origin ips and optionally the port if its not default 1935. Uncomment origin entries-->
      <!-- <value>0.0.0.1</value> -->
      <!-- <value>0.0.0.2:1935</value> -->
    </list>
  </property>

  <!-- edge/origin link cluster password -->
  <property name="password" value="changeme" />

  <!-- THIS SERVER'S public ip used for round-robin at the origin server. -->
  <property name="publicIp" value="0.0.0.0" />

  <!-- THIS SERVER'S public RTMP port.-->
  <property name="publicPort" value="1935" />

  <!-- edges and origin can be excluded from round robin queries. -->
  <!-- set PrivateInstance to true to exclude this instance in round robin subscriber queries. -->
  <property name="privateInstance" value="false" />

  <!-- For an edge, it is the interval in seconds to retry to connect to origin if the origin becomes unavailable. -->
  <property name="retryDuration" value="30" />
</bean>

Red5 Pro Default Clustering Functions

The main purpose of clustering is content distribution.

Usage

The following capabilities provided by the Red5 Pro Server can benefit from clustering:

• Streaming

Every live stream published at the Origin is repeated by the Edges. There are no configurations to specify specific stream names eligible for repeating.

• Second Screen

Second Screen functions are treated as live content so local pairing works with hosts and clients using different Edges.

• Applications

The ApplicationAdapter events - other than those mentioned above - are not broadcast between edges and origins. App joins, room starts, and other often used server events are not distributed.


Extending

There is an API to send service calls through the clustered instances. You can specify handlers at runtime, and call those handlers from your application. This is the same API that the Second Screen SDK uses to operate in the clustered environment.

If you need to broadcast session events for your clients - such as room join - each instance sets the business handler locally and at runtime calls it remotely.

The application adapter can query the instance to find if it is an edge or an origin. The application adapter can optionally broadcast user messages to all other clustered instances, all edges, all origins, a specific instance, or all instances excluding one or more.


Use cases

The following sections describe several use cases for utilizing clusters.

Two Servers

A most common use case for small productions and events is to have one Origin and one Edge. The Origin is included in the round robin and because of that, both servers receive subscribers.

Both the Origin and Edge cluster.xml files are edited to show privateInstance is false.

The Edge is also configured with the Origin IP. The Edge initiates the connection and reconnection if the Origin becomes unavailable.

Publishers

Stream providers use the Origin IP as the server to connect to. The streams are then repeated at the Edge.

Subscribers

Subscribing clients use the Origin IP to call the round robin service and receive either the edge or the Origin IP. They then subscribe to the stream at the provided host IP.

Two Server Clustering


One Origin and Three Edges

Another common use case is to have multiple Edges and one Origin where subscribers are given only the Edge IP.

The Origin cluster.xml file is edited to show privateInstance is true. This removes it from round robin sequencing. Edges are all set so privateInstance is false.

The Edges are configured with the Origins IP, and they will initiate the connection to the Origin and reconnection if the Origin becomes unavailable.

Publishers

Stream providers use the Origin IP as the server to connect to. The streams are then repeated at the Edges.

Subscribers

Stream consumers use the Origin IP for calling the round robin servlet. They then subscribe to the stream at the provided host IP.

One Origin and Three Edges


Multiple Origins

After setting up three Edges and One origin as described, you can add another Origin. All Edges are given two Origin IPs in the cluster.xml page.

Restart the Edges and the new Origin.

Publishers

Publishers can use either origin and all streams will be received by all edges.

Subscribers

Subscribers can use the round robin service at either Origin and receive an active Edge IP.

On Origin and Three Edges