Upgrading Stream Manager (general)

This document assumes that you have already set up an autoscaling environment per Deploying Stream Manager and Autoscaling on Amazon Web Services or Deploying Stream Manager and Autoscaling on Google Cloud Compute.

1. Create New Disk Image/AMI

Follow Upgrading Red5 Pro for Autoscaling Nodes and make a note of the disk image name to use in the launchconfig for the Stream Manager update.

Update Stream Manager

Per "How to Upgrade Red5 Pro Server," the cleanest way to upgrade the Stream Manager will be to:

  1. Download the latest Red5 Pro Server distribution and the latest Red5 Pro Autoscaling Library Extensions (for AWS or for Google Cloud) from https://account.red5pro.com/download to your local machine.
  2. Copy both files up to the Stream Manager.
  3. Copy and unzip the new distribution into the same directory where the original red5pro folder was (e.g. /usr/local/).
  4. Copy the new cloud controller jar file to the webapps/streammanager/WEB-INF/lib/ directory of the new server distribution.
  5. Remove /conf/autoscale.xml, /plugins/red5pro-autoscale-plugin-###.jar, and /plugins/red5pro-webrtc-plugin-###.jar from the new server distribution.
  6. Note the values in the previous {red5prohome}/webapps/streammanager/WEB-INF/red5-web.properties file and edit the new webapps/streammanager/WEB-INF/red5-web.properties per those values.
  7. Edit the Launch Config and Scaling Files as described for AWS, or for Google Cloud.

When you are ready to upgrade

  1. Rename your existing red5pro folder to bak.red5pro (sudo mv red5pro bak.red5pro).
  2. Rename the new server distribution directory to red5pro (sudo mv red5pro-server-* red5pro).
  3. Kill the Red5 Pro Service (ps aux | grep java to get the Red5 Pro PID, then sudo kill -9 <PID>).
  4. Start the Red5 Pro Service sudo systemctl start red5pro.
  5. Following the API Basics below, create a new nodegroup and add an origin server.
  6. Once the new nodegroup is up, terminate the old nodegroup.

Upgrading Stream Manager 2.0 or 3.0 to 3.1

If you are upgrading from Stream Manager v2.0 (Red5 Pro < v5.0.0), or v3.0 (v5.0.0, v5.1.1) to Stream Manager v3.1, you will need to follow a more complicated upgrade path. Note that this will require downtime.

Update Database

Due to significant changes in the autoscaling database schema and logic, it is not possible to simply upgrade the database in place. The best solution is to setup a separate database instance, parallel to the one which is live.

Follow your cloud database setup instructions to setup a new database for autoscaling. Use the same instance configuration and credentials as your live database instance.

IMPORTANT: Make a note of this new database IP address (for Google Cloud) or Endpoint url (for AWS). This will be used in red5-web.properties - config.dbHost property when configuring stream manager.

  • Configure the the new SQL instance to allow access from the stream manager IP (both public and private IP for AWS).

Create New Disk Image/AMI

You will need to use the latest Red5 Pro Server build to take advantage of the improvements in the Stream Manager API 3.1. Follow Upgrading Red5 Pro for Autoscaling Nodes and make a note of the disk image name to use in the launchconfig policy for the Stream Manager update.

Upgrade Stream Manager

Per "How to Upgrade Red5 Pro Server," the cleanest way to upgrade the Stream Manager will be to:

  1. Download the latest Red5 Pro Server distribution and the latest Red5 Pro Autoscaling Library Extensions (for AWS or for Google Cloud) from https://account.red5pro.com/download to your local machine.
  2. Copy both files up to the Stream Manager.
  3. Copy and unzip the new distribution into the same directory where the original red5pro folder was (e.g. /usr/local/).
  4. Copy the cloud controller jar file to the webapps/streammanager/WEB-INF/lib/ directory of the new server distribution.
  5. Remove /conf/autoscale.xml and /plugins/red5pro-autoscale-plugin-*.jar from the new server distribution.
  6. Note the values in the previous {red5prohome}/webapps/streammanager/WEB-INF/red5-web.properties file and edit the new webapps/streammanager/WEB-INF/red5-web.properties per those values - with the exception of using the new database IP or Endpoint URL for the config.dbHost property. There are several new values in the latest red5-web.properties file, so you can't just copy this from the old build.
  7. Starting with Stream Manager API v3.0, there are no longer launchconfig and scaleconfig policy files. Instead, these policies are created and managed via the API.

When you are ready to upgrade to 3.1

If you take care of all of the above steps first, you will need about 15-30 minutes of downtime, depending on how many nodegroups and nodes you have, to switch over.

  1. Delete any existing nodegroups; you will create new ones once the Stream Manager is upgraded. If you skip this step, then when you restart the Stream Manager, it will tear down the existing nodes because they are not in the new database.
  2. Rename your existing red5pro folder to bak.red5pro (sudo mv red5pro bak.red5pro).
  3. Rename the new server distribution directory to red5pro (sudo mv red5pro-server-* red5pro).
  4. Kill the Red5 Pro Service (ps aux | grep java to get the Red5 Pro PID, then sudo kill -9 <PID>).
  5. Start the Red5 Pro Service sudo systemctl start red5pro.
  6. Following the API Basics below, create a nodegroup to match the previous setup and add an origin server. Note that in the new version of autoscaling, a nodegroup can have multiple origin servers. In addition, you can specify different server types for origin and edge servers.

Adding Backwards Compatibility for Broadcast/Subscribe Clients

Note: this is critical if you have a Mobile application that is referencing a previous API version.

If you have existing streaming clients (mobile or webapp) referencing the previous version of Stream Manager API, upgrading to the latest Stream Manager will make your streammanager API service inaccessible to them unless you add in redirection support.

The process of changing a URL request endpoint dynamically at runtime on a server is known as URLRewriting. To ensure that the API 2.0 or API 3.0 event-stream requests are automatically sent to the new API 3.1 request path we need to rewrite the requests as /api/3.1 requests.

Red5 Pro runs on Tomcat engine, so we need to use a Tomcat supported solution for our URL-rewriting. The following solution uses UrlRewriteFilter, a popular URL-rewrite library for apache tomcat:

STEP 1: Download the url-rewrite library

Download the latest version of the UrlRewriteFilter library (jar) from here.

STEP 2: Copy the library into streammanager

Copy the downloaded jar into {red5prohome}/webapps/streammanager/WEB-INF/lib/

STEP 3: Register the library in streammanager to intercept and handle web requests

  • Edit {red5prohome}/webapps/streammanager/WEB-INF/web.xml with the editor of your choice, adding the UrlRewriteFilter filter xml snippet at the end (after the authFilter):-
<filter>
<filter-name>UrlRewriteFilter</filter-name>
<filter-class>org.tuckey.web.filters.urlrewrite.UrlRewriteFilter</filter-class>

<init-param>
<param-name>statusEnabled</param-name>
<param-value>false</param-value>
</init-param>
</filter>

<filter-mapping>
<filter-name>UrlRewriteFilter</filter-name>
<url-pattern>/*</url-pattern>
<dispatcher>REQUEST</dispatcher>
<dispatcher>FORWARD</dispatcher>
</filter-mapping>

STEP 4: Activate URLRewriting for event broadcast & subscribe requests

  • Create a file named urlrewrite.xml in {red5prohome}/webapps/streammanager/WEB-INF/, with the following code (note that this contains redirects for both 2.0 and 3.0 clients to the new 3.1 api:
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE urlrewrite PUBLIC "-//tuckey.org//DTD UrlRewrite 4.0//EN"
"http://www.tuckey.org/res/dtds/urlrewrite4.0.dtd">

<urlrewrite use-context="true" use-query-string="true">

<rule match-type="regex">
<from>/api/2.0/event/(.*)$</from>
<to type="redirect">/api/3.1/event/$1</to>
</rule>

<rule match-type="regex">
<from>/api/3.0/event/(.*)$</from>
<to type="redirect">/api/3.1/event/$1</to>
</rule>

</urlrewrite>

This file contains the re-writing rule which is used by UrlRewriteFilter to decide when to and how to redirect URL request paths.

STEP 5: Restart server

Once all of the above steps are done, you need to restart the Red5 Pro service on the Stream Manager for changes to take effect.


VERIFYING

Once your deployment is active, you can test the UrlRewriteFilter by making a simple GET request at the old api 2.0 or 3.0 endpoint from the browser.

Example: (This is a test endpoint)

REQIUEST

Url

<streammanagerurl>/streammanager/api/2.0/event/test?action=broadcast

Method: GET

RESPONSE

Success: HTTP CODE 200

DATA

welcome to stream manager event route

Once you enter the URL in the browser and hit "Enter", you should see your URL automatically changing to <streammanagerurl>/streammanager/api/3.1/event/test?action=broadcast in the browser's address bar.


API 3.1 Basics, to get you started

(for full API, see Red5 Pro Stream Manager API)

Verify Stream Manager is Using the Correct Controller (GET)

GET call: https://<streammanagerurl>/streammanager/api/3.1/admin/debug/cloudcontroller?accessToken=<rest.administratorToken from red5-web.properties file>

should return: Google Compute for Google Cloud or Amazon Web Services for AWS.

Create a New Scale Policy (POST)

POST call:

https://<streammanagerurl>/streammanager/api/3.1/admin/configurations/scalepolicy?accessToken=<rest.administratorToken>

json data (example):

{
      "policy": {
          "name": "allnodetypes-v3",
          "description": "This is a sample 3rd generation scale policy file with all four node types",
          "type": "com.red5pro.services.autoscaling.model.ScalePolicyMaster",
          "version": "0.0.3",
          "targets": {
              "region": [
                  {
                      "name": "default",
                      "target": [
                          {
                              "role": "edge",
                              "estimatedWarmUpTime": 150000,
                              "maxLimit": 20,
                              "scaleAdjustment": 1,
                              "coolDownPeriod": 180000,
                              "minLimit": 2
                          },
                          {
                              "role": "origin",
                              "estimatedWarmUpTime": 150000,
                              "maxLimit": 10,
                              "scaleAdjustment": 1,
                              "coolDownPeriod": 180000,
                              "minLimit": 2
                          },
                                                    {
                              "role": "relay",
                              "estimatedWarmUpTime": 150000,
                              "maxLimit": 10,
                              "scaleAdjustment": 1,
                              "coolDownPeriod": 180000,
                              "minLimit": 2
                          },
                                                    {
                              "role": "transcoder",
                              "estimatedWarmUpTime": 150000,
                              "maxLimit": 10,
                              "scaleAdjustment": 1,
                              "coolDownPeriod": 180000,
                              "minLimit": 2
                          }
                      ]
                  }

                      ]
                  }
      }
  }

Create a New Launch Policy (POST)

POST call:

https://<streammanagerurl>/streammanager/api/3.1/admin/configurations/launchconfig?accessToken=<rest.administratorToken>

json data (example):

    {
    "launchconfig": {
      "name": "allnodetypes",
      "description": "This is a sample aws launch configuration with all four node types",
      "image": "<ami name>",
      "version": “0.0.2”,

      "targets": {
          "target": [
        {
          "role": "origin",
          "instanceType": "c5.large",
          "connectionCapacity": "2000"
        },
        {
          "role": "edge",
          "instanceType": "c5.large",
          "connectionCapacity": "2000"
        },
        {
          "role": "relay",
          "instanceType": "c5.large",
          "connectionCapacity": "2000"
        },
        {
          "role": "transcoder",
          "instanceType": "c5.xlarge",
          "connectionCapacity": "2000"
        }
         ]
        },

        "properties": {
          "property": [
            {
              "name": "property-name",
              "value": "property-value"
            }
          ]
        },
        "metadata": {
          "meta": [
            {
              "key": "meta-name",
              "value": "meta-value"
            }
          ]
        }
      }}

Create a New Node Group (POST)

Using a tool like Postman, create a new node group via the API. baseCapacity is the Minimum subscriber connections that this group should support. (This parameter helps the scale-in process decide when to scale down an edge).

POST call: https://<streammanagerurl>/streammanager/api/3.1/admin/nodegroup?accessToken=<rest.administratorToken from red5-web.properties file>

Data (make sure to select JSON as the body type):

{
 "regions": [
   "<google region, e.g.: us-east1; or aws region, e.g. us-east-1>"
 ],
 "launchConfig": "default-v2",
 "scalePolicy": "default-v2"
}

Note the cluster “name” that is returned by the above call. It will be used to create a new Origin server.

Launch New Origin (POST)

After you create a node group, create the origin server. Creating an origin server will generate all of the minimum designated nodes, per scaling policy min limit.

POST call:

https://<streammanagerurl>/streammanager/api/3.1/admin/nodegroup/<cluster-name>/node/origin?accessToken=<rest.administratorToken from red5-web.properties file>

Set Alarm Threshold (POST)

By default, the alarm threshold (the capacity percentage at which the cluster will scale up) is set to 60%. To modify this, POST the following:

FOR EDGE:

https://<streammanagerurl>/streammanager/api/3.1/admin/alarm/scaleout/default?type=edge&threshold=<threshold>&accessToken=<rest.administratorToken from red5-web.properties file>

FOR ORIGIN:

https://<streammanagerurl>/streammanager/api/3.1/admin/alarm/scaleout/default?type=origin&threshold=<threshold>&accessToken=<rest.administratorToken from red5-web.properties file>

LIST GROUPS (GET)

https://<streammanagerurl>/streammanager/api/3.1/admin/nodegroup?accessToken=<rest.administratorToken from red5-web.properties file>