Deploying Stream Manager and Autoscaling on Google Cloud Platform

Overview:

This document assumes that you have a basic knowledge of operating on google cloud platform and some basic linux administration skills. If you need more detailed information, please contact us.

Before you proceed with the setup make sure you have the following prerequisite steps:

  • You have already set up your GCP account with valid billing information.
  • You should have a valid acceessible Project in your GCP account with an active billing account associated.
  • You should have installed and configured the associated SDK for management (https://cloud.google.com/sdk/).

In order to use the Red5 Pro Stream Manager service you will need the following:

  1. Latest Red5 Pro Server build
  2. The google-cloud-controller.jar, from the Red5 Pro Autoscaling Library Extensions section
  3. An active Red5 Pro license key (needed for clustering). REGISTER HERE
  4. A Google Cloud Platform Developer project ID

Before you Begin:

You will want to keep a record of the usernames, passwords, IP addresses, and other information generated during the setup process, as you will need the information for stream manager configuration and future operations via the API. Click here to download a handy list for tracking all of your Red5 Pro Autoscaling values.

1. Reserve Static IP for Stream Manager

It is critical that the Stream Manager have a static IP address, so that in the event that the instance is rebooted, it will retain the same public IP address. You can reserve a static IP address via the Google Cloud SDK. Choose the region in which you wish to host the stream manager:

gcloud compute --project <your-project-id> addresses create streammanagerip --region <preferred-region>

You can also reserve a static IP address via the Google Cloud Console. Navigate to VPC Networks, External IP Addresses and assign an IP.

Reserve Static IP

Google cloud offers two types of static IP addresses- Premium and Standard. The Premium IP addresses (default selection) are optimized for better routing using google's own backbone. The shell command specified above will reserve a Premium IP address. To know more about IP addresses and their differences take a look at official documentation on External IP addresses

Make a note of the assigned IP address. You will need this for Database permissions and Stream Manager configuration.

2. Create Google Cloud SQL Instance

  1. Select the SQL menu option in the sliding menu pane on left.
  2. Click on "Create instance" option.
  3. Click on "Choose MySQL". sql setup
  4. Set a instance ID such as streammanager-db.
  5. Set a root password (you can let GCP generate the password).
  6. Select the region where you reserved the static IP for the stream manager.
  7. Click and expand Configuration options.
  8. Select a machine type, storage type according to estimated usage load.
  9. Under Set Connectivity, make sure Public IP is selected. Below it click on +Add Networks and add the stream manager IP address. Stream Manager access Note: You might also want to add your IP address to be able to manage the stream manager database.
  10. Leave the other options to default settings.
  11. Under Set maintenance schedule, select an appropriate time for possible maintainence. Select a day which falls outside your streaming schedule.
  12. Press "Create" button to create the instance (see screenshots below).

sql config sql config

First Generation (Legacy)

From the Google Cloud Developer Console:

  1. Select the SQL menu option in the sliding menu pane on left.
  2. Click on "Create instance" option.
  3. Click on "For First Generation MySQL instances, click here".
  4. Set a instance ID such as streammanager-db.
  5. Select the region where you reserved the static IP for the stream manager.
  6. Select a memory capacity according to estimated usage load.
  7. expand “advanced options” - choose defaults except where noted below.
  8. Choose billing plan based on your expected use.
  9. Set activation policy to "Always on".
  10. Enable the "Assign an IPv4 address to my Cloud SQL instance" option.
  11. Under Authorized Networks +Add Item.
  12. enter the IP address that you reserved for the stream manager.
  13. Press "Create" button to create the instance (see screenshots below).

sql config sql config

3. Configure SQL Database for Stream Manager

Click on the newly created database to go to the Instance Details screen.

Create the “cluster” Database

  1. Click on Databases tab
  2. Under MySQL Databases, click on Create database
  3. Name the new database: cluster
  4. Leave the default character set (utf8)
  5. Click Add to finish

sql setup

Add a Database User Account

  1. Click on Access Control (note, you should see your stream manager listed under Connection tab) Note: If you dont already have, you might also want to add the development system IP address to be able to initialize the instance with the stream manager database.
  2. Click on Users
  3. Click on Create user account
  4. Set a username and password
  5. NOTE: these will be used by the Stream Manager, so make sure to note the username/password you’ve created; For security, do not use the same password as you set for the root account.
  6. Allow any host
  7. Create to add the user account
  8. NOTE: you will not be able to alter permissions once you’ve created the user, but you can update the password through google cloud dashboard

Make a note of the following for stream manager configuration:

  1. SQL Host IP Address (Can be seen under the Overview tab)
  2. Database access username and password (from step 4)

Populate the Cluster Database Tables

You will find cluster.sql in the {red5prohome}/webapps/streammanager/WEB-INF/sql directory of the latest Red5 Pro server distribution. This file should be used to populate the cluster database. You can do this using MySQL Workbench, or even via the Google Developer Console by:

Using Google Developer Console

  1. Under Storage Tab, Storage menu option in the sliding menu pane on left, CREATE BUCKET (for example: autoscaling). Click on your new bucket, then UPLOAD FILES, to add the cluster.sql file to the bucket.
  2. Import the file through the Google Cloud SQL management tool:
  3. From left-hand navigation, choose SQL
  4. Click on your database instance, then click on IMPORT (format of import - SQ6.
  5. Browse to the cluster.sql file. Show advanced options, and enter Database name: "cluster"
  6. Then click on Import. storage

Using MySQL Workbench

Another easy and flexible way to populate the database instance is using the MySQL Workbench which can be downloaded from https://dev.mysql.com/downloads/workbench/.

  • Launch MySQL Workbench tool on your system
  • Click on the “+” icon to the right of MySQL Connections to open the new connection configuration window.
  • Enter a connection name - (for example, “google-mysql”)
  • Hostname: the database IP which we obtained from setting up the instance earlier
  • Username: the Database username configured above
  • Password: the Database password configured above
  • Default Schema: cluster (database schema for streammanager)
  • *Click on Test Connection to verify you can connect.
  • Click ok to save configuration and close window.
  • Download the Red5 Pro Server Distribution, and unzip to your desktop.
  • Double click on the new connection you created to open database workspace view.
  • Download the Red5 Pro Server Distribution, and unzip to your desktop.
  • From menu click => File => Open SQL Script, and browse the server distribution to find the cluster.sql file {red5prohome}/webapps/streammanager/WEB-INF/sql/cluster.sql
  • Click the execute icon to execute the sql script which populates the cluster database execute clusterdbpopulated

4. Create Firewall Rules

You will need to set up a disk Image that will be used by the Autoscaler when it initiates new server instances. Node Groups will be deployed to the default network, so you will need to enable incoming access to the Red5 Pro server ports on that network.

Note: If you are going to be using load-balanced stream managers, then you need to use the loadbalancer IP in the autoscale.xml file of your disk image.

  • Under VPC Network tab -> Firewall Rules, Click on CREATE FIREWALL RULE. Add a new firewall rule to the default network profile.
  • Name: Specify a name for the rule such as red5pro-ports
  • Description: Add a description
  • Logs: Leave to default (off)
  • Network: Make sure default network is selected
  • Priority: Leave to default
  • Direction of traffic: Leave to default (Ingress)
  • Action on match: Leave to default (Allow)
  • Targets: All instances in the network
  • Source Filter: IP Ranges
  • Source IP Ranges: 0.0.0.0/0 (Implies Any)
  • Second Source Filter: Leave to default
  • Protocols & Ports: Select Specified Protocols & Ports and enter the following ports for mentioned protocols.

TCP: 5080,443,1935,8554,8081,8083,6262,22 UDP: 40000-65535

  • Click Create to create rule
Port Description Protocol
22 SSH TCP
5080 default web access of Red5 Pro/Websockets for WebRTC / Stream Manager TCP
443 modified https access of Red5 Pro; secure websockets for WebRTC / Stream Manager TCP
1935 default Red5 Pro RTMP port TCP
8554 default RTSP port TCP
6262 websockets for HLS TCP
8081 websockets for WebRTC (severs earlier than 5.4.0) TCP
8083 secure websockets for WebRTC (severs earlier than 5.4.0) TCP
40000-65535 TURN/STUN/ICE port range for WebRTC UDP

NOTE: As of release 5.4.0, websockets automatically use the same ports as http/https as defined in red5.properties, so you do not need to open up ports 8081/8083.

networking

networks

firewall

5. Create Red5 Pro Disk Image

  • From Compute Engine, VM, create an Ubuntu (recommended for the small footprint of the OS) virtual machine instance, using the network you created in your default zone.

vmiimage

It is recommeended that you select a machine type with a minimum of 2 vcpu and 4 GiB Memory for acceptable performance.

  • When selecting the Boot Disk, make sure to set the size to atleast 16 GB

boot disk

  • Once the instance has spun up, use the Google Cloud SDK to copy the latest version of Red5 Pro server to the instance:

    • gcloud compute scp red5pro-server.zip red5pro-image-01:/tmp/

The google cloud SDK will help you generate SSH keys that are required to access the compute instances for your project.Usually the keys are associated with your login account when generated using the cloud SDK. Also checkout Connecting to Instance.

  • SSH into the server and install java and unzip so that you can set up the server:

You can also use the Red5 Pro installer to automate your installation and save time. The installer will install all the necessary dependencies for a regular setup and then subsequently allow you to install Red5 Pro from your Red5 Pro account. For more information check out Red5 Pro Installer on Github.

Also, make sure that the server time zone is UTC (this should be the default on Google Cloud instances). Type date at the prompt. This will return the date, time and zone (eg:Tue Dec 13 20:21:49 UTC 2016). If you need to switch to UTC, execute sudo dpkg-reconfigure tzdata, scroll to the bottom of the Continents list and select None of the above; in the second list, select UTC.

Configure Autoscaling on the Instance:

Navigate to the location where you extracted the red5pro files, and edit the file {red5prohome}/conf/autoscale.xml

    <bean name="config" class="com.red5pro.clustering.autoscale.Configuration" >
        <property name="active" value="false"/>

        <!--Stream manager hosted uri. use the host of your stream manager.  -->
                <property name="cloudWatchHost" value="http://0.0.0.0:5080/streammanager/cloudwatch"/>
  • Change the active value from “false” to “true
  • Replace 0.0.0.0 with the static ip (elastic ip) you reserved for stream manager previously or the host name associated with the IP address.

NOTE: if you are going to be using a secure URL for pointing to a load balancer or for standard WebRTC usage, then you should use that (for example: <property name="cloudWatchHost" value="https://my.streammanager.com/streammanager/cloudwatch"/>)

  • Save and close the file

Set a Unique Cluster Password:

Edit the file {red5prohome}/conf/cluster.xml and replace the default <!-- edge/origin link cluster password --> password from changeme to sometime unique. Make a note of this password, to be used in the Stream Manager red5.properties file.

NOTE: The cluster password can not contain any CAPITAL LETTERS at this time

Remove Unnecessary Webapps:

To ensure a faster startup time for your server, we suggest deleting any webapps that you will not be using from the {red5pro}/webapps directory (for example: secondscreen, template, vod, streammanager; do not remove root). For ease of setup for the Stream Manager, leave a zipped copy of the server distribution on the server.

Verifying Red5 Pro on Instance:

Start Red5 Pro service sudo systemctl start red5pro

Once you have started the Red5 Pro Server, you can verify that it is running and available by opening a web browser and navigating to http://xxx.<instance-ip>.com:5080, where <instance-ip> is the Public IP address of your instance.

After you've verified, stop the Red5 Pro service sudo systemctl stop red5pro

Create Disk Image

To create a disk image, you will need to delete the instance you just created, while retaining the boot disk. From the Google Cloud Dashboard, edit the VM instance and deselect “Delete boot disk when instance is deleted," then click on Save to save this new configuration.

bootdisk

Now, delete the instance and confirm that you are not deleting the boot disk - leave the checkbox empty.

bootdisk2

From Compute Engine, Images:

  • Click on [+] CREATE IMAGE
  • Give the image a name and make a note of it in you Google Cloud Compute AutoScaling Checklist. This will be used in the launchconfig default.json. We suggest naming it per the build version number for easy reference. Note that the name can only contain letters, numbers and hyphens.
  • Source = Disk.
  • Source disk: use the pull-down to select the disk from the instance you just configured and deleted.
  • Click on Create

10acreateimage

5. Stream Manager Configuration

Create a new VM instance using the image you created above.

It is recommeended that you select a machine type with a minimum of 4 vcpu and 8 GiB Memory for acceptable performance for the Stream Manager.

From Compute Engine, VM Instances:

  • Create an instance
  • Choose the same Zone where your MySQL database was assigned
  • Boot disk - Change - Your image tab, select the server image you just created
  • Boot disk type: Standard persistent disk
  • IMPORTANT: - Compute Engine default service account, Allow full access to all Cloud APIs
  • On networking tab:
  • You can choose the default network profile that you set up for Red5 Pro, or if you want to be more restrictive, the Stream Manager only needs ports 22, for ssh access, and 5080 open. If you are using the Stream Manager as an SSL proxy, then you also need to open port 8083.
  • Under External IP, choose the static IP that you reserved from the Networking tab.

createsm

Copy the google-cloud-controller.jar file up to the server (gcloud compute scp google-cloud-controller.jar stream-manager-01:/tmp/)

The google cloud SDK will help you generate SSH keys that are required to access the compute instances for your project.Usually the keys are associated with your login account when generated using the cloud SDK. Also checkout Connecting to Instance.

SSH into the stream manager instance (gcloud compute ssh stream-manager-01)

Stop the Red5 Pro service (sudo systemctl stop red5pro)

Install NTP (network time protocol)

NTP is necessary to ensure that the Stream Manager and all nodes are in sync.

sudo apt-get install ntp

The service should start automatically once installed.

Also, make sure that the server time zone is UTC (this should be the default on Google Cloud instances). Type date at the prompt. This will return the date, time and zone (eg:Tue Dec 13 20:21:49 UTC 2016). If you need to switch to UTC, execute sudo dpkg-reconfigure tzdata, scroll to the bottom of the Continents list and select None of the above; in the second list, select UTC.

Remove Autoscale Files and WebRTC Plugin

Delete the following files:

  • {red5prohome}/conf/autoscale.xml
  • {red5prohome}/plugins/red5pro-autoscale-plugin-<version>.jar
  • {red5prohome}/plugins/red5pro-webrtc-plugin-<version>.jar

Import Cloud Controller and Activate

Copy the google-cloud-controller.jar into {red5prohome}/webapps/streammanager/WEB-INF/lib/

Edit the applicationContext.xml file, located at {red5prohome}/webapps/streammanager/WEB-INF/applicationContext.xml,

Locate the google controller “bean” and uncomment it as shown below (note: do not modify the values, only uncomment the bean configuration to make it active):

    <!-- GOOGLE COMPUTE CONTROLLER -->
    <bean id="apiBridge" class="com.red5pro.services.cloud.google.component.ComputeInstanceController" init-method="initialize">
       <property name="project" value="${compute.project}"/>
       <property name="defaultZone" value="${compute.defaultzone}"/>
       <property name="defaultDiskType" value="${compute.defaultdisk}"/>
       <property name="operationTimeoutMilliseconds" value="${compute.operationTimeoutMilliseconds}"/>
    </bean>

Comment out (or delete the entry for) the default controller as shown below to disable it:

<!-- Default CONTROLLER -->
<! --
<bean id="apiBridge" class="com.red5pro.services.cloud.sample.component.DummyCloudController" init-method="initialize">
</bean>
 -->

Modify Stream Manager App Properties (red5-web.properties)

(For detailed information, see Red5 Pro Stream Manager User Guide)

The Stream Manager’s configuration details are stored in the red5-web.properties file, found in: {red5prohome}/webapps/streammanager/WEB-INF/red5-web.properties. This is where streammanager reads all its settings from. Each configurable setting is organized into its own section.

You will need to modify the following values:

DATABASE CONFIGURATION SECTIONu>

  • config.dbHost={host} -- the IP address of your MySQL server instance
  • config.dbUser={username} -- username you set to connect to the MySQL instance
  • config.dbPass={password} -- password used to connect to the MySQL instance

NODE CONTROLLER CONFIGURATION SECTION - MILLISECONDS

  • instancecontroller.replaceDeadClusters=false -- by default, any cluster that is becomes unresponsive will be shut down. Setting this value to true will automatically replace any clusters that have failed
  • instancecontroller.deleteDeadGroupNodesOnCleanUp=true -- by default, any unresponsive nodes will be deleted from the dashboard. Setting this value to false will stop the instances, but not delete them.

CLUSTER CONFIGURATION INFORMATION

  • cluster.password=changeme -- modify this to be the same as the password that you set in the cluster.xml file on your disk image.

LOADBALANCING CONFIGURATION

  • streammanager.ip={streammanager-static-ip} -- The static IP address used for stream manager. This parameter is optional for a single stream manager setup. This is required when you wish to setup multiple stream managers behind a load balancer.

GOOGLE COMPUTE CLOUD CONTROLLER CONFIGURATION

You will need to comment out the following entries:

  • compute.project={project-id} -- your Google Cloud project ID
  • compute.defaultzone={zone-id} -- the default zone for your Google Cloud project
  • compute.defaultdisk=pd-standard -- do not modify this value
  • compute.operationTimeoutMilliseconds=20000 -- estimated time to start a new VM. We do not recommend modifying this value.

REST SECURITY SECTION

  • rest.administratorToken= -- You need to set a valid password string here before you start using streammanager. This is the password that you will use to execute API commands

RED5PRO NODE DEFAULT APPLICATION

  • webapp which the stream manager uses for checking node cluster status. The default value is live

WEBSOCKET PROXY SECTION

  • proxy.enabled set to true enables, or set to false disables the websocket proxy service. You must use the proxy if you are using WebRTC with Red5 Pro autoscaling.

DEBUGGING CONFIGURATION SECTION

  • debug.logaccess -- Set to true if you want to allow access to log files via REST API. This can be specially useful during development on cloud. With log access enabled you can use the Stream Manager REST api to download log files with using SSH. For more info on how to use the log access api refer to the Stream Manager Rest API.

Please note that if you modify any of the above values after your initial deployment, you will need to restart the Red5 Pro service.

ALARM THRESHOLD (no longer in the properties file)

The autoscaling alarm threshold is no longer set in the red5-web.properties file. Instead, the default value is 60%. If you want to modify this value, do so directly after node group creation using the Rest API for alarms calls. You can set different thresholds for origins and edges via the rest API.

Sample red5-web.properties file content:

 ## RED5 APP CONFIGURATION SECTION - Do Not Tamper
webapp.contextPath=/streammanager
webapp.virtualHosts=*

## DATABASE CONFIGURATION SECTION
config.dbHost=192.168.0.100
config.dbPort=3306
config.dbUser=admin
config.dbPass=aBcD12345EfGhijk

## DATA STORE MANAGEMENT CONFIGURATION SECTION
store.usageStatsDiscardThresholdDays=7

## NODE CONTROLLER CONFIGURATION SECTION - MILLISECONDS
instancecontroller.newNodePingTimeThreshold=150000
instancecontroller.replaceDeadClusters=true
instancecontroller.deleteDeadGroupNodesOnCleanUp=true
instancecontroller.instanceNamePrefix=node
instancecontroller.nodeGroupStateToleranceTime=180000
instancecontroller.nodeStateToleranceTime=180000
instancecontroller.cloudCleanupInterval=180000
instancecontroller.blackListCleanUpTime=600000
instancecontroller.pathMonitorInterval=30000
instancecontroller.minimumNodeFreeMemory=50

## METRIC WEIGHTS FOR BEST NODE EVALUATION SECTION
instanceevaluator.streams.metricweight=30
instanceevaluator.connections.metricweight=15
instanceevaluator.subscribers.metricweight=60
instanceevaluator.memory.metricweight=20
instanceevaluator.restreamer.metricweight=35

## CLUSTER CONFIGURATION INFORMATION
cluster.password=changeme
cluster.publicPort=1935
cluster.accessPort=5080
cluster.reportingSpeed=10000
cluster.retryDuration=30
cluster.mode=auto
cluster.idleClusterPathThreshold=30000

## LOADBALANCING CONFIGURATION
streammanager.ip=

## LOCATIONAWARE CONFIGURATION
location.region=
location.geozone=
location.strict=false

## CLOUD CONTROLLER CONFIGURATION SECTION  - MILLISECONDS

## AWS CLOUD CONTROLLER CONFIGURATION ##
#aws.defaultzone={default-region}
#aws.operationTimeoutMilliseconds=200000
#aws.accessKey={account-accessKey}
#aws.accessSecret={account-accessSecret}
#aws.ec2KeyPairName={keyPairName}
#aws.ec2SecurityGroup={securityGroupName}
#aws.defaultVPC={boolean}
#aws.vpcName={vpcname}
#aws.faultZoneBlockMilliseconds=3600000
#aws.forUsGovRegions=false

## AZURE CLOUD CONTROLLER CONFIGURATION ##
#az.resourceGroupName={master-resourcegroup}
#az.resourceGroupRegion={master-resourcegroup-region}
#az.resourceNamePrefix={resource-name-prefix}
#az.clientId={azure-ad-application-id}
#az.clientKey={azure-ad-application-key}
#az.tenantId={azure-ad-id}
#az.subscriptionId={azure-ad-subscription-id}
#az.vmUsername=ubuntu
#az.vmPassword={password-to-set-for-dynamic-instances}
#az.defaultSubnetName=default
#az.operationTimeoutMilliseconds=120000
#az.quickOperationResponse=true
#az.quickResponseCheckInitialDelay=20000
#az.apiLogLevel=BASIC

## GOOGLE COMPUTE CLOUD CONTROLLER CONFIGURATION ##
compute.project=root-random-131129
compute.defaultzone=us-east1
compute.defaultdisk=pd-standard
compute.operationTimeoutMilliseconds=20000

## SIMULATED-CLOUD CONTROLLER CONFIGURATION ##
#managed.regionNames={region-name}
#managed.availabilityZoneNames={zone-name}
#managed.operationTimeoutMilliseconds=20000
#managed.recycleDeadNodes=true

## LIMELIGHT-CLOUD CONTROLLER CONFIGURATION ##
#limelight.regionNames={region-name}
#limelight.availabilityZoneNames={zone-name}
#limelight.operationTimeoutMilliseconds=20000
#limelight.port=
#limelight.user=
#limelight.pwd=
#limelight.recycleDeadNodes=true

## REST SECURITY SECTION
rest.administratorToken=xyz123

## RED5PRO NODE DEFAULT APPLICATION
default.server.application=live

## RED5PRO NODE SERVER API SECTION
serverapi.port=5080
serverapi.protocol=http
serverapi.version=v1
serverapi.accessToken=xyz123

## DEBUGGING CONFIGURATION SECTION
debug.logaccess=true
debug.logcachexpiretime=60000

## WEBSOCKET PROXY SECTION
proxy.enabled=true

Start Red5 Pro Service to Use the Stream Manager

sudo systemctl start red5pro

Optional: Load-Balance Multiple Stream Managers

Prerequisites:

  • One reserved elastic IP address for each Stream Manager.
  • A registered Domain Name to associate with the reserved Load Balancer IP address.
  • Create the first Stream Manager per the above instructions, then create a snapshot from that instance. Build the second stream manager from that snapshot. It is essential that the config files be identical between the two stream managers with one exception:
  • Edit red5pro/webapps/streammanager/WEB-INF/red5-web.properties and modify ## LOADBALANCING CONFIGURATION streammanager.ip=, adding the Assigned IP address of the individual Stream Manager instance you are modifying.
  • Add all Stream Manager public IP addresses to the Database security group.

Under Networking tab choose Load balancing.

loadbalancer01

Click on + CREATE LOAD BALANCER

Choose TCP Load Balancing, click on Start configuration.

loadbalancer02

Internet facing or internal only - choose From internet to my VMs; Connection termination - Do you want to offload SSL processing to the Load Balancer? - choose Yes (SSL Proxy) if you have an SSL cert; otherwise choose No (TCP). Click on Continue

loadbalancer03

Name your load balancer (eg, streammanager-loadbalancer), then click on Backend configuration.

loadbalancer04

Backend configuration: Choose the region where your stream managers are from the pull-down. Click on Select existing instances tab and add your two stream managers.

loadbalancer06

Create a health check: name your health check, and modify HTTP to use port 5080 (Red5 Pro default). You can make the healthy/unhealty threshold (to remove/re-add servers in the pool accordingly) as aggressive as you like.

loadbalancer05

Frontend configuration: click on Create IP address to Reserve a new static IP for the Load Balancer

loadbalancer07 loadbalancer08

Review and finalize: look over the details, then click on Create

IMPORTANT You will need to create a new disk image - create a new VM from the original disk image, and modify {red5prohome}/conf/autoscale.xml to point to the Load Balancer IP address, then create a new disk image from this VM to use for your nodes.

Set Scaling Policy

With Stream Manager API v3.0, the Scaling and Launch policies are set via the API (see scale policy management and launch policy management).

Using a tool like Postman, you must set a scaling policy before creating any node groups. The policy supports two optional attributes per role target - scaleInWaitTime & scaleOutWaitTime. These are optional parameters denoting delayed scale-in/scale-out time in milliseconds. The attributes require positive values (>=0). If the attribute is omitted it defaults to 0. For more information see scale policy description.

You will need to create a scaling policy that includes each node type that you want in your setup. If you want to take advantage of the new features (multi-bitrate streaming, adaptive bitrate subscribing, and dynamic clustering) you need to have a minimum of two each origin, edge, and relay nodes. If you want to include VP8 transcoding support, then you also need two transcoder nodes.

POST call: https://<streammanager_URL>/streammanager/api/3.1/admin/configurations/scalepolicy?accessToken=<accessToken>

Data (make sure to select JSON as the body type):

   {
    "policy": {
        "name": "<policy-name>",
        "description": "<policy-description>>",
        "version": "<policy-version>",
        "type": "<policy-type>",
    "targets": {
      "target": [
        {
          "role": "<role>",
          "minLimit": "<min-node-count>",
          "maxLimit": "<max-node-count>",
          "scaleAdjustment": "<node-scale-adjustmant>"
        }
      ]
    }
    }}

Example:

REQUEST: https://<streammanager_URL>/streammanager/api/3.1/admin/configurations/scalepolicy?accessToken=xyz123

Data:

   {
      "policy": {
          "name": "default-v3",
          "description": "Sample Scale Config policy with all node types",
          "type": "com.red5pro.services.autoscaling.model.ScalePolicyMaster",
          "version": "0.0.3",
          "targets": {
              "region": [
                  {
                      "name": "default",
                      "target": [
                          {
                              "role": "edge",
                              "maxLimit": 20,
                              "scaleAdjustment": 1,
                              "minLimit": 2
                          },
                          {
                              "role": "origin",
                              "maxLimit": 10,
                              "scaleAdjustment": 1,
                              "minLimit": 2
                          },
                          {
                              "role": "relay",
                              "maxLimit": 10,
                              "scaleAdjustment": 1,
                              "minLimit": 2
                          },
                          {
                              "role": "transcoder",
                              "maxLimit": 10,
                              "scaleAdjustment": 1,
                              "minLimit": 2
                          }
                      ]
                  }

Set Launch Configuration Policy

You will need to create a launch policy that includes each node type that you want in your setup. If you want to take advantage of the new features (multi-bitrate streaming, adaptive bitrate subscribing, and dynamic clustering) you need to have a minimum of two each origin, edge, and relay nodes. If you want to include VP8 transcoding support, then you also need two transcoder nodes. You will need the name of the ami that you created in step 7 above. You can choose different instance types and capacities for each node type if you wish.

POST call: https://<streammanager_URL>/streammanager/api/3.1/admin/configurations/launchconfig?accessToken=<accessToken>

Data (make sure to select JSON as the body type):

   {
    "launchconfig": {
      "name": "<configuration-name>",
      "description": "<configuration-descrption>",
      "image": "<red5pro-image>",
      "version": "0.0.2",

    "targets": {
        "target": [
      {
        "role": "<role>",
        "instanceType": "<instance-type>",
        "connectionCapacity": "<instance-capacity>"
      }
      ]
      },

      "properties": {
        "property": [
          {
            "name": "<property-name>",
            "value": "<property-value>"
          }
        ]
      },
      "metadata": {
        "meta": [
          {
            "key": "<meta-name>",
            "value": "<meta-value>"
          }
        ]
      }
    }}

Example:

REQUEST: https://<streammanager_URL>/streammanager/api/3.1/admin/configurations/launchpolicy?accessToken=xyz123

Data:

     {
    "launchconfig": {
      "name": "all-nodes-launch",
      "description": "Sample Launch Config with all four nodetypes",
      "image": "red5pro-image-name",
      "version": "0.0.3",

      "targets": {
          "target": [
        {
          "role": "origin",
          "instanceType": "n1-standard-2",
          "connectionCapacity": "2000"
        },
        {
          "role": "edge",
          "instanceType": "n1-standard-2",
          "connectionCapacity": "2000"
        },
        {
          "role": "relay",
          "instanceType": "n1-standard-2",
          "connectionCapacity": "2000"
        },
        {
          "role": "transcoder",
          "instanceType": "n1-standard-4",
          "connectionCapacity": "2000"
        }
         ]
        },

        "properties": {
          "property": [
            {
              "name": "property-name",
              "value": "property-value"
            }
          ]
        },
        "metadata": {
          "meta": [
            {
              "key": "meta-name",
              "value": "meta-value"
            }
          ]
        }
      }}

API Basics, to get you started

(for full API, see Red5 Pro Stream Manager API)

For the POST calls, we recommend using a tool like Postman.

Verify Stream Manager is Using the Correct Controller

GET call: https://<streammanager_URL>/streammanager/api/3.1/admin/debug/cloudcontroller?accessToken=<rest.administratorToken>

should return: Amazon Web Services

Create a Scale Policy (post)

  • Make sure to include a role section for each node type you want to include (origin, edge, relay, transcoder):

POST call: https://<streammanager_URL>/streammanager/api/3.1/admin/configurations/scalepolicy?accessToken=<rest.administratorToken>

Data (make sure to select JSON as the body type):

   {
  "policy": {
          "name": "<policy-name>",
          "description": "<policy-description>>",
          "type": "<policy-type>",
          "version": "<policy-version>",
          "targets": {
              "region": [
                  {
                      "name": "default",
                      "target": [
        {
          "role": "<role>",
          "minLimit": "<min-node-count>",
          "maxLimit": "<max-node-count>",
          "scaleAdjustment": "<node-scale-adjustment>"
        }
      ]
    }
    }}

Create a Launch Policy (post)

For ease of administration, we recommend naming the policy something easily identifiable and descriptive, such as <imagename>-<nodetypes> (e.g., red5pro4.6.0-allnodetypes)

POST call: https://<streammanager_URL>/streammanager/api/3.1/admin/configurations/launchconfig?accessToken=<rest.administratorToken>

Data (make sure to select JSON as the body type):

   {
    "launchconfig": {
      "name": "<configuration-name>",
      "description": "<configuration-descrption>",
      "image": "<red5pro-image>",
      "version": "0.0.3",

    "targets": {
        "target": [
      {
        "role": "<role>",
        "instanceType": "<instance-type>",
        "connectionCapacity": "<instance-capacity>"
      }
      ]
      },

      "properties": {
        "property": [
          {
            "name": "<property-name>",
            "value": "<property-value>"
          }
        ]
      },
      "metadata": {
        "meta": [
          {
            "key": "<meta-name>",
            "value": "<meta-value>"
          }
        ]
      }
    }}

Create a New Node Group (post)

POST call: https://<streammanager_URL>/streammanager/api/3.1/admin/nodegroup?accessToken=<rest.administratorToken>

Data (make sure to select JSON as the body type):

{
 "regions": [
   "<region, eg: us-west2>"
 ],
 "launchConfig": "<launchconfig-created-above",
 "scalePolicy": "<scaleconfig-created-above>"
}

Note the cluster “name” that is returned by the above call. It will be used to create a new Origin server.

Launch New Origin (post)

After you create a node group, create the origin server. Creating an origin server will also generate at least one edge, per scaling policy min limit.

https://<streammanager_URL>/streammanager/api/3.1/admin/nodegroup/<cluster-name>/node/origin?accessToken=<rest.administratorToken from red5-web.properties file>

NOTE: If you wish to launch more than one origin, you can repeat the call. The maximum origins allowed will depend on the maxLimit attribute of the 'origin' object described in scale policy. If 'origin' is omitted from the scale policy then the default value for maximum origins is 1.

Set Alarm Threshold (POST)

By default, the alarm threshold (the capacity percentage at which the cluster will scale up any given node type) is set to 60%. To modify this, POST the following:

FOR EDGE:

https://<streammanager_URL>/streammanager/api/3.1/admin/alarm/scaleout/default?type=edge&threshold=<threshold>&accessToken=<rest.administratorToken>

FOR ORIGIN:

https://<streammanager_URL>/streammanager/api/3.1/admin/alarm/scaleout/default?type=origin&threshold=<threshold>&accessToken=<rest.administratorToken>

FOR RELAY:

https://<streammanager_URL>/streammanager/api/3.1/admin/alarm/scaleout/default?type=relay&threshold=<threshold>&accessToken=<rest.administratorToken>

FOR TRANSCODER:

https://<streammanager_URL>/streammanager/api/3.1/admin/alarm/scaleout/default?type=transcoder&threshold=<threshold>&accessToken=<rest.administratorToken>

LIST GROUPS (get)

https://<streammanager_URL>/streammanager/api/3.1/admin/nodegroup?accessToken=<rest.administratorToken from red5-web.properties file>


Stream Manager Publish and Subscribe Examples

Stream Manager Proxy Publish and Subscribe Examples

With the latest release, the live webapp includes two examples: proxy-publisher.html and proxy-subscriber.html. These examples will take the following query parameters:

Name Description Default Value
host hostname or IP window.location.hostname
protocol protocol which Stream Manager is served over (http or https) window.location.protocol
port port number that Stream Maager is served on window.location.port
app webapp name to stream to on the server live
streamName The unique stream name to broadcast with or subscribe to None. Required
verbose Flag to enable verbose logging in Dev Console None. optional
view Target broadcast tech (rtc, rtmp or hls) None. Optional

Example URI: https://streammanager.test.com/live/proxy-publisher.html?streamName=stream1&verbose=1

Red5 Pro HTML5 SDK Examples:

Stream Manager examples

Note: the streaming-html5 examples testbed is included with the Red5 Pro server distribution, and can be accessed via your stream manager at https://your.server.url/webrtcexamples/.

Red5 Pro iOS SDK Examples:

Publish - Stream Manager

Subscribe - Stream Manager

Red5 Pro Android SDK Examples:

Publish - Stream Manager

Subscribe - Stream Manager

Troubleshooting

  1. You can see the role of a node in the Custom metadata on an instance's property page custommeta
  2. If you have created a new nodegroup, but adding an Origin server doesn't automatically spin up an accompanying Edge server, check the following on your node image:

  3. Verify that red5pro/conf/autoscale.xml in your Red5Pro server image points to the IP address of your stream manager (replace 0.0.0.0 in <property name="cloudWatchHost" value="http://0.0.0.0:5080/streammanager/cloudwatch"/>); make sure that you the change active value from default “false” to “true” (<property name="active" value="true"/>)

  4. Check {red5pro}/conf/cluster.xml - make sure that the password set for <!-- edge/origin link cluster password --> matches the cluster.password= in the stream manager red5-web.properties file.
  5. Make sure that your have your LICENSE.KEY file in the {red5pro} root directory; clustering (and in turn, autoscaling) will not work without a valid license key.
  6. If all of these check out, then start up red5pro service on your ami instance, and tail the Stream Manager {red5pro}/log/red5.log file - you should see messages about an instance pinging the Stream Manager. If you don't, then check for inbound port access to your stream manager (or outbound port access from your node security group).