Deploying Stream Manager and Autoscaling on Amazon Web Services (AWS)

Overview:

This document assumes that you have some basic knowledge of AWS EC2 management. It also assumes that you have some basic linux administration skills. If you need more detailed information, please contact us.

In order to use the Red5 Pro Stream Manager service you will need the following:

  1. Latest Red5 Pro Server build
  2. The aws-cloud-controller.jar, from the Red5 Pro Autoscaling Library Extensions section
  3. An active Red5 Pro license key (needed for clustering) REGISTER HERE
  4. An active AWS account

Before you Begin:

NOTE: Because of the structure of the AWS EC2 many of the steps in this process need to be executed against every region in which you choose to include autoscaling.

You will want to keep a record of the usernames, passwords, IP addresses, and other information generated during the setup process, as you will need the information for stream manager configuration and future operations via the API. Click here to download a handy list for tracking all of your Red5 Pro Autoscaling values

1. Get Security Credentials (IAM)

IAM

Obtain security credentials for stream manager AWS account access:

  • Go to the Amazon Identity and Access Management Dashboard
  • Click on “Create individual IAM Users” => “Manage Users” createindividual
  • Click on “Create New Users”
  • Enter the username "streammanager" - make sure that the "generate access key for each user" box is checked
  • Click “Create” button to create credentials showusercreds
  • Click “Show User Security Credentials” to reveal the access and secret keys generated. It is critical that you make a note of these, as you will not be able to retrieve them from the AWS console if you happen to lose them.
  • Click “download Credentials” button to download a csv file with the credentials to your computer. You will not have this option again, so keep this file in a safe place!
  • Click “close” to close the window and return to IAM Users screen.
  • You should be able to see the user you recently created - “streammanager” in the list on the page.

Give the stream manager account full access to EC2:

  • Click on the new user you just created (streammanager).
  • Click on Permissions tab.
  • Click on Attach Policy.
  • Select the Policy Name - AmazonEC2FullAccess, and click on Attach Policy.
  • The credentials you created earlier can now be used for EC2 specific operations. policy

2. Create EC2 SSH2-RSA Keypair

This Keypair will be used for Instance Authorizing Operations Across All Regions

Amazon EC2 key pairs are region-specific, so you will need to manually create an SSH2-RSA key pair to be used by the stream manager across all regions.

Generate Your SSH2-RSA Key Pair:

On Mac/Linux:

  • Open Terminal
  • Type: ssh-keygen -t rsa
  • Accept the default path and modify the file name if you wish (Enter file in which to save the key (~/.ssh/id_dsa)
  • Enter a passphrase and confirm (make sure this is SECURE, and noted somewhere for future reference)
  • Private and public keys will be generated

On Windows:

  • Download and run PuTTYgen
  • Click on “Generate” button and follow instructions displayed to help PuTTYgen generate your private public key-pair.
  • Once PuTTYgen finishes generating keys it will display the public key and other details in the application window.
  • Enter a passphrase and confirm. (make sure this is SECURE, and noted somewhere for future reference)
  • Click “Save public key” and save the key with a filename (ex: red5pro_node) on your file system in a secure location (or a common standard folder such as ~/.ssh/ under your user profile). Provide an extension of .pub to the file (public key).
  • Click “Save private key” and save the key with the same filename as the public key (ex: red5pro_node) on your file system in a secure location (or a common standard folder such as ~/.ssh/ under your user profile). Provide an extension of .ppk to the file (private key).

Import Public Key to EC2 Regions:</U>

Note: you will need to repeat this process in every region you wish to include in your autoscaling solution.

To Import your public key:

  1. Navigate to the EC2 Dashboard
  2. Copy your public key to a non-hidden directory (you will not be able to find the .ssh directory in the browse to import step)
  3. In the left-side navigation, under NETWORK & SECURITY click on Key Pairs
  4. Click on Import Key Pair button to open import window. Browse and select the public key (.pub) you just created.
  5. Click import button to finish importing the key pair into your current region.
  6. Navigate to each other region that you wish to include in autoscaling and repeat steps 3-5

importkey

3. Reserve Elastic (Static) IP for Stream Manager

It is critical that the Stream Manager have a static IP address, so that in the event that the instance is rebooted, it will retain the same public IP address.

To reserve an elastic IP address:

  • Navigate to the EC2 Dashboard
  • Ensure that you are in the region where you wish to setup Stream Manager (you can change region from the region selector located in top right section of your AWS console).
  • In the left-side navigation, under NETWORK & SECURITY, click on Elastic IPs
  • Click on Allocate New Address to bring up confirmation dialog box. Choose EIP used in: VPC
  • Click “Yes, Allocate” to reserve an IP address.
  • After an IP is reserved the confirmation dialog will display it. Make a note of this IP address, to be used by the stream manager. eipinvpc

4. Create VPCs and Security Groups

To be used for Red5 Pro Autoscaling Node Groups

Note: you will need to repeat this entire four-step process in every region you wish to include in your autoscaling solution.

Create VPC

  • Navigate to the VPC Dashboard
  • Click on Start VPC Wizard
  • Step 1: Select a VPC Configuration - Choose the first/default option (VPC with a Single Public Subnet) and click on Select
  • Step 2: VPC with a Single Public Subnet - Accept all defaults. Give your VPC a name to identify it.
  • Availability zone: choose the first one (you will add the others to this VPC shortly) createVPC
  • Click on Create VPC

Modify the Subnet Properties

  • From left-hand navigation, under Virtual Private Cloud, choose Subnets
  • Chose the subnet that was just created for your VPC
  • Click on Subnet Actions, and choose “Modify Auto-Assign Public IP” - add a checkmark in the box, and click on Save
  • Click on the Route Table tab for this subnet, and make sure there is an entry for 0.0.0.0/0 - the target should be igw-(8 alpha-numeric characters)
  • Make a note of this route table id - you will need to use it for the other zones that you add to your VPC routetable

Add Additional Subnets to the VPC, for Each Zone

Repeat the following for all zones that are available in the selected region:

  • From the left-hand navigation, under Virtual Private Cloud, choose Subnets
  • Click on Create Subnet
  • Name Tag: Type in a name tag for your reference
  • VPC: choose the VPC you just created
  • Availability Zone: choose the next availability zone in your region
  • CIDR block: if the first public subnet you created was 10.0.0.0/24, then make this next one 10.0.1.0/24, the next one 10.0.2.0/24, etc.
  • Click on Subnet Actions, and choose “Modify Auto-Assign Public IP” - add a checkmark in the box, and click on Save
  • Click on the Route Table tab for this new subnet
  • Click on Edit
  • "Change to:" select the route table id of the first subnet (this will add the igw-xxxxxxxx target to this subnet)
  • Click on Save

addzones changeroute

Create Security Group with Red5 Pro Ports

IMPORTANT: The security group name will need to be unique within an individual region, and then repeated across all of the regions you with to include in your autoscaling solution. The stream manager uses the security group name to identify where to create a new node.

  • From left-hand navigation, under Security, choose Security Groups
  • Click on Create Security Group
  • Fill in the the group name / name tag to be used across all regions (this will be the aws.ec2SecurityGroup in the Stream Manager’s red5-web.properties file)
  • VPC - select the VPC that you just created.
  • Click on Yes, Create newsecuritygroup
  • Select the security group you just created, and click on the Inbound Rules tab
  • Click on Edit
  • Add the following ports for Red5 Pro functionality. Optionally, you can also add port 22 if you wish to be able to SSH into instances (for troubleshooting)
  • For source range, type in 0.0.0.0/0 for each of the following ports:
Port Description
5080 default web access of Red5 Pro
1935 default RTMP port
8554 default RTSP port
6262 websockets (for HLS)
22 SSH (optional)
8088 second screen client (optional)
8081 websocket (for RTC subscriber)
  • Click Save to save the updates

inboundrules

5. Prepare MySQL Database

NOTE: For Optimal performance, the Stream Manager and Database should be in the same Region

To Create the Database for Autoscaling:

  • Open the AWS RDS console
  • From left-hand navigation, under RDS Dashboard, click on Instances
  • Click on Launch DB Instance
  • Step 1: Select Engine - Click on the MySQL tab and click Select selectenginemysql
  • Step 2: Production? - Select MySQL for production environment use and click the “Next Step” button. (Note: if you are setting this up for development/testing, you can choose that option, but you will need to set up a different Production instance when you go live). production
  • Step 3: Specify DB Details - Set the DB Instance Class as per your traffic needs, keeping in mind that higher configurations account for higher billing charges by AWS. For Storage Type anything other than magnetic helps in performance boost. Stream Manager doesn't not require much storage, but depending on your use-case you may want to allocate more or less space accordingly. The default settings will likely be sufficient for your autoscaling solution.
  • Finally set a database instance identifier and database credentials for access by the Stream Manager. Make a note of the username and password. mysqlconfig
  • Step 4: Configure Advanced Settings -

Network & Security:

  • VPC - select the VPC that you created for autoscaling
  • Subnet Group: Create new DB Subnet Group (or use default)
  • Publicly Accessible: Yes
  • Availability Zone: No Preference
  • VPC Security Group(s): Create new Security Group (We will be configuring the security group for RDS next)

Database Options:

  • Database Name: cluster
  • Leave other options as their default values.
  • Backup, Monitoring, and Maintenance options should be configured per your operations' requirements.
  • Click Launch DB Instance dbadvanced
  • Click on the View your DB Instances.
  • You should now be able to see your db instance being created. It will take several minutes (you can stretch your legs now) for the initialization process to be completed. creating
  • When Status has changed from "creating" to "available" select your database instance and click on configuration details tab icon (middle icon).
  • Make a note of the Endpoint url (will be something like streammanager-db.abcdefghijk123.us-west-1.rds.amazonaws.com) of the mysql database since this will be used for the config.dbHost variable in the stream manager red5-web.properties file.
  • On this tab, locate and click on the Security Group for this instance to navigate to the security group configuration page. configdetails

Configure Stream Manager access to this security group:

  • Select the security group (Description will be "Created from the RDS Management Console") and click on “Actions” => “Edit inbound rules” option.
  • Click Add Rule
  • Select - “MySql/Aurora” for Type (port 3306)
  • Select - “Custom IP” in source and enter the stream manager's elastic IP address with its CIDR. (typing a “ / ” after the ip address will auto suggest CIDR values for you select)
  • Additionally, you will need to add the private IP address of the Stream Manager to the security group - this will be assigned after the Stream Manager instance is associated with the reserved IP address.
  • For source, select “My IP” to add your IP address to the whitelist of the RDS security group. This will ensure that you can connect to MySql from your desktop and populate the database schema using a RDS client. When you select My IP in source amazon will auto fill in your CIDR too. (note: AWS may have already added this by default)
  • Click “Save” to apply changes to this security group. Your stream manager instance will now be able to connect to the mysql instance after it has been deployed.

6. Connect to MySQL and Add Cluster Schema

To connect to the mysql instance which we just created, we suggest using the freeware MySQL workbench tool, and can be downloaded from https://dev.mysql.com/downloads/workbench/.

  • Launch MySQL Workbench tool on your system
  • Click on the “+” icon to the right of MySQL Connections to open the new connection configuration window.
  • Enter a connection name - (for example, “aws-mysql”)
  • Hostname: the database endpoint URL which we obtained from RDS instance details tab
  • Username: the Database Master Username configured above
  • Password: the Database Master password configured above
  • Default Schema: cluster (database schema for streammanager)
  • Click on Test Connection to verify you can connect. - if you are not on the system that was added as "my IP" when you created the RDS security policy, this may fail. mysqlworkbench
  • Click “ok” to save configuration and close window.
  • Double click on the new connection you created to open database workspace view.
  • Download the Red5 Pro Server Distribution, and unzip to your desktop.
  • From menu click => File => Open SQL Script, and browse the server distribution to find the cluster.sql file {red5prohome}/webapps/streammanager/WEB-INF/sql/cluster.sql
  • Click the execute icon to execute the sql script which populates the cluster database execute clusterdbpopulated

7.Prepare Red5 Pro AMI for Nodes

Note: After creating the server AMI, you will need to copy it to every region you wish to include in your autoscaling solution, per https://aws.amazon.com/blogs/aws/ec2-ami-copy-between-regions/

Note #2: The AWS platform does allow multiple images of same name. Please make sure to use a unique name when creating the image to be used for autoscaling. If your region has more than one resource of the same name, the first one in the fetch images api operation will be used.

Note #3: If you are going to be using load-balanced stream managers, then you need to use the loadbalancer dns name in the autoscale.xml file of your AMI.

Launch an Instance and Install Red5Pro

Start a Basic Instance:

  • Navigate to the EC2 Dashboard
  • From left-hand navigation, under INSTANCES, click on Instances, then Launch Instance
  • Step 1: Choose an Amazon Machine Image (AMI) - Select the “Ubuntu Server 16.04 LTS (HVM)”
  • Step 2: Choose an Instance Type - Select any machine type (free-tier is fine). Click on Next: Configure Instance Details
  • Step 3: Configure Instance Details - Network: choose the VPC that your created for autoscaling in that region. Ensure that “Auto-assign Public IP” is set to Enable; accept all other defaults.
  • Click on Next: Add storage; click on Next: Tag Instance; click on Next: Configure Security Group
  • Step 6: Configure Security Group - choose “Select an existing security group” and choose the group that you created above.
  • Click Review and Launch
  • Click Launch
  • In the final step when prompted, Choose an existing key pair, and select the public key that was created earlier reviewinstance

Install Red5Pro on the Instance:

  • Follow the Installing Red5Pro on an Ubuntu Linux Server instructions
  • Make sure that Red5 Pro service is set to auto-start on launch
  • Connect to your instance using the private key from the key-pair that you created
  • ssh -i ~/.ssh/keys/red5prokey ubuntu@<server_ip_address>
  • Red5 Pro Clustering does not work without a license. You will need to purchase a Red5 Pro Professional license from http://account.red5pro.com. Add a file named LICENSE.KEY with your Pro license key (16 characters, plus dashes, like: YOUR-LIC-ENSE-KEY1) to the root of the Red5 Pro Server installation ({red5prohome}/LICENSE.KEY)

Install NTP (network time protocol)

NTP is necessary to ensure that the Stream Manager and all nodes are in sync.

sudo apt-get install ntp

The service should start automatically once installed.

Also, make sure that the server time zone is UTC (this should be the default on AWS instances). Type date at the prompt. This will return the date, time and zone (eg:Tue Dec 13 20:21:49 UTC 2016). If you need to switch to UTC, execute sudo dpkg-reconfigure tzdata, scroll to the bottom of the Continents list and select None of the above; in the second list, select UTC.

Configure Autoscaling on the Instance:

Navigate to the location where you extracted the red5pro files, and edit the file {red5prohome}/conf/autoscale.xml

    <bean name="config" class="com.red5pro.clustering.autoscale.Configuration" >
        <property name="active" value="false"/>

        <!--Stream manager hosted uri. use the host of your stream manager.  -->
                <property name="cloudWatchHost" value="http://0.0.0.0:5080/streammanager/cloudwatch"/>
  • Change the active value from “false” to “true
  • Replace 0.0.0.0 with the static ip (elastic ip) you reserved for stream manager previously

NOTE: if you are going to be using a secure URL for pointing to a load balancer, then you should use that (for example: <property name="cloudWatchHost" value="https://my.streammanager.com/streammanager/cloudwatch"/>)

  • Save and close the file

Enable Clustering and Set a Unique Cluster Password:

Edit the file {red5prohome}/conf/cluster.xml and replace the default <!-- edge/origin link cluster password --> password from changeme to sometime unique. Make a note of this password, to be used in the Stream Manager red5.properties file.

Edit the {red5pro}/webapps/red5-default.xmlfile. Remove the lines above and below the ClusterServiceResolver line to uncomment it:

 <!-- Uncomment as part of the clustering configuration
 <bean id="clusterServiceResolver" class="com.red5pro.cluster.plugin.ClusterServiceResolver"/>
 -->

Remove Unnecessary Webapps:

To ensure a faster startup time for your server, we suggest deleting any webapps that you will not be using from the {red5pro}/webapps directory (for example: secondscreen, template, vod, streammanager; do not remove root). For ease of setup for the Stream Manager, leave a zipped copy of the server distribution on the server.

Verifying Red5 Pro on Instance:

Start Red5 Pro service sudo /etc/init.d/red5pro start

Once you have started the Red5 Pro Server, you can verify that it is running and available by opening a web browser and navigating to http://xxx.compute-1.amazonaws.com:5080, where xxx.compute-1.amazonaws.com is the Public DNS value of your instance.

After you've verified, stop the Red5 Pro service sudo /etc/init.d/red5pro stop

Create AMI Image from Instance

To create an AMI, to be used by autoscaling for node creation:

  • Navigate to the EC2 Dashboard, click on Running Instances, and select your instance.
  • Click on “Actions” => “Image” => “Create Image”
  • In the “Create Image” popup window enter a unique image name and description and click create image. Leave additional default settings. Make a note of the image name - you will need this for the stream manger launchconfig.
  • You can now stop (or terminate) the instance (NOT the AMI). You will not be charged for a server instance that is live for shorter than 15 minutes.

Copy Red5 Pro AMI to Other Regions

You will need repeat the following steps in each region where you wish to run Red5 Pro autoscaling node groups.

  • Navigate to the EC2 Dashboard
  • From the left-hand navigation, under IMAGES, click on AMIs. You should be able to see your image with status set to “available” (by the name you specified).
  • Select your AMI and click on Actions, Copy AMI
  • Select the Destination Region from the drop-down.
  • By default, the backing snapshot of an AMI will be copied with its original encryption status, so leave that box unselected
  • Click on Copy AMI
  • Choose the Destination Region to copy to
  • Accept the other defaults
  • Click on Copy AMI copyAMI

8.Configure and Deploy Stream Manager on AWS

Note: after the Stream Manager instance is started up, you will need to add its Private IP address to the Database Security group, per the DB security step above

New Server Instance:

  • Navigate to the EC2 Dashboard, in the region where you reserved the streammanager IP address
  • Click on Launch Instance.
  • Step 1: Choose an Amazon Machine Image (AMI) - click on My AMIs, and select the AMI that you just created for the nodes.
  • Step 2: Choose an Instance Type - we suggest selecting a multi-cpu instance type with high network performance (for example, m4.large). Click on Next: Configure Instance Details
  • Step 3: Configure Instance Details - Network: chose the VPC that you set up earlier for this region; accept other defaults. Click on Next: Add Storage.
  • Step 4: Add Storage - the default disk size should be sufficient. Click on Next: Tag Instance, then Next: Configure Security Group
  • Step 6: Configure Security Group - Create a new security group with the following ports allowed from anywhere (optionally, you can restrict SSH access to your network):
Port Description
22 SSH
5080 default web access of Red5Pro / Stream Manager
8083 secure websocket port, necessary if using SSL Proxy
  • Click on Review And Launch, then Launch - Select the keypair that you added, or create a new one to use exclusively on this machine.
  • Make a note of the instance ID - will need this to associate the Elastic IP address.

Associate the reserved IP address with this instance:

  • From the left-hand navigation, under NETWORK & SECURITY, select Elastic IPs
  • Choose the IP address that you reserved
  • Actions, Associate Address
  • Select the Stream Manager instance from the list. Accept the defaults. Click on Associate

Configure the Stream Manager Instance

  • SSH into the Stream Manager instance
  • Stop the Red5 Pro service (sudo /etc/init.d/red5pro stop)

Install NTP (network time protocol)

NTP is necessary to ensure that the Stream Manager and all nodes are in sync.

sudo apt-get install ntp

The service should start automatically once installed.

Also, make sure that the server time zone is UTC (this should be the default on AWS instances). Type date at the prompt. This will return the date, time and zone (eg:Tue Dec 13 20:21:49 UTC 2016). If you need to switch to UTC, execute sudo dpkg-reconfigure tzdata, scroll to the bottom of the Continents list and select None of the above; in the second list, select UTC.

  • Navigate to the directory where you installed Red5 Pro (e.g. /usr/local/red5pro)
  • Delete the following files:
    • {red5prohome}/conf/autoscale.xml
    • {red5prohome}/plugins/red5pro-autoscale-plugin-<version>.jar
    • {red5prohome}/plugins/red5pro-webrtc-plugin-<version>.jar

Import and Activate AWS Cloud Controller

Copy the aws-cloud-controller.jar into {red5prohome}/webapps/streammanager/WEB-INF/lib/

Edit the applicationContext.xml file, located at {red5prohome}/webapps/streammanager/WEB-INF/applicationContext.xml per the following:

  • Locate the AWS controller “bean” and uncomment it as shown below (note: do not modify the values, only uncomment the bean configuration to make it active):
<!-- AWS CONTROLLER -->
<bean id="apiBridge" class="com.red5pro.services.cloud.aws.component.AWSInstanceController" init-method="initialize">
<property name="accessKey" value="${aws.accessKey}"/>
<property name="accessSecret" value="${aws.accessSecret}"/>
<property name="ec2KeyPairName" value="${aws.ec2KeyPairName}"/>
<property name="ec2SecurityGroup" value="${aws.ec2SecurityGroup}"/>
<property name="defaultZone" value="${aws.defaultzone}"/>
<property name="operationTimeoutMilliseconds" value="${aws.operationTimeoutMilliseconds}"/>
</bean>

Comment out (or delete the entry for) the default controller as shown below to disable it:

<!-- Default CONTROLLER -->
<! --
<bean id="apiBridge" class="com.red5pro.services.cloud.sample.component.DummyCloudController" init-method="initialize">
</bean>
 -->

Modify Stream Manager App Properties (red5-web.properties)

Refer to your AWSAutoScalingChecklist.rtf for the data needed to complete this section. For more details on the red5-web.properties file sections, please read here.

The Stream Manager’s configuration details are stored in the red5-web.properties file, found in: {red5prohome}/webapps/streammanager/WEB-INF/red5-web.properties. This is where streammanager reads all its settings from. Each configurable setting is organized into its own section.

You will need to modify the following values:

DATABASE CONFIGURATION SECTION

  • config.dbHost={host} -- the DB Endpoint URL of your MySQL server instance
  • config.dbUser={username} -- Database Master Username
  • config.dbPass={password} -- Database Master Password

NODE CONTROLLER CONFIGURATION SECTION - MILLISECONDS

  • instancecontroller.replaceDeadClusters=false -- by default, any cluster that becomes unresponsive will be shut down. Setting this value to true will automatically replace any clusters that have failed.
  • instancecontroller.deleteDeadGroupNodesOnCleanUp=true -- by default, any unresponsive nodes will be deleted from the dashboard. Setting this value to false will stop the instances, but not delete them.

CLUSTER CONFIGURATION INFORMATION

  • cluster.password=changeme -- modify this to be the same as the password that you set in the cluster.xml file on your disk image.

LOADBALANCING CONFIGURATION

  • streammanager.ip={streammanager-static-ip} -- The static IP address used for Stream Manager. This is required when you wish to setup multiple Stream Managers behind a load balancer. This parameter is optional for a single Stream Manager setup.

AWS CLOUD CONTROLLER CONFIGURATION

You will need to un-comment and configure the following entries:

  • aws.defaultzone={default-region} -- the default zone for your autoscaling
  • aws.operationTimeoutMilliseconds=20000 -- estimated time to start a new VM. We do not recommend modifying this value
  • aws.accessKey = {account-accessKey} -- IAM access key
  • aws.accessSecret = {account-accessSecret} -- IAM access secret
  • aws.ec2KeyPairName = {keyPairName} -- SSH keypair name for connecting to nodes
  • aws.ec2SecurityGroup ={securityGroupName} -- security group name
  • aws.defaultVPC={boolean} -- true if using the default VPC, or false if not; (AWS accounts created in the last few years have default VPCs in each region; older accounts do not).

REST SECURITY SECTION

  • rest.administratorToken= -- You need to set a valid password string here before you start using streammanager. This is the password that you will use to execute API commands

WEBSOCKET PROXY SECTION

Please note that if you modify any of the above values after your initial deployment, you will need to restart the Red5 Pro service.

ALARM THRESHOLD (no longer in the properties file)

The autoscaling alarm threshold is no longer set in the red5-web.properties file. Instead, the default value is 60%. If you want to modify this value, do so directly after node group creation using the Rest API for alarms calls. You can set different thresholds for origins and edges via the rest API.

Sample red5-web.properties file content:

 ## RED5 APP CONFIGURATION SECTION - Do Not Tamper
webapp.contextPath=/streammanager
webapp.virtualHosts=localhost, 127.0.0.1

 ## DATABASE CONFIGURATION SECTION
config.dbHost=streammanager-db.abcdefghi.us-west-1.rds.amazonaws.com
config.dbPort=3306
config.dbUser=streammanager
config.dbPass=123445ffff

 ## NODE CONTROLLER CONFIGURATION SECTION - MILLISECONDS
instancecontroller.newNodePingTimeThreshold=150000
instancecontroller.replaceDeadClusters=true
instancecontroller.deleteDeadGroupNodesOnCleanUp=false
instancecontroller.instanceNamePrefix=node
instancecontroller.nodeGroupStateToleranceTime=180000
instancecontroller.nodeStateToleranceTime=180000
instancecontroller.cloudCleanupInterval=180000
instancecontroller.blackListCleanUpTime=600000

## BEST ORIGIN EVALUATION SECTION
instanceevaluator.streams.metricweight=30
instanceevaluator.connections.metricweight=15
instanceevaluator.subscribers.metricweight=60

## CLUSTER CONFIGURATION INFORMATION
cluster.password=changeme
cluster.publicPort=1935
cluster.accessPort=5080
cluster.reportingSpeed=10000
cluster.retryDuration=30

## LOADBALANCING CONFIGURATION
streammanager.ip=

## CLOUD CONTROLLER CONFIGURATION SECTION  - MILLISECONDS

## AWS CLOUD CONTROLLER CONFIGURATION ##
aws.defaultzone=us-west-1b
aws.operationTimeoutMilliseconds=20000
aws.accessKey=ABCDEFGHIJKLMNOPQ
aws.accessSecret=12344556778abcdeFGHIJKLMNOPabud12345
aws.ec2KeyPairName=red5proautoscaling
aws.ec2SecurityGroup=red5pro-nodes
aws.defaultVPC=false

## GOOGLE COMPUTE CLOUD CONTROLLER CONFIGURATION ##
#compute.project={project-id}
#compute.defaultzone={zone-id}
#compute.defaultdisk=pd-standard
#compute.operationTimeoutMilliseconds={operation-timeout}

## Autoscaler SCALE POLICY SECTION - MINUTES
scalepolicy.store=scalingpolicies
scalepolicy.filescandelay=60

## LAUNCH CONFIGURATION SECTION - MINUTES
launchconfig.store=launchconfigs
launchconfig.filescandelay=60

## AUTOSCALING MANAGEMENT SECTION
autoscale.scaleout.mode=competitive

## REST SECURITY SECTION
rest.administratorToken=xyz123

## WEBSOCKET PROXY SECTION
proxy.enabled=false

Edit Launch Configuration File

Launch Configuration JSON files are stored in {red5prohome}/webapps/streammanager/WEB-INF/launchconfigs

NOTE: Not all server types are available in all EC2 regions. You may need to create special launch config files for some regions.

Sample Launch Configuration File, default-v2.json:

{
  "launchconfig": {
    "name": "aws-default-v2",
    "description": "This is a sample version launch configuration for second generation autoscaling for aws",
    "image": "red5pro-server-v110",
    "version": "0.0.2",

    "targets": {
      "target": [
        {
            "role": "origin",
            "instanceType": "c4.xlarge",
            "connectionCapacity": "2000"
        },
        {
            "role": "edge",
            "instanceType": "c4.large",
            "connectionCapacity": "2000"
        }
     ]
    },

    "properties": {
      "property": [
        {
          "name": "network",
          "value": "default"
        }
      ]
    },
    "metadata": {
      "meta": [
        {
          "key": "role",
          "value": "unassigned"
        }
      ]
    }
  }
}

The only values that you need to edit are:

  • name - the name of the configuration file (for example, for default.json this is "default").
  • image - the Red5 Pro AMI that you built, to be used for node creation.
  • instanceType - the AWS machine type https://aws.amazon.com/ec2/instance-types/
  • connectionCapacity - The number of concurrent connections that can be supported for one instance. This number will be referred to by the autoscale scaleout threshold percentage. The connection capacity will vary depending on your broadcast quality, client connection type, and server type. For QA purposes, you can set this value low (e.g., to 20), and set the Alarm Threshold lower than the default 60%, so that you can test the autoscaling feature without having to simulate 2,000 concurrent connections.

Edit Scaling Policy File

Scale Policy JSON files are stored in {red5prohome}/webapps/streammanager/WEB-INF/scalingpolicies

Sample Scaling Policy File, default-v2.json

[ Second generation autoscaling scale policy ]

{
    "policy": {
        "name": "default-v2",
        "description": "This is a sample scale policy file",
        "version": "0.0.2",
        "type": "com.red5pro.services.autoscaling.model.ScalePolicySchema",
        "targets": {
            "target": [
                {
                    "role": "edge",
                    "coolDownPeriod": "180000",
                    "estimatedWarmUpTime": "120000",
                    "minLimit": "2",
                    "maxLimit": "10",
                    "scaleAdjustment": "1"
                },
                {
                    "role": "origin",
                    "coolDownPeriod": "180000",
                    "estimatedWarmUpTime": "120000",
                    "minLimit": "1",
                    "maxLimit": "2",
                    "scaleAdjustment": "1"
                }
            ]
        }
    }
}

NOTE: The entire origin configuration object can be omitted if you do not wish to configure origin maxLimit. In that case it will default to 1.

The only values that you should edit are:

  • edge, minLimit - the minimum number of edges to be in a node group (if you set this to 2, for example, then when you add an origin to a nodegroup, two edge servers will be spun up as well).
  • edge, maxLimit - the maximum number of edges to be in a node group
  • origin, minLimit - the minimum number of origins to be in a node group, (if you set this to 2, for example, then when you add an origin to a nodegroup, a second origin server will be spun up as well).
  • origin, maxLimit - the maximum number of origins to be in a node group .

Start Red5 Pro Service to Use the Stream Manager

sudo /etc/init.d/red5pro start

Optional: Load-Balance Multiple Stream Managers

Prerequisites:

  • One reserved elastic IP address for each Stream Manager.
  • A registered Domain C-Name to associate with the Load Balancer DNS.
  • Create the first Stream Manager per the above instructions, then create an AMI from that instance. Build the second stream manager from that image in a different subnet that then first one, for better redundancy. It is essential that the config files be identical between the two stream managers with one exception:
  • Edit red5pro/webapps/streammanager/WEB-INF/red5-web.properties and modify ## LOADBALANCING CONFIGURATION streammanager.ip=, adding the Assigned IP address of the individual Stream Manager instance you are modifying.
  • Add all Stream Manager public and private IP addresses to the Database security group.
  • Request a new certificate from ACM. This will send an email requst to the domain owner to approve.

Create Load Balancer

  • Navigate to the EC2 Dashboard, in the region where you reserved the streammanager IP addresses.
  • From the left-hand navigation, under LOAD BALANCING, click on Load Balancers
  • Click on Create Load Balancer and choose Classic Load Balancer, then click Continue.

classiclb

  • Step 1: Define Load Balancer:
    • Give the load balancer a name (only alphanumeric characters and - are allowed).
    • Create LB inside: select the VPC that you created for autoscaling.
    • Listener Configuration: add the following port maps:
Load Balancer Protocol Load Balancer Port Instance Protocol Instance Port
HTTP 80 HTTP 5080
HTTPS 443 HTTP 5080
SSL 8083 TCP 8081

portmaps

Note: The port 8083 mapping is essential if you are going to be broadcasting with WebRTC via the Stream Manager SSL Proxy.

  • Step 2: Select Subnets - Select at least two subnets in the availability zone, where your stream managers live. Click the + symbol next to each subnet to add.
  • Click on Next: Assign Security Groups.
  • Choose the Stream Manager security group that you created above. Click on Next: Configure Security Settings.
  • Step 3: Configure Security Settings, Select a Certificate.
    • Select "Choose an existing certificate from AWS Certificate Manager (ACM), and select the certificate you created earlier, which should be in the drop-down menu next to Certificate:. This certificate will be used both for HTTPS and Secure Websockets.
    • Unless you have some specific security needs, accept the Predefined Security Policy, and click on Next: Configure Health Check.

existingcert

  • Step 4: Configure Health Check - Ping Protocol: HTTP, Ping Port: 5080 (unless you have Red5 Pro running on a different port). Advanced details: You can accept the defaults, but we recommend at least for startup that you reduce the Healthy Threshold from 10 to 2, to get things up and running. Click on Next: Add EC2 Instances
  • Step 5: Add EC2 Instances - Select your Stream Managers from the list of servers in the selected VPC. Click on Next: Add Tags (optional); click on Review and Create
  • Step 7: Review. Make sure everyting is set up as you wish, then click on Create to launch the load balancer.

IMPORTANT You will need to create a new disk image - create a new VM from the original AMI, and modify {red5prohome}/conf/autoscale.xml to point to the Load Balancer DNS name on port 5080, then create a new AMI from this VM to use for your nodes.


API Basics, to get you started

(for full API, see Red5 Pro Stream Manager API)

Verify Stream Manager is Using the Correct Controller

GET call:

http://<streammanager_IP>:5080/streammanager/api/2.0/admin/debug/cloudcontroller?accessToken=<rest.administratorToken from red5-web.properties file>

should return: Amazon Web Services

Create a New Node Group (post)

Using a tool like Postman, create a new node group via the API. baseCapacity is the Minimum subscriber connections that this group should support. (This parameter helps the scale-in process decide when to scale down an edge).

POST call: http://<streammanager_IP>:5080/streammanager/api/2.0/admin/nodegroup?accessToken=<rest.administratorToken from red5-web.properties file>

Data (make sure to select JSON as the body type):

{
 "regions": [
   "us-west-1"
 ],
 "launchConfig": "default-v2",
 "scalePolicy": "default-v2"
}

postman

Note the cluster “name” that is returned by the above call. It will be used to create a new Origin server.

Launch New Origin (post) After you create a node group, create the origin server. Creating an origin server will also generate at least one edge, per scaling policy min limit.

http://<streammanager_IP>:5080/streammanager/api/2.0/admin/nodegroup/<cluster-name>/node/origin?accessToken=<rest.administratorToken from red5-web.properties file>

NOTE: If you wish to launch more than one origin, you can repeat the call. The maximum origins allowed will depend on the maxLimit attribute of the 'origin' object described in scale policy. If 'origin' is omitted from the scale policy then the default value for maximum origins is 1.

Set Alarm Threshold (POST)

By default, the alarm threshold (the capacity percentage at which the cluster will scale up) is set to 60%. To modify this, POST the following:

FOR EDGE:

http://<streammanager_IP>:5080/streammanager/api/2.0/admin/alarm/scaleout/default?type=edge&threshold=<threshold>&accessToken=<rest.administratorToken from red5-web.properties file>

FOR ORIGIN:

http://<streammanager_IP>:5080/streammanager/api/2.0/admin/alarm/scaleout/default?type=origin&threshold=<threshold>&accessToken=<rest.administratorToken from red5-web.properties file>

LIST GROUPS (get)

http://<streammanager_IP>:5080/streammanager/api/2.0/admin/nodegroup?accessToken=<rest.administratorToken from red5-web.properties file>


Stream Manager Publish and Subscribe Examples

Red5 Pro HTML5 SDK Examples:

Publish - Stream Manager

Subscribe - Stream Manager

If you are publishing/subscribing using the Stream Manager SSL Proxy, use:

Publish - Stream Manager Proxy

Subscribe - Stream Manager Proxy

Note: the streaming-html5 examples testbed is included with the Red5 Pro server distribution, and can be accessed via your stream manager at https://your.server.url/webrtcexamples/.

Red5 Pro iOS SDK Examples:

Publish - Stream Manager

Subscribe - Stream Manager

Red5 Pro Android SDK Examples:

Publish - Stream Manager

Subscribe - Stream Manager

Troubleshooting

  1. If you have created a new nodegroup, but adding an Origin server doesn't automatically spin up an accompanying Edge server, then you probably didn't modify the red5pro/conf/autoscale.xml in your Red5Pro server image to point to the IP address of your stream manager (replace 0.0.0.0 in <property name="cloudWatchHost" value="http://0.0.0.0:5080/streammanager/cloudwatch"/>) and/or change active value from default “false” to “true” (<property name="active" value="true"/>)
  2. You can see the role of a node on the Tags tab in the EC2 console instancetags