Deploying Stream Manager and Autoscaling on Microsoft Azure

Overview:

This document assumes that you have some basic operational knowledge of Microsoft Azure platform management and have administrative access to the Microsoft portal with a valid subscription. It also assumes that you have some basic linux administration skills. If you need more detailed information, please contact us.

In order to use the Red5 Pro Stream Manager service you will need the following:

  1. Latest Red5 Pro Server build
  2. The azure-cloud-controller.jar, from the Red5 Pro Autoscaling Library Extensions section
  3. An active Red5 Pro license key (needed for clustering) REGISTER HERE
  4. An active Azure account.

A Free Trial subscription might be good to try things out but it is not recommended due to restrictions put in place by Microsoft.

Before you begin

You will want to keep a record of the usernames, passwords, IP addresses, and other information generated during the setup process, as you will need the information for stream manager configuration and future operations via the API. Click here to download a handy list for tracking all of your Red5 Pro Autoscaling values
Optional: download and install the Azure CLI

For ease of administration, you may want to customize the Favorites menu (left hand navigation bar).

  • Click on More services > to expand
  • Click in the Star (to change from grey to gold) for the following:

    • Azure Active Directory
    • Subscriptions
    • Resource Groups
    • Public IP Addresses
    • Virtual Networks
    • Network Security Groups
    • Azure Database for MySQL servers
    • Virtual Machines
    • Images

    Favorites

For some additional information on Red5 Pro Autoscaling on the Azure platform, check out Autoscaling on Microsoft Azure - Overview

1. Get credentials for Stream Manager

These credentials will be used to access Azure resources (Tenant ID, Client ID, Client Secret and Subscription ID),

Register Stream Manager application in your Azure active directory

  • Go to the Azure portal
  • Navigate to the Active Directory Menu Blade. You can select the pushpin icon in the top right to pin the menu to your dashboard.
  • If you have multiple active directories, make sure to switch to the correct one first (from the top left corner of the portal) .
  • With the directory selected click on “App registrations” menu item. Register Application
  • Click on New application registration and fill in the details for identifying the web application.
    • Name : streammanager
    • Application type: Select Web app/API
    • Sign-on URL: Enter the DNS name of the web app if you have it otherwise you can fill in a temporary URL that can be changed later. Usually since Stream Manager uses SSL cert for WebRTC, You should use the domain name associated with the SSL, but you can use any placeholder. This is not used for Red5 Pro Autoscaling.

Enter Application Info

  • When you are ready, click on Create
  • Once the application registered in the AD it appears in the clients list.Application Registered
  • Now click on the application and you will see the Application-ID of the web app listed under Essentials. This is the az.clientId value. You should copy this to a safe place for future reference.

Get Client ID

  • With the application selected, select the Keys menu item from the right side panel. Use this to generate a secret key for the Stream Manager web app. After clicking on Keys, enter a key description and select Never Expires for the key.

Generate Client Secret

  • Click the Save button and the key will be generated automatically. Make sure to copy the key value to a safe location for future reference. This is our client key/secret which is required as a part of authentication info.
  • Now select Azure Active Directory from the left pane once again and select the Properties menu item. Here you will find the active directory ID. Copy this to a safe location for future reference. This is the az.tenantId value which is a essential part of the authentication info.

Get Tenant ID

Now we must assign a access role to the application so that it has the right permissions to carry out the operations that it is intended for. With the “Azure Active Directory” selected:

  • Navigate to Subscriptions.
  • Click and select the subscription account that the autoscaling setup will be using
  • Click on Access Control (IAM)
  • Click the Add button to add a new role and ensure the following values for the form.

    • Role: Select an appropriate role with sufficient permissions (ex: contributor)
    • Assign access to: Azure AD user, group or application
    • Select: Type in the application name (streammanager)
    • Once the application is shown, click to select it
    • Click Save to save the IAM settings for the application.

    IAM Settings

  • With the subscription selected, click on Properties to find the Subscription ID. Copy this to your Azure AutoScaling Checklist for future reference. This is part of the authentication info that is required by the Stream Manager web app.

Subscription Properties

2. Create an SSH2-RSA Keypair for instances

This Keypair will be used for Instance Authorizing Operations Across All Regions

The public key is used to start manual instances, such as for Red5 Pro node image creation and Stream Manager instance setup. The dynamic instances will use a username and a password for securing the instances.

Generate Your SSH2-RSA Key Pair:

On Mac/Linux:

  • Open Terminal
  • Type: ssh-keygen -t rsa
  • Accept the default path and modify the file name if you wish (Enter file in which to save the key (~/.ssh/id_dsa)
  • Enter a passphrase and confirm (make sure this is SECURE, and noted somewhere for future reference)
  • Private and public keys will be generated

On Windows:

  • Download and run PuTTYgen
  • Click on “Generate” button and follow instructions displayed to help PuTTYgen generate your private public key-pair.
  • Once PuTTYgen finishes generating keys it will display the public key and other details in the application window.
  • Enter a passphrase and confirm. (make sure this is SECURE, and noted somewhere for future reference)
  • Click “Save public key” and save the key with a filename (ex: red5pro_node) on your file system in a secure location (or a common standard folder such as ~/.ssh/ under your user profile). Provide an extension of .pub to the file (public key).
  • Click “Save private key” and save the key with the same filename as the public key (ex: red5pro_node) on your file system in a secure location (or a common standard folder such as ~/.ssh/ under your user profile). Provide an extension of .ppk to the file (private key).

3. Create the Master Resource Group for the Autoscaling Setup

Create a Resource Group in your region of choice. This will be the default region for the group. A Resource group can hold resources from different regions, so there should not be an issue with cross region resources management.

To create a master resource group:

  • Navigate to Resource Groups menu
  • Click on +Add
  • Fill in the form displayed:
    • Enter Resource group name
    • Select a subscription (Use the same subscription that is used for AD access authentication)
    • Select the location of the Resource Group
  • Click Create to create the resource group

Autoscaling Resource Group

4. Reserve A Static IP Address for Stream Manager

Note: Public IP resources are not the same as Reserved Virtual Addresses (Azure classic deployment).

  • Navigate to the Azure portal
  • Navigate to Public IP Addresses.
  • Click on +Add
  • Fill in the relevant information for creating a new IP public addresses
    • Name: Name to associate with your Stream Manager
    • IP Version: Select IPv4
    • IP address assignment: Static
    • Idle Timeout: 4 (Leave at default)
    • DNS name label: None (Leave at default)
    • Create an IPv6 address: Unchecked (Leave at default)
    • Subscription: Select your subscription (Use the same subscription that is used for AD access authentication)
    • Resource group: Select “Use existing” and select your autoscaling resource group.
    • Location: Select the region where the IP address will be reserved.
    • Click Create to create the IP address. It will take a few seconds to create the IP address resource.

Reserve IP

How to delete a reserved public IP address using CLI or the portal’s GUI

USING PORTAL GUI

  • Navigate to All Resources or Public IP Addresses in the portal.
  • Identify the Public IP address resource and delete it.

USING AZURE CLI

az network public-ip delete -n <static-ip-resource-name> -g <resource-group-name>

5. Create Virtual Networks and Network Security Groups

Note: you will need to repeat this entire four-step process in every region you wish to include in your autoscaling solution. Follow the recommended naming convention to create regional resources.

Create Virtual Network

A virtual network is a logical separation of the actual network for operational, security and management purpose. Virtual networks are location-based resources. Thus we need to create a virtual network in each region that we wish to target with autoscaling. Each virtual network can have one or more subnets. For our purpose a single subnet is sufficient.

To create a virtual network:

  • Navigate to Virtual Networks in the left pane in your Azure portal dashboard and click on +Add.
  • Fill in the Virtual network information in the form shown, following the naming convention mentioned earlier to help Stream Manager locate resources across regions easily - <prefix>-<region>-vnet.

If you do not know the region name you can use the following azure CLI command: az account list-locations

Avaliable regions at the time of this publication

  • eastasia
  • southeastasia
  • centralus
  • eastus
  • eastus2
  • westus
  • northcentralus
  • southcentralus
  • northeurope
  • westeurope
  • japanwest
  • japaneast
  • brazilsouth
  • australiaeast
  • australiasoutheast
  • southindia
  • centralindia
  • westindia
  • canadacentral
  • canadaeast
  • uksouth
  • ukwest
  • westcentralus
  • westus2
  • koreacentral
  • koreasouth

The following creates a virtual network called red5proautoscaling-eastus-vnet in eastus region, in the red5proautoscaling master resource group

  • Name: Fill in the name of the virtual network as red5proautoscaling-eastus-vnet
  • Address Space: Leave the settings to default
  • Subscription: Select your subscription (Use the same subscription that is used for AD access authentication)
  • Resource Group: Select Use existing then select our master autoscaling resource group created previously from the drop down.
  • Location: Select the region where the virtual network will be created
  • Subnet
    • Name: Leave it to default
    • Address Range: Leave it to default setting
  • Service endpoints: Disabled

Click Create to create the virtual network. It takes a few seconds to create the resource. You can refresh the page to see updated list of Virtual Networks.

New Virtual Network

New Virtual Network Created

Currently Red5 Pro autoscaling on Azure supports a single subnet. This subnet must be named default.

Create Network Security Group For Nodes

A network security group defines the firewall rules for a Virtual machines running in a virtual network. An Azure network security group is similar to a AWS security group in many aspects.

Similar to the Virtual Network, the Network Security Group is also a regional resource. Meaning that you need to create a security group for each region that you wish to target with autoscaling.

A network security group defines inbound and outbound rules with port and protocol combinations. Additionally Azure allows you to specify the importance of the rule with a rank number. A lower rank implies that the rule is more important than others. There should be a difference between jump of ranks.

To create a network security group:

  • Navigate to Network Security Groups.

  • If upon selecting the Network Security Group item you see a new window, select a deployment model, select Resource manager and click Create.

  • Click on +Add, and fill in the basic necessary details of the Network Security Group:
    • Name: The network security group name. Use the following naming convention: <prefix>-<region>-nsg.
    • Subscription: Select your subscription (Use the same subscription that is used for AD access authentication)
    • Resource Group: Select Use existing then select our master resource group created previously.
    • Location: Select the region where the virtual network will be created (USEAST for example).
  • Click Create to create the Network Security Group. It takes a few seconds to create the resource. You can refresh the page to see updated list of Network Security Groups or you can check out your resource group content from the Resource Group menu item.

Once the resource is ready, click on your Network Resource Group. Select Inbound Security Rules from the left and +Add Red5 Pro ports rules as shown below. Note that there are three network rules by default. Our new rules are meant to have higher priority than the default ones.

  • Add the following ports. You can either add each port as an individual rule, or add one comma-separated list for all of the TCP ports, and one rule for the UDP port range. Source/destination for each should be Any; Source port range should be *:
Priority Port Description Protocol
100 22 SSH Access TCP
200 5080 default web access of Red5 Pro TCP
300 1935 default Red5 Pro RTMP port TCP
400 8554 default RTSP port TCP
500 6262 websockets for HLS TCP
600 8081 websockets for WebRTC TCP
700 40000-65000 TURN/STUN/ICE port range for WebRTC UDP

Inbound Rules List

Network Security Group inbound rules

  • Next, select Outbound Security Rules and create a rule to allow all traffic as shown below. Note that there are three network rules by default. Our new rules are meant to have higher priority than the default ones.

Network Security Group outbound rules

Create Network Security Group For Stream Manager

  • Add the following ports. You can either add each port as an individual rule, or add one comma-separated list for all of the TCP ports, and one rule for the UDP port range. Source/destination for each should be Any; Source port range should be *:
Priority Port Description Protocol
100 22 SSH Access TCP
200 5080 default web access of Red5 Pro TCP
300 443 HTTPS TCP
400 8081 websockets for WebRTC TCP
500 8083 secure websockets for WebRTC TCP
  • Next, select Outbound Security Rules and create a rule to allow all traffic as above.

Note: if you want more restrictions on your Autoscaling setup, please refer to the Securing Red5 Pro Autoscaling document.
You need to repeat the above steps for each region that you wish to target. This will allow you to run Red5 Pro VMs across regions.

6. Prepare MySQL Database

Azure provides two types of MySQL database offerings to help developers integrate MySQL with their web apps - Azure database for MySQL servers and ClearDB. Both types of offerings work with Stream Manager web application. However based on price and performance we recommend using Azure database for MySQL servers.

NOTE: For Optimal performance, the Stream Manager and Database should be in the same Region.

Make a note of database credentials while creating the database for later use.

To Create the Database for Autoscaling:

  • Navigate to Azure Database For MYSQL Servers.
  • Click on Create Azure Database For MySQL users.
  • Fill in the form displayed as necessary for new database provisioning.
    • Server Name: The database name following the naming convention. e.g.: red5proautoscaling-eastus-db
    • Subscription: Select a subscription account
    • Resource Group: Select Use existing and select your Red5 Pro autoscaling master resource group
    • Server admin login name: The admin user name
    • Password: The admin password
    • Confirm the password
    • Location: Select a region for the database server (recommend the same region as your stream manager)
    • Version: Select the latest MySQL version (5.7 at the time of writing)
    • Pricing tier: You can edit the pricing plan of the database offering or reconfigure the selected plan.

Currently Azure MySQL offering does not offer many options for configuration. You can configure the COMPUTE UNITS to adjust performance. For more information, check out: Azure Database for MySQL pricing

Click Create to create the database server.

New Azure MySQL Database Server

It may take a few minutes for the database to get created. You can refresh the page once the deployment is complete to see the instance in the listing.

  • You can click the database server to see the details view.

Database Details

  • Next we need to configure the Security for the database server. To configure security, with the database server selected :
    • Click on Connection Security menu item.
    • Disable the SSL settings (SSL access to database server from Stream Manager is currently not supported).
  • Configure firewall rules:
    • +Add My IP so you can administer the database via MySQL Workbench.
    • Add the Stream Manager's reserved IP address (for Start IP and End IP) to allow it to access the database server.
    • Click Save from top menu.

Database Connection Security

This completes the security configuration. We can now access the server using MySQL workbench.

Database connection properties

To be able to connect to the database you should know the database host, the admin username and the password; in addition, your IP should be registered in the firewall rules. You can find the database host and username in the Properties menu after clicking on the database server. The password is not shown. You should have noted it down while creating the database server itself. If you lose the password, you can reset it from the database Overview section.

Database Connection Properties

7. Connect to MySQL and Add Cluster Schema

To connect to the mysql instance which we just created, we suggest using the freeware MySQL workbench tool, and can be downloaded from https://dev.mysql.com/downloads/workbench/.

  • Launch MySQL Workbench tool on your system
  • Click on the “+” icon to the right of MySQL Connections to open the new connection configuration window.
  • Enter a connection name - (for example, “azure-mysql”)
  • Hostname: the database endpoint URL which we obtained from RDS instance details tab
  • Username: the Database Master Username as configured
  • Password: the Database Master password as configured
  • Default Schema: Leave blank for now
  • Click on Test Connection to verify you can connect. : If you have trouble connecting, make sure that the credentials are correct and that your IP address is added in the database firewall settings as described earlier. MySQL Workbench
  • Click "ok" to save configuration and close window.
  • Double click on the new connection you created to open database workspace view.
  • In the "SCHEMAS" section, right-click and select Create schema to create a new schema.
  • Schema Name: type in cluster and click Apply, then Apply again in the REview the SQL Script popup.
  • Right-click on the cluster schema and choose "Set as Default Schema".
  • Download the Red5 Pro Server Distribution, and unzip to your desktop.
  • Click “ok” to save configuration and close window.
  • Download the Red5 Pro Server Distribution, and unzip to your desktop.
  • From menu click => File => Open SQL Script, and browse the server distribution to find the cluster.sql file {red5prohome}/webapps/streammanager/WEB-INF/sql/cluster.sql
  • Make sure that the cluster database is selected, then click the execute icon to execute the sql script which populates the cluster database clusterdbpopulated

8.Prepare Red5 Pro Image for Nodes

Make sure to follow the recommended naming convention for image resources as mentioned earlier.

Launch an Instance and Install Red5 Pro

Start a Basic Instance:

  • Navigate to Virtual Machines. Click on +Add
  • From the list of displayed Image options, click on Ubuntu Server and select Ubuntu Server 16.04 LTS. Make sure Resource Manager is selected, then click the Create button to start the VM creation wizard.

New Ubuntu VM

  • Basics: Fill in the basic information to configure the VM as mentioned below:
    • Name: The instance name. e.g.: red5pronode
    • VM Disk type: HDD (the default SSD type seems to limit the machine types available across regions)
    • Authentication type: Select SSH Public Key.
    • Username: ubuntu
    • SSH Public Key: Paste the content of your public key (From the RSA key pair created earlier) here.
    • Subscription: Your subscription account
    • Resource Group: Select Use existing then select your resource group created previously.
    • Location: Select the region where the VM will be created.

Click Ok to move to next step in the wizard.

New VM Wizard - Basic

NOTE: Do not create a separate resource group for the instance. The VM should exist in the master resource group for the final image to exist in the same resource group.

  • Size: Select a instance type for the VM. Instance types on Azure come with different configuration and pricing policy. Since we are going to simply create an image, the A1 Standard VM configuration is sufficient to get started.
    • Change the Supported disks type drop down to select HDD.
    • Click on the A1 Standard size type.
    • Click Select to save selection and continue.

New VM Wizard - Size

  • Settings : This is the last step of the wizard. Here we configure the launch specific configuration such as network, security etc. Set the option as mentioned below.

    • Availability Set: None
    • Use Managed Disks: Yes
    • Virtual Network: Select the virtual network created for the target region Ex: <prefix>-<region>-vnet.
    • Subnet: default
    • Public IP Address: Leave settings to default (new) as the resource is auto-generated. But keep note of the resource name as we will be deleting it after we are done creating the image.
    • Network Security Group: Select the Network Security Group that you created for the target region e.g. <prefix>-<region>-nsg
    • Extensions: No extensions.
    • Auto-shutdown: Off.
    • Disable Boot diagnostics and Guest OS diagnostics under the Monitoring option.
    • Once you are done, click ok to move to the final step.
  • Summary: The final step checks your parameters and validates them for VM parameters. Click on Create to start creating the VM.

This operation can take a couple of minutes. If you navigate to the “Virtual Machines” menu you can see the VM being created.

  • After a few minutes the VM switches state from “creating” to “Running”. The instance is now ready for Red5 Pro installation.

Install Red5Pro on the Instance:

  • Connect to your instance using the private key from the key-pair that you createdL ssh -i ~/.ssh/keys/red5prokey ubuntu@<server_ip_address>
  • Follow the Installing Red5Pro on a Linux Server instructions
  • Make sure that Red5 Pro service is set to auto-start on launch.
  • Red5 Pro Clustering does not work without a license. You will need to purchase a Red5 Pro Professional license from http://account.red5pro.com. Add a file named LICENSE.KEY with your Pro license key (16 characters, plus dashes, like: YOUR-LIC-ENSE-KEY1) to the root of the Red5 Pro Server installation ({red5prohome}/LICENSE.KEY)

Install NTP (network time protocol)

NTP is necessary to ensure that the Stream Manager and all nodes are in sync.

sudo apt-get install ntp

The service should start automatically once installed.

Also, make sure that the server time zone is UTC. Type date at the prompt. This will return the date, time and zone (eg:Tue Dec 13 20:21:49 UTC 2016). If you need to switch to UTC, execute sudo dpkg-reconfigure tzdata, scroll to the bottom of the Continents list and select None of the above; in the second list, select UTC.

Configure Autoscaling on the Instance:

Navigate to the location where you extracted the red5pro files, and edit the file {red5prohome}/conf/autoscale.xml

    <bean name="config" class="com.red5pro.clustering.autoscale.Configuration" >
        <property name="active" value="false"/>

        <!--Stream manager hosted uri. use the host of your stream manager.  -->
                <property name="cloudWatchHost" value="http://0.0.0.0:5080/streammanager/cloudwatch"/>
  • Change the active value from “false” to “true
  • Replace 0.0.0.0 with the static ip (elastic ip) you reserved for stream manager previously

NOTE: If you plan on supporting WebRTC, you will need to install SSL certificate on the Stream Manager instance. Thus you will have a hostname associated with the SSL cert instead of an IP address. Make sure to use the host name instead of the IP address in the autoscale configuration shown above.

  • Save and close the file

Set a Unique Cluster Password:

Edit the file {red5prohome}/conf/cluster.xml and replace the default <!-- edge/origin link cluster password --> password from changeme to sometime unique. Make a note of this password, to be used in the Stream Manager red5.properties file.

NOTE: The cluster password can not contain any CAPITAL LETTERS at this time

Remove Unnecessary Webapps:

To ensure a faster startup time for your server, we suggest deleting any webapps that you will not be using from the {red5pro}/webapps directory (for example: secondscreen, template, vod, streammanager; do not remove root). For ease of setup for the Stream Manager, leave a zipped copy of the server distribution on the server.

Verifying Red5 Pro on Instance:

Start Red5 Pro service sudo systemctl start red5pro

Once you have started the Red5 Pro Server, you can verify that it is running and available by opening a web browser and navigating to http://{instance-IP}}:5080, where {instance-ip} is the Public IP address of your vm.

After you've verified, stop the Red5 Pro service: sudo systemctl stop red5pro

Create Image from Instance

After you have created the VM, select it from the list and click on Overview. Stop the instance, then click on Capture to create an image.

Capture

  • Name: It is essential to name the image according to the conventions referenced in the Resource Naming Convention grid. For the server image, use <imagename>-<serverversion>-<region>-img (for example: autoscalenode01-red5pro430-eastus-img). If you don't include the region-img then the Stream Manager will not be able to find the image to create nodes.
  • Resource Group: Choose Use existing and select the group that you set up.
  • Do not select Automatically delete this virtual machine after creating the image; doing so will add a lot of extra time to the image creation.
    • You will want to delete the VM after you've created the image, as doing so generalizes the VM and it won't be useable.
  • Click on Create

Create Image

Copy Red5 Pro Image to Other Regions

There does not seem to be a simple way to copy images or even snapshots to other regions using the Azure portal. There is however a method to copy the snapshot to another region, create a managed disk from it, create a VM from the managed disk nd then finally => create a image from the VM. The process is not discussed here because of its scope. If you are an Azure administrator, you can always create automation scripts using CLI for this.

The fastest recommended way of making a Red5 Pro image available for another region would be to create an image fresh from start using a new Ubuntu Server 16.04 LTS image as described earlier. You may use your own standard shell script automation for installing Red5 Pro and configuring the instance for autoscaling.

9. Configure and Deploy Stream Manager on Azure

Note: after the Stream Manager instance is started up, you will need to add its IP address to the database firewall settings if you have not done that already

Launch an Instance and Install Red5Pro

You can use the image that you created for the nodes above as a base for the Stream Manager.

  • Navigate to the Images list
  • Select the image that you created, and click on Create VM

Create VM

  • Basic : Fill in the basic information to configure the VM as per the following:
    • Name: The instance name. Ex: streammanager-eastus-vm
    • VM Disk type: HDD
    • Authentication type : Select SSH Public Key.
    • Username: ubuntu
    • SSH Public Key: Paste the content of your public key (From the RSA key pair created earlier) here.
    • Subscription: Your subscription account
    • Resource Group: Select Use existing then select our resource group created previously.
    • Location: Select the region where the Stream Manager VM will be created.

Click Ok to move to next step in the wizard.

  • Size: Select an appropriate instance size for the VM. Instance types on Azure come with different configuration and pricing policy.

    • Change the Supported disks type drop down to select HDD.
    • Click on an appropriate instance size type size type. You can get more information on their configuration and performance from the Azure Sizes for Cloud Services page. We suggest selecting a multi-vcpu instance type with high network performance. Azure provides a wide range of machine sizes to select from.
    • Click Select to save selection and continue.
  • Settings: Select configuration options as mentioned below.

    • Availability Set: None
    • Use Managed Disks: Yes
    • Virtual Network: Select the virtual network created for the target region Ex: <prefix>-<region>-vnet.
    • Subnet: default
    • Public IP Address: Click to select the static public IP address (by name) that we created for Stream Manager web app earlier. EX: streammanager-eastus-pip.
    • Network Security Group: Select the Network Security Group created for the target region Ex: <prefix>-<region>-nsg
    • Ensure to turn off Boot diagnostics under Monitoring option

    If you added ports for Stream Manager to your network security group for nodes you should use the same one for Stream Manager. Otherwise you can create a new network security group for Stream Manager with all the necessary ports listed earlier.

Once you are done, click ok to move to the final step.

  • Summary: The final step checks your parameters and validates them for VM parameters. Click on Create to start creating the VM.

This operation can take a couple of minutes. If you navigate to the “Virtual Machines” menu you can see the VM being created.

  • After a few minutes the VM switches state from Creating to Running. The instance is now ready for Red5 Pro installation.

Associating the static IP address with this VM at a later stage:

You can also assign the static public IP address to the Stream Manager after it has started. However it is recommended to assign the public IP address at configuration stage. Once the new instance is created it will have a dynamic IP associated with it. If you wish to associate the static IP address at this this you will need to do it via the network interface of the Stream Manager VM.

  • Navigate to Virtual Machines content window using the left pane
  • Locate the Stream Manager virtual machine, and click on it to open the detail view.
  • Click on the Networking option, locate the Network interface associated with the virtual machine and “click” on it. This will take you to the Network Interface detail view window for the particular network interface.

Network Interface

  • Click on IP Configuration under settings to show IP configuration for this network interface. Each network interface can have more than one IP configurations. But for simplicity we will be working with a single one.
  • Click on the Primary IP Configuration listed in the window. This will show the IP configuration detail view.
  • Click on IP address and select the static public IP address (by name) that was created earlier.
  • You can leave the Private IP address settings untouched.
  • Click Save to save changed to the IP configuration for the network interface. (It takes a few seconds for changes to be saved).
  • You can now access the VM using the static IP address instead of the old one.
  • Delete the older dynamic Public IP address resource which is now disassociated from the network interface by locating it in the All Resources menu.

Once you have a static IP assigned to your instance You can also install SSL certificate on the Stream Manager instance.

Configure the Stream Manager Instance

  • SSH into the Stream Manager instance
  • Stop the Red5 Pro service (sudo systemctl stop red5pro)

Install NTP (network time protocol)

NTP is necessary to ensure that the Stream Manager and all nodes are in sync.

sudo apt-get install ntp

The service should start automatically once installed.

Also, make sure that the server time zone is UTC. Type date at the prompt. This will return the date, time and zone (eg:Tue Dec 13 20:21:49 UTC 2016). If you need to switch to UTC, execute sudo dpkg-reconfigure tzdata, scroll to the bottom of the Continents list and select None of the above; in the second list, select UTC.

  • Navigate to the directory where you installed Red5 Pro (e.g. /usr/local/red5pro)
  • Remove the red5pro directory that was configured for the node; unzip the distribution and rename the unzipped directory red5pro.
  • Delete the following files:
    • {red5prohome}/conf/autoscale.xml
    • {red5prohome}/plugins/red5pro-autoscale-plugin-<version>.jar
    • {red5prohome}/plugins/red5pro-webrtc-plugin-<version>.jar

Import and Activate Azure Cloud Controller

Copy the azure-cloud-controller.jar file into {red5prohome}/webapps/streammanager/WEB-INF/lib/

Edit the applicationContext.xml file, located at {red5prohome}/webapps/streammanager/WEB-INF/applicationContext.xml per the following:

  • Locate the Azure controller “bean” and uncomment it as shown below (note: do not modify the values, only uncomment the bean configuration to make it active):
<!-- AZURE CONTROLLER -->
<bean id="apiBridge" class="com.red5pro.services.cloud.microsoft.component.AzureComputeController" init-method="initialize">
   <property name="resourceGroupName" value="${az.resourceGroupName}"/>
   <property name="resourceGroupRegion" value="${az.resourceGroupRegion}"/>
   <property name="resourceNamePrefix" value="${az.resourceNamePrefix}"/>
   <property name="instanceNamePrefix" value="${instancecontroller.instanceNamePrefix}"/>
   <property name="clientId" value="${az.clientId}"/>
   <property name="clientKey" value="${az.clientKey}"/>
   <property name="tenantId" value="${az.tenantId}"/>
   <property name="subscriptionId" value="${az.subscriptionId}"/>
   <property name="vmUsername" value="${az.vmUsername}"/>
   <property name="vmPassword" value="${az.vmPassword}"/>
   <property name="defaultSubnetName" value="${az.defaultSubnetName}"/>
   <property name="operationTimeoutMilliseconds" value="${az.operationTimeoutMilliseconds}"/>
   <property name="quickOperationResponse" value="${az.quickOperationResponse}"/>
   <property name="quickResponseCheckInitialDelay" value="${az.quickResponseCheckInitialDelay}"/>
   <property name="apiLogLevel" value="${az.apiLogLevel}"/>
</bean>

Comment out (or delete the entry for) the default controller as shown below to disable it:

<!-- Default CONTROLLER -->
<! --
<bean id="apiBridge" class="com.red5pro.services.cloud.sample.component.DummyCloudController" init-method="initialize">
</bean>
 -->

Modify Stream Manager App Properties (red5-web.properties)

Refer to your Azure Red5 Pro Auto Scaling Checklist for the data needed to complete this section. For more details on the red5-web.properties file sections, please read here.

You will need to modify the following values:

DATABASE CONFIGURATION SECTION

  • config.dbHost={host} -- the DB Endpoint URL of your MySQL server instance
  • config.dbUser={username} -- Database Master Username
  • config.dbPass={password} -- Database Master Password

NODE CONTROLLER CONFIGURATION SECTION - MILLISECONDS

  • instancecontroller.replaceDeadClusters=false -- by default, any cluster that becomes unresponsive will be shut down. Setting this value to true will automatically replace any clusters that have failed.
  • instancecontroller.deleteDeadGroupNodesOnCleanUp=true -- by default, any unresponsive nodes will be deleted from the dashboard. Setting this value to false will stop the instances, but not delete them.
  • instancecontroller.newNodePingTimeThreshold=200000 --This is the max calculated time taken by a new Red5 Pro node to get into running state and ping Stream Manager. On Azure it takes longer than most of the other supported cloud platforms. Make sure to set this value to 200000
  • instancecontroller.nodeGroupStateToleranceTime=500000 -- This is the maximum time for which Stream Manager allows a node group to be in a inconsistent state. After this it is removed from the system forcibly. Attributing to the long startup time of Red5 Pro node on Azure, this value must be set to 500000
  • instancecontroller.nodeStateToleranceTime=500000 -- This is the maximum time for which Stream Manager allows a node to be in a inconsistent state. After this it is removed from the system forcibly. Attributing to the long startup time of Red5 Pro node on Azure, this value must be set to 500000

CLUSTER CONFIGURATION INFORMATION

  • cluster.password=changeme -- modify this to be the same as the password that you set in the cluster.xml file on your disk image.

AZURE CLOUD CONTROLLER CONFIGURATION

You will need to un-comment all the az name-spaced attributes and configure the following entries:

  • az.resourceGroupName -- Name of the master autoscaling resource group
  • az.resourceGroupRegion -- Default region of the master autoscaling resource group
  • az.resourceNamePrefix -- Resource prefix used for name-spacing and resolving resources on the Azure platform
  • az.clientId -- Azure active directory client ID is application access ID received after web app registration.
  • az.clientKey -- Azure active directory client ID is the application access secret received after web app registration.
  • az.tenantId -- The ID of the Azure active directory.
  • az.subscriptionId -- The ID of the subscription plan that is used for autoscaling setup
  • az.vmUsername -- Dynamic virtual machine username to be created for ssh . Recommended a minimum of 6 characters.
  • az.vmPassword -- Dynamic virtual machine password to be created for ssh access. Password must be between 6-72 characters long and must satisfy at least 3 of password complexity requirements from the following:

    1. Contains an uppercase character
    2. Contains a lowercase character
    3. Contains a numeric digit
    4. Contains a special character
    5. Control characters are not allowed
  • az.defaultSubnetName -- The default subnet name of virtual network(s) used. This should always be default.

  • az.quickOperationResponse -- Boolean flag indicating whether to use quickresponse mode for delete operations. This is set to true by default. Setting it to false would mean that Stream Manager will wait for all resources to be deleted. This can take upto 400 seconds or more and can be blocking on autoscale operations. With the value set to true The controller returns a success to Stream Manager as soon as the virtual machine reaches Deleting state.
  • az.quickResponseCheckInitialDelay -- Initial wait time before checking for virtual machine Deleting state. This shoudl default to 20000.
  • az.apiLogLevel -- The Azure SDK logging level for API operations. Default is BASIC - which will provide the most logging. Other options, from most to least, are WARNING, ERROR and OFF.

REST SECURITY SECTION

  • rest.administratorToken= -- You need to set a valid password string here before you start using streammanager. This is the password that you will use to execute API commands

WEBSOCKET PROXY SECTION

  • proxy.enabled -- set to true enables, or set to false disables the websocket proxy service. You must set up SSL on the Stream Manager and use the proxy if you are streaming WebRTC with Red5 Pro autoscaling.

DEBUGGING CONFIGURATION SECTION

  • debug.logaccess -- Set to true if you want to allow access to log files via REST API. This can be especially useful during development on cloud. With log access enabled you can use the Stream Manager REST API to download log files without using SSH. For more info on how to use the log access API refer to the Stream Manager Rest API.

WARNING! Please note that the log files may contain sensitive information such as your license key. If you enable log access for REST API, make sure that the administrator token is not accessible to unauthorized personnel. Please note that if you modify any of the above values after your initial deployment, you will need to restart the Red5 Pro service.

ALARM THRESHOLD (no longer in the properties file)

The autoscaling alarm threshold is no longer set in the red5-web.properties file. Instead, the default value is 60%. If you want to modify this value, do so directly after node group creation using the Rest API for alarms calls. You can set different thresholds for origins and edges via the rest API.

Sample red5-web.properties file content:

 ## RED5 APP CONFIGURATION SECTION - Do Not Tamper
webapp.contextPath=/streammanager
webapp.virtualHosts=localhost, 127.0.0.1

## DATABASE CONFIGURATION SECTION
config.dbHost=red5proautoscaling-streammanager-eastus-db.mysql.database.azure.com
config.dbPort=3306
config.dbUser=user@red5proautoscaling-streammanager-eastus-db
config.dbPass=xyz@123456

 ## NODE CONTROLLER CONFIGURATION SECTION - MILLISECONDS
instancecontroller.newNodePingTimeThreshold=200000
instancecontroller.replaceDeadClusters=true
instancecontroller.deleteDeadGroupNodesOnCleanUp=false
instancecontroller.instanceNamePrefix=node
instancecontroller.nodeGroupStateToleranceTime=500000
instancecontroller.nodeStateToleranceTime=500000
instancecontroller.cloudCleanupInterval=180000
instancecontroller.blackListCleanUpTime=600000

## BEST ORIGIN EVALUATION SECTION
instanceevaluator.streams.metricweight=30
instanceevaluator.connections.metricweight=15
instanceevaluator.subscribers.metricweight=60

## CLUSTER CONFIGURATION INFORMATION
cluster.password=changeme
cluster.publicPort=1935
cluster.accessPort=5080
cluster.reportingSpeed=10000
cluster.retryDuration=30

## LOADBALANCING CONFIGURATION
streammanager.ip=

## CLOUD CONTROLLER CONFIGURATION SECTION  - MILLISECONDS

## AZURE CLOUD CONTROLLER CONFIGURATION ##
az.resourceGroupName=red5proautoscaling
az.resourceGroupRegion=eastus
az.resourceNamePrefix=autoscaling
az.clientId=0abr72fc-d552-4a0e-8ecb-4712758560a5
az.clientKey=poaX/MiFmBpVWDpm4ywB1YYtsxcSptZQwgmiM4yciPM=
az.tenantId=25fb21dd-0bqf-4f16-8e26-3f1e6g30b2fc
az.subscriptionId=7c234f4e-6r34-45f4-9cf5-e8e8t87e9d70
az.vmUsername=ubuntu
az.vmPassword=xyz@123456789
az.defaultSubnetName=default
az.operationTimeoutMilliseconds=120000
az.quickOperationResponse=true
az.quickResponseCheckInitialDelay=20000
az.apiLogLevel=BASIC

## AWS CLOUD CONTROLLER CONFIGURATION ##
#aws.defaultzone={default-region}
#aws.operationTimeoutMilliseconds=200000
#aws.accessKey={account-accessKey}
#aws.accessSecret={account-accessSecret}
#aws.ec2KeyPairName={keyPairName}
#aws.ec2SecurityGroup={securityGroupName}
#aws.defaultVPC={boolean}
#aws.vpcName={vpc-name}
#aws.faultZoneBlockMilliseconds=3600000

## GOOGLE COMPUTE CLOUD CONTROLLER CONFIGURATION ##
#compute.project={project-id}
#compute.defaultzone={zone-id}
#compute.defaultdisk=pd-standard
#compute.operationTimeoutMilliseconds={operation-timeout}

## SIMULATED-CLOUD CONTROLLER CONFIGURATION ##
#managed.regionNames={custom-region}
#managed.availabilityZoneNames={custom-region-zone}
#managed.operationTimeoutMilliseconds=20000
#managed.recycleDeadNodes=true

## Autoscaler SCALE POLICY SECTION - MINUTES
scalepolicy.store=scalingpolicies
scalepolicy.filescandelay=60

## LAUNCH CONFIGURATION SECTION - MINUTES
launchconfig.store=launchconfigs
launchconfig.filescandelay=60

## AUTOSCALING MANAGEMENT SECTION
autoscale.scaleout.mode=competitive

## REST SECURITY SECTION
rest.administratorToken=xyz123

## DEBUGGING CONFIGURATION SECTION
debug.logaccess=false
debug.logcachexpiretime=60000

## WEBSOCKET PROXY SECTION
proxy.enabled=true

Edit Launch Configuration File

Launch Configuration JSON files are stored in {red5prohome}/webapps/streammanager/WEB-INF/launchconfigs. Please refer to Sizes for Linux virtual machines in Azure to select the desired instance type. You will want to choose an instance type with at least 2 CPUs and 4GB RAM. In addition, depending on your anticipated traffic, you will probably want high network throughput.

NOTE: Not all server types are available in all regions. You may need to configure the launch config files with the vmSizeFallbacks option.

Sample Launch Configuration File, az-default-v2.json:


{
"launchconfig": {
"name": "az-default-v2",
"description": "This is a sample version launch configuration for second generation autoscaling for azure",
"image": "autoscalenode",
"version": "0.0.2",
"targets": {
"target": [{
"role": "origin",
"instanceType": "Standard_DS1",
"connectionCapacity": "500"
},
{
"role": "edge",
"instanceType": "Standard_DS1",
"connectionCapacity": "500"
}
]
},
"properties": {
"property": [{
"name": "vmSizeFallbacks",
"value": [
"Standard_D1",
"Standard_D2",
"Standard_D3"
]
}]
},
"metadata": {
"meta": [{
"key": "meta-name",
"value": "meta-value"
}]
}
}
}

The only values that you need to edit are:

  • name - the name of the configuration file (for example, for default.json this is "default").
  • image - the Red5 Pro image that you built, to be used for node creation. Note that if you correctly followed the naming convention, then your image name is something like : <imagename>-<region>-img. In that case the value for image should be just <imagename>. The controller will resolve the resource by adding the other parts of the name to it.
  • instanceType - The Azure VM size type. Sizes for Linux virtual machines in Azure
  • connectionCapacity - The number of concurrent connections that can be supported for one instance. This number will be referred to by the autoscale scaleout threshold percentage. The connection capacity will vary depending on your broadcast quality, client connection type, and server type. For QA purposes, you can set this value low (e.g., to 20), and set the Alarm Threshold lower than the default 60%, so that you can test the autoscaling feature without having to simulate 2,000 concurrent connections.

The vmSizeFallbacks property

There are restrictions on availability of VM size types across regions. A specific VM size type on Azure platform, may or may not be available in another region. This can cause failures in autoscaling across regions. To counter this problem, the Stream Manager launch configuration file allows you to specify alternate VM size types to use when the default one (instanceType) is not available in that region. The launch configuration file is capable of holding any number of arbitrary properties. The fallback sizes is defined as a property called vmSizeFallbacks which is an array of alternative vm size types. Ex: ["Standard_D1","Standard_D2","Standard_D3"]. Fallback alternatives will be looked up exactly in the order they have been specified. So for example if the specified type Standard_DS1 for a role is not available, it will fall back to Standard_D1 etc.

Fallbacks cannot be specified per role type. It is the responsibility of the system admin to ensure that the default instanceType being requested for a role is available in the regions that are being targeted through autoscaling.

NOTE : This property field is mandatory for Azure controller. So even if you do not have any fallbacks planned you should have the property in the launch configuration file with an empty array as values.

Edit Scaling Policy File

Scale Policy JSON files are stored in {red5prohome}/webapps/streammanager/WEB-INF/scalingpolicies

Sample Scaling Policy File, default-v2.json

[ Second generation autoscaling scale policy ]

{
    "policy": {
        "name": "default-v2",
        "description": "This is a sample scale policy file",
        "version": "0.0.2",
        "type": "com.red5pro.services.autoscaling.model.ScalePolicySchema",
        "targets": {
            "target": [
                {
                    "role": "edge",
                    "coolDownPeriod": "180000",
                    "estimatedWarmUpTime": "120000",
                    "minLimit": "2",
                    "maxLimit": "10",
                    "scaleAdjustment": "1"
                },
                {
                    "role": "origin",
                    "coolDownPeriod": "180000",
                    "estimatedWarmUpTime": "120000",
                    "minLimit": "1",
                    "maxLimit": "2",
                    "scaleAdjustment": "1"
                }
            ]
        }
    }
}

The only values that you should edit are:

  • edge, minLimit - the minimum number of edges to be in a node group (if you set this to 2, for example, then when you add an origin to a nodegroup, two edge servers will be spun up as well).
  • edge, maxLimit - the maximum number of edges to be in a node group
  • origin, minLimit - the minimum number of origins to be in a node group, (if you set this to 2, for example, then when you add an origin to a nodegroup, a second origin server will be spun up as well).
  • origin, maxLimit - the maximum number of origins to be in a node group .

Start Red5 Pro Service to Use the Stream Manager

sudo systemctl start red5pro

Configure Stream Manager with SSL

If you want to publish via WebRTC, or iOS, you must set up the Stream Manager with a valid SSL certificate and use the Stream Manager SSL Proxy feature. Please refer to this document to configure SSL on your Red5 Pro Stream Manager.

API Basics, to get you started

(for full API, see Red5 Pro Stream Manager API)

Verify Stream Manager is Using the Correct Controller

GET call:

http://<streammanager_IP>:5080/streammanager/api/2.0/admin/debug/cloudcontroller?accessToken=<rest.administratorToken from red5-web.properties file>

should return: Microoft Azure

Create a New Node Group (post)

Using a tool like Postman, create a new node group via the API.

Not all regions may support virtual machines based on the nature of your subscription. You would need to check with Azure support about the limitations of your subscription.

POST call: http://<streammanager_IP>:5080/streammanager/api/2.0/admin/nodegroup?accessToken=<rest.administratorToken from red5-web.properties file>

Data (make sure to select JSON as the body type):

{
 "regions": [
   "eastus"
 ],
 "launchConfig": "az-default-v2",
 "scalePolicy": "default-v2"
}

Note the cluster “name” that is returned by the above call. It will be used to create a new Origin server.

Launch New Origin (post) After you create a node group, create the origin server. Creating an origin server will also generate at least one edge, per scaling policy min limit.

http://<streammanager_IP>:5080/streammanager/api/2.0/admin/nodegroup/<cluster-name>/node/origin?accessToken=<rest.administratorToken from red5-web.properties file>

NOTE: If you wish to launch more than one origin, you can repeat the call. The maximum origins allowed will depend on the maxLimit attribute of the 'origin' object described in scale policy. If 'origin' is omitted from the scale policy then the default value for maximum origins is 1.

Set Alarm Threshold (POST)

By default, the alarm threshold (the capacity percentage at which the cluster will scale up) is set to 60%. To modify this, POST the following:

FOR EDGE:

http://<streammanager_IP>:5080/streammanager/api/2.0/admin/alarm/scaleout/default?type=edge&threshold=<threshold>&accessToken=<rest.administratorToken from red5-web.properties file>

FOR ORIGIN:

http://<streammanager_IP>:5080/streammanager/api/2.0/admin/alarm/scaleout/default?type=origin&threshold=<threshold>&accessToken=<rest.administratorToken from red5-web.properties file>

LIST GROUPS (get)

http://<streammanager_IP>:5080/streammanager/api/2.0/admin/nodegroup?accessToken=<rest.administratorToken from red5-web.properties file>


Stream Manager Publish and Subscribe Examples

Stream Manager Proxy Publish and Subscribe Examples

With the latest release, the live webapp includes two examples: proxy-publisher.html and proxy-subscriber.html. These examples will take the following query parameters:

Name Description Default Value
host hostname or IP window.location.hostname
protocol protocol which Stream Manager is served over (http or https) window.location.protocol
port port number that Stream Maager is served on window.location.port
app webapp name to stream to on the server live
streamName The unique stream name to broadcast with or subscribe to None. Required
verbose Flag to enable verbose logging in Dev Console None. optional
view Target broadcast tech (rtc, rtmp or hls) None. Optional

Example URI: https://streammanager.test.com/live/proxy-publisher.html?streamName=stream1&verbose=1

Red5 Pro HTML5 SDK Examples:

Publish - Stream Manager

Subscribe - Stream Manager

If you are publishing/subscribing using the Stream Manager SSL Proxy, use:

Publish - Stream Manager Proxy

Subscribe - Stream Manager Proxy

Note: the streaming-html5 examples testbed is included with the Red5 Pro server distribution, and can be accessed via your stream manager at https://your.server.url/webrtcexamples/.

Red5 Pro iOS SDK Examples:

Publish - Stream Manager

Subscribe - Stream Manager

Red5 Pro Android SDK Examples:

Publish - Stream Manager

Subscribe - Stream Manager

Troubleshooting

  1. If you have created a new nodegroup, but adding an Origin server doesn't automatically spin up an accompanying Edge server, then you probably didn't modify the red5pro/conf/autoscale.xml in your Red5Pro server image to point to the IP address of your stream manager (replace 0.0.0.0 in <property name="cloudWatchHost" value="http://0.0.0.0:5080/streammanager/cloudwatch"/>) and/or change active value from default “false” to “true” (<property name="active" value="true"/>)
  2. You can see the role of a node on the Tags tab in the VM details instancetags

Special Operational Notes for Azure Cloud

  • Always create a separate resource group for Red5 Pro autoscaling projects.
  • Always use a short and unique value for the resource name prefix.
  • When creating a new cluster on Azure, remember that the time taken for a node to start up is about 200 seconds. So if you need the cluster operational for an event, make sure to start creating the cluster well ahead of time.
  • Node deletion can take an equally long time (up to ~400 seconds) since it involves deleting the sub-resources of a VM. Stream Manager runs in a quickresponse mode by default. This means when the controller detects that the node is in a Deleting state it will return a positive to the core. This mode ensures that autoscaling does not stall due to a delete operation.
  • Due to the quick response mode, when you delete a node group do not stop Stream Manager for at least five minutes after the API response. This will make sure that the controller code has cleaned up all vm resources properly.
  • Before targeting a region for autoscaling make sure to use the portal UI to check what VM sizes are supported in that region and configure the launch configuration file (as well as the vmSizeFallbacks option) carefully.
  • SSL security on Azure MySQL database server must be turned off.
  • Some subscriptions won't allow you to create virtual machines in certain regions. Make sure that you check out each targeted region using the portal before you try to use Stream Manager to launch a node there.