/

Recommended - Use AWS autoscaling for Stream Managers


The solution discussed in this document is based on the AWS Auto Scaling mechanism. Please note that AWS's Auto Scaling feature is completely independent of Red5 Pro Autoscaling, and is only used for load-balanced Stream Managers.

The AWS Load Balancer and Autoscale Group will remain static, while the AWS Launch Configuration will be changed with each new Stream Manager build.

DB ACCESS

In order for the stream managers to be able to write/read to/from the database, you will need to give the stream manager security group access to the database inbound mysql/aurora port (3306). Make sure this permission is in place before you start up the stream manager autoscale group.

1. Prepare a Stream Manager AMI

To create an AMI from the Stream Manager instance:

  • Navigate to the EC2 Dashboard and select your Stream Manager instance. Stop the VM for the cleanest image.
  • Click on “Actions” => “Image” => “Create Image”
  • In the “Create Image” popup window enter a unique image name and description and click create an image. Leave additional default settings.
  • Navigate to the AMIs listing, and copy the AMI ID (for use in the launch configuration)

2. Create an EC2 Launch Configuration

In this step we will define a launch configuration for AWS autoscaling to be used by the AWS autoscaling group created below.

Navigate to the EC2 Dashboard, click on Launch configurations, and click on Create launch configuration to start wizard.

Launch configuration name

It is helpful to include easily interpreted description, for example including details such as Red5 Pro Server Release Version, instance type used by the AMI, and creation date (e.g. SM-red5pro-server-9.0.0-june28-m5large-LC)

Amazon machine image (AMI)

Type in the AMI ID that you just created

Instance type

Select the instance type used for creating your Stream Manager image. example: m5.large.

Additional configuration

Under Addtional configuration expand the Advanced Details section. Scroll down to the User data variable. Select As text and enter the following shell script in the text area.

#!/bin/bash

uid=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 16 | head -n 1 | tr '[:upper:]' '[:lower:]')
smhost=$(echo "$uid.com")

sed -i "s/\(streammanager\.ip=\).*\$/\1${smhost}/" /usr/local/red5pro/webapps/streammanager/WEB-INF/red5-web.properties

The above script provided as UserData will run automatically when a new instance is created. It is programmed to edit the Stream Manager instance's configuration file - red5-web.properties and set a unique value for the property - streammanager.ip (it will be something like streammanager.ip=z2go1nhorcpsxlxy.com). This property needs to be set to a unique identifier string such as an IP address or a hostname when using multiple Stream Managers behind a load balancer.

For IP Address Type, select Assign a public IP address to every instance. Since the Stream Managers will be accessed via a load balancer and Red5 Pro nodes also communicate to Stream Managers via the load balancer, having a public IP address is optional, but it is recommended in case any administration or troubleshooting is necessary.

Storage (volumes)

  • Make sure that the storage Size is set to at least 16 Gb or more.
  • Leave other options to default

Security Groups

  • Click on Select an existing security group
  • Select the Security Group created for Stream Manager instances

Key pair

Choose the keypair that you loaded earlier (or generate a new one if you wish), to be used for SSH access.

then click on Create launch configuration

3. Set up an Application Load Balancer

The next step is to create an Application Load Balancer. The load balancer will be responsible for distributing traffic to one or more Stream Manager instances efficiently, taking into account the instance health status.

Create Load Balancer

  • Navigate to the EC2 Dashboard
  • From the left-hand navigation, under LOAD BALANCING, click on Load Balancers
  • Click on Create Load Balancer and choose Application Load Balancer, then click Create.

Step 1: Configure Load Balancer

  • Name: Give the load balancer a name (only alphanumeric characters and - are allowed).
  • *Scheme**: internet-facing
  • IP address type - ipv4
  • Listener Configuration: add the following listener ports:
Load Balancer ProtocolLoad Balancer Port
HTTP5080
HTTPS443
  • Availability Zones:

    • Choose your Red5 Pro autoscaling VPC, and select all of the availability or the ones where you plan to make your Stream Manager(s) available (advance).
  • Click Next: Configure Security Settings to move to the next step

Step 2: Configure Security Settings

  • Select "Choose an existing certificate from AWS Certificate Manager (ACM), and select the certificate you created earlier, which should be in the drop-down menu next to Certificate:. This certificate will be used both for HTTPS and Secure Websockets.

    You can use ACM to request a new certificate for your domain, subdomain etc or even upload your own certificate information that you might have purchased from elsewhere. For more information check out ACM documentation

    • Unless you have some specific security needs, accept the Predefined Security Policy.
    • Click on Next: Configure Security Groups to move to next step

Step 3: Configure Security Groups

  • Choose the security group intended for Stream Manager that you set up previously while setting up Stream Manager.

Step 4: Configure Routing

  • Target Group: New Target Group (give it a name); target type: Instance. Protocol HTTP, port 5080.
  • Health checks: protocol: HTTP, path /.
  • Click on Next: Register Targets to move to next step

By this configuration, we will be routing every traffic received on port 5080 and 443 of the load balancer on the port 5080 of instances registered in our target group.

Step 5: Register targets

  • Skip this step, since we won't be registering any instances directly here.
  • Click on Next: Review

Step 6: Review

  • Review your configuration edits to make sure everything is set up as you wish, then click on Create to launch the load balancer

Wait a while as the load balancer turns from provisioning to active.

Creating a DNS CNAME Record

At this point, you will have a load balancer with an empty target group. We will now assign a read friendly alias to the load balancer so that it is easy to remember and use.

  • Navigate to the load balancer main page and select your newly created load balancer; copy the DNS string (will be something like red5pro-autoscale-1662349126.us-east-1.elb.amazonaws.com).
  • Go to the domain management interface of your business domain (The domain that you wish to create the alias with).
  • Create DNS record of type CNAME with an alias host having the value of the load balancer DNS.

NOTE: If you are updating an existing domain name, DNS propagation can take up to 24 hours in worst cases. Once the propagation is complete, you can access the load balancer using the alias.

4. Create an AWS Autoscaling Group

The final component of the setup is to create an AWS Autoscaling Group. The autoscaling group will be responsible for adding/removing Stream Manager instance(s) based on configuration provided. The group will use the launch configuration in the first part of the step and it will make the instances accessible via the Load Balancer created in the second part of the setup.

  • Navigate to the EC2 Dashboard
  • From the left-hand navigation, under AUTO SCALING, click on Auto Scaling Groups
  • Click on Create Auto Scaling group to start the wizard

Choose launch template or configuration

  • Give your Autoscale Group an easily-identifiable name (e.g. Red5Pro-dev-Autoscale-Group)
  • Click on Switch to launch configuration and from the pull-down menu, select the launch configuration which was created earlier.
  • Click on Next

Configure settings

  • Network: Select your Red5 Pro autoscaling VPC
  • Subnet: Select all of the subnets which you created for your VPC.
  • click on Next

Configure advanced options

  • Load Balancing: Select Attach to an existing load balancer
  • Classic Load Balancers: Leave empty (since are using Application Load Balancer)
  • Target Groups: Click for drop down list, and choose the group that was generated in step 5 of the loadbalancer setup
  • Health Check Type: Select ELB
  • Health Check Grace Period: Leave at default setting (300)
  • Click on next

Configure scaling policies

Group size - set your initial min/max number of Stream Managers (this can be modified at any point in the future, so it is recommended that you start with 1/1)

  • Select the option - Target tracking scaling policy
  • Choose Average CPU utilization and set a target value of 50% (default)

With the datachannel release (server v8.0.0), websocket connections are no longer persisted on the stream managers, so you can allow AWS to scale stream managers in and out.

Click on Next. You can Add Notifications for adding or removing EC2 instances if you wish. Next add tags if you wish. Then click on Next to review.

Once the group is ready it will take a short while for the scaling policy to kick in and automatically create the first Stream Manager instance. The group will work towards maintaining the requested minimum desired capacity at all times. Subsequently, as the load balancer health check completes, the instance is registered as an active InService target. You can now access the Stream Manager(s) using the CNAME.

Remove Instances / Stop Auto Scaling

To remove all instances from an AWS autoscale group, do not manually delete instances from the EC2 console!!. Instead edit the autoscale group and set the minimum & desired instances to zero. The Autoscaling mechanism will automatically remove instance(s) and no new instances will be added.

Updating Stream Manager Image for Autoscaling

When you need to update the stream manager, if you are using AWS Autoscaling, you will need to follow this process:

  1. Create a new instance using the current Stream Manager AMI
  2. Modify that AMI with the updated release per this doc
  3. Create an image from that new instance
  4. Create a new AWS Launch Configuration
  5. Modify your Auto Scaling group - select the group, and choose Actions, edit. Change the Launch Configuration (pull-down) to the new one that you just created.
  6. You can manually delete one of the existing stream managers at a time; or navigate to the Instance refresh tab of your Autoscaling Group, and click on Start instance refresh to set the minimum healthy percentage for replacing.

Tips and Takeaways

  • When creating an Auto Scaling group, subnets should match the load balancer targeted subnets.
  • The AWS Load balancer setup should not register any targets initially. The target will be assigned by the Auto Scale group.
  • Instance warmup time is the net time taken for the Red5 Pro service to be available over port 5080. This includes the time taken for the VM to start and then the service to start. For Red5 Pro the approximate warmup time with safe buffer can be about 180 seconds.
  • If you plan to use WebRTC via Stream Manager, ensure that Instance Protection option is set for the Auto Scale group.
  • When configuring scaling policies remember that a scale-in action should be faster & more responsive than a scale-out action. Use this as a rule of thumb when setting metric threshold and monitoring period.
  • Notifications are optional but a very useful feature to stay on top of your Autoscale Group events.

References: