/

9. AWS Elastic File System

Integrating Autoscaling with Amazon Elastic File System


You may want to mount an EFS drive if you are recording files, instead of using Amazon S3 for storage. This option is a little cleaner, as you can bypass the post-process step involved in using S3 storage. The only limitation currently is that EFS is VPC-based, so you would need a different EFS created for each region.

Create A New Elastic File System

Access the EFS Dashboard here, and click on the Create file system button.

  1. Configure file system access: Select the VPC that you are using for your autoscaling nodegroups from the dropdown list.
  2. Create mount targets: The subnets that you created should be automatically selected. Under security groups locate the policy that you set up for your nodes and add that. Click on Next Step
  3. Configure optional settings: at this time we suggest keeping the defaults (General Purpose performance mode and no encryption added)
  4. Review and create: click on Create File System after reviewing the values.
  5. Make note of the File system ID and DNS name of your EFS. You will need this information for connection.

It is necessary to keep the following in mind while creating a new File System:

  • EFS is VPC based, so make sure to select the correct VPC (the one you are going to use for autoscaling).
  • The VPC should have a subnet in each availability zone and each subnet should be selected as a mount target for the File System. This will ensure that the EFS will be mountable in any of the availability zones for that region.
  • Ensure that the security group for the EFS is properly configured to allow access from EC2 instances.
  • In the Configure optional settingsscreen, select General Purpose (default) as the performance mode.
  • Do not select Enable encryption of data at rest.

General Settings for EFS Security Group

  • Inbound: Add a Custom TCP Rule for port 2049 allowing NFS read/write. For a simple but less secure configuration use 0.0.0.0/0 for Source. If you know the addressing scheme and IP ranges for your VPC you can try specifying that was well.
  • Outbound: Allow all traffic for all port ranges and any destination(defaults).

Modifying the Node AMI to use EFS via NFS

Install the NFS client

sudo apt-get install nfs-common

Testing EFS mount manually

Once you have installed the NFS client and your EFS is configured with proper access security, you should test the connection.

sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport <File-System-DNS-Name>:/ <Mount-Destination-Path>

Example

sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-de56fa55.efs.us-east-2.amazonaws.com:/ /usr/local/red5pro/webapps/live/streams

To verify that the filesystem is mounted, run df -h and you should see the path listed at the bottom, for example:

Filesystem                                 Size  Used Avail Use% Mounted on
udev                                       1.9G     0  1.9G   0% /dev
tmpfs                                      372M  5.2M  367M   2% /run
/dev/nvme0n1p1                              16G  1.5G   14G  10% /
tmpfs                                      1.9G     0  1.9G   0% /dev/shm
tmpfs                                      5.0M     0  5.0M   0% /run/lock
tmpfs                                      1.9G     0  1.9G   0% /sys/fs/cgroup
tmpfs                                      372M     0  372M   0% /run/user/1000
fs-de56fa55.efs.us-east-2.amazonaws.com:/  8.0E     0  8.0E   0% /usr/local/red5pro/webapps/live/streams

Configure AMI to mount EFS on VM startup

You will want to create a shell script to run the mount command, and then add a Crontab job to mount on reboot.

Example script

nfsmount.sh added to the ubuntu home directory:

#! /bin/bash
sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-de56fa55.efs.us-east-2.amazonaws.com:/ /usr/local/red5pro/webapps/live/streams

To create a crontab entry, crontab -e and select your preferred editor. Add the following:

# m h  dom mon dow   command
@reboot /home/ubuntu/nfsmount.sh
  • To verify your script/crontab, restart the VM and run df -h to check that the nfs mount is included. Then create an AMI from this VM.