Discover the Power of Cloud Computing with AWS and Salt

This article is the first one of a series showing how to use cloud-computing in media and entertainment (M&E workflow).
Cloud-computing represents the future of infrastructure.

Every day, dozens of companies are moving from a traditional infrastructure to a cloud-computing infrastructure. Even the biggest ones, such as Netflix, are moving all their computing to the Amazon or other clouds infrastructure.

AWS, Amazon Web Services, is a cloud-computing platform offered by Amazon. It provides the power, flexibility and reliability your company needs.

However, it’s easy to get lost in this jungle; here’s a quick guide to get started on AWS and automate the setup and management of your cloud infrastructure with SaltStack.

AWS Console

Start an EC2 instance in your VPC

After creating an account, you will face the diversity of AWS. It covers everything from computing to storage. Today we are interested in creating a virtual server in the cloud, known as an EC2 Instance. Click on it to get started.

Create your instance

On this screen, you will find a summary of your instances : your default VPC (Virtual Private Cloud), how much are running, adding a new one, etc.
We will create a new instance, click on “Launch Instance”

Pick the distribution you are going to use

On this page, you will choose the image for your machine. You have the choice between the officials images provided by Amazon (mostly base operating systems) or more specific images provided by the community.

In our case, we will choose an Ubuntu image.

Decide the size of the instance

Depending on your needs, you have to choose a size for your instance. For test purposes, a t2.micro is enough. If you want to have more information like details and monthly pricing of the different instances, you can check this website http://www.ec2instances.info/?cost=monthly.

The flexibility of Amazon allows you to change the size of your instance whenever you want without reinstalling your application.

Launch this new instance

We will tune the other settings later, click on Review and Launch. A page will summarize your new instance, click on Launch.

Select the way to authenticate against your EC2 instance

In order to connect and authenticate against your instance, you need to generate a key pair. Select “create a new key pair”, give it a name and download it.

Assign a static IP

We need to assign an elastic ip to your server. Like that even if you decide to stop and start your instance, you will still have the same static IP address.

Click on Elastic ip and confirm that you want a new elastic ip. Do a right click on your ip and choose Associate ip. Fill the form with your instance, network interface and Private ip address range.

Your instance is now attached to your VPC. You can edit the configuration and do approximately everything from this dashboard. Let’s take some interest in the security group. Click on Security groups on the left panel.

Security groups management

AWS also ensures the security of your machines. Instead of using a traditional firewall on your router, Amazon provides you security groups.

Go into security groups in the left panel. You will see 2 entrees in the list. The first one represents the security rules of your VPC; by default, all the traffic (inbound and outbound) is allowed, we will leave it like that. The second group represents the rules applied to your instance, by default just inboud ssh is allowed.

Imagine you just added a web server to your instance. All you need is just to open the port 80 (http) inbound. It’s a very efficient way to manage traffic security.

Command line and bastion instance setup

ssh access

Now it’s time to use the key you previously downloaded. Open a terminal in the directory where you stored the key and run the commands below.

ssh -i youkey.pem ubuntu@YOUR_INSTANCE_IP
sudo -i

SaltStack setup

SaltStack is an open source configuration management, it will be your best friend to manage your Amazon private cloud. It offers you the ability to quickly create a new instance and deploy a configuration on it. One server is defined as the salt master, and the other ones are minions who are getting the configuration from their master.

Repository configuration

This commands below will setup the SaltStack official repository for Ubuntu 14.04 and install the salt packages.

wget -O - https://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest/SALTSTACK-GPG-KEY.pub | sudo apt-key add -
echo "deb http://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest trusty main" > /etc/apt/sources.list.d/saltstack.list
apt-get update
apt-get install salt-master salt-cloud

SaltStack configuration deployment

We will configure a salt master on our bastion server. Let’s configure a salt profile, which will set up the bash configuration and a user profile for all our users. Today we will use this configuration only for our bastion / jump server but in the future when we will add more servers we will just have to apply to them the exact same configuration we did here.

mkdir -p /srv/salt/generic && mkdir -p /srv/salt/templates/etc/skel
cd /srv/salt
nano top.sls

hosts profile definitions

In Salt, the file which contains a mapping between groups of machines on a network and the configuration roles that should be applied to them is called a top file.

Right now we want only to apply the base configuration and deploy our users to all our hosts (‘*’). This will include bastion and all the instances we will launch in the future.

Content of our /srv/salt/top.sls:

base:
  '*':
    - generic.base
    - generic.users

basic hosts configuration

This bases.sls file is going to define the basic configuration we want on all our machines like:

• shell profiles (.bashrc)
• motd messages
• pre-installed packages
• …

Content of /srv/salt/generic/base.sls:

{% set config_files = [
    "/etc/skel/.bashrc"
]
%}

{% for config_file in config_files %}
{{ config_file }}:
    file.managed:
        - source: salt://templates/{{ config_file }}
        - user: root
        - group: root
        - mode: 666
        - template: jinja
        - makedirs: True
{% endfor %}

/root/.bashrc:
    file.managed:
        - source: salt://templates/etc/skel/.bashrc
        - template: jinja

bash configuration file

Content of /srv/salt/templates/etc/skel/.bashrc:

# System-wide .bashrc file for interactive bash(1) shells.

# To enable the settings / commands in this file for login shells as well,
# this file has to be sourced in /etc/profile.

# If not running interactively, don't do anything
[ -z "$PS1" ] && return

# check the window size after each command and, if necessary,
# update the values of LINES and COLUMNS.
shopt -s checkwinsize

# set variable identifying the chroot you work in (used in the prompt below)
if [ -z "${debian_chroot:-}" ] && [ -r /etc/debian_chroot ]; then
 debian_chroot=$(cat /etc/debian_chroot)
fi

if [[ EUID -eq 0 ]]; then
 export PS1='\[\e[0;31m\]\u@\h\[\e[m\] \[\e[1;34m\]\w\[\e[m\] \[\e[0;31m\]\$ \[\e[m\]\[\e[0m\]'
else
 export PS1='\[\e[0;32m\]\u@\h\[\e[m\] \[\e[1;34m\]\w\[\e[m\] \[\e[1;32m\]\$ \[\e[m\]\[\e[0m\]'
fi

# enable color support of ls and also add handy aliases
if [ -x /usr/bin/dircolors ]; then
 test -r ~/.dircolors && eval "$(dircolors -b ~/.dircolors)" || eval "$(dircolors -b)"
 alias ls='ls --color=auto'
 alias grep='grep --color=auto'
 alias fgrep='fgrep --color=auto'
 alias egrep='egrep --color=auto'
fi

# some more ls aliases
alias ll='ls -alF --color=auto'
alias la='ls -A --color=auto'
alias l='ls -CF --color=auto'
alias scl='salt-call --local'
alias hf='hostname -f'
alias t="tail -fn100"

# if the command-not-found package is installed, use it
if [ -x /usr/lib/command-not-found -o -x /usr/share/command-not-found/command-not-found ]; then
 function command_not_found_handle {
 # check because c-n-f could've been removed in the meantime
  if [ -x /usr/lib/command-not-found ]; then
   /usr/lib/command-not-found -- "$1"
   return $?
  elif [ -x /usr/share/command-not-found/command-not-found ]; then
   /usr/share/command-not-found/command-not-found -- "$1"
   return $?
  else
   printf "%s: command not found\n" "$1" >&2
   return 127
  fi
 }
fi

Users definitions

One of the big advantage of salt is the fact you manage your user authentication with the usual local files like /etc/passwd and /etc/shadow.

In this case we will:

• create / update the linux user
• set the password
• setup a ssh public key for this account

Content of /srv/salt/generic/users.sls:

{%
    set user_list = [
        {
            "id": "msol",
            "uid": 1001,
            "fullname": "MSolution",
            "password": "$1$5Wm4UFH3$gcWZBd1qxxBT23BT565Dj1",
            "envs": []
        }
    ]
%}

sudo:
  group.present:
    - system: True

{% for user in user_list %}
{{ user["id"] }}:
    user.present:
        - fullname: {{ user["fullname"] }}
        - shell: /bin/bash
        - home: /home/{{ user["id"] }}
        - createhome: true
        - uid: {{ user["uid"] }}
        - groups:
            - sudo
        - gid_from_name: true
        - password: {{ user["password"] }}
    file.absent:
        - name: /home/{{ user["id"] }}/.ssh/authorized_keys
    ssh_auth:
        - require:
            - user: {{ user["id"] }}
            - file: /home/{{ user["id"] }}/.ssh/authorized_keys
        - present
        - user: {{ user["id"] }}
        - source: salt://ssh-keys/{{ user["id"] }}.pub
{% endfor %}

This configuration requires a ssh key, let’s add ours into /srv/salt/ssh-keys/msol.pub

Now, we need to add this to the top.sls configuration

You can see the following rule who will set all users in our top.sls with this specific line:

- generic.users

Running a “salt-call –local state.highstate” will apply the configuration to the master:

• setting up the base system
• creating local users

salt-call --local state.highstate

Now our bastion host also acting as a salt-master is up and running we will be able to use salt from there in the future to deploy and maintain new instances.

The salt configuration options are infinite and you can basically create a template for anything.

Next!

Now you have your EC2 bastion host up and running. In the next article, you will be able to setup your own transcoding instance through salt-cloud. Salt-cloud is a tool for automatically provisioning and managing cloud servers within and across supported cloud providers (AWS, rackspace, joyent…).

About TrackIt

TrackIt is an international AWS cloud consulting, systems integration, and software development firm headquartered in Marina del Rey, CA.

We have built our reputation on helping media companies architect and implement cost-effective, reliable, and scalable Media & Entertainment workflows in the cloud. These include streaming and on-demand video solutions, media asset management, and archiving, incorporating the latest AI technology to build bespoke media solutions tailored to customer requirements.

Cloud-native software development is at the foundation of what we do. We specialize in Application Modernization, Containerization, Infrastructure as Code and event-driven serverless architectures by leveraging the latest AWS services. Along with our Managed Services offerings which provide 24/7 cloud infrastructure maintenance and support, we are able to provide complete solutions for the media industry.