This article is the first one of a series showing how to use cloud-computing in media and entertainment (M&E workflow).
Cloud-computing represents the future of infrastructure.
Every day, dozens of companies are moving from a traditional infrastructure to a cloud-computing infrastructure. Even the biggest ones, such as Netflix, are moving all their computing to the Amazon or other clouds infrastructure.
AWS, Amazon Web Services, is a cloud-computing platform offered by Amazon. It provides the power, flexibility and reliability your company needs.
However, it’s easy to get lost in this jungle; here’s a quick guide to get started on AWS and automate the setup and management of your cloud infrastructure with SaltStack.
AWS Console
Start an EC2 instance in your VPC
After creating an account, you will face the diversity of AWS. It covers everything from computing to storage. Today we are interested in creating a virtual server in the cloud, known as an EC2 Instance. Click on it to get started.
Create your instance
On this screen, you will find a summary of your instances : your default VPC (Virtual Private Cloud), how much are running, adding a new one, etc.
We will create a new instance, click on “Launch Instance”
Pick the distribution you are going to use
On this page, you will choose the image for your machine. You have the choice between the officials images provided by Amazon (mostly base operating systems) or more specific images provided by the community.
In our case, we will choose an Ubuntu image.
Decide the size of the instance
Depending on your needs, you have to choose a size for your instance. For test purposes, a t2.micro is enough. If you want to have more information like details and monthly pricing of the different instances, you can check this website http://www.ec2instances.info/?cost=monthly.
The flexibility of Amazon allows you to change the size of your instance whenever you want without reinstalling your application.
Launch this new instance
We will tune the other settings later, click on Review and Launch. A page will summarize your new instance, click on Launch.
Select the way to authenticate against your EC2 instance
In order to connect and authenticate against your instance, you need to generate a key pair. Select “create a new key pair”, give it a name and download it.
Assign a static IP
We need to assign an elastic ip to your server. Like that even if you decide to stop and start your instance, you will still have the same static IP address.
Click on Elastic ip and confirm that you want a new elastic ip. Do a right click on your ip and choose Associate ip. Fill the form with your instance, network interface and Private ip address range.
Your instance is now attached to your VPC. You can edit the configuration and do approximately everything from this dashboard. Let’s take some interest in the security group. Click on Security groups on the left panel.
Security groups management
AWS also ensures the security of your machines. Instead of using a traditional firewall on your router, Amazon provides you security groups.
Go into security groups in the left panel. You will see 2 entrees in the list. The first one represents the security rules of your VPC; by default, all the traffic (inbound and outbound) is allowed, we will leave it like that. The second group represents the rules applied to your instance, by default just inboud ssh is allowed.
Imagine you just added a web server to your instance. All you need is just to open the port 80 (http) inbound. It’s a very efficient way to manage traffic security.
Command line and bastion instance setup
ssh access
Now it’s time to use the key you previously downloaded. Open a terminal in the directory where you stored the key and run the commands below.
ssh -i youkey.pem ubuntu@YOUR_INSTANCE_IP sudo -i
SaltStack setup
SaltStack is an open source configuration management, it will be your best friend to manage your Amazon private cloud. It offers you the ability to quickly create a new instance and deploy a configuration on it. One server is defined as the salt master, and the other ones are minions who are getting the configuration from their master.
Repository configuration
This commands below will setup the SaltStack official repository for Ubuntu 14.04 and install the salt packages.
wget -O - https://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest/SALTSTACK-GPG-KEY.pub | sudo apt-key add - echo "deb http://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest trusty main" > /etc/apt/sources.list.d/saltstack.list apt-get update apt-get install salt-master salt-cloud
SaltStack configuration deployment
We will configure a salt master on our bastion server. Let’s configure a salt profile, which will set up the bash configuration and a user profile for all our users. Today we will use this configuration only for our bastion / jump server but in the future when we will add more servers we will just have to apply to them the exact same configuration we did here.
mkdir -p /srv/salt/generic && mkdir -p /srv/salt/templates/etc/skel cd /srv/salt nano top.sls
hosts profile definitions
In Salt, the file which contains a mapping between groups of machines on a network and the configuration roles that should be applied to them is called a top file.
Right now we want only to apply the base configuration and deploy our users to all our hosts (‘*’). This will include bastion and all the instances we will launch in the future.
Content of our /srv/salt/top.sls:
base:
'*':
- generic.base
- generic.users
basic hosts configuration
This bases.sls file is going to define the basic configuration we want on all our machines like:
- shell profiles (.bashrc)
- motd messages
- pre-installed packages
- …
Content of /srv/salt/generic/base.sls:
{% set config_files = [
"/etc/skel/.bashrc"
]
%}
{% for config_file in config_files %}
{{ config_file }}:
file.managed:
- source: salt://templates/{{ config_file }}
- user: root
- group: root
- mode: 666
- template: jinja
- makedirs: True
{% endfor %}
/root/.bashrc:
file.managed:
- source: salt://templates/etc/skel/.bashrc
- template: jinja
bash configuration file
Content of /srv/salt/templates/etc/skel/.bashrc:
# System-wide .bashrc file for interactive bash(1) shells.
# To enable the settings / commands in this file for login shells as well,
# this file has to be sourced in /etc/profile.
# If not running interactively, don't do anything
[ -z "$PS1" ] && return
# check the window size after each command and, if necessary,
# update the values of LINES and COLUMNS.
shopt -s checkwinsize
# set variable identifying the chroot you work in (used in the prompt below)
if [ -z "${debian_chroot:-}" ] && [ -r /etc/debian_chroot ]; then
debian_chroot=$(cat /etc/debian_chroot)
fi
if [[ EUID -eq 0 ]]; then
export PS1='\[\e[0;31m\]\u@\h\[\e[m\] \[\e[1;34m\]\w\[\e[m\] \[\e[0;31m\]\$ \[\e[m\]\[\e[0m\]'
else
export PS1='\[\e[0;32m\]\u@\h\[\e[m\] \[\e[1;34m\]\w\[\e[m\] \[\e[1;32m\]\$ \[\e[m\]\[\e[0m\]'
fi
# enable color support of ls and also add handy aliases
if [ -x /usr/bin/dircolors ]; then
test -r ~/.dircolors && eval "$(dircolors -b ~/.dircolors)" || eval "$(dircolors -b)"
alias ls='ls --color=auto'
alias grep='grep --color=auto'
alias fgrep='fgrep --color=auto'
alias egrep='egrep --color=auto'
fi
# some more ls aliases
alias ll='ls -alF --color=auto'
alias la='ls -A --color=auto'
alias l='ls -CF --color=auto'
alias scl='salt-call --local'
alias hf='hostname -f'
alias t="tail -fn100"
# if the command-not-found package is installed, use it
if [ -x /usr/lib/command-not-found -o -x /usr/share/command-not-found/command-not-found ]; then
function command_not_found_handle {
# check because c-n-f could've been removed in the meantime
if [ -x /usr/lib/command-not-found ]; then
/usr/lib/command-not-found -- "$1"
return $?
elif [ -x /usr/share/command-not-found/command-not-found ]; then
/usr/share/command-not-found/command-not-found -- "$1"
return $?
else
printf "%s: command not found\n" "$1" >&2
return 127
fi
}
fi
Users definitions
One of the big advantage of salt is the fact you manage your user authentication with the usual local files like /etc/passwd and /etc/shadow.
In this case we will:
- create / update the linux user
- set the password
- setup a ssh public key for this account
Content of /srv/salt/generic/users.sls:
{%
set user_list = [
{
"id": "msol",
"uid": 1001,
"fullname": "MSolution",
"password": "$1$5Wm4UFH3$gcWZBd1qxxBT23BT565Dj1",
"envs": []
}
]
%}
sudo:
group.present:
- system: True
{% for user in user_list %}
{{ user["id"] }}:
user.present:
- fullname: {{ user["fullname"] }}
- shell: /bin/bash
- home: /home/{{ user["id"] }}
- createhome: true
- uid: {{ user["uid"] }}
- groups:
- sudo
- gid_from_name: true
- password: {{ user["password"] }}
file.absent:
- name: /home/{{ user["id"] }}/.ssh/authorized_keys
ssh_auth:
- require:
- user: {{ user["id"] }}
- file: /home/{{ user["id"] }}/.ssh/authorized_keys
- present
- user: {{ user["id"] }}
- source: salt://ssh-keys/{{ user["id"] }}.pub
{% endfor %}
This configuration requires a ssh key, let’s add ours into /srv/salt/ssh-keys/msol.pub
Now, we need to add this to the top.sls configuration
You can see the following rule who will set all users in our top.sls with this specific line:
- generic.users
Running a “salt-call –local state.highstate” will apply the configuration to the master:
- setting up the base system
- creating local users
salt-call --local state.highstate
Now our bastion host also acting as a salt-master is up and running we will be able to use salt from there in the future to deploy and maintain new instances.
The salt configuration options are infinite and you can basically create a template for anything.
Next!
Now you have your EC2 bastion host up and running. In the next article, you will be able to setup your own transcoding instance through salt-cloud. Salt-cloud is a tool for automatically provisioning and managing cloud servers within and across supported cloud providers (AWS, rackspace, joyent…).