CI/CD Pipelines, GitHub Actions, and CircleCI

CI/CD pipelines are a set of automated processes that help to build, test, and deploy software applications. The acronym CI/CD stands for Continuous Integration and Continuous Deployment. The goal of CI/CD is to continuously integrate and test code changes as they are made, and to automatically deploy the code to production if it passes all tests.

GitHub Actions and CircleCI are two popular continuous integration and continuous delivery (CI/CD) platforms that provide developers with the tools and functionality to automate the software development process, including building, testing, and deploying code changes.

Terragrunt

Terragrunt is a tool for building, changing, and versioning infrastructure safely and efficiently. It is a thin wrapper for Terraform that provides extra tools for keeping Terraform configurations DRY (Don’t Repeat Yourself principle of writing once and preventing repetitions), working with multiple Terraform modules, and managing remote state. Terragrunt enables companies to keep Terraform code organized and reusable across multiple environments and teams.

CI/CD pipelines built with Terragrunt allow developers to safely and efficiently manage infrastructure as code by automatically testing, building, and deploying infrastructure changes. This enables teams to quickly and easily roll out new updates and features while maintaining a high level of control and visibility over infrastructure.

Below are two step-by-step tutorials on how to set up a CI/CD pipeline with Terragrunt, AWS, and GitHub Actions or CircleCI.

Tutorial #1 – How to Set Up a CI/CD Pipeline with Terragrunt, AWS, and GitHub Actions

Github Repository of Tutorial

1. Setup Terragrunt Repository

A. File Structure

$ tree -r . ├── terragrunt.hcl └── tenant     ├── us-west-2     │   └── dev     │       ├── vpc     │       │   └── terragrunt.hcl     │       └── env.hcl     └── account.hcl 4 directories, 4 files

B. Explain VPC Terragrunt Configuration

i – Explain Github Base Source URL (for private Terraform module repositories)

In tenant/us-west-2/dev/vpc there is the Terragrunt configuration for the VPC that needs to be deployed.

In the locals constants there are 3 variables for the source URL:

• gh_source_url
• source_url
• base_source_url

The gh_source_url should be used only when the Terraform modules are in private GitHub repositories. In this case, a PAT is required so that Terragrunt can pull the other repositories from GitHub Actions. Readers can learn more about this in the section titled ‘Configure Github Action’.

If users don’t need to have access to private repositories, they can simply remove the gh_source_url variable and replace the current base_source_url variable by the source_url variable:

locals {   […]   ref             = “?ref=main”   gh_source_url   = “git::https://github.com/TrackIt/tf-modules.git//vpc_example${local.ref}”   source_url      = “git::git@github.com:TrackIt/tf-modules.git//vpc_example${local.ref}”   base_source_url = get_env(“IS_GH”, “FALSE”) == “TRUE” ? local.gh_source_url : local.source_url }

The get_env function from Terragrunt allows users to get an environment variable and set a value by default if it doesn’t already exist. For instance, when trying to access the IS_GH environment variable, it is set to FALSE if the variable doesn’t exist. If this environment variable is set to TRUE it will use the gh_source_url as the base_source_url in the flow.

2. Create New GitHub Action

The next step is to create a GitHub Action which will plan the dev infrastructure in the tenant/us-west-2/dev folder.

A. Create folder `.github/worklows`

$ mkdir -p .github/workflows

B. Create `dev-plan.yml` file in `.github/worklows`

$ touch dev-plan.yml

3. Configure GitHub Action

A. Setup Workflow (i.e.: on workflow dispatch)

When configuring Github Actions, users have to specify what triggers the Github Action. In this example, the Workflow Dispatch event is being used to manually trigger deployments in the GitHub Action dashboard. Readers can find the available event list here.

The following is the configuration for the Workflow Dispatch:

name: ‘Dev Plan’ on: workflow_dispatch:

B. Setup Environment Variables (credentials, working directory)

To deploy the infrastructure, users have to set up environment variables:

[…] env: WORKING_DIRECTORY: ‘./tenant/us-west-2/dev’ AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} IS_GH: “TRUE”

• WORKING_DIRECTORY: This is the directory where users are going to deploy from. In this example, all dev resources are deployed in the us-west-2 region from the tenant account.
• AWS_ACCESS_KEY_ID: the AWS Secret Key ID used to deploy the infrastructure (the user who deploys the infrastructure needs to have the correct permissions, see IAM permissions). A Github Action secret is also created for this value so that it can be protected. 
• AWS_SECRET_ACCESS_KEY: the AWS Secret Access Key corresponding to the Access Key ID. A GitHub Action secret has also been created for this value so that it can be protected.
• IS_GH: This environment variable is required to use private Terraform module repositories. It must be set to TRUE.

C. Configure Job

Now that the Github Action has been globally configured, the next step is setting up the jobs. In this example, the job being created is going to init and plan Terragrunt.

[…] jobs: plan:   name: ‘Terragrunt’   runs-on: ubuntu-latest   steps:     – name: ‘Checkout’       uses: actions/checkout@master

i. Setup Terragrunt / Terraform steps

First, users need to install Terraform and Terragrunt:

[…] steps: […] – uses: hashicorp/setup-terraform@v2   with:     terraform_version: 1.0.0     terraform_wrapper: false – name: Setup Terragrunt   uses: autero1/action-terragrunt@v1.1.0   with:     terragrunt_version: 0.37.1

ii. Setup Git Global Config (for private Terraform module repositories)

Readers who are not using private Terraform module repositories can skip this step. Otherwise, git global config is set up using a PAT to access those repositories. The PAT can be saved as a secret; in this example, a GitHub Action secret called TF_MODULES_TOKEN has been created.

[…] steps: […] – name: ‘Setup Git Global Config’   run: |   git config –global url.”https://${{ secrets.TF_MODULES_TOKEN }}@github”.insteadOf https://github

iii. Terragrunt Init and Plan

Users can now interact with Terragrunt to init, plan, or even apply. In this example, only init and plan are being executed but the same can be done to apply infrastructure:

[…] steps: […] – name: ‘Terragrunt Init’   run: terragrunt run-all init –terragrunt-non-interactive   working-directory: ${{ env.WORKING_DIRECTORY }} – name: ‘Terragrunt Plan’   run: terragrunt run-all plan –terragrunt-non-interactive   working-directory: ${{ env.WORKING_DIRECTORY }}

As seen above, the WORKING_DIRECTORY environment variable is being used to run the Terragrunt commands only in the correct directory. Users can now init and plan resources using GitHub Actions and Terragrunt.

Tutorial #2 – How to Set Up a CI/CD Pipeline with Terragrunt, AWS, and CircleCI

1. Setup Workflow

By default, CircleCI will automatically test, build and deploy infrastructure changes when the GitHub repository contains a CircleCI configuration.

2. Create Config File

The CircleCi configuration file must be stored in .circleci/config.yml

It will contain the definition of the different steps of the build.

An orb (plug-in) is also required to enable access to the AWS CLI.

version: 2.1orbs:  aws-cli: circleci/aws-cli@3.1.1jobs:  […]workflows:  […]

3. Set Up Environment Variables

Environment variables are configured on the CircleCI console under Project Settings -> Environment variables

Set up the following:

• AWS_ACCESS_KEY_ID: the AWS Secret Key ID used to deploy the infrastructure (the user who deploys the infrastructure needs to have the correct permissions, see IAM permissions). 
• AWS_SECRET_ACCESS_KEY: the AWS Secret Access Key corresponding to the Access Key ID. 
• GITHUB_TOKEN:  the GitHub secret token to clone the modules repository

4. Build Steps for Terraform Plan and Apply

[…]jobs:  init_plan:    docker:      – image: alpine/terragrunt:1.1.0    working_directory: ~/infrastructure-live/admin    steps:      – checkout:           path: ~/infrastructure-live      – aws-cli/setup:         aws-access-key-id: AWS_ACCESS_KEY         aws-secret-access-key: AWS_ACCESS_SECRET         aws-region: AWS_REGION      – run:          name: Terragrunt init          command: |            terragrunt run-all init      – run:          name: Terragrunt validate          command: |            terragrunt run-all validate      – run:          name: Terragrunt plan          command: |            terragrunt run-all plan  deploy:    docker:      – image: alpine/terragrunt:1.1.0    working_directory: ~/infrastructure-live/admin    steps:      – checkout:           path: ~/infrastructure-live      – aws-cli/setup:         aws-access-key-id: AWS_ACCESS_KEY         aws-secret-access-key: AWS_ACCESS_SECRET         aws-region: us-east-1      – run:          name: Terragrunt apply          command: |            terragrunt run-all apply –terragrunt-non-interactiveworkflows:  version: 2  deploy_changes:    jobs:      – init_plan      – deploy

With this configuration, CircleCI will execute 2 steps, one for plan and one for apply. This configuration can be upgraded with the addition of a manual validation of the plan if needed.

How TrackIt Can Help

Companies in need of assistance can partner with an AWS-recognized partner like TrackIt to automate infrastructure provisioning by integrating Terragrunt into their CI/CD pipelines. TrackIt can also assist in the integration of AWS services like AWS Elastic Container Service (ECS), AWS Elastic Kubernetes Service, and AWS Elastic Beanstalk to help achieve high availability, scalability, and security.

About TrackIt

TrackIt is an international AWS cloud consulting, systems integration, and software development firm headquartered in Marina del Rey, CA.

We have built our reputation on helping media companies architect and implement cost-effective, reliable, and scalable Media & Entertainment workflows in the cloud. These include streaming and on-demand video solutions, media asset management, and archiving, incorporating the latest AI technology to build bespoke media solutions tailored to customer requirements.

Cloud-native software development is at the foundation of what we do. We specialize in Application Modernization, Containerization, Infrastructure as Code and event-driven serverless architectures by leveraging the latest AWS services. Along with our Managed Services offerings which provide 24/7 cloud infrastructure maintenance and support, we are able to provide complete solutions for the media industry.