Building a CI/CD Workflow with CircleCI

Leveraging CircleCI to Build a CI/CD Pipeline For The Healthline Media Activation Platform

Healthline Media, a TrackIt AWS Managed DevOps customer, was in need of a CI/CD pipeline for its Healthline Activation Platform packaged in a Windows Docker container.

Healthline leverages CircleCI for all of its existing CI/CD pipelines and wanted the Activation Platform (which up until that point was using Jenkins) to conform to the same standards in order to maintain its normal processes and maximize efficiencies.

“We wanted to streamline the CI/CD pipeline so that it matches what we do everywhere else and also remove the need for an in-house maintained Jenkins server. That was the main driver for us to switch from Jenkins to CircleCI.” — Pierre Grandin, Lead Infrastructure Engineer, Healthline Media

Having already collaborated with TrackIt on previous projects, Pierre Grandin, Lead Infrastructure Engineer at Healthline Media was confident seeking the TrackIt team’s expertise in building this pipeline despite the challenging nature of a Windows container project.

“Victor [DevOps engineer at TrackIt] has worked on a few different projects for us and it went well every time! He was already quite familiar with the project itself, the technologies involved, and the team involved with it. All of this made it very easy for him to start working on the project. For me, it was a no-brainer to ask Victor to work on this for us.” — Pierre Grandin, Lead Infrastructure Engineer, Healthline Media

Strategies Considered to Build the CI/CD Pipeline

Due to the novelty of Windows containers, the Healthline and TrackIt teams considered two separate strategies to build the pipeline — each with its own advantages and disadvantages:

• Self-hosted build server (Jenkins)

Advantages: The server is highly-customizable and can be kept dry so no update breaks the build in an unforeseen way.

Disadvantages: The server requires maintenance and does not conform to the build pattern used by all the other CircleCI-based projects.

• SaaS solution (CircleCI)

Advantages: A SaaS solution is low maintenance and requires a smaller initial effort to deploy

Disadvantages: Windows containers are relatively new and come with a few caveats (which we’ll describe in the following sections) .

The teams decided to go with the CircleCI CI/CD SaaS solution approach since it fit homogeneously into the build pipeline for all of Healthline’s projects.

Challenges Faced

• Challenge #1 — CircleCI’s Lack of Documentation For Windows Containers

CircleCI’s lack of documentation for Windows containers was an obstacle that the TrackIt team had to overcome. A considerable amount of trial and error was required to resolve several issues due to a lack of adequate documentation.

The available documentation only consisted of a “hello world” pipeline and a list of installed software. The following is the example pipeline that was provided:

version: 2.1

orbs:
  win: circleci/windows@2.2.0

jobs:
  build:
    executor:
      name: win/default
    steps:
      # default shell is Powershell
      - run:            
         command: $(echo hello | Out-Host; $?) -and $(echo world | Out-Host; $?)
         shell: powershell.exe
      - run:
         command: echo hello && echo world
         shell: bash.exe
      - run:
         command: echo hello & echo world
         shell: cmd.exe

• Challenge #2 — PATH env var: Windows and bash

One of the challenges encountered was figuring out how the PATH env var worked between Windows and bash, since from within bash the path to C:\ is not formatted the same way as Windows PATH env var. For example, here’s how to add a dependency to the path from within bash:

export PATH=$PATH:/c/Python38

• Challenge #3 — Version Dependency of Windows Containers

The TrackIt team also had to navigate through another issue that has more to do with Windows Containers than CircleCI — the fact that Windows containers are version dependent. This means that the Windows instance (in our case, an ECS Windows instance node) must be at least up to date with the version of Windows where the container was built. This version dependency could also pose problems in the future if CircleCI was to update its version of Windows — since the containers would not start with an older version of Windows.

This issue was addressed by looking at the build number of the Docker image created by CircleCI and matching it with the corresponding AMI on the Windows ECS cluster nodes. AWS provides a list of all Windows AMI ids, matching them with Windows build versions: https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/windows-ami-version-history.html.

Since CircleCI doesn’t provide the exact Windows build number, it had to be extracted from the Docker image.

Diagram of the Build Pipeline

Breakdown of the Implemented Pipeline

• When a commit is pushed, either on the “staging” or “master” branch, the CircleCI build is triggered. This is done automatically by CircleCI, via a webhook added on the GitHub repository :

• When the pipeline starts it automatically clones the repository to the correct reference
• The first step of the build is to install the dependencies needed. These include Visual Studio test agent, terraform and other miscellaneous tools. One of the limitations of Windows on CircleCI is that dependencies have to be installed each time, and cannot be baked into the base image. The following are the necessary dependencies for the build process :

- run:
  name: "Install dependencies"
  command: |
    choco install -y wget
    choco install -y terragrunt --version 0.18.6
    choco install -y terraform --allow-downgrade --version 0.11.14
    choco install -y python
    choco install -y visualstudio2019testagent --package-parameters "--includeRecommended --includeOptional"

• The second step in the build pipeline installs the project dependencies (which are different from the system dependencies in the last step). This is done with the NuGet program, which is a dependency manager for Windows environments:

• The third step in the pipeline involves decrypting the secrets in the config file (database passwords). This allows the configuration to be kept in the GitHub repository without compromising security by having secrets stored in plaintext. This is what the config looks like in the repository:

• The fourth step of the pipeline builds and compiles the project. It does a clean build by making sure to rebuild all binaries. This is important because a “normal” build (without explicitly specifying a full rebuild of the application) wouldn’t re-compile the binaries used by the testing pipeline, and as such the tests would fail to run as they wouldn’t be up to date with other components in the application.

• The fifth step temporarily authorizes ingress connections to the test database of the CircleCI IP. This is achieved using the following AWS CLI command, after assuming the IAM user provided in the env variables of the build :

• The sixth step of the pipeline runs the unit tests of the project. If any of the tests fail the pipeline stops and no code is deployed.
• The seventh step removes the temporary authorization to the test database.
• The eighth and last step in the pipeline builds the docker container, pushes it to Amazon ECR, and finally deploys it on the ECS cluster. The following is the output of the final step of the build using Terraform to deploy the new docker image:

This step takes the longest amount of time (about 8–9 minutes) since the base docker image needs to be pulled each time (about 2–3 GB to download) and is not cached by CircleCI.

Breakdown of Time Taken by Each Step in the Build Pipeline

“In our ongoing engagement with Healthline, TrackIt has successfully completed the streamlining of challenging workflows that not only improve Healthline’s operating environment but also expands TrackIt’s own expertise. It has been mutually very successful and we look forward to our continued partnership” Ludovic Francois, TrackIt CEO

About Healthline Media

Healthline is the fastest growing health information brand with offices in San Francisco, New York, and the UK. Our vision is to create a stronger, healthier world. Through our three properties — Healthline, Medical News Today, and Greatist, every member of our team works to provide the best news, tools, and information to our consumers. As the #1 health site in the US (per Comscore), we reach over 250 million people a month globally.

About TrackIt

TrackIt is an international AWS cloud consulting, systems integration, and software development firm headquartered in Marina del Rey, CA.

We have built our reputation on helping media companies architect and implement cost-effective, reliable, and scalable Media & Entertainment workflows in the cloud. These include streaming and on-demand video solutions, media asset management, and archiving, incorporating the latest AI technology to build bespoke media solutions tailored to customer requirements.

Cloud-native software development is at the foundation of what we do. We specialize in Application Modernization, Containerization, Infrastructure as Code and event-driven serverless architectures by leveraging the latest AWS services. Along with our Managed Services offerings which provide 24/7 cloud infrastructure maintenance and support, we are able to provide complete solutions for the media industry.