Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

On-Demand Test Environments With Ansible and Shippable

DZone's Guide to

On-Demand Test Environments With Ansible and Shippable

Learn how to set up on-demand test environments using Ansible and Shippable as part of implementing an end-to-end Continuous Delivery pipeline.

· DevOps Zone ·
Free Resource

DevOps involves integrating development, testing, deployment and release cycles into a collaborative process. Learn more about the 4 steps to an effective DevSecOps infrastructure.

One of the biggest challenges to implementing an end-to-end Continuous Delivery pipeline is making sure adequate test automation is in place. However, even if you have automated across your entire Test Suite, there is a second challenge: How do you manage Test infrastructure and environments without breaking the bank?

If you want to move towards Continuous Delivery, you need to execute a majority of your tests for each code change in a pristine environment that is as close to your production environment as possible. This ensures that code defects are identified immediately and every code change is therefore "shippable." However, creating these environments and updating them with each new application version or every time the config changes adds a lot of overhead. If you're testing an application with many tiers or microservices, the complexity increases since each tier might need to be tested independently in its own environment against specific versions of other tiers.

The utopia of Test Automation is the following:

  • Environment definitions are represented by infrastructure-as-code tools like Ansible, Terraform, Puppet, or Chef. The provisioning scripts are committed to source control and versioned, so you can go back to an earlier state if needed.
  • All (or at least a good majority) of your tests are automated and either committed to source control or hosted on services such as Nouvola, Sauce, etc.
  • You have a completely automated deployment pipeline that automatically spins up a production-like Test environment for every code change, triggers your automation, and if all tests succeed, destroys the environment. If tests fail, the right folks are notified and the environment is kept live until someone can debug the failures.

The first step is already happening in most organizations. The DevOps movement encouraged Ops teams to start writing scripts to provision and manage environments and infrastructure, and multiple vendors support this effort quite effectively. The second step is still a challenge in many organizations, but this is really something that needs executive buy-in and a commitment to automation, even if it slows down product development for a while.

This whitepaper presents a method of implementing the third step - spinning up test environments on-demand and destroying them automatically after the automated test suite is executed.

The Scenario

To make things simpler, we'll skip the CI step which builds and tests the application Docker image and pushes it to Amazon ECR. This can be accomplished by following instructions for CI: Run CI for a sample app.

Our example follows the steps below:

1. A service definition, aka manifest, is created, including the Docker image and some options

2. A test environment is provisioned using Ansible under the cover. Ansible config files are templatized using environment variables defined in

Shippable, allowing Ansible config to become highly reusable to provision multiple test clusters if needed.

3. The manifest is deployed to the Test environment and functional test suite is triggered

4. If tests pass, the test environment is destroyed using Ansible and test owner is notified.

5. If tests fail, the test owner is notified and the environment is not destroyed. The Test owner can always destroy the environment manually after he/she has extracted the information they need about the failure.

Before We Start

You will need the following to implement this scenario:

  • A GitHub or Bitbucket account that you will use to login to Shippable
  • An AWS account
  • A Docker Hub account (or Amazon ECR/GCR/Quay)
  • Ideally, some familiarity with Ansible is desirable, though not required.

If you're not familiar with Shippable, here are some basic concepts you should know before you start:

  • Configuration: The Assembly Lines configuration for Shippable resides in a shippable.yml file. The repository that contains this config in your source control is called a Sync Repository, aka syncRepo. A syncRepo is added through your Shippable UI to add your Assembly Line.
  • Jobs are executable units of your pipeline and can perform any activity such as CI, provisioning an environment, deploying your application, or running pretty much any custom script. A simple way to think of it is, if something can execute in the shell of your laptop, it can execute as a Job.
  • To execute, such as credentials, pointer to Resources typically contain information needed for a Jobs cluster on a Container Engine or an image on a Hub, or any key-value pairs. Resources are also used to store information produced by a job which can be then accessed by downstream jobs.
  • Integrations are used to configure connections to third-party services, such as AWS, Docker Hub, GKE, Artifactory, etc.
  • The Single Pane of Glass view shows a real-time, interactive view of your Assembly Line(s).

How the Sample Application Is Structured

Our sample repositories are on GitHub:

  • The sample application that we will run functional tests on is a voting app that is built using Python Flask and Redis. The source for the front end (Flask) can be found in the vote_fe repository and the backend (Redis) in the vote_be repository. The shippable.yml in these repositories contains the CI configuration to build and deploy their Docker images to their public repositories on Docker Hub.
  • devops-recipes/on_demand_test_environments contains the Shippable configuration and required Ansible playbooks for this scenario. The sections below explain in detail how the Shippable configuration is built.

Step 1: Enabling CI for the Sample Application

  • Fork the vote_fe and the vote_be repositories into your SCM.
  • Login to Shippable with your SCM account and enable CI using these steps.
  • Create a Docker Registry integration using these steps and call it drship_dockerhub. If you use a different integration name, replace drship_dockerhub in the shippable.yml file.
  • Specify your Docker repository and account in the shippable.yml file and commit the file.
  • Trigger CI for these repositories using these steps.

At the end of Step 1, you should have two images published in your Docker registry integration.

Step 2: Create the Service Definition

A. Define the Resource in the shippable.yml File

The shippable.yml file can be committed in one of the app repositories or to a separate repository. We have used a different repository, devops-recipes/on_demand_test_environments, in our sample. The repository containing your jobs and resources ymls is called a Sync repository and represents your workflow configuration.

resources: 
###---------------------------------------------------------------# 
###----------------------- BUILD/CI Resources --------------------# 
###---------------------------------------------------------------# 
# Back-end image 
  - name: vote_be_odte 
    type: image 
    # replace dr-dockerhub with your docker registry integration name 
    integration: dr-dockerhub pointer: 
    # replace devopsrecipes/vote_be with your repository 
      sourceName: "devopsrecipes/vote_be" 
      seed: 
      # specify the latest tag of the image in your docker registry 
        versionName: "master.2" 

# Front-end image 
  - name: vote_fe_odte 
    type: image 
    # replace dr-dockerhub with your docker registry integration name 
    integration: dr-dockerhub 
    pointer: 
    # replace devopsrecipes/vote_fe with your repository 
      sourceName: "devopsrecipes/vote_fe" 
    seed: 
    # specify the latest tag of the image in your docker registry 
    versionName: "master.3"

# Docker options to expose port 80 on the front-end container and link the redis container 
  - name: vote_fe_options_odte 
    type: dockerOptions 
    version: 
      memory: 128 
      portMappings: 
        - "80:5000/tcp" 
      links: 
        - vote_be_odte:redis

B. Define the Jobs in the shippable.yml File

create_app_man_odteis a manifest job that defines all the containers that run in the ECS cluster. This definition is versioned and each version is immutable.

Add the following to your shippable.yml file and commit it.

jobs: 
#---------------------------------------------------------------#
#------------------- BUILD/CI with SHIPPABLE CI ----------------#
#---------------------------------------------------------------#

# CI job definition. The image that is pushed to Docker hub is specified in an OUT image resource. 
# This image resource becomes an IN to the manifest job and triggers the manifest job whenever 
# a new image version (tag) is created. 

  - name: vote_be_runCI 
    type: runCI 
    steps: 
      - OUT: vote_be_odte 

  - name: vote_fe_runCI 
    type: runCI 
    steps: 
      - OUT: vote_fe_odte 

# Application service definition 

  - name: create_app_man_odte 
    type: manifest 
    steps: 
       - IN: vote_fe_odte 
       - IN: vote_fe_options_odte 
         applyTo: 
           - vote_fe_odte 
       - IN: vote_be_odte

Step 3: Provision the Test Environment

We use an Ansible playbook to create the ECS cluster, whose implementation can be found here.

We templatize the Ansible configuration files to make them flexible.The configuration in then defined by Shippable generated environment variables and resources.

A. Ansible.cfg File

Here we use the SCRIPTS_REPO_ODTE_STATE environment variable to point to the root of the repository when the playbook is run in a Shippable node.

[defaults] 
# update, as needed, for your scenario 
host_key_checking=false 
inventory = ${SCRIPTS_REPO_ODTE_STATE}/infra/provision-ecs-ansible/inventory/ 

[ssh_connection] 
# for running on Ubuntu 
control_path=%(directory)s/%%h-%%r

B. Group Variables

All the variables used by Ansible modules to create the cluster are defined as placeholders. These placeholders are replaced at runtime by values defined in a params resource.

ec2_instance_type: "${EC2_INSTANCE_TYPE}" 
ec2_image: "${EC2_IMAGE}" 
ec2_keypair: "${EC2_KEYPAIR}" 
ec2_user_data: "#!/bin/bash \n echo ECS_CLUSTER=\"${ECS_CLUSTER_NAME}\" >> /etc/ecs/ecs.config" 
ec2_region: "${EC2_REGION}" 
ec2_tag_Role: "${EC2_TAG_ROLE}" 
ec2_tag_Type: "${EC2_TAG_TYPE}" 
ec2_volume_size: ${EC2_VOLUME_SIZE} 
ec2_count: ${EC2_COUNT} 
STATE_RES_NAME: "${STATE_RES_NAME}" 
ec2_security_group: "${TEST_PUBLIC_SG_ID}" 
ec2_subnet_ids: ["${TEST_PUBLIC_SN_01_ID}","${TEST_PUBLIC_SN_02_ID}"] 
ec2_tag_Environment: "${ENVIRONMENT}" 
ECS_CLUSTER_NAME: "${ECS_CLUSTER_NAME}"

C. Define Ansible Configuration in the shippable.yml File

resources: 

#---------------------------------------------------------------# 
#-------------------- Common INFRA Resources -------------------# 
#---------------------------------------------------------------# 

# Ansible scripts repository 
  - name: scripts_repo_odte 
    type: gitRepo 
    integration: "dr-github" 
    pointer: 
      sourceName: "devops-recipes/on-demand-test-environments" 
      branch: master 

# AWS integration that sets up the AWS CLI environment used by Ansible playbook 
  - name: aws_cli_config_odte 
    type: cliConfig 
    integration: dr-aws-keys 
    pointer: 
      region: us-east-1 

# SecOps approved AMI 
  - name: ami_sec_approved_odte 
    type: params 
    version: 
      params: 
        AMI_ID: "ami-9eb4b1e5" 

#---------------------------------------------------------------# 
#----------------------- TEST VPC Resources --------------------# 
#---------------------------------------------------------------# 

# TEST environment config 
  - name: test_conf_odte 
    type: params 
    version: 
      params: 
        EC2_REGION: "us-east-1" 
        EC2_TAG_ROLE: "dr-on-demand-test-environments" 
        EC2_TAG_TYPE: "ecs-container-instance" 
        EC2_VOLUME_SIZE: 30 
        EC2_COUNT: 1 
        STATE_RES_NAME: "test_info_odte" 
        ECS_CLUSTER_NAME: "test_env_ecs_odte" 
        ENVIRONMENT: "test" 
        EC2_INSTANCE_TYPE: "t2.large" 
        EC2_IMAGE: "ami-9eb4b1e5" 
        EC2_KEYPAIR: "ambarish-useast1" 

# Test VPC Info 
  - name: test_vpc_conf_odte 
    type: params 
    version: 
      params: 
        TEST_VPC_ID: "vpc-a36912da" 
        TEST_PUBLIC_SG_ID: "sg-c30fc8b6" 
        TEST_PUBLIC_SN_01_ID: "subnet-34378e50" 
        TEST_PUBLIC_SN_02_ID: "subnet-34378e50" 
        REGION: "us-east-1" 

# Output of Test ECS Provisioning 
  - name: test_info_odte 
    type: params 
    version: 
      params: 
        SEED: "initial_version" 

# Reference to ECS Test Cluster 
  - name: test_env_ecs_odte 
  type: cluster 
  integration: "dr-aws-keys" 
  pointer: 
     sourceName : "test_env_ecs_odte" 
     region: "us-east-1"

D. Augment the Ansible Playbook ansible-ecs-provision That Provisions the ECS Cluster

After the cluster is created, we use Shippable platform resources and API to persist important cluster metadata such as the ARN and public IP of the cluster in a params resource test_info_odte and the cluster resource test_env_ecs_odte.

The ansible-ecs-provision playbook calls two roles to provision the ECS cluster.

--- 
### provision AWS ECS cluster 
- hosts: localhost 
  connection: local 
  gather_facts: false 
  user: root 
  pre_tasks: 
    - include_vars: group_vars/ecs-cluster-vars.yml 
  roles: 
    - ecs-cluster-provision 
    - ec2-container-inst-provision 
  post_tasks: 
    - name: refresh hosts inventory list 
      meta: refresh_inventory
  • ecs-cluster-provision
    --- 
    # update Shippable resource state with this job number 
    - name: run cmd 
      shell: | 
        shipctl post_resource_state "" versionName "build-${BUILD_NUMBER}" 
    
    # provision ECS cluster 
    - name: Create ECS Cluster 
      ecs_cluster: 
        name: "" 
        state: present 
      register: ecs 
    
    # update shippable resource state with provisioned cluster_arn 
    - name: run cmd 
      shell: | 
        shipctl put_resource_state "" CLUSTER_ARN "" 
        shipctl put_resource_state "" TEST_ECS_CLUSTER_ID ""
  • ecs-cluster-provision
    --- 
    - name: Provision instances with tag 
      local_action: 
        module: ec2 
        key_name: "" 
        group_id: "" 
        instance_type: "" 
        instance_profile_name: "ecsInstanceRole" 
        image: "" 
        user_data: "" 
        vpc_subnet_id: "" 
        region: "" 
        instance_tags: '{"Name":"","Role":"","Type":"","Environment":""}' 
        assign_public_ip: yes 
        wait: true 
        exact_count: "" 
        count_tag: 
          Role: "" 
        volumes: 
          - device_name: /dev/xvda 
          volume_type: gp2 
          volume_size: "" 
          delete_on_termination: true 
       register: ec2 
    
    - add_host: 
        name: "{{item.public_ip}}" 
        groups: tag_Type_,tag_Environment_ 
        ec2_region: "" 
        ec2_tag_Name: "" 
        ec2_tag_Role: "" 
        ec2_tag_Type: "" 
        ec2_tag_Environment: "" 
        ec2_ip_address: "{{item.public_ip}}" 
      with_items: "" 
    
    - name: Wait for the instances to boot by checking the ssh port 
      wait_for: host={{item.public_ip}} port=22 delay=15 timeout=300 state=started 
      with_items: "" 
    
    # update shippable resource state 
      - name: run cmd 
        shell: | 
          shipctl put_resource_state "" "INST_{{item.ami_launch_index}}_PUBLIC_IP" "{{item.public_ip}}" 
          shipctl put_resource_state "" "INST_{{item.ami_launch_index}}_ID" "{{item.id}}" 
          shipctl put_resource_state "" "REGION" "" 
          shipctl put_resource_state "" "INST_{{item.ami_launch_index}}_PUBLIC_IP" "{{item.public_ip}}" 
          shipctl put_resource_state "" "INST_{{item.ami_launch_index}}_ID" "{{item.id}}" 
        with_items: ""

shipctl provides a comprehensive library of utilities that can be used to extract and persist useful data in a Shippable param or state resource. This data can then be used by jobs downstream.

E. Define the Shippable Job That Runs the Ansible Playbook ansible-ecs-provision.yml

Add the following to shippable.yml file and commit it.

--- 
- name: Provision instances with tag 
  local_action: 
    module: ec2 
    key_name: "" 
    group_id: "" 
    instance_type: "" 
    instance_profile_name: "ecsInstanceRole" 
    image: "" 
    user_data: "" 
    vpc_subnet_id: "" 
    region: "" 
    instance_tags: '{"Name":"","Role":"","Type":"","Environment":""}' 
    assign_public_ip: yes 
    wait: true 
    exact_count: "" 
    count_tag: 
      Role: "" 
    volumes: 
       - device_name: /dev/xvda 
         volume_type: gp2 
         volume_size: "" 
         delete_on_termination: true 
   register: ec2 

- add_host: 
    name: "{{item.public_ip}}" 
    groups: tag_Type_,tag_Environment_ 
    ec2_region: "" 
    ec2_tag_Name: "" 
    ec2_tag_Role: "" 
    ec2_tag_Type: "" 
    ec2_tag_Environment: "" 
    ec2_ip_address: "{{item.public_ip}}" 
  with_items: "" 

- name: Wait for the instances to boot by checking the ssh port 
  wait_for: host={{item.public_ip}} port=22 delay=15 timeout=300 state=started 
  with_items: "" 

- name: display ecs cluster 
  debug: 
    msg: "{{item}}" 
  with_items: "" 

# update shippable resource state 
- name: run cmd 
  shell: | 
    shipctl put_resource_state "" "INST_{{item.ami_launch_index}}_PUBLIC_IP" "{{item.public_ip}}" 
    shipctl put_resource_state "" "INST_{{item.ami_launch_index}}_ID" "{{item.id}}" 
    shipctl put_resource_state "" "INST_{{item.ami_launch_index}}_PUBLIC_DNS" "{{item.public_dns_name}}" 
    shipctl put_resource_state "" "REGION" "" 
    shipctl put_resource_state "" "INST_{{item.ami_launch_index}}_PUBLIC_IP" "{{item.public_ip}}" 
    shipctl put_resource_state "" "INST_{{item.ami_launch_index}}_ID" "{{item.id}}" 
  with_items: ""

Step 4: Deploy the Application to the Test ECS Environment

deploy_app_test_odte is a deploy job, which creates the service and task definition in the ECS cluster and starts the service. Since it needs to run after the ECS cluster is created in the workflow, prov_test_vpc_odte is specified as an input.

Add the following to the shippable.yml file and commit it.

jobs: 

##---------------------------------------------------------------# 
##-------------------- App Release Automation -------------------# 
##---------------------------------------------------------------# 

# DEPLOY to TEST environment 
  - name: deploy_app_test_odte 
    type: deploy 
    steps: 
      - IN: create_app_man_odte 
        switch: off 
      - IN: prov_test_vpc_odte 
      - IN: test_env_ecs_odte 
        switch: off 
      - TASK: managed

Step 5: Run Functional Tests on the Test Cluster

Add thedeploy_app_test_odte job to your shippable.yml file. This job extracts the public DNS of the ECS cluster from the test_info_odte params resource and passes it to the script that runs some tests using the public DNS.

It is a runSh job that lets you run any shell script. Since it needs to run after the application is deployed in the workflow, test_env_ecs_odte is specified as an input. In addition, we also provide the manifest job as an input to the job.

jobs: 

# RUN System Integration Testing 
  - name: sit_odte 
    type: runSh 
    steps: 
      - IN: scripts_repo_odte 
        switch: off 
      - IN: deploy_app_test_odte 
      - TASK: 
        # Run tests 
        - script: | 
            pushd $(shipctl get_resource_state "scripts_repo_odte")/tests 
              PARAMS_JSON=$(shipctl get_resource_version_key test_info_odte params) 
              CLUSTER_DNS=$(echo $PARAMS_JSON | jq -r .INST_0_PUBLIC_DNS) 
              echo "ECS Cluster DNS: "$CLUSTER_DNS 
              ./run-tests.sh $CLUSTER_DNS 
            popd 
  on_success: 
    - script: echo "SUCCESS" 
  on_failure: 
    - script: echo "FAILURE"

Step 5: Deprovision the Cluster

A. Add thedeprov_test_infra_ode Job to Your shippable.yml file

It is a runSh job that lets you run any shell script. Since it needs to run after the system integrations tests are run, sit_odte is specified as an input.

jobs: 

#---------------------------------------------------------------# 
#----------------------- Deprov Test Infra----------------------# 
#---------------------------------------------------------------# 

# DEPROV TEST Infra with Ansible 
  - name: deprov_test_infra_odte 
    type: runSh 
    steps: 
      - IN: sit_odte 
      - IN: aws_cli_config_odte 
        switch: off 
      - IN: test_vpc_conf_odte 
        switch: off 
      - IN: test_conf_odte 
        switch: off 
      - IN: test_info_odte 
        switch: off 
      - IN: scripts_repo_odte 
        switch: off 
      - IN: ami_sec_approved_odte 
        switch: off 
      - TASK: 
        - script: shipctl replace 
            $SCRIPTS_REPO_ODTE_STATE/infra/provision-ecs-ansible/ansible.cfg 
            $SCRIPTS_REPO_ODTE_STATE/infra/provision-ecs-ansible/group_vars/ecs-cluster-vars.yml 
        - script: sudo pip install boto3 
        - script: | 
            cd $SCRIPTS_REPO_ODTE_STATE/infra/provision-ecs-ansible 
            ansible-playbook -v ansible-ecs-terminate.yml 
    on_success: 
      - script: echo "SUCCESS" 
    on_failure: 
      - script: echo "FAILURE"

B. Commit Shippable.Yml and Create a Sync Repo in Your Shippable Account

Follow these instructions to import your configuration files into your Shippable account.

Your pipeline should now look like this in the SPOG view.


Different sections of the pipeline expanded.

6. Trigger Your Pipeline

Right click on create_app_man_odtein the SPOG and click on Build Job. This will trigger the entire pipeline.


Screenshot of the Manifest Job

Screenshot of the Cluster Provision Job

Screenshot of Resources Populated by the Cluster Provision Job


Screenshot of the Deploy Job

Screenshot of the Deprovision Job

Read the 4-part DevOps testing eBook to learn how to detect problems earlier in your DevOps testing processes.

Topics:
testing ,test automation ,ansible ,devops ,docker ,continuous delivery

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}