Automated Testing Lifecycle

This is an article from DZone's 2023 Automated Testing Trend Report.

For more:

Read the Report

As per the reports of Global Market Insight, the automation testing market size surpassed $20 billion (USD) in 2022 and is projected to witness over 15% CAGR from 2023 to 2032. This can be attributed to the willingness of organizations to use sophisticated test automation techniques as part of the quality assurance operations (QAOps) process. By reducing the time required to automate functionalities, it accelerates the commercialization of software solutions. It also offers quick bug extermination and post-deployment debugging, and it helps the integrity of the software through early notifications of unforeseen changes.

 Figure 1: QAOps cycle 

What Is the Automated Testing Lifecycle?

The automation testing lifecycle is a multi-stage process that covers the process of documentation, planning, strategy, and design. The cycle also involves development of the use cases using technology and deploying it to an isolated system that could run on specific events or based on a schedule. 

Phases of the Automated Testing Lifecycle

There are six different phases of the automated testing lifecycle: 

1. Determining the scope of automation
2. Architecting the approach for test automation (tools, libraries, delivery, version control, CI, other integrations) 
3. Setting the right test plan, test strategy, and test design 
4. Automation environment setup
5. Test script development and execution
6. Analysis and generation of test reports

Figure 2: Automated testing lifecycle

Architecture

Architecture is an important part of the automation lifecycle that leads to defining the strategy required to start automation. In this phase of the lifecycle, the people involved need to have a clear understanding of the workflows, executions, and required integrations with the framework. 

Tools of the Trade

In today’s automation trends, the new buzzword is "codeless automation," which helps accelerate test execution. There are a few open-source libraries as well, such as Playwright, which use codeless automation features like codegen.

Developing a Framework

When collaborating in a team, a structured design technique is required. This helps create better code quality and reusability. If the framework is intended to deliver the automation of a web application, then the team of automation testers need to follow a specific design pattern for writing the code.

Execution of Tests in Docker

One important factor in today’s software test automation is that the code needs to be run on Docker in isolation every time the test runs are executed. There are a couple of advantages to using Docker. It helps set up the entire testing environment from scratch by removing flaky situations. Running automation tests on containers can also eliminate any browser instances that might have been suspended because of test failures. Also, many CI tools support Docker through plugins, and thus running test builds by spinning a Docker instance each time can be easily done.

Continuous Testing Through CI

When it comes to testing in the QAOps process, CI plays an important role in the software release process. CI is a multi-stage process that runs hand in hand when a commit is being made to a version control system to better diagnose the quality and the stability of a software application ready for deployment. 

Thus, CI provides an important aspect in today’s era of software testing. It helps to recover integration bugs, detect them as early as possible, and keep track of the application's stability over a period of time. Setting up a CI process can be achieved through tools like Jenkins and CircleCI. 

Determining the Scope of Test Automation

Defining the feasibility for automation is the first step of the automation lifecycle. This defines the scope and automates the required functionality. 

Test Case Management

Test case management is a technique to prioritize or select the broader scenarios from a group of test cases for automation that could cover a feature/module or a service as a functionality. In order to ensure the top quality of products, it is important that complexity of test case management can scale to meet application complexity and the number of test cases. 

The Right Test Plan, Test Strategy, and Test Design

Selecting a test automation framework is the first step in the test strategy phase of an automated testing lifecycle, and it depends on a thorough understanding of the product. 

In the test planning phase, the testing team decides the: 

• Test procedure creation, standards, and guidelines
• Hardware
• Software and network to support a test environment
• Preliminary test schedule
• Test data requirements
• Defect tracking procedure and the associated tracking tool

Automation Environment Setup

The build script to set up the automation environment can be initiated using a GitHub webhook. The GitHub webhook can be used to trigger an event in the CI pipeline that would run the build scripts and the test execution script. The build script can be executed in the CI pipeline using Docker Compose and Docker scripts.

docker-compose.yml:

version: "3.3"
services:
  test:
    build: ./
    environment:
      slack_hook: ${slack_hook}
      s3_bucket: ${s3_bucket}
      aws_access_key_id: ${aws_access_key_id}
      aws_secret_access_key: ${aws_secret_access_key}
      aws_region: ${aws_region}
    command: ./execute.sh --regression

FROM ubuntu:20.04

ENV DEBIAN_FRONTEND noninteractive

# Install updates to base image
RUN apt-get -y update && \
    apt-get -y install --no-install-recommends tzdata && \
rm -rf /var/lib/apt/lists/*

# Install required packages
ENV TZ=Australia/Melbourne
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
RUN dpkg-reconfigure --frontend noninteractive tzdata
RUN apt-get -y update && \
    apt-get install -y --no-install-recommends software-properties-common \
apt-utils \
curl \
wget \
unzip \
libxss1 \
libappindicator1 \
libindicator7 \
libasound2 \
libgconf-2-4 \
libnspr4 \
libnss3 \
libpango1.0-0 \
fonts-liberation \
xdg-utils \
gpg-agent \
git && \
rm -rf /var/lib/apt/lists/*
RUN add-apt-repository ppa:deadsnakes/ppa

# Install chrome
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN sh -c 'echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/
google-chrome.list'
RUN apt-get -y update \
    && apt-get install -y --no-install-recommends google-chrome-stable \
    && rm -rf /var/lib/apt/lists/*

# Install firefox
RUN apt-get install -y --no-install-recommends firefox

# Install python version 3.0+
RUN add-apt-repository universe
RUN apt-get -y update && \
    apt-get install -y --no-install-recommends python3.8 \
python3-pip && \
rm -rf /var/lib/apt/lists/*

RUN mkdir app && mkdir drivers

# Copy drivers directory and app module to the machine
COPY app/requirements.txt /app/

# Upgrade pip and Install dependencies
RUN pip3 install --upgrade pip \
                 -r /app/requirements.txt
COPY app /app
COPY drivers /drivers

# Execute test
ADD execute.sh .
RUN chmod +x execute.sh
ENTRYPOINT ["/bin/bash"]

Seeding Test Data in the Database

Seed data can be populated for a particular model or can be done using a migration script or a database dump. For example, Django has a single-line loader function that helps seeding data from a YML file. 

The script to seed the database can be written in a bash script and can be executed once every time a container is created. Take the following code blocks as examples.

entrypoint.sh:

#!/bin/bash
set -e

python manage.py loaddata maps/fixtures/country_data.yaml

exec "$@"

FROM python:3.7-slim

RUN apt-get update && apt-get install

RUN apt-get install -y libmariadb-dev-compat libmariadb-dev
RUN apt-get update \
    && apt-get install -y --no-install-recommends gcc \
    && rm -rf /var/lib/apt/lists/*

RUN python -m pip install --upgrade pip
RUN mkdir -p /app/

WORKDIR /app/

COPY requirements.txt requirements.txt
RUN python -m pip install -r requirements.txt

COPY entrypoint.sh /app/
COPY . /app/
RUN chmod +x entrypoint.sh

ENTRYPOINT ["/app/entrypoint.sh"]

Setting up the Workflow Using Pipeline as Code

Nowadays, it is easy to run builds and execute Docker from CI using Docker plugins. The best way to set up the workflow from CI is by using pipeline as code. A pipeline-as-code file specifies actions and stages for a CI pipeline to perform. Because every organization uses a version control system, changes in pipeline code can be tested in branches for the corresponding changes in the application to be deployed. The following code block is an example of pipeline as code.

config.yml:

steps:
- label: ":docker: automation pipeline"
env:
  VERSION: "$BUILD_ID"
timeout_in_minutes: 60
plugins:
   - docker-compose#v3.7.0:
      run: test
retry:
  automatic:
     - exit_status: "*"
       limit: 1

Checklist for Test Environment Setup

• Test data
• List of all the systems, modules, and applications to test
• Application under test access and valid credentials
• An isolated database server for the staging environment
• Tests across multiple browsers
• All documentation and guidelines required for setting up the environment and workflows
• Tool licenses, if required
• Automation framework implementation

Development and Execution of Automated Tests

To ensure test scripts run accordingly, the development of test scripts based on the test cases requires focusing on: 

• Selection of the test cases
• Creating reusable functions
• Structured and easy scripts for increased code readability 
• Peer reviews to check for code quality
• Use of reporting tools/libraries/dashboards

Execution of Automated Tests in CI

Figure 3 is a basic workflow that defines how a scalable automation process can work. In my experience, the very basic need to run a scalable automation script in the CI pipeline is met by using a trigger that would help set up the test dependencies within Docker and execute tests accordingly based on the need. 

Figure 3: Bird's eye view of automation process

For example, a test pipeline may run a regression script, whereas another pipeline may run the API scripts. These cases can be handled from a single script that acts as the trigger to the test scripts. 

execute.sh:

#!/bin/bash
set -eu

# Check if csv_reports, logs directory, html_reports, screenshots is present
mkdir app/csv_reports app/logs mkdir app/html_reports/screenshots

# Validate that if an argument is passed or not
if [ $# -eq 0 ]; then echo "No option is passed as argument"; fi

# Parse command line argument to run tests accordingly
for i in "$@"; do
    case $i in
        --regression) pytest -p no:randomly app/test/ -m regression --browser firefox --headless
true --html=app/html_reports/"$(date '+%F_%H:%M:%S')_regression".html --log-file app/logs/"$(date
'+%F_%H:%M:%S')".log
            break
            ;;
        --smoke) pytest app/test -m smoke
            break
            ;;
        --sanity) pytest app/test -m sanity --browser chrome --headless true --html=app/html_reports/
sanity_reports.html --log-file app/logs/"$(date '+%F_%H:%M:%S')".log
            break
            ;;
        --apitest) npm run apitest
            break
            ;;
        --debug) pytest app/test -m debug --browser chrome --headless true --html=app/html_reports/
report.html --log-file app/logs/"$(date '+%F_%H:%M:%S')".log
            break
            ;;
        *) echo "Option not available"
            ;;
    esac

done

test_exit_status=$?

exit $test_exit_status

Analysis of Test Reports

By analyzing test reports, testing teams are able to determine whether additional testing is needed, if the scripts used can accurately identify errors, and how well the tested application(s) can withstand challenges. Reports can be represented either using static HTML or dynamic dashboard. Dashboards can help stakeholders in understanding trends in the test execution by comparing the current data with the past data of execution. For example, allure reporting creates a concise dashboard with the test outcomes and represents it using data collected from test execution. 

Conclusion

Automated testing lifecycle is a curated process that helps testing applications meet specific goals within appropriate timelines. Furthermore, it is very important for the QAOps process to gel properly with the SDLC and rapid application development. When completed correctly, the six phases of the lifecycle will achieve better outcomes and delivery. 

Additional Reading: 

• Cloud-Based Automated Testing Essentials Refcard by Justin Albano
• "Introduction to App Automation for Better Productivity and Scaling" by Soumyajit Basu

This is an article from DZone's 2023 Automated Testing Trend Report.

For more:

Read the Report