DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Last call! Secure your stack and shape the future! Help dev teams across the globe navigate their software supply chain security challenges.

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Releasing software shouldn't be stressful or risky. Learn how to leverage progressive delivery techniques to ensure safer deployments.

Avoid machine learning mistakes and boost model performance! Discover key ML patterns, anti-patterns, data strategies, and more.

Related

  • Test Automation for Mobile Apps: Strategy for Improved Testing Results
  • Getting Started With Microsoft Tool Playwright for Automated Testing
  • Developing and Scaling a Microservice
  • Containerization and AI: Streamlining the Deployment of Machine Learning Models

Trending

  • Docker Model Runner: Streamlining AI Deployment for Developers
  • A Simple, Convenience Package for the Azure Cosmos DB Go SDK
  • Designing a Java Connector for Software Integrations
  • AWS to Azure Migration: A Cloudy Journey of Challenges and Triumphs
  1. DZone
  2. Testing, Deployment, and Maintenance
  3. Testing, Tools, and Frameworks
  4. Automated Testing Lifecycle: A Guide To Using Automation to Enhance Application Delivery and Scalability

Automated Testing Lifecycle: A Guide To Using Automation to Enhance Application Delivery and Scalability

Enhance application delivery and scalability and curate a process that helps meet specific goals with testing applications in this guide to using automation.

By 
Soumyajit Basu user avatar
Soumyajit Basu
DZone Core CORE ·
Sep. 14, 23 · Analysis
Likes (7)
Comment
Save
Tweet
Share
6.5K Views

Join the DZone community and get the full member experience.

Join For Free

This is an article from DZone's 2023 Automated Testing Trend Report.

For more:


Read the Report

As per the reports of Global Market Insight, the automation testing market size surpassed $20 billion (USD) in 2022 and is projected to witness over 15% CAGR from 2023 to 2032. This can be attributed to the willingness of organizations to use sophisticated test automation techniques as part of the quality assurance operations (QAOps) process. By reducing the time required to automate functionalities, it accelerates the commercialization of software solutions. It also offers quick bug extermination and post-deployment debugging, and it helps the integrity of the software through early notifications of unforeseen changes.

QAOps cycle

 Figure 1: QAOps cycle 

What Is the Automated Testing Lifecycle?

The automation testing lifecycle is a multi-stage process that covers the process of documentation, planning, strategy, and design. The cycle also involves development of the use cases using technology and deploying it to an isolated system that could run on specific events or based on a schedule. 

Phases of the Automated Testing Lifecycle

There are six different phases of the automated testing lifecycle: 

  1. Determining the scope of automation
  2. Architecting the approach for test automation (tools, libraries, delivery, version control, CI, other integrations) 
  3. Setting the right test plan, test strategy, and test design 
  4. Automation environment setup
  5. Test script development and execution
  6. Analysis and generation of test reports

Automated testing lifecycle

Figure 2: Automated testing lifecycle

Architecture

Architecture is an important part of the automation lifecycle that leads to defining the strategy required to start automation. In this phase of the lifecycle, the people involved need to have a clear understanding of the workflows, executions, and required integrations with the framework. 

Tools of the Trade

In today’s automation trends, the new buzzword is "codeless automation," which helps accelerate test execution. There are a few open-source libraries as well, such as Playwright, which use codeless automation features like codegen.

Developing a Framework

When collaborating in a team, a structured design technique is required. This helps create better code quality and reusability. If the framework is intended to deliver the automation of a web application, then the team of automation testers need to follow a specific design pattern for writing the code.

Execution of Tests in Docker

One important factor in today’s software test automation is that the code needs to be run on Docker in isolation every time the test runs are executed. There are a couple of advantages to using Docker. It helps set up the entire testing environment from scratch by removing flaky situations. Running automation tests on containers can also eliminate any browser instances that might have been suspended because of test failures. Also, many CI tools support Docker through plugins, and thus running test builds by spinning a Docker instance each time can be easily done.

Continuous Testing Through CI

When it comes to testing in the QAOps process, CI plays an important role in the software release process. CI is a multi-stage process that runs hand in hand when a commit is being made to a version control system to better diagnose the quality and the stability of a software application ready for deployment. 

Thus, CI provides an important aspect in today’s era of software testing. It helps to recover integration bugs, detect them as early as possible, and keep track of the application's stability over a period of time. Setting up a CI process can be achieved through tools like Jenkins and CircleCI. 

Determining the Scope of Test Automation

Defining the feasibility for automation is the first step of the automation lifecycle. This defines the scope and automates the required functionality. 

Test Case Management

Test case management is a technique to prioritize or select the broader scenarios from a group of test cases for automation that could cover a feature/module or a service as a functionality. In order to ensure the top quality of products, it is important that complexity of test case management can scale to meet application complexity and the number of test cases. 

The Right Test Plan, Test Strategy, and Test Design

Selecting a test automation framework is the first step in the test strategy phase of an automated testing lifecycle, and it depends on a thorough understanding of the product. 

In the test planning phase, the testing team decides the: 

  • Test procedure creation, standards, and guidelines
  • Hardware
  • Software and network to support a test environment
  • Preliminary test schedule
  • Test data requirements
  • Defect tracking procedure and the associated tracking tool

Automation Environment Setup

The build script to set up the automation environment can be initiated using a GitHub webhook. The GitHub webhook can be used to trigger an event in the CI pipeline that would run the build scripts and the test execution script. The build script can be executed in the CI pipeline using Docker Compose and Docker scripts.

docker-compose.yml:

 
version: "3.3"
services:
  test:
    build: ./
    environment:
      slack_hook: ${slack_hook}
      s3_bucket: ${s3_bucket}
      aws_access_key_id: ${aws_access_key_id}
      aws_secret_access_key: ${aws_secret_access_key}
      aws_region: ${aws_region}
    command: ./execute.sh --regression


Dockerfile
 
FROM ubuntu:20.04

ENV DEBIAN_FRONTEND noninteractive

# Install updates to base image
RUN apt-get -y update && \
    apt-get -y install --no-install-recommends tzdata && \
rm -rf /var/lib/apt/lists/*

# Install required packages
ENV TZ=Australia/Melbourne
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
RUN dpkg-reconfigure --frontend noninteractive tzdata
RUN apt-get -y update && \
    apt-get install -y --no-install-recommends software-properties-common \
apt-utils \
curl \
wget \
unzip \
libxss1 \
libappindicator1 \
libindicator7 \
libasound2 \
libgconf-2-4 \
libnspr4 \
libnss3 \
libpango1.0-0 \
fonts-liberation \
xdg-utils \
gpg-agent \
git && \
rm -rf /var/lib/apt/lists/*
RUN add-apt-repository ppa:deadsnakes/ppa

# Install chrome
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN sh -c 'echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/
google-chrome.list'
RUN apt-get -y update \
    && apt-get install -y --no-install-recommends google-chrome-stable \
    && rm -rf /var/lib/apt/lists/*

# Install firefox
RUN apt-get install -y --no-install-recommends firefox

# Install python version 3.0+
RUN add-apt-repository universe
RUN apt-get -y update && \
    apt-get install -y --no-install-recommends python3.8 \
python3-pip && \
rm -rf /var/lib/apt/lists/*

RUN mkdir app && mkdir drivers

# Copy drivers directory and app module to the machine
COPY app/requirements.txt /app/

# Upgrade pip and Install dependencies
RUN pip3 install --upgrade pip \
                 -r /app/requirements.txt
COPY app /app
COPY drivers /drivers

# Execute test
ADD execute.sh .
RUN chmod +x execute.sh
ENTRYPOINT ["/bin/bash"]


Seeding Test Data in the Database

Seed data can be populated for a particular model or can be done using a migration script or a database dump. For example, Django has a single-line loader function that helps seeding data from a YML file. 

The script to seed the database can be written in a bash script and can be executed once every time a container is created. Take the following code blocks as examples.

entrypoint.sh:

 
#!/bin/bash
set -e

python manage.py loaddata maps/fixtures/country_data.yaml

exec "$@"


Dockerfile
 
FROM python:3.7-slim

RUN apt-get update && apt-get install

RUN apt-get install -y libmariadb-dev-compat libmariadb-dev
RUN apt-get update \
    && apt-get install -y --no-install-recommends gcc \
    && rm -rf /var/lib/apt/lists/*

RUN python -m pip install --upgrade pip
RUN mkdir -p /app/

WORKDIR /app/

COPY requirements.txt requirements.txt
RUN python -m pip install -r requirements.txt

COPY entrypoint.sh /app/
COPY . /app/
RUN chmod +x entrypoint.sh

ENTRYPOINT ["/app/entrypoint.sh"]


Setting up the Workflow Using Pipeline as Code

Nowadays, it is easy to run builds and execute Docker from CI using Docker plugins. The best way to set up the workflow from CI is by using pipeline as code. A pipeline-as-code file specifies actions and stages for a CI pipeline to perform. Because every organization uses a version control system, changes in pipeline code can be tested in branches for the corresponding changes in the application to be deployed. The following code block is an example of pipeline as code.

config.yml:

 
steps:
- label: ":docker: automation pipeline"
env:
  VERSION: "$BUILD_ID"
timeout_in_minutes: 60
plugins:
   - docker-compose#v3.7.0:
      run: test
retry:
  automatic:
     - exit_status: "*"
       limit: 1


Checklist for Test Environment Setup

  • Test data
  • List of all the systems, modules, and applications to test
  • Application under test access and valid credentials
  • An isolated database server for the staging environment
  • Tests across multiple browsers
  • All documentation and guidelines required for setting up the environment and workflows
  • Tool licenses, if required
  • Automation framework implementation

Development and Execution of Automated Tests

To ensure test scripts run accordingly, the development of test scripts based on the test cases requires focusing on: 

  • Selection of the test cases
  • Creating reusable functions
  • Structured and easy scripts for increased code readability 
  • Peer reviews to check for code quality
  • Use of reporting tools/libraries/dashboards

Execution of Automated Tests in CI

Figure 3 is a basic workflow that defines how a scalable automation process can work. In my experience, the very basic need to run a scalable automation script in the CI pipeline is met by using a trigger that would help set up the test dependencies within Docker and execute tests accordingly based on the need. 

Bird's eye view of automation process

Figure 3: Bird's eye view of automation process

For example, a test pipeline may run a regression script, whereas another pipeline may run the API scripts. These cases can be handled from a single script that acts as the trigger to the test scripts. 

execute.sh:

 
#!/bin/bash
set -eu

# Check if csv_reports, logs directory, html_reports, screenshots is present
mkdir app/csv_reports app/logs mkdir app/html_reports/screenshots

# Validate that if an argument is passed or not
if [ $# -eq 0 ]; then echo "No option is passed as argument"; fi

# Parse command line argument to run tests accordingly
for i in "$@"; do
    case $i in
        --regression) pytest -p no:randomly app/test/ -m regression --browser firefox --headless
true --html=app/html_reports/"$(date '+%F_%H:%M:%S')_regression".html --log-file app/logs/"$(date
'+%F_%H:%M:%S')".log
            break
            ;;
        --smoke) pytest app/test -m smoke
            break
            ;;
        --sanity) pytest app/test -m sanity --browser chrome --headless true --html=app/html_reports/
sanity_reports.html --log-file app/logs/"$(date '+%F_%H:%M:%S')".log
            break
            ;;
        --apitest) npm run apitest
            break
            ;;
        --debug) pytest app/test -m debug --browser chrome --headless true --html=app/html_reports/
report.html --log-file app/logs/"$(date '+%F_%H:%M:%S')".log
            break
            ;;
        *) echo "Option not available"
            ;;
    esac

done

test_exit_status=$?

exit $test_exit_status


Analysis of Test Reports

By analyzing test reports, testing teams are able to determine whether additional testing is needed, if the scripts used can accurately identify errors, and how well the tested application(s) can withstand challenges. Reports can be represented either using static HTML or dynamic dashboard. Dashboards can help stakeholders in understanding trends in the test execution by comparing the current data with the past data of execution. For example, allure reporting creates a concise dashboard with the test outcomes and represents it using data collected from test execution. 

Conclusion

Automated testing lifecycle is a curated process that helps testing applications meet specific goals within appropriate timelines. Furthermore, it is very important for the QAOps process to gel properly with the SDLC and rapid application development. When completed correctly, the six phases of the lifecycle will achieve better outcomes and delivery. 

Additional Reading: 

  • Cloud-Based Automated Testing Essentials Refcard by Justin Albano
  • "Introduction to App Automation for Better Productivity and Scaling" by Soumyajit Basu

This is an article from DZone's 2023 Automated Testing Trend Report.

For more:


Read the Report

Test automation Test case Version control Docker (software)

Opinions expressed by DZone contributors are their own.

Related

  • Test Automation for Mobile Apps: Strategy for Improved Testing Results
  • Getting Started With Microsoft Tool Playwright for Automated Testing
  • Developing and Scaling a Microservice
  • Containerization and AI: Streamlining the Deployment of Machine Learning Models

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!