DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
View Events Video Library
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Integrating PostgreSQL Databases with ANF: Join this workshop to learn how to create a PostgreSQL server using Instaclustr’s managed service

Mobile Database Essentials: Assess data needs, storage requirements, and more when leveraging databases for cloud and edge applications.

Monitoring and Observability for LLMs: Datadog and Google Cloud discuss how to achieve optimal AI model performance.

Automated Testing: The latest on architecture, TDD, and the benefits of AI and low-code tools.

Related

  • Continuous Testing for Agile and DevOps Using Testsigma
  • DevOps Pipeline and Its Essential Tools
  • Legacy Code Refactoring: Tips, Steps, and Best Practices
  • How Does Continuous Testing Improve DevOps?

Trending

  • The Agile Architect: Mastering Architectural Observability To Slay Technical Debt
  • Decoding the Differences: Continuous Integration, Delivery and Deployment
  • Unleashing the Power of Microservices With Spring Cloud
  • Continuous Integration vs. Continuous Deployment
  1. DZone
  2. Testing, Deployment, and Maintenance
  3. DevOps and CI/CD
  4. Continuous Testing at the Speed of DevOps

Continuous Testing at the Speed of DevOps

A review of what continuous testing is, and how to move from automated testing to continuous testing.

Erika  Barron-Delgado user avatar by
Erika Barron-Delgado
·
Dec. 18, 15 · Opinion
Like (4)
Save
Tweet
Share
5.55K Views

Join the DZone community and get the full member experience.

Join For Free
"Even if you've got a Maserati, you need a good driver who knows how to drive it. Speed is important, but safety and accuracy are also key" – Bob Aiello

Parasoft recently hosted a "Continuous Testing in the DevOps World" webinar featuring Bob Aiello, Technical Editor for CM Crossroads and Wayne Ariola, Parasoft Chief Strategy Officer and co-author of Continuous Testing. Since the webinar generated such overwhelming attendance and response, we thought we'd highlight some of the key points here.

What is DevOps?

DevOps is a set of principles and practices that helps you communicate and collaborate more effectively. Why? Because developers know a lot of very important information. So do QA testers… and so does Operations. If your organization is going to be successful, you need to be able to harness all of that knowledge and experience. A key part of DevOps is creating the "automated deployment pipeline" –the ability to deploy changes (bug fixes, new features, infrastructure changes, etc.) as often as needed, with absolute reliability. The ability to get it right each and every time is really key. Companies that master this gain a very important strategic advantage. Those that don't incur a huge amount of risk.

What is Continuous Testing?

Per the Continuous Testing Wikipedia page:

"Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate. For Continuous Testing, the scope of testing extends from validating bottom-up requirements or user stories to assessing the system requirements associated with overarching business goals."

Are We Done Testing? Wrong Question!

When assessing the risk of a release candidate, most software teams ask the question:  “Are we done testing?”  Fundamentally, this is the wrong question. 

Especially with DevOps and Continuous Delivery, releasing with both speed and confidence requires having immediate feedback on the business risks associated with a software release candidate. Given the rising cost and impact of software failures, you can't afford to unleash a release that could disrupt the existing user experience or introduce new features that expose the organization to new security, reliability, or compliance risks. To prevent this, the organization needs to extend from validating bottom-up requirements to assessing the system requirements associated with overarching business goals.

Instead of “Are we done testing,” software development teams should be asking: “Does the release candidate have an acceptable level of business risk?” If we can answer this question, we can determine if the application is truly ready to progress through the delivery pipeline at any given time.

 What Will the Next Generation of Testing Look Like?

The overarching goal will be to close the gap between business expectations and dev/test activities. First off, we'll have to bring operations (runtime) data associated with the actual user experience into our testing environment. We'll also use simulation to enable testing to begin early and continue to run at the necessary frequency. We won't stop testing requirements or performing bottom-up verification of changes. This still needs to occur, but it needs to be designed into a context so that it's integrated into a continuous regression suite that's validated vs. the user experience. Understanding how new functionality impacts the broader system is key for accurately assessing the total risk of the release candidate.

All tests need to be placed into a regression suite that measures business objectives, then "process intelligence" can be used to better understand the impact of change. This will guide us to two things. First, better exploratory testing, where corner cases and more investigative testing can happen on more variable outcomes. Second, automated acceptance testing. This is designed to ensure that there's no negative impact to today's user experience.

What Are Key Factors in Moving From Automated Testing to Continuous Testing?

Automated testing involves automated, CI-driven execution of whatever set of tests the team has accumulated. However, if one of these tests fails, what does that really mean: does it indicate a critical business risk, or just a violation of some naming standard that nobody is really committed to following anyway? And what happens when it fails? Is there a clear workflow for prioritizing defects vs. business risks and addressing the most critical ones first? And for each defect that warrants fixing, is there a process for exposing all similar defects that might already have been introduced, as well as preventing this same problem from recurring in the future?  This is where the difference between automated and continuous becomes evident.

To evolve from automated to continuous, you need the following:

  1. Clearly defined business expectations, with business risks identified per application, team, and release.
  2. Defects automatically prioritized versus the business drivers and knowing how to mitigate those risks before the release candidate goes live.
  3. Testing in complete test environments continuously using simulation—this is critical for protecting the current user experience from the impact of change.
  4. Feedback loop for defect prevention—looking for patterns that emerge and using this as an opportunity to design and implement defect prevention practices that prevent similar defects from being introduced.

How Does Service Virtualization Fit Into DevOps and Continuous Testing?

After organizations start accelerating their software delivery pipeline, they often reach the point where they need to test, but can't exercise the AUT because a complete test environment is not yet ready. A lot of teams use simulation technologies such as service virtualization to get around these roadblocks. For example, say your mainframe is only available for 2 hours on Saturday night. You can record your AUT's interactions with it, then capture this as a virtual asset. You can do the same for a database, 3rd-party application, SAP, etc.

Simulation technologies are not perfect. But in order for us to start to exercise the "big blocks" and get those risks out of the way, we need to begin exploratory testing . . . and we can't do this without simulation. To truly protect the end user experience, we need to aggressively test and defend the end user's experience across key end-to-end transactions. With today's systems, those transactions pass through a high number of different components, so it's very difficult to accommodate that in a single staged test environment — cloud or not. Simulation helps us get around this.  For the most realistic simulated environment, we need to really understand how components are working in an operational environment and transfer this to the simulation.

Pipeline (software) DevOps Software development Testing User experience Release (agency)

Published at DZone with permission of Erika Barron-Delgado, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Continuous Testing for Agile and DevOps Using Testsigma
  • DevOps Pipeline and Its Essential Tools
  • Legacy Code Refactoring: Tips, Steps, and Best Practices
  • How Does Continuous Testing Improve DevOps?

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: