DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
  1. DZone
  2. Testing, Deployment, and Maintenance
  3. DevOps and CI/CD
  4. DevOps Pipeline Quality Gates: A Double-Edged Sword

DevOps Pipeline Quality Gates: A Double-Edged Sword

In theory, quality gates seem like a no-brainer, but it does come with a catch.

Tracy Ragan user avatar by
Tracy Ragan
·
Jeff Pierce user avatar by
Jeff Pierce
·
Aug. 23, 19 · Analysis
Like (4)
Save
Tweet
Share
15.03K Views

Join the DZone community and get the full member experience.

Join For Free

There has been much discussion about the many benefits of "moving testing left," and our experts will tell you that doing so by having automated testing (Quality Gates) integrated into your build pipelines is a critical success factor for the rapid build and deploy process automation necessary to truly reap the benefits of Agile. That said, there are significant costs to the organization for implementing automated Quality Gates, which must be weighed carefully and optimized to retain positive Return On Investment (ROI) for the implementation.

Quality Gates are based upon the stage-gate system initially presented in 1986 and originally applied to quality control processes in the automotive industry. The concept is simple: you have tests, or gates, that validate each step in your overall process. If the step passes the test, the process proceeds to the next step. Upon failure, the process is stopped and corrective actions are initiated to identify and resolve the issue. Cost savings are increased if defects are found and corrected earlier in the process. Note: It is not necessary for the gates to be serial, and in fact you will likely need to run parallel processes to remain within time constraints for the overall process (more on this idea later).

On my most recent client project, we helped them incrementally implement automated Continuous Integration and Continuous Delivery (CI/CD) pipelines (Jenkins Pipeline in multi-branch mode) that not only performed build and deployment, leveraging Nexus build artifact repos, Nginx, Kafka, Redis, etc. but also executed unit testing, static analysis, security scanning (Fortinet). We eventually also added dynamic and performance tests (Jmeter and Gridlastic) by leveraging test automation we built for regression, performance, and portability testing. The results were astounding: we identified numerous bugs with every build before expending testing time and energy on defective builds. The resulting time savings and increases in build quality provided more efficiency in the testing process, new opportunities for increased testing coverage, and a resulting product that was more reliable and had increased quality.

So, all of this sounds great so far, right? What is the catch? It's a common saying in product development that "you can have it good, you can have it fast, or you can have it cheap...Pick any two." What we have done here is focus on "good and fas,t but there is a cost that comes in several forms, including not only the build-out time, but also in the form of complexity and time, which has direct impacts to your development staff and process, and Cloud server resources.

Building and deploying software systems is a complex process involving a multitude of steps that all must be done correctly for a successful outcome. You cannot decrease the processes that have to happen, and your codebase is growing daily, so to get the process completed more quickly, you need to both automate the build (for repeatability) and deployment process steps and have more steps executed at the same time (parallelism). Building the scripts to automate build and deployment is a complex undertaking that requires specialized tools and skills. Building parallelism into the pipelines is an additional layer of complexity. (see image below)



On our recent client project, in order to both enable parallelization in build and test execution, and reduce time needed to spin up temporary services needed to accomplish the pipeline tasks, we leveraged containerization (Docker containers on Kubernetes clusters and pods) and AWS Cloud Formation templates. Time became a significant constraint, as every test (or Quality Gate) we added resulted in increased pipeline execution time, which in turn cost more developer wait time. The additional complexity of parallelization had another dimension as well: debugging any build failures became much harder, as a direct result of the implementation of build automation and parallelism that had been introduced to the pipelines to increase efficiency. Developers had to be trained on the pipeline architecture and tooling to be able to debug build and test failures. Tools had to be configured to allow log file access, and logging strategies developed to balance what needed to be logged and retained versus space and time needed for creating and storing the logs.

Having lived this project, I offer some recommendations to successfully implement Quality Gates, to include:

  1. Careful and deliberate decisions with respect to which tests (or Quality Gates) to include in your process, and regarding balancing the ROI against the cost, time and complexity;
  2. Coordinating incremental change, and leading the rollout of each change with training as the first step; and
  3. Embedding DevOps engineers into your development scrum teams to increase communication and collaboration, and to provide the necessary support developers will need in both understanding the process, and debugging issues that arise;

In our case, we implemented the base pipelines repo by repo, training and supporting each of the affected development teams as we rolled out the changes. As we added new steps and Quality Gates, we repeated the cycle of training and incremental rollout, minimizing disruption and developer down time, while incrementally improving the pipelines.

Change is constant, so set up your teams and processes to encourage collaboration and to improve overall quality.

Pipeline (software) Continuous Integration/Deployment DevOps SWORD (protocol)

Published at DZone with permission of Tracy Ragan, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Steel Threads Are a Technique That Will Make You a Better Engineer
  • Tracking Software Architecture Decisions
  • The Power of Zero-Knowledge Proofs: Exploring the New ConsenSys zkEVM
  • Authenticate With OpenID Connect and Apache APISIX

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: