DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Testing, Deployment, and Maintenance
  3. DevOps and CI/CD
  4. DevOps: Building Better Pipelines

DevOps: Building Better Pipelines

Learn about upgrading your traditional DevOps pipeline with insights from this All Day DevOps talk, for better security, flexibility, and more.

Derek Weeks user avatar by
Derek Weeks
·
Mar. 22, 18 · Presentation
Like (5)
Save
Tweet
Share
7.94K Views

Join the DZone community and get the full member experience.

Join For Free

"The deployment pipeline takes value from development and delivers it to clients."

This is one of the better summations of a development pipeline that I have heard, and it came from Dan Barker (@barkerd427) during his recent All Day DevOps conference talk, " Becoming a Plumber: Building Deployment Pipelines." Dan's talk focuses on the value of pipelines, how to build them well, and how to build both development and infrastructure pipelines.

Traditional Pipelines

The traditional pipeline: development-->test-->production servers is a good foundation, and, like the plumbing in your home, is helpful to generally get us where we want to go. However, as anyone who has raised boys knows, a toilet doesn't guarantee 100% compliance. The user isn't perfect.

It is similar to basic development pipelines. Things don't go quite as planned. As Dan mentioned in his talk, as a developer, he strides to keep his code all in dev, and then push to test, and then push to production. However, that doesn't always happen. And there are other inconsistencies that inadvertently crop up (such as tab vs. space), despite everyone's best intentions.

Dan is a chief architect and works in highly-regulated industries -- financial services, health care, and insurance -- so precision, compliance, audit trails, etc. are all the more important. He architects pipelines to ensure human error is minimized.

What's in a Pipeline?

But why all of the fuss over pipelines? Well, first Dan touched on some of the values pipelines provide:

  • Abstract audit and compliance
  • Trivialities eliminated (spaces vs. tabs)
  • Security checks occur early/often
  • Tests all the things
  • Nimble security
  • Common artifact repositories
  • Standardized approval system
  • Apps become secure by default

But how do you architect a pipeline to deliver these values?

In the illustration below, you can see Dan's pipeline architecture where he offers key points that all of us should consider applying:

  • Production data is separated from development data. You can only get there through the pipeline
  • You interact with the Platform, and it configures the levels below it
  • You can duplicate containers as much as you want to scale horizontally, and can replicate that in production
  • The configuration must come from the environment

As he dug in, he started with covering the advantages of two types of pipelines, recognizing you need to select this first:

Scripted:

  • Very Groovy!
  • More powerful
  • Provides greatest level of flexibility

Declarative:

  • Only a little Groovy
  • Simpler to maintain
  • Easier to read and understand

Dan then dug into the technical nitty-gritty, such as shared libraries, setups, software to manage pipelines, fabric8, Kubernetes, Yahoo! Screwdriver, etc. and then made the observation that he thinks that deployment pipelines have fallen behind. In his real-life example, he and his team decided to take a more holistic approach to their pipeline (think: the first way of DevOps) so they could build and deploy apps and also build and deploy the infrastructure.

Sharing Knowledge

They built a system that is especially applicable to those in highly-regulated industries, and they hope to open-source it soon, so keep an eye on@barkerd427 for more info. In the meantime, he describes in detail the process it follows, software and libraries it uses, etc.

If you are looking to build better pipelines, if you want to find out how Dan built the dual pipeline for the apps and the infrastructure, or you are just interested in his thoughts on better pipelines and the advantages and disadvantages of different software packages, Dan's full talk is available below or you can watch it here.


Yearning for more? You can watch any of the 2017 AllDayDevOps sessions free-of-charge.

All Day DevOps 2018

All Day DevOps 2018 is just around the corner! Registration is available here.

The free, online conference goes live on October 17th, offering 100 different practitioner-led sessions, each one 30-minutes long. With 5 separate tracks: CI/CD, Cloud-Native Infrastructure, DevSecOps, Cultural Transformations, & Site Reliability Engineering, and 100 speakers, there's sure to be something for everyone.

And speaking of everyone, if you're part of an organization with 20+ people that want to attend the conference (again, it's free!) then you should consider joining the Club 20 program so that you might get your company logo added to the ADDO site. Check out some of the Club 20 participants here and consider joining them.

Hope to see you online at the show!

Pipeline (software) DevOps Continuous Integration/Deployment

Published at DZone with permission of Derek Weeks, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Event Driven 2.0
  • The 12 Biggest Android App Development Trends in 2023
  • API Design Patterns Review
  • Tech Layoffs [Comic]

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: