Programmable Infrastructure in Your Organization
Programmable Infrastructure in Your Organization
We discuss how development teams can integrate different tools and tech stacks to create a better continuous integration flow.
Join the DZone community and get the full member experience.Join For Free
At Atlassian, we believe in releasing quality products frequently and predictably to our customers. A continuous delivery pipeline is a classic example of programmable infrastructure that helps execute on that vision. Read about continuous delivery and its business value for more context.
We, Atlassian, are the makers of Bitbucket and I love how source code, commits, branches, pull requests, pipelines, and deployments are organized in our "one-stop shop" Bitbucket UI. See the left panel in the screenshot below.
In this article, I'll highlight how to leverage this programmable infrastructure to ship your releases in an auditable and traceable manner. I'll get into the weeds right away. So, fasten your seatbelts!
Build Your Pipeline One "Step" at a Time
A continuous delivery pipeline is an incarnation of the "continuous everything" paradigm and Bitbucket Pipelines is one of the solutions available in the market.
You can configure Bitbucket Pipelines using a YAML (Yet Another Markup Language) file called "bitbucket-pipelines.yml." This is a classic example of configuration-as-code and demonstrates the beauty of defining pipelines using key-value pairs.
Bitbucket makes it quick and easy so that, instead of getting buried in shipping logistics, you can focus on building great products that wow the world. Commit this YAML at the root of your source code repository. Your pipelines will run in the cloud and the YAML's simplicity will get you started in no time. No mess.
The "image" keyword points to a Docker image used to execute the pipeline. I am playing with a Node.js application flowing through the pipeline, and hence I point to a node image below - "node 8.6" to be specific.
image: node:8.6 options: # Maximum time a step can execute in minutes max-time: 5
If you write your application in Java, you would point to a Java image like below:
Your particular configuration depends on your tech stack, and here are language guides to jumpstart yourself.
The "max-time" keyword denotes the maximum time (in minutes) a step can execute for. I use this as my first line of defense to prevent the pipeline from going for a toss. When I run lengthy integration or performance tests, this keyword acts like a timeout in case the test suite cannot return the control back to the pipeline (zero return code for success and non-zero return code for failure). While patience is a virtue, the team may not feel very virtuous with deadlines looming.
Integrate early and integrate often. Let's create a build step in this pipeline with the "step" keyword. For the most part, the keywords and their values are self-explanatory, and here's a guide for your reference.
Use your application's build command with the "script" keyword. Since I am playing around with a Node.js application at this time, I have used "npm install."
pipelines: default: - step: name: Build caches: - node script: - export FORCE_COLOR=1 - npm install
In the snippet above, the "pipelines" keyword announces the beginning of all pipeline definitions. Since this is the first major keyword, it doesn't have whitespaces in front of it. Don't forget that the whitespace in front of all the other keywords you see are spaces and not tabs. Bitbucket provides an online YAML validator since a bad YAML would break the pipeline just as much as bad code will. Validate YAML syntax before you commit since broken pipelines inconvenience the entire team and diminish throughput.
Practice trunk-based development where possible. If you can't due to business reasons, feature branches should be short-lived. The "default" keyword signifies that the pipeline definition under it applies to all branches. In case you have more than one branch, use the "branches" keyword to apply different pipeline definitions to different branches.
Downloading dependencies can be time-consuming and can inflate cycle time and time-2-market. I use the "caches" keyword since it prevents pipelines from downloading dependencies every time they execute.
Without continuous testing, teams can shoot erroneous code into production, only faster! While velocity is a great attribute, by velocity we mean responsible speed and not suicidal speed.
Let's add a second test step that will validate the artifact built in the first step. I use "npm test" with the "script" keyword to invoke automated tests that give fast feedback. Depending on your tech stack, use your own test invocation command here.
- step: name: Test caches: - node script: - npm test
Some things to remember:
a) Design your test suites to return zero (0) for success and non-zero integers for failure so that pipelines know whether to proceed or abort.
b) Also, if your tests can execute in parallel, the test cycle time is going to be a function of the longest test. Use "parallel" steps in Bitbucket Pipelines to execute tests simultaneously and get fast feedback. You could design your suites in a way that all tests execute no matter what, and then the pipeline aborts if there was at least one failed test. Or, you could design it such that the first failed test aborts the pipeline.
c) Test code, scripts, data, and configuration are all versioned artifacts in your source code repository and keep them co-located with your code.
d) Last but not least, integrate all types of tests - unit tests, static code analysis, functional, integration, performance, and security - with the pipeline. "Shift Left" advocates for checks to be pulled left, or in other words, earlier in the pipeline.
Build your automated tests such that when something passes through your pipelines, you will feel good about it!
As per business needs, deliver your product from test to staging to production in an automated fashion.
The publication of versioned artifacts helps establish an audit trail and facilitates rollback to the last working version. Although rolling forward is the right thing to do, it can require discipline and practice, and in the meantime rolling back keeps things real for teams who are working to get there.
Publish artifacts only when enough tests have passed since there is no point in distributing a dysfunctional artifact for downstream consumption. My sample publication script is in Python and invokes the AWS CLI (command line interface) to archive artifacts in version-enabled S3 buckets. It references environment variables like $APPLICATION_NAME, $S3_BUCKET, and $S3_BUCKET_FOLDER that handle the communication between Bitbucket and AWS.
You could define whatever publication mechanism suits you best. Check your organization's artifact retention policy and make sure you comply with audit guidelines.
import os os.system("apt-get update && \ apt-get install -y python-dev && \ curl -O https://bootstrap.pypa.io/get-pip.py && \ python get-pip.py && \ pip install awscli && \ aws deploy push --application-name $APPLICATION_NAME --s3-location s3://$S3_BUCKET/$S3_BUCKET_FOLDER/$APPLICATION_NAME-$BITBUCKET_BUILD_NUMBER.tar --ignore-hidden-files")
Add a step to the same pipeline to publish versioned artifacts to an artifact repository. Inside the "step" keyword, feel free to override the pipeline image if you need to support a script written in a different language. In this example, I have a Python script that needs to execute within the step and I ran it inside a Python container.
- step: name: Upload to S3 caches: - node image: python:3.5.1 #Image inside a step script: - python publish_artifact_to_s3.py
"Works for me!" doesn't work for me at all. Or for that matter, "works for me!" doesn't work for anyone.
Artifacts should be downloaded and deployed to test, staging, and production with the exact same deployment mechanism, whatever your mechanism may be. Use the "deployment" keyword to define your environments as Test, Staging, and Production, and these will serve as milestones for your team.
Note the manual step to deploy in production in line #14 implemented with the "trigger" keyword.
- step: name: Deploy to test deployment: Test script: - <Use the same deployment mechanism for test, staging, and production> - step: name: Deploy to staging deployment: Staging script: - <Use the same deployment mechanism for test, staging, and production> - step: name: Deploy to production deployment: Production trigger: manual #Manual step for production deployment script: - <Use the same deployment mechanism for test, staging, and production>
Once you have your environments defined in the YAML, you will see them under "Deployments" in the left panel. This view provides visibility into the versions deployed in each environment and the deployment history.
Remove the manual step from line #14 by commenting out a single line of configuration in the YAML. Only when you are ready though! And that's the major difference between continuous delivery and continuous deployment. The manual step, or the lack of it.
- step: name: Deploy to test deployment: Test script: - <Use the same deployment mechanism for test, staging, and production> - step: name: Deploy to staging deployment: Staging script: - <Use the same deployment mechanism for test, staging, and production> - step: name: Deploy to production deployment: Production #trigger: manual (commenting out the manual step) script: - <Use the same deployment mechanism for test, staging, and production>
Continue to build out your pipeline, one step at a time, till your pipeline represents the value stream map of your release process. Notice how the names of the steps show up in the pipeline visualizer under the "Pipelines" tab. When you click on a step, you can see the logs on the right panel. Everything is traceable and you can go back in time from Pipeline #46 to #45 to #44 and so on. You can download the raw log using the "Download raw" link at the top right corner.
And with that, we have touched upon the four pillars of the continuous paradigm - continuous integration, continuous testing, continuous delivery, and continuous deployment. In the interest of time and words, we can't get into all the details. However, feel free to reach out to the Atlassian community if you have questions.
This is a good segway to zoom out and discuss the anatomy of a typical continuous delivery pipeline that ships a Node.js application from test to staging to production. Use this for reference, and design a customized approach for your business.
Anatomy of a Continuous Delivery Pipeline
The anatomy could vary depending on, among other things, your product architecture, tech stack, infrastructural choices between on-prem and cloud, toolchain, skills, and people's mindset. Multiple vendors need to play ball and I have articulated a bunch for your convenience. The Atlassian Bitbucket marketplace offers integrations that help organizations achieve their goals.
Here's the anatomy of a typical continuous delivery pipeline that ships a Node.js application. This by no means is an exhaustive list, and you can integrate with whoever satisfies your engineering and business use cases.
A typical pipeline looks like:
1) Checkout code from Git.
Here's some Git reference material for your convenience.
2) Build an artifact using npm.
3) Run unit tests using and report code coverage.
Unit tests are your best friend if you have to refactor and one option is Jasmine. Try not to make a big deal with your code coverage numbers, since I have seen some teams with a higher code coverage cause more production outages than teams with lower code coverage. You can measure code coverage with Cobertura.
4) Run static code analysis.
Static analyzers detect issues in code without executing it. This is an inexpensive way to identify violations in coding best practices, potential memory leaks, security vulnerabilities, etc. Popular choices include ESLint and Coverity.
5) Publish versioned artifact to an artifact repository.
An artifact repository retains your versioned artifacts for a specified duration. This preserves an audit trail, and also helps to rollback to the last working version in cases where rolling forward isn't feasible. Popular artifact repository choices are Artifactory, Nexus repository, or version-enabled S3 buckets.
6) Download the latest artifact and deploy to a test environment.
Your environment could be IaaS solutions like AWS and Azure, or PaaS solutions like Heroku and CloudFoundry. This could also be VMs in your Data Center, although we recommend you consider moving your on-prem data centers to the cloud.
7) Run functional tests, performance tests, and security tests in parallel. File a defect automatically if a test fails.
SauceLabs is like Selenium in the cloud, BlazeMeter is like JMeter in the cloud, and OWASP ZAP can be containerized and hosted in the cloud to run security tests. Test in the cloud so that you can focus on the test coverage and automation, instead of the test environment.
Run tests using "parallel" steps, so that you can optimize test cycle time. The total execution time should not exceed the time taken to run your longest test.
Use Jira as your automated defect tracker and you could transition defects through their life cycle to reflect they are [open | in progress | resolved | closed | ...]. You could use native integration with Jira if it is there, Jira plug-ins within your orchestrator, or Jira's REST APIs directly.
8) Deploy to staging.
Once again, this is the IaaS/PaaS of your choice.
9) Run integration, performance, and security tests in parallel. File a defect automatically if a test fails.
Focus on testing out the interfaces and the network this time.
10) File a change request and transition it throughout its life cycle.
Jira allows you to create an audit trail for all changes headed for our customers. You could transition change requests through their life cycle to reflect they are [open | in progress | resolved | closed | ...].
11) Deploy to production.
Once again, this is the IaaS/PaaS of your choice. The important thing is to keep your test, staging, and production environments alike.
12) Run smoke tests in production. File a defect automatically if a test fails.
This could be a subset of your existing integration, performance, and security tests.
A/B tests can provide a ton of insights on how your new change might affect your customers and Optimizely is one option. Try to do this even earlier, meaning, in staging. These could be executed as part of a manual gate.
13) Go live!
You have moved the bits with the deployment steps. Now that smoke tests have passed, turn them on. Understand that moving bits and turning them on are two different things, often brought up in the context of explaining the difference between a "deploy" and a "launch". LaunchDarkly helps teams release with feature flags/toggles.
14) Last but not least, at any point in the pipeline workflow, generate alerts for the team if needed.
Use a team communication/collaboration tool to send notifications. This is especially helpful for remote folks. Understand that notification fatigue can happen if you are not mindful of the number of times people get pinged. Also, people tend to ignore notices when there are too many false positives. It's tricky, but we need to find the right balance.
It's a Learning Game
Have fun. Continuous delivery is neither a checklist that you can rush through nor a destination that you can run to. Remember, continuous improvement is at the heart of the continuous everything paradigm, and keep using your learnings from the previous Sprint to groom and refine your single prioritized backlog.
Then 1-peat, 2-peat, 3-peat, and repeat. Continuous, right?
Opinions expressed by DZone contributors are their own.