{{announcement.body}}
{{announcement.title}}

The DevOps Toolchain

DZone 's Guide to

The DevOps Toolchain

We’ll be looking at the core technology categories and specific technologies that help you realize your DevOps objectives.

· DevOps Zone ·
Free Resource

Introduction

We’ll be looking at the core technology categories and specific technologies that help you realize your DevOps objectives.

About DevOps and Tools

Things to remember about DevOps and its corresponding tools:

  • The goal is continuous improvement
  • You can’t buy DevOps
  • Tool adoption happens in stages

DevOps Technology Categories

Planning

Why Does Planning Matter for DevOps?

  • Shared purpose
  • Transparency
  • Empowerment

Examples of Planning Tools:

GitLab

GitLab is a web-based DevOps lifecycle tool. It provides a Git-repository manager, which provides a wiki, issue tracking, and CI/CD pipeline features, using an open-source license developed by GitLab Inc.

GitLab has some tools for actually planning things out and being able to look at the scope of the work.

Tasktop

Tasktop allows all of these tools to be added to the entire range of Agile and application lifecycle management (ALM), PPM, and ITSM tools supported by Tasktop, enabling unprecedented visibility across the entire lifecycle.

CollabNet’s VersionOne

VersionOne supports Scrum, Kanban, XP, SAFe, and hybrid development methodologies and makes it easier to plan, track, and report across all teams, programs, software portfolios, and the enterprise.

Pivotal Tracker

The Agile project-management tool of choice for developers around the world for real-time collaboration around a shared, prioritized backlog.

Trello

Trello is a web-based Kanban-style list-making application that’s a subsidiary of Atlassian.

It’s used by many teams as they plan out individual sprints of work.

Azure Boards

Track your software projects and plan better with Agile tools, including scrum boards, Kanban boards, and dashboards for any Agile methodology.

And, again, these are ways to visualize work, share plans, track progress, and make sure you’re moving toward the goal.

Issue Tracking

Why Does it Matter for DevOps?

  • Customer responsiveness
  • Limited knowledge waste
  • Feedback loop

Examples of Issue-Tracking Tools:

Atlassian’s Jira

Jira is a proprietary issue-tracking product developed by Atlassian that allows bug tracking and Agile project management.

JetBrains’ YouTrack

YouTrack is a proprietary, commercial browser-based bug tracker, issue-tracking system, and project-management software developed by JetBrains. It focuses on query-based issue searching — with autocompleting, manipulating issues in batches, customizing the set of issue attributes, and creating custom workflows.

Zendesk

Zendesk makes it simple to track every customer issue from all of your channels. Many companies, including Uber and Airbnb, use Zendesk Support as their issue-tracking software.

Source Control

Why Does it Matter for DevOps?

  • Asset control
  • Limit transportation waste
  • Empower teams

Examples of SCM Tools:

Git

Git is a distributed version-control system for tracking changes in source code during software development. It’s designed for coordinating work among programmers, but it can be used to track changes in any set of files. Its goals include speed, data integrity, and support for distributed, nonlinear workflows.

GitHub

GitHub offers the distributed-version control and source code–management functionality of Git, plus its own features.

GitLab

Version control in GitLab helps your development team share, collaborate, and maximize their productivity with world-class source-code management.

Bitbucket

Bitbucket is a web-based version control-repository hosting service owned by Atlassian for source code and development projects that use either Mercurial or Git revision-control systems.

Subversion

Apache Subversion is a software-versioning and revision-control system distributed as open source under the Apache License. Software developers use Subversion to maintain current and historical versions of files, such as source code, web pages, and documentation.

Build Tools

Why Do They Matter for DevOps?

  • Consistent packaging
  • Automates an error-prone activity
  • Yields early quality signals

Examples of Build Tools:

Maven/Gradle

Maven is a build-automation tool used primarily for Java projects. Maven can also be used to build and manage projects written in C#, Ruby, Scala, and other languages.

Gradle is an open-source build-automation system that builds upon the concepts of Apache Ant and Apache Maven and introduces a Groovy-based domain-specific language instead of the XML form used by Apache Maven for declaring the project configuration.

MSBuild

Microsoft Build Engine, better known as MSBuild, is a free and open-source build tool set for managed code as well as native C++ code and was part of .NET Framework. Visual Studio depends on MSBuild (but not vice versa).

Rake

Rake is a software task-management and build-automation tool. It allows the user to specify tasks and describe dependencies. You can also do this for group tasks in a namespace.

JFrog Artifactory

JFrog Artifactory is a tool designed to store the binary output of the build process for use in distribution and deployment. Artifactory provides support for a number of package formats, such as Maven, Debian, npm, Helm, Ruby, Python, and Docker.

JFrog offers high availability, replication, disaster recovery, scalability, and works with many on-prem and cloud-storage offerings.

Sonatype Nexus

Sonatype Nexus is a repository manager. It allows you to proxy, collect, and manage your dependencies so you’re not constantly juggling a collection of JARs. It makes it easy to distribute your software.

Internally, you configure your build to publish artifacts to Nexus, and they then become available to other developers.

NuGet

NuGet is the package manager for .NET. The NuGet client tools provide the ability to produce and consume packages.

The NuGet Gallery is the central package repository used by all package authors and consumers.

Testing Tools

Why Do They Matter for DevOps?

  • Puts focus on built-in quality
  • Creates confidence in deployed artifacts

Examples of Testing Tools:

JUnit

JUnit is a unit-testing framework for the Java programming language. JUnit has been important in the development of test-driven development and is one of a family of unit-testing frameworks, collectively known as xUnit, that originated with SUnit.

xUnit.net

xUnit.net is an open-source unit-testing tool for the .NET Framework written by the original author of NUnit.

Selenium

Selenium is a portable framework for testing web applications. Selenium provides a playback tool for authoring functional tests without the need to learn a test-scripting language.

Jasmine

Jasmine is an open-source testing framework for JavaScript. It aims to run on any JavaScript-enabled platform, to not intrude on the application or the IDE, and to have an easy-to-read syntax. It’s heavily influenced by other unit-testing frameworks, such as Screw.Unit, JSSpec, JSpec, and RSpec.

Cucumber

Cucumber is a software tool that supports behavior-driven development. Central to the Cucumber BDD approach is its ordinary-language parser called Gherkin. It allows expected software behaviors to be specified in a logical language that customers can understand.

Continuous Integration and Deployment (CI/CD)

Why Do They Matter for DevOps?

  • Fast feedback
  • Reduce defect waste and waiting waste

Examples of CI Tools:

Jenkins

Jenkins is a free and open-source automation server. Jenkins helps to automate the nonhuman part of the software-development process, with CI and facilitating technical aspects of CD. It’s a server-based system that runs in servlet containers, such as Apache Tomcat.

CircleCI

CircleCI is the world’s largest shared CI/CD platform and the central hub where code moves from idea to delivery.

As one of the most-used DevOps tools, processing more than 1 million builds a day, CircleCI has unique access to data on how engineering teams work and how their code runs. Companies like Spotify, Coinbase, Stitch Fix, and BuzzFeed use CircleCI to improve engineering-team productivity, release better products, and get to et faster.

Travis CI

Travis CI is a hosted-CI service used to build and test software projects hosted on GitHub. Travis CI provides various paid plans for private projects and a free plan open source. TravisPro provides custom deployments of a proprietary version on the customer’s own hardware.

Concourse

Concourse is an automation system written in Go. It’s most commonly used for CI/CD and is built to scale to any kind of automation pipeline, from simple to complex.

AWS CodePipeline

AWS CodePipeline is a fully managed CD service that helps you automate your release pipelines for fast and reliable application and infrastructure updates.

CodePipeline automates the build, test, and deploy phases of your release process every time there’s a code change, based on the release model you define. This enables you to rapidly and reliably deliver features and updates.

You can easily integrate AWS CodePipeline with third-party services, such as GitHub, or with your own custom plugin. With AWS CodePipeline, you only pay for what you use. There are no upfront fees or long-term commitments.

Azure Pipelines

Get cloud-hosted pipelines for Linux, macOS, and Windows. Build web, desktop, and mobile applications. Deploy to the cloud or on premises.

Automate your builds and deployments with Pipelines so you spend less time with the nuts and bolts and more time being creative.

CD Tools

Why Do They Matter for DevOps?

  • Limit your preproduction inventory
  • Automate complex pipelines
  • Unify team around getting value to customers

Examples of CD Tools:

Spinnaker

Spinnaker is a free and open-source CD software platform originally developed by Netflix and quickly picked up and extended by Google.

Spinnaker is a multicloud CD platform for releasing software changes with high velocity and confidence. It combines a powerful and flexible pipeline-management system with integrations to the major cloud providers.

Octopus Deploy

Octopus Deploy is an automated deployment and release-management tool used by leading CD teams worldwide.

Octopus is a toolset that can drastically streamline any DevOps process for continuous testing and deployment of numerous microservices or applications via cloud or virtual machines.

AWS CodeDeploy

AWS CodeDeploy is a fully managed deployment service that automates software deployments to a variety of compute services, such as Amazon EC2, AWS Fargate, AWS Lambda, and your on-premises servers.

You can use AWS CodeDeploy to automate software deployments, eliminating the need for error-prone manual operations.

Configuration-Management Tools

Why Do They Matter for DevOps?

  • Enforce consistency
  • Treat infrastructure as code

Examples of Configuration-Management Tools:

Terraform

Terraform is an open-source infrastructure-as-code (IaC) software tool created by HashiCorp. It enables users to define and provision a data-center infrastructure using a high-level configuration language known as Hashicorp Configuration Language or, optionally, JSON.

BOSH

BOSH is a project that unifies release engineering, deployment, and the lifecycle management of small and large-scale cloud software. BOSH can provision and deploy software over hundreds of VMs. It also performs monitoring, failure recovery, and software updates with zero-to-minimal downtime.

While BOSH was developed to deploy Cloud Foundry, it can also be used to deploy almost any other software (Hadoop, for instance). BOSH is particularly well-suited for large distributed systems.

In addition, BOSH supports multiple infrastructure-as-a-service (IaaS) providers like VMware vSphere, Google Cloud Platform, Amazon Web Services EC2, Microsoft Azure, OpenStack, and Alibaba Cloud. There’s a Cloud Provider Interface (CPI) that enables users to extend BOSH to support additional IaaS providers, such as Apache CloudStack and VirtualBox.

Chef

Chef is a configuration-management tool for dealing with machine setup on physical servers, virtual machines, and in the cloud. Many companies use Chef software to control and manage their infrastructure, including Facebook, Etsy, Cheezburger, and Indiegogo.

Chef is the leader in continuous-automation software, an innovator in application automation, and one of the founders of the DevOps movement. Chef works with more than 1,000 of the most innovative companies around the world to deliver their vision of digital transformation, providing the practices and platform to deliver software at speed

Ansible

Ansible is an open-source software-provisioning, configuration-management, and application-deployment tool. It runs on many Unix-like systems and can configure both Unix-like systems as well as Microsoft Windows. It includes its own declarative language to describe system configuration.

Puppet

Puppet provides the ability to define which software and configuration a system requires and then maintains a specified state after an initial setup.

You use a declarative domain-specific language (DSL) that’s similar to Ruby to define configuration parameters for a specific environment or infrastructure. Puppet discovers information about a system by using a utility called Facter, which is installed when you install the Puppet software package.

The Puppet master is the system that manages important configuration information for all of the nodes that it controls by using manifests.

The nodes that the master controls are those that have Puppet installed on them and are running the Puppet agent, which is a daemon. The configuration information that the agent collects about a node is sent to the Puppet master. The Puppet master then complies a catalog based on how the node should be configured. Each node uses that information to apply any necessary configuration updates to itself.

Puppet works by using a pull mode, where agents poll the master at regular intervals to retrieve site-specific and node-specific configurations. In this infrastructure, managed nodes run the Puppet agent application, typically as a background service.

Google Cloud Deployment Manager

Google Cloud Deployment Manager is an infrastructure-management service that makes it simple to create, deploy, and manage Google Cloud Platform resources.

Cloud Platforms

Why Do They Matter for DevOps?

  • Automation friendly
  • Observable runtimes for software

Examples of Cloud Platforms:

Amazon Web Service

Amazon Web Services (AWS) is a secure cloud-services platform, offering compute power, database storage, content delivery, and other functionality to help businesses scale and grow.

In simple words, AWS allows you with the following things:

  • Running web and application servers in the cloud to host dynamic websites
  • Securely store all your files on the cloud so you can access them from anywhere
  • Using managed databases, like MySQL, PostgreSQL, Oracle, or SQL Server, to store information
  • Deliver static and dynamic files quickly around the world using a content delivery network (CDN)
  • Send bulk email to your customers

Microsoft Azure

Microsoft Azure is a cloud-computing service created by Microsoft for building, testing, deploying, and managing applications and services through Microsoft-managed data centers.

Microsoft Azure, formerly known as Windows Azure, is Microsoft’s public cloud-computing platform. It provides a range of cloud services, including compute, analytics, storage, and networking. Users can pick and choose from these services to develop and scale new applications or run existing applications in the public cloud.

Google Cloud Platform

Google Cloud Platform, offered by Google, is a suite of cloud-computing services that run on the same infrastructure that Google uses internally for its end-user products, such as Google Search, Gmail, and YouTube.

Google Cloud Platform is a provider of computing resources for deploying and operating applications on the web. Its specialty is providing a place for individuals and enterprises to build and run software, and it uses the web to connect to the users of that software.

Meet your business challenges head on with cloud-computing services from Google, including data management, hybrid and multicloud, and AI and ML.

Pivotal Cloud Foundry

Pivotal Cloud Foundry (PCF) is an open-source, multi-cloud application platform as a service (PaaS) governed by the Cloud Foundry Foundation, a 501 organization. The software was originally developed by VMware but was transferred to Pivotal Software. It was brought back into VMware at the end of 2019 with VMware’s takeover of Pivotal.

PCF is a multicloud platform for the deployment, management, and continuous delivery of applications, containers, and functions. PCF allows developers to deploy and deliver software quickly, without needing to manage the underlying infrastructure.

Heroku

Heroku is a cloud PaaS supporting several programming languages. One of the first cloud platforms, Heroku has been in development since June 2007, when it supported only the Ruby programming language, but now supports Java, Node.js, Scala, Clojure, Python, PHP, and Go.

Heroku is a container-based cloud PaaS. Developers use Heroku to deploy, manage, and scale modern apps. Our platform is elegant, flexible, and easy to use, offering developers the simplest path to getting their apps to et.

Container schedulers

The main task of a container scheduler is to start containers on the most appropriate host and connect them together. It has to handle failures by doing automatic failovers, and it needs to be able to scale containers when there’s too much data to process/compute for a single instance.

The three most popular container schedulers are Docker Swarm, Apache Mesos, and Google Kubernetes.

Docker Swarm

Docker Swarm is a container scheduler developed by Docker. The development of this cluster solution being managed by Docker offers advantages like the use of the standard Docker API.

The purpose of Mesos is to build a scalable and efficient system that supports a wide array of both current and future frameworks. This is also the main issue: frameworks like Hadoop and MPI are developed independently — thus it’s not possible to do fine-grained sharing across frameworks.

The proposition of Mesos is to add a thin resource-sharing layer, giving frameworks a common interface for accessing cluster resources. Mesos is delegating control over scheduling to the frameworks because many frameworks already implement sophisticated scheduling.

Four types of frameworks exist depending on the type of jobs you want to run on your cluster, and some of them have native Docker support like Marathon. The support of Docker containers has been added in Mesos 0.20.0.

We’ll focus on the use of Mesos with Marathon because it’s a framework actively maintained by Mesosphere that offers many features in terms of scheduling, such as constraints, health checks, service discovery, and load balancing.

Apache Mesos architecture using Marathon, © Adrian Mouat

As we can see on the image, there are four elements in the cluster. ZooKeeper helps Marathon to look up the address of the Mesos master — multiple instances are available to handle failure.

Marathon starts, monitors, and scales the containers. The Mesos master sends the tasks assigned to a node and makes offers to Marathon when a node has some free CPU/RAM. The Mesos slaves run the container and submit a list of their available resources.

Google Kubernetes

Kubernetes is an orchestration system for Docker containers using the concepts of labels and pods to group containers into logical units. Pods are the main difference between Kubernetes and the two other solutions — they’re collections of colocated containers forming a service deployed and scheduled together. This approach simplifies the management of the cluster — compared to an Affinity-based coscheduling of containers (like Swarm and Mesos).

The Kubernetes scheduler’s task is to watch for pods having an empty PodSpec. NodeName attributes a value to schedule the container somewhere in the cluster.

This is a difference compared to Swarm and Mesos, as Kubernetes allows developers to bypass the scheduler by defining a PodSpec.NodeName when running the pod.

The scheduler uses predicates and priorities to define which nodes a pod should run. The default values of these parameters can be overridden using a new scheduler policy configuration.

By using the command-line flag — policy-config-file pointing to a JSON file describing the predicates and priorities to use when starting Kubernetes, the scheduler will use the policy defined by the administrator.

Kubernetes architecture (containers in gray, pods in color), © Google Inc.

Monitoring and Logging Tools

Why Do They Matter for DevOps?

  • Fast recovery
  • Responsiveness
  • Transparency
  • Limited human involvement during incidents

Examples of Monitoring/Logging Tools:

ELK Stack

The ELK Stack is a collection of three open-source products — Elasticsearch, Logstash, and Kibana. They’re all developed, managed, and maintained by the company Elastic.

  • E stands for ElasticSearch — used for storing logs
  • L stands for Logstash — used for both shipping as well as processing and storing logs
  • K stands for Kibana — a visutalization tool (a web interface)

Datadog

Datadog is a monitoring service for cloud-scale applications, providing monitoring of servers, databases, tools, and services through a SaaS-based data-analytics platform.

Datadog Application Performance Monitoring (APM or tracing) provides you with deep insight into your application’s performance — from automatically generated dashboards for monitoring key metrics, like request volume and latency, to detailed traces of individual requests — side by side with your logs and infrastructure monitoring.

When a request is made to an application, Datadog can see the traces across a distributed system and show you systematic data about precisely what is happening to this request.

New Relic

New Relic is a San Francisco, California–based technology company that develops cloud-based software to help website and application owners track the performances of their services.

New Relic’s software analytics product for application-performance monitoring (APM) delivers real-time and trending data about your web application’s performance and the level of satisfaction your end users experience.

Prometheus

Prometheus is a free software application used for event monitoring and alerting. It records real-time metrics in a time-series database, built using a HTTP pull model with flexible queries and real-time alerting.

The Prometheus server works on the principle of scraping — i.e., invoking the metrics endpoints of the various nodes that it’s configured to monitor. It collects these metrics at regular intervals and stores them locally. The nodes expose these over the endpoints that the Prometheus server scrapes.

Zipkin

Zipkin is a distributed-tracing system. It helps gather timing data needed to troubleshoot latency problems in service architectures. Features include both the collection and lookup of this data.

If you have a trace ID in a log file, you can jump directly to it. Otherwise, you can query based on attributes such as service, operation name, tagsm and duration. Some interesting data will be summarized for you, such as the percentage of time spent in a service and whether or not operations have failed.

Azure Monitor

Azure Monitor maximizes the availability and performance of your applications and services by delivering a comprehensive solution for collecting, analyzing, and acting on telemetry from your cloud and on-premises environments.

It helps you understand how your applications are performing and proactively identifies issues affecting them and the resources they depend on.

Just a few examples of what you can do with Azure Monitor include:

  • Detect and diagnose issues across applications and dependencies with Application Insights
  • Correlate infrastructure issues with Azure Monitor for VMs and Azure Monitor for Containers
  • Drill into your monitoring data with Log Analytics for troubleshooting and deep diagnostics
  • Support operations at scale with smart alerts and automated actions
  • Create visualizations with Azure dashboards and workbooks

Communication Tools

Why Do They Matter for DevOps?

  • Connect teams
  • Limit waiting waste
  • Improve collaboration

Examples of Communication Tools:

Slack

Slack is a proprietary instant-messaging platform developed by Slack Technologies.

Slack is essentially a chat room for your whole company, designed to replace email as your primary method of communication and sharing. Its work spaces allow you to organize communications by channels for group discussions and allows for private messages to share information, files, and more all in one place.

Microsoft Teams

Microsoft Teams is a unified communication and collaboration platform that combines persistent workplace chat, video meetings, file storage, and application integration.

Teams is a chat-based collaboration tool that provides global, remote, and dispersed teams with the ability to work together and share information via a common space. You can utilize cool features like document collaboration, one-on-one chat, team chat, and more.

Google Hangouts

Google Hangouts is a communication software product developed by Google. Originally a feature of Google+, Hangouts became a stand-alone product in 2013, when Google also began integrating features from Google+ Messenger and Google Talk into Hangouts.

Hangouts Chat is a powerful way to communicate with people in your organization. Start a message with one or more coworkers, create a room for ongoing conversations with groups of people, and use bots to help automate your work. Chat works in your computer browser and in mobile apps for Android and iOS.

Zoom

Zoom Video Communications is a remote conferencing-services company headquartered in San Jose, California. It provides a remote conferencing service that combines video conferencing, online meetings, chat, and mobile collaboration.

Skype is better for teams looking for a holistic business solution. Zoom is a better fit for teams that have frequent video chats and meetings. However, neither tool was specifically built for remote work.

Knowledge-Sharing Tools

Why Do They Matter for DevOps?

  • Reduce knowledge waste
  • Increase new-hire productivity
  • Limit repeat mistakes

Examples of Knowledge-Sharing Tools:

GitHub Pages

GitHub Pages is a static site-hosting service that takes HTML, CSS, and JavaScript files straight from a repository on GitHub, optionally runs the files through a build process, and publishes a website. You can see examples of GitHub Pages sites in the GitHub Pages examples collection.

Confluence

Confluence is a collaboration software program developed and published by Atlassian. Atlassian wrote Confluence in the Java programming language and first published it in 2004.

Confluence is a collaboration wiki tool used to help teams to collaborate and share knowledge efficiently. With Confluence, you can capture project requirements, assign tasks to specific users, and manage several calendars at once with the help of the Team Calendars add-on.

Jekyll

Jekyll is a simple, blog-aware, static-site generator for personal, project, or organizational sites. Written in Ruby by Tom Preston-Werner, GitHub’s co-founder, it’s distributed under the open-source MIT license.

Jekyll is a parsing engine, bundled as a Ruby gem, used to build static websites from dynamic components, such as templates, partials, liquid code, down, etc. Jekyll is known as “a simple, blog-aware, static-site generator.”

Google Sites

Google Sites is a structured wiki- and webpage-creation tool offered by Google. The declared goal of Google Sites is for anyone to be able to create simple web sites that support collaboration between different editors.

Collaboratively create engaging, high-quality sites for your team, project, or event. The sites look great on every screen, from desktop to smartphone — all without learning design or programming.

Summary

I covered all these different categories that come together to collectively help you do DevOps. Any one of these is helpful, but I’d argue all of these are required for you to be in the place where you’re continuously delivering value to the customer in a way that’s sustainable to the team, as well as helping you differentiate.

I covered the overall DevOps tool chain — things like planning tools, issue-tracking tools, source-control management, building and testing your code and being successful with that, continuously integrating and deploying your source code, managing configurations, using cloud platforms, and then monitoring and logging before doing some of the softer things like communication and knowledge sharing.

I hope you enjoyed this article. DevOps is an exciting way to work. It’s not going to be the only way or even the best way. Something better may come along in the future, but for right now, it seems to represent a good way for teams to collectively work together to ship value that matters in a way that’s sustainable for the people on the team, while also helping them focus on the customer, not on IT efficiency — and not just on internal things but on speed, on learning, and on the things that actually matter as companies are differentiating now based on how good they are at technology.

Thank you for reading to this. I hope you enjoyed the article.

Topics:
devops, docker, git, jira, tools

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}