Managing the Plethora Of DevOps Tools

I have been thinking about DevOps tools a lot, and discussions about tools often distract from the real problems. But what are the right DevOps tools? Well I will not go into specific tools, but instead I will tell what I am looking for in DevOps tools beyond the functionality they provide. In my experience you can build good DevOps toolchains with just about any tool, but some tools might take more effort to integrate than others. I will also provide some guidance on how to manage tools at the enterprise level.

It seems that new DevOps tools appear on the market every month. This is extenuated by the fact that it is difficult to classify all the tools in the DevOps toolbox. One of the best reference guides is the XebiaLabs periodic table of DevOps tools, which is well worth checking out.

Before I go into the details of what characteristics a good DevOps tool should have, I want to address one other aspect: Should you have one tool or many in an organization?

In general, in large organizations it makes sense to have a minimal set of tools to support for several reasons:

• Optimize license cost
• Leveraging skills across the organization
• Minimizing complexity of integration

Yet on the other side some tools are much better for specific contexts than others (e.g. your .NET tooling might be very different from your mainframe tooling). And then there are new tools coming out all the time. So how do you deal with this? Here is my preferred approach:

• Start with a small set of standard tools in your organization.
• Allow a certain percentage of teams to diverge from the standard for a period of time (3-6 months perhaps).
• At the end of the “trial-period’ gather the evidence and decide what to do with the tool in a discussion:
  • Replace the current standard tool
  • Get it added as an additional tool for specific contexts
  • Discard the tool and the team transitions back to the standard tool

Obviously DevOps tools should support DevOps practices and promote the right culture. This means the tools should not be a “fenced garden” and only work within their own ecosystem. It is very unlikely anyway that a company uses only tools from one vendor or ecosystem. Hence, the most important quality of tools is the ability to integrate it with other tools (and yes possibly be able to replace it which is important in such a fast moving market.)

1. So then the first check is how well APIs are supported. Can you trigger all functionality that is available through the UI via an API (command line or programming language based)?
2. We should treat our tools just like any other application in the organization, which means we want to version control it. The second check is whether all configurations of the tool can be version controlled in an externalized configuration file (not just in the application itself)?
3. Related to the second point, does the functionality support multiple environments for the tool ( e.g. Dev vs Prod)? How easy is it to promote configuration? How can you merge configuration of different environments (code lines)?
4. We want everyone in the company to be able to use the same tool. This has implications for the license model that is appropriate. Of course open source works for us in this case, but what about commercial tools? They are not necessarily bad. What is important is that they don’t discourage usage. For example, tools that require agents should not price for every agent, as this means people will be tempted to not use it everywhere. Negotiate an enterprise license or ‘buckets of agents’ so that each usage will not require a business case.
5. Visualization and analytics are important aspects of every DevOps toolchain. To make this work we need easy access to the underlying data, and that means we likely want to export or query data. If your data is stored in an obscure data model or you have no way to access the underlying data and export it for analysis and visualization then you will require additional overhead to get good data. Dashboards and reports within the tool are no replacement as you likely want to aggregate and analyse across tools.

I hope these criteria are all relatively clear. What is surprising is how few tools adhere to these. I hope tool vendors will start to realize that if they want to provide DevOps tools they need to adhere to the cultural values of DevOps to be accepted in the community.

Hopefully the tools you are using are adhering to many of these points. Let me know what you think in the comments.