Unit Testing and DevOps
Unit Testing and DevOps
Unit tests are arguably mandatory. Get a comprehensive overview of libraries, tools, and strategies for DevOps unit testing.
Join the DZone community and get the full member experience.Join For Free
Open source vulnerabilities are on the rise. Read here how to tackle them effectively.
I should start with some background on why I did this talk in the first place. Unit testing (and integration and end-to-end) is something familiar to even a junior developer. It’s definitely an extensive and hotly debated topic. Anything from TDD to the quality of tests to the significance of coverage to prioritizing integration over unit tests, etc, etc. These are interesting subjects, but they are for a different post.
Unfortunately, unit testing is often a foreign concept in many DevOps/syseng/ops teams. The way I look at systems engineering or DevOps (I hate the term when used as a job description, though I think I lost that battle) is that in a modern organization, they are fundamentally software development teams with software engineers. It just so happens that their domain of expertise is systems/observability/scalability, etc and their product is often the tooling or the glue or the platform that makes all of this happen.
But if your job is primarily writing software and you take infrastructure-as-code approach, then tests are absolutely mandatory. My goal with this talk was to give a quick intro to testing and lay out a few options that are available to folks in this world. A lot of details are in the slides, which I am not going to rehash here, but some of the things that can be useful:
- moto - excellent library for mocking out boto and a lot of AWS services. Having python scripts and tools is fairly common in ops and boto library is quite good. I often use it for testing in addition to unittest and others.
- bats, roundup, and shunit2 - bash scripts can also use some love and testing.
For config management tools, there are plenty of testing frameworks available as well. My teams tended to settle on Ansible so I am more intimately familiar with it, but a lot of these work across the board. The tricky part is deciding where to draw the line and what can be reasonably tested. A few general things that will typically be important to test are:
- Complex conditionals – sometimes these are unavoidable. These often come paired with somewhat complex data structures that are being evaluated. You want to make sure you check your logic.
- Logic in templating engines (Jinja2 in Ansible) – Something very complicated there is often a code smell, but there are exceptions to every rule.
- Variable inheritance – In Ansible, you can define variables in 24 places and the precedence is sometimes unclear. When the same variable is being set in multiple places, you need to test that the precedence is correct.
More specifically, with Ansible, you can use a combination of:
- Assert module - this works particularly well with variable testing.
- Serverspec, inspec, rolespec, testinfra, goss, chefspec, ansible_spec, test-kitchen – these all work reasonably well. I prefer goss, but that’s mostly a matter of opinion.
- Molecule - a decent tool that glues docker/vagrant and your testing framework of choice. Will save you from writing a lot of scripts.
As a reference example, this is a workflow that a few of my teams have used in the past. It’s an Ansible example, but other config management systems will be reasonably similar. A typical role folder/file structure will look like this:
r-role_name .bumpversion.cfg molecule.yml playbook.yml defaults/ main.yml files/ goss/ test.yml handlers/ main.yml tasks/ main.yml templates/ main.yml tests/ test.yml vars/ main.yml
A majority of the directory structure is the same as you would see with ansible-galaxy init. The rest is:
- bumpversion – used for auto-versioning roles by the build system.
- molecule.yml – has Molecule configuration in there. This is where we’d specify the docker container, test sequences, and the tool used for testing among other things.
- files/goss/test.yml – this will have a set of tests for a particular role. It will generally include some combination of these tests if using goss and/or additional custom testing code.
When engineer commits changes to the role, the build system will trigger a "build," execute the tests defined for it and increment the version if the tests pass. Especially in cases when dealing with complex roles that handle many scenarios, this can be very helpful. Someone might be making a small modification and the overall impact is possibly unclear. Assuming tests are well written and cover main use cases, this will help an engineer catch an error early.
No organization is going to be perfect and have 100% test coverage that captures every corner case. But adopting some structure/process, setting aspirational goals will go a long way, especially for larger teams. What’s not acceptable is ignoring testing altogether or saying that it can’t be done.
Published at DZone with permission of Arthur Freyman . See the original article here.
Opinions expressed by DZone contributors are their own.