Flip on the Testing Switch for Ops-DevOps Engineers

If you are an operations or infrastructure type engineer, testing may mean traditional “QA”, UAT, or the newer test automation. An alternate way to think about tests is that you are asking a question about the software or infrastructure. The question is open-ended — there is no definitive answer. What will happen when the instance runs out of disk space? It will trigger an alert. Will it? For example, why did the bedroom alarm clock not ring on the day of your important meeting? If the alert is triggered, will the people who are paged know how to respond? If we have a separate operations team, will they understand the application to troubleshoot the alert? What are the different situations for when the instance can run out of space? Let’s call these type of questions tests.

There is another type of question for which we know the outcome. I verify that the new instance of WordPress is working by logging into the application using the username, ‘admin’, and the password, ‘password’. I could use different types of usernames and passwords. I could maybe try different lengths of strings and different characters. When you know the outcome of the question, we can call these checks.

Whether it’s infrastructure or applications, I find it useful to think about tests and checks as distinct. Note that this is a thought experiment; this is a way of thinking. What, if anything, you do with this is up to you. When I test, I hold myself accountable for the questions (tests and checks) that I didn’t ask.

Other Thoughts to Consider 

• What is the value of thinking of checks and tests?
• How would you use this for infrastructure/DevOps?
• What tests do you use when commissioning a new server?
• Are Serverspec scripts considered tests?
• Are Python unit considered tests (not related to infrastructure)?
• Are UAT scripts considered tests?
• Are code reviews considered tests?
• Is an AWS Well-Architected Review a series of checks, or are they tests?
• Is it easier to spot a check compared to spotting a test?
• Are tests more common when working with infrastructure?
• Can you repeat tests? How about checks? Should you repeat tests?
• Can you write tests?
• Can you run tests?

Engineers who have a development background, as opposed to operations, will be familiar with unit tests and test automation. They may also be familiar with the idea of "checks". Engineers from an infrastructure/hardware background (Ops-DevOps) may not be as familiar with tests and might find the idea of ‘checks’ useful. If you are a developer who is very familiar with unit tests and test automation, I recommend thinking of checks when you write your tests (checks?).

Tests and checks may not be binary, as described here. However, if you haven’t heard of checks, you can start with thinking of tests and checks as distinct. I’ll leave it as homework for you to figure out if they are distinct. I hope you don’t dismiss the concept just because they aren’t binary.

The concept of checks was proposed in 2009.

There is a strong school of thought that considers testing as critical thinking. A good starting point to learn more is by checking out the Association of Software testing.