You Can’t Test All the Things: API, IoT, ROI TBD

I have three words for everyone in software testing: prioritize, prioritize, and prioritize.

You can’t test every possible permutation of your software, especially so with APIs and IoT devices where you’re placing much of the user experience in the hands of integrators to your core products and services. You just can’t, so we should throw our hands up now and just give up, right?

What Sort of Tester Are You Anyway?

I’m not; at best I’m an automation engineer. I don’t nearly have the mind or patience that is required of a full-time, dedicated testing professional. But I talk to them and lots of other people every day, and I’m still a developer by heart, so I feel very kin to everyone involved in the software delivery process.

You don’t have to be a tester to understand the constraints of being human. Humans make mistakes and only have so much time per day. Developers who are starting to experience a shift-left in testing responsibilities might make sure that all functions and procedures are unit tested, but often don’t go much further simply because there isn’t enough time. Not enough people, not enough time…two things I hear an awful lot and part of the reason why I’m in the automated testing tools business with Ready! API.

Side note: I would also like to take this opportunity to say that “no budget” and “not a priority” are the worst reasons not to invest in a better future, one where automation gives you back time to build better software and where getting your delivery pipeline right is critical to moving forward with rapid iteration.

But Shouldn’t We Always Test More?

No. Well, yes. Actually, sort of and not always. There’s an aspect of diminishing return to introducing more testing in that if you don’t prioritize the most business-critical components and workflows to be tested first, you fail your job, your business, and yourself as a professional tester.

We simply can test everything, we just don’t have the time, and we also need to stay nimble. If I were a tester, I’d probably get fired for wanting to confirm that what I was told to work on actually affects the bottom line of the business in the most impactful way at this moment in time. I would not sit there creating tests just so that we could report that we have 79% instead of 78% testing coverage today. I would want to see ROI, even though apparently “there ain’t no ROI in software testing”.

What About All the New Stuff, Shouldn’t That Be Tested?

I got a question after my talk at APIdays IoT San Francisco 2015 in July: “If you don’t know how people are going to combine your devices and APIs, how can you ensure that you’ve tested everything?” In no disrespectful way, this kind of question underlines the immaturity of the IoT space. We’ve got a very young startup culture (think 20-somethings with multiple PhDs) attacking big problems, most usually around how the digital world imbues itself back into the physical world, as quickly as possible for as cheap as possible.

There’s going to be some broken things, incomplete thoughts, and definitely failures to learn from. I saw this first hand at O’Reilly Solid the week after APIdays too. The speakers were experienced industry leaders, and it was good that those people were on stage speaking to the young engineers building all sorts of IoT devices without a thought to protocol or integration standards. The median booth sponsor was grad school geniuses that are practically DevOps natives building our future…so busy implementing that they may not even know to ask the more important question, “should we”?

What Do You Have Against Startups?

Nothing personal, I’ve been involved in a few myself. The “startup kids” are working incredibly hard, many harder than I’ve ever worked, to make it or break it in the cutthroat technology market that we’ve built for ourselves. And when I say “kids”, I mean figuratively; folks who bring a variety of backgrounds, skills, and experience to a startup still often have a lot to learn about running a lean operation before things lock in and start whirring.

Some things take time to learn (thanks to our lizard brains) and we (the technology industry) would do well to remember that people are still naturally slow to accept and properly utilize new technologies (like for instance the slow but increasing industry adoption of lightweight service virtualization tools).

If there was one thing I’d take issue with related to startup culture, it’s that some important corners can get cut when you’re so busy accelerating that you don’t have time to ask “should we?”. For instance, API security is a huge deal now because of a number of very public API security failures have occurred in the past few years. Moonpig, Uber, Tinder, and others could have avoided this just by applying free knowledge to their designs.

The stakes get bigger too the closer to money and safety your software is, like as in financial and medical institutions, or worse defense and air travel; proactively identifying vulnerabilities to critical services is a fundamental reason for regulatory audits, and keep omissions in safety introduced by startup culture thinking from having a massive negative affect on everyday life for the average digital citizen.

Then How Do I Know When I’ve Tested Enough?

That’s up to you, it’s your software, it’s your customers and their expectations that should help you know when enough is enough. In the absence of formal UX and operational metrics, you may need to set some goals starting with your technical team. You should also be talking directly with your consumers, getting their feedback and input on what they expect from your software.

Go read some James Bach or Michael Bolton books if you want to learn about testing in general, but when it comes down to specifics like SLAs and reporting to management, there are very tangible questions to ask about quality. For instance, a slightly more matured set of questions around testing IoT and other multiples of design complexity (like Microservices, Hypermedia, etc.) might include:

• Should my goal be to test all the things and permutations?
• What do the user-experience metrics say about what my customers see as a priority?
• What technical omissions contribute to our worst business deficits?
• Do I need to build/test/ship this, right now, at this very moment?
• What do we gain and/or lose by shipping what we have, such as it is?

Okay, So I Don’t Have to Test More?

Don’t we all wish; sorry, no, this isn’t a free lunch. Yes, you should be testing more, almost universally. Most software vendors and engineers know that automated testing is the only way to achieve high quality with less manual intervention, but haven’t seen the pain that a lack of proper testing causes.

Good engineers and technicians have consciences and know when something needs more attention, which is why I leave it up to you to answer the question “how much testing do I need?” You don’t need your inner voice telling you that you need to test and monitor something that only you know would cripple your business if it went down for any amount of time. So in those moments, I humbly submit to you that if you see or feel the future pain, it is your responsibility to write the test, or at least put it in the priority backlog, or something that your team is sure to follow up on.

Often backlogged is “test coverage”, a useful assessment tool, but also just a broad-spectrum approach to getting visibility over technical gaps in software delivery. It helps you see when new things are implemented without the proper team communication, but should not be a goal on its own right. When was the last time your customer said “if you don’t have at least 82% testing coverage, I don’t want to use your software”? Right. Most consumers care about the downstream effects of bad software like app crashes, not the things you do to prevent bugs from getting to production. Coverage is not a golden idol, it’s just a method of assessing where your weaknesses lie.

Like security testing, test coverage is a measurement of only one aspect of readiness that should be part of a larger perspective on customer-focused software quality. The connection between testing and customer value shows up in other areas like beta acceptance testing, performance monitoring, and consumer feedback. It’s there for the taking when we want it.

That’s my experience with testing “all the things”, but what’s yours? I’d love to hear your thoughts.