When to Automate Mobile Tests

There are a few speed bumps in the path of releasing mobile products, even for teams with the most current XP/Agile/DevOps process. New software is usually delivered through iTunes or the Google Play store. Before getting into the store, there are approval and testing procedures that companies must follow. Getting a new version out if there are bugs in production is more complicated than rerunning a deploy script, and there is no guarantee that everyone will take the update.

Companies use automation to reduce some of that risk. Finding the right balance for mobile automation can be a challenge.

In the API

This is a little counter-intuitive, but the first place I like to start with mobile test automation is in the API. Most things someone will perform in a user interface- submit a form, navigate, upload and download files- call a bit of code underneath the surface. Testing that code directly means one less complicated piece of code to test a user interface, and potentially one less thing testers have to remember to focus on before a release. For example, consider an eCommerce app with a product page to order a t-shirt. That shirt has two cuts, one for women and one for men, 5 sizes and 5 colors for each size. Testing all of those permutations by hand means no one's going to do that in UI test, either with code by hand; they'll make a small random sample. At the API level, you could create one script that passes data to the /buy endpoint, have a separate file for each data combination and loop through them all in five or six seconds if the API is slow. A fast API in the same data center should run all those checks in under a second.

The API is an ideal place to automate data testing and some minor workflows.

On the User Interface 

Automating the user interface for a mobile app requires an automation framework, a proxy like Appium to run the tests, and environment considerations like whether you want to run on simulators or real devices, public or private clouds, and in-house or through a service. Despite those challenges, there is a lot of value in building these tests. The user interface is a great place to automate crucial workflows that the team would want to know are broken as quickly as possible, and cursory testing of each page. 

Referring back to our shirt buying example, you might create a test that covers making a complete purchase and one that attempts to buy an option that is out of stock. This covers basic functionality while hitting a lot of different aspects of the app. Since the majority of data testing is already handled underneath the covers in your API, your UI tests can focus on workflows and the most important aspects of your app. After all, we now expect the path to purchase to succeed at the UI level for one combination, and the API to succeed for all combinations, that's a pretty decent amount of coverage for not that much effort. 

The Fine Line of Exploration

There is a movement in the testing community that likes to draw lines between automation and testing. The idea being that testing is something performed by a person. Humans have values, goals, can learn, and will constantly revise and change their testing approach based on what they just learned. Automation, on the other hand, is an algorithm that performs exactly the same way every single time. To me, that seems reasonable. Yet the process of building good automation is itself a kind of learning journey - and exploration.

Usually, when I open a mobile app with a test to automate in mind, it goes something like this.

First I write a line of code and run it. It doesn't work, and I don't know why. So I open the app and perform those steps by hand, noticing that a button is missing. Is that right? Should that button be there? Then it's off to talk with a product manager and get an update. Removing the reference to that button, I discover a problem with the alignment of a text field.

Each line of code requires exploration, learning; some require the author to pivot, changing the plan.

A person that is good at testing will probably be pretty good at making test automation ... after some time learning the tools.