Testing the Internet of Things

The mobile revolution flipped testing on its side. Suddenly teams and testers had to think about connectivity in new ways, power sources, the impact of touch technology, system interruptions, bandwidth, and dropped connections — which was barely on the radar screen before.

Testing is on the cusp of a new revolution; this time, it is the internet of things, or IoT.

What Is the IoT, and Why Does It Matter

While phones and tablets are mobile, they are essentially still “just” personal computers in a new size. The Internet of Things covers other things connected to the internet, like your thermostat, automobile, or washing machine. All of these tools existed before the internet; we are adding the internet to old things. The smart watch is a great example of an internet of things device; people are using them to track not only the run, but also their heartbeat during the run. The watch uses a GPS and a wireless connection to upload the data to the internet, providing graphs and analysis later.

Two years ago the smart watch was something that appeared in Dick Tracy comics. A year ago, it was an expensive luxury that conveyed social status. Today you can buy a smart watch at WalMart for about a hundred dollars, and a fitbit for a little less.

The IoT is quickly becoming something the world relies on to drive, report, and optimize performance for everything, from your 5K run to the mileage in your car. And it needs to be tested.

Start with Usability

Done well, the IoT is deceptively simple; your wrist band might have a single start/stop button to record a run. Things have to be this way because while the devices vary in size, the displays and buttons are generally very small – somewhere between four square inches and a quarter of a square inch. That makes usability supremely important. While software that runs on a browser can give hints and subtle clues about how to use it, IoT devices have no room for subtlety.

When I first started looking at usability, the topic seemed like a mystery, more art than science. Explanations of usability seem easy to understand, even intuitive, but that does not provide a method to actually do usability testing on a new product. The ability to almost instantly recognize functionality and how to use it is crucial for consumer products. There is limited display and functional space, that translates to a need to make things obvious.

Minimalism is one thing that you can look for, more specifically, you can look for the things that are missing. Minimal user interfaces are free of extra information that might be in a normal web page — tool tips, formatting examples, general help.

A lot of these principles are described in depth in the book The Design of Everyday Things by Donald Norman.

I have a Nest Thermostat in my house. The thermostat is incredibly simple on the outside. There are two main ways a person can use the Nest; you can turn the knob, and you can click it. With these two functions, you can do everything you need to do. And you don’t need guidance to figure it out.

On the inside, the Nest is collecting data about local weather conditions, controlling the temperature in my house, and sending and receiving data about energy usage. It is all neatly hidden away behind a slick, usable interface.

Devices, Devices Everywhere

There are thousands, maybe hundreds of thousands of different internet connected products now. All of them are collecting and sending and displaying data all the time. Some companies create and sell their own devices, like the Nest. For these companies, it’s not a big deal to have all or most of the important physical devices on hand. Others create applications or serve data to many different devices, and even if they could identify every hardware / operating system possibility, they couldn’t possibly have them all available for testing.

Companies in this second category have the challenge of finding a minimal reference set to test on, or rotate what devices are used for testing. One way to figure out what to test on is to get a prototype or early version out fast, even if it is just to employees. Once the device is out, get usage information (Google Analytics is free) to figure out what devices are most popular; then select the top devices to test on.

With real information from meaningful customers, you can make educated choices about which devices are important to have, and which might be OK to leave out of the testing rotation for now. Companies that have usage information (or can make informed guesses), can make a list of supported devices, just like software in the 1980’s used to have a list of supported computers.

Sometimes, like with Netflix, you have no control over the devices a service is used on. Communication with the people consuming your service is important in these cases so they can stay in sync with the API and data you provide.

Connectivity

Internet connectivity is what makes this network of things possible, but in many cases that network is intermittent or unreliable. It is important to know that your data will be saved and stored correctly if you unexpectedly lose a connection, and is delivered correctly when your connection is restored. You can simulate this by using airplane mode or temporarily disabling the internet connection on the device, sure.

Even better than simulation is taking a walk in and out of wifi connections and seeing what really happens when you really lose a connection.

Growing Security Concerns

According to this article, 70 percent of devices in the Internet of Things are vulnerable to security problems — missing data encryption, minimal password requirements, and access to the user interface without passwords. Security through obscurity, the idea that being secretive about how a product is implemented, may have been good enough when product lines first emerged. The IoT isn’t exactly obscure anymore, and many devices have large customer bases.

Don’t wait till there is a headline news story to understand your products’ security short comings.

Tailor Your Approach

There is incredible variety in devices, software, the people that use them, and the needs these products are trying to meet. ‘It depends…’ isn’t a great answer to the question of what you should focus on when testing the the things of the Internet of Things, but it won’t lead you astray. Starting your test strategy with an understanding of what your customers value will be a powerful first step.

Some devices like internet connected storage will have you focusing on performance and security, for medical devices precision of data collection and delivery is important, and gaming consoles should be entertaining. Never forget who your customers are.

The IoT is still a developing market; things are going to change a lot in the next few years, especially in the area of security.