How Testlio Brought Crowd Testing to IoT

When I hear about a successful startup with founders from Estonia, I tend to take notice. It’s a country that’s birthed a range of interesting tech companies including Skype, TransferWise, pipedrive and Jobbatical.  It practices open governance. Data is open to citizens via the Internet, you can start a business, pay your taxes, and vote online. Even if you’re not from Estonia, you can still benefit — with the option to become an e-resident to enable non-EU members to do business in Europe.

So when I heard of an app testing company that was started in Tallinn, I had to find out more. Kristel Kruustük was just 23 when she became disillusioned by how app testers were treated. She wanted to build a platform that would appreciate the work of testers and change the way QA was done. She shared her idea with then-boyfriend, later-cofounder, now-husband Marko Kruustük and the two entered the world’s largest hackathon: Angelhack. Together, they took home first place, winning a $25,000 seed investment and their first paying customer.

Testlio provides an online mobile app testing solution that allows offloading QA with flexible requirements. It offers online platform, full testing, integration, reporting, functional testing, and localization services. Profitable from the get-go, Testlio handles QA for many successful apps including Microsoft, Lyft, Salesforce, CBS Interactive, Flipboard, Strava, OfferUp, Voxer, and GoDaddy. Last year they announced their $6.25M Series A and expanded their position in the $60B app economy. I spoke to CEO and founder Kristel Kruustük to learn more about Testlio and the role their testing plays in IoT.

A Network of Valuable Testers

At Testlio testing is carried out through a global freelancer network, made up of over 300 verified testers from 40 different countries who assist customers in 36 different languages. Of all applicants who apply to be part of the team, only 3% are successful. Over 50% of the team are women and minorities.

I asked what it takes to be a good tester. She replied:

“Oh, I think one key word here is curiosity. I know what it was like to be a tester myself. You need curiosity and courage, courage to be not afraid of asking questions. Like ‘what are we doing it this way? Why would it work this way? How can we break the software? Right. And I think that’s the number one thing that makes for a great tester."

What Are the Challenges of IoT Testing?

The sheer diversity of IoT devices and applications means they require strong test capabilities with a high performance across all devices that exceeds user expectations. Each IoT device has its own hardware and software as well as different versions of firmware and operating systems. Due to the sheer number of options, it may not be possible to test all popular combinations of hardware and software. Thus, the tester must ask the client which devices and software versions they’re using and determine the most popular combinations.

Connectivity is a significant issue when you consider the communication requirements of IoT devices. The network infrastructure can cause problems due to overburdened Wi-Fi channels, poor network hardware, and an unreliable Internet connection. This requires that IoT devices and applications are tested across these different conditions to reduce the risk of data loss when offline. 

Then, of course, there's the persistent elephant in the room whenever you talk of IoT, security. A white paper on IoT testing released this year by Qualitest highlights the need for companies to include multichannel and behavior driven testing models and approaches as well as focused platform migration testing as part of their security testing. It's particularly crucial to have strong links with a test automation framework, connected world test strategies, end user performance analysis, and competition benchmarking capabilities.

However, nothing can be achieved if companies don't invest in IoT testing. According to research in CapGemini, HPE and Sogeti’s World Quality Report 2016-17, which surveyed 1600 CIOs across 32 countries, businesses are now spending on average  31% of their IT budget on QA and Testing, compared with 18% in 2012. 85% of participants said IoT products are part of their business operations and with it come a whole new world of connectivity and functionality that must be validated and verified. However, two-thirds of these organizations did not have a testing strategy specific to IoT.

Functionality Testing Is Key

A white paper on IoT testing released this year by Qualitest, reports that testers need to focus on the disruptive nature of digital technologies by paying close attention to customer experience-based testing. The key to successfully executing this new approach is to look for service offerings that feature an integrated test delivery platform and encompass omnichannel test automation frameworks, mobile testing strategies, and crowd testing. According to Kruustük, the biggest customer need is functionality testing, noting that both testers and clients pay attention to customer reviews:

"These days customers really try out the products. And if they don’t like something they will give a negative rating online or in an app store or Facebook and show their frustration against that product so you have to really react to that feedback and maintain the quality that your users really want."   

In regard to future tech, Kruustük believes that  machine learning and IA are going to be the next big thing in IoT and mobile testing: 

"The IoT and mobile space are very competitive. You need to keep up with your competitors while building new features and fixing your issues. I’ve seen a trend over the past couple of years in the building of IA embedded products that help us analyze how the end users use connected products so we can make better decisions on what we need for the QA team and what the testers should really focus on. "

As connected devices continue their onslaught into the mainstream, the role of the tester has never been more important. With many IoT-connected products having lifespans that may exceed a decade (such as white goods), we can expect the role of the tester to become more important and more complex. Today's and tomorrow's customers need the software that accompanies their connected devices to be only interoperable and secure but also adaptable to future updates and iterations, some of which will only become apparent with the passing of time. Something tells me that the team at Testlio is ready for the challenge.