User-Driven Testing Is the New Normal

As Capgemini recently stated in its new World Quality Report for 2015-2016, "consumer usage patterns must drive test scenarios." We agree. The transition to digital is allowing organizations to better engage with their users from any digital screen. 

Many test professionals are already aware of this gap and are acting on it. But challenges remain in embracing full user-centric testing, including: 

1) Determining which conditions to test for and which traits represent an organization's most important users. 

2) How to create these conditions in a controlled, repeatable and consistent manner. Do I need to get my IT and operations teams involved? 

3) Collecting sufficient data to point to the root cause of failure and correcting it. 

4) Spotting points of failure early in the test cycle so I can react and correct issues quickly before heading to production. 

As we try to address these challenges, and encourage businesses to improve their application's readiness for the real world, we've adopted the mantra "Bring your users to the lab." 

This new approach to include real user conditions in your test coverage is called Wind Tunnel™. It's a new capability built on top of our existing cloud-based Continuous Quality Lab. 

So what does Wind Tunnel™ actually do? 

To start, Wind Tunnel™ offers a set of user profiles that assist in defining what real user conditions need to be emulated during testing. Our objective was to align the testing methods to the actual user for whom you're creating the application. 

For instance, a hotel app may primarily target the traveling sales person. This sales person can be described by several demographic attributes. For example, she uses the latest iPhone and iPad devices and the latest OS version. She's hopping onto flights, using on-board Wi-Fi and then going on cellular. She is rapidly changing cellular connections, and has many social and productivity apps running in the background. And she'll have a lot of interruptions in the form of calls, texts and Twitter alerts. 

Below is an example of a "corporate road warrior" user profile, named Georgia.Perfecto will offer a number of preset user-profiles like this to choose from, and will also offer the ability to customize user profiles to your needs. 

User Profile Infographic: Are You Testing For All Kinds of People? 

Here is a code snippet to show how easy it is to add a user profile to your script: 

capabilities.setCapability("windTunnelPersona", "Persona Name"); 
For example: capabilities.setCapability("windTunnelPersona", "Georgia"); 

The user profile is defined by the following settings during the launch of the test:

• Location (through setting address or Latitude/Longtitude)
• Any network (3G, 4G – Poor, Good, etc.)
• Any background app name(s)
• Screen orientation
• Any device and OS (New, Legacy)

Injecting Wind Tunnel™ into functional testing

The above capabilities come on top of existing automation frameworks -- such as Selenium in the cloud -- which allows customers to achieve user-driven functional test automation.The goal is to find failures in the test that only reveal themselves when including the user profile in the test.Once the user profile is defined, Wind Tunnel™ will set up the environment in which the test will run. The device will be injected with the simulated location and the app network traffic will go through a simulation of real-world conditions. These conditions can easily be applied to a Cloud-SaaS managed environment. Perfecto enhanced its cloud offering to allow these real world aspects to be present in parallel to your test. This way, your user profile test can run anytime -- it's repeatable and consistent. 

Analyzing User-driven Testing With Wind Tunnel™

The main objective with Wind Tunnel is to discover points of failure or degraded user experience and allow the DevTesters to quickly troubleshoot the cause. Given the complex and fragile nature of the app on the device, we want to provide as many data points for what happened on the device, inside and around the app. The Wind Tunnel™ collects and reports UI assets (such as screenshots and video), and measures transaction timers based on the actual device UI and allows actionable validation of the test run. 

In addition, a Wind Tunnel™ report is designed to quickly capture the attention of the tester when analyzing test results. 

In the below example, a timer of 5.2 seconds crossed the threshold for a specific user profile. But data analysis shows that this is the result of a spike in the application CPU caused by a high-definition video playback.As organizations ready themselves for the new age of digital transformation, meeting user expectations will be a mandatory requirement for success. The Wind Tunnel™ is the first of its kind to test for real user conditions and expose real glitches before they can reach users.