You have an amazing new mobile application and you can’t wait to release it! You build your code and try to find all the bugs in there. Maybe even use some continuous integration tool to automate the developing process and eliminate human mistakes that can occur during manual testing. However, questions remain: “Can we trust simulator/emulator testing enough? How much testing is necessary to be 100% sure your application runs smoothly without any errors? Do we need to bring real devices into the testing game and is it really necessary to double check on real handsets to ensure the wellbeing of your app?” Let’s dig in into this topic now.
Before we can answer these complex questions, let’s take a closer look at the main strengths and shortcomings of testing on simulators /emulators and how these raise the need for checking an app on real devices.
Testing on Simulators (iOS)
- Fast (simulates only the software)
- Relatively easy to set up
- Follows the behavior of your app
- Doesn’t take into consideration some hardware-related factors
- Apps don’t run exactly as they do on real hardware
- Expensive to set up a fully comprehensive simulator
Testing on Mobile Device Emulators (Android)
- Emulates both software and hardware
- App runs unmodified
- Helps you diagnose unexpected behavior
- Can be connected to an IDE for early testing during development
- Very slow (emulates both hardware and software)
- Doesn’t take into consideration some hardware-related factors
- May not be fully compatible with your app and require patching
- Emulators may support only certain OS versions
Bringing in a Real-world Testing Environment
While testing on emulators/simulators is a reasonable start, one has to remember that users are running apps on real devices, not in a virtual environment. There are bugs that you just can’t find on emulators and simulators, and that’s one of the biggest reasons that people use them in conjunction with real devices. Let’s look at what emulators/simulators are missing.
Hardware: The Physical Device
The vast variety of mobile devices, all with their own specifics, can fool the emulator/ simulator, however testing on physical handsets will alert you to any device specific bugs. One such characteristic that emulators/simulators cannot adequately represent is the memory. Every developer’s goal is to provide users with the best quality UI and amazing effects, which often use a great amount of memory and might cause device crashes. A virtual environment with limited dedicated resources simply does not take into account all processes and functions that might be in play on a low-end system.
Closely linked to memory is device battery. An app’s power consumption cannot be adequately tested in virtual environments. By contrast, real handsets will alert you immediately if the power consumption of a mobile app is crossing a defined threshold. If your “hero sized” app is burning out device battery in an hour, the likelihood of finding any users is close to zero.
Tests on real devices give you a great amount of information about the display regarding to brightness and saturation etc. This allows you to get an accurate idea of the exact color displays in the sunlight or in a dark environment. Particularly within the Android ecosystem, the availability of a myriad of distinct devices from different manufacturers raises a variety of chipset and sensor issues.
Software: Your App + Mobile Device Software = ❤ ?
Integrating your mobile app code into an already pre-installed software might bring up some unforeseen surprises. Does your app adapts with the selected platform and run smoothly? Given that many applications have integral dependencies on other software, application and the workflow just can’t be tested via emulators/simulators. Last but not least are OEM + carrier customizations. For customer satisfaction, it is crucial that an app works well not only on pristine versions of an operating system but also on devices that will actually be in users’ hands.
Usability: Experience the Application
The biggest advantage of real devices is that they allow you to use your app rather than simply test it. After hours of clicking and typing your code you can actually see and feel how your app works on a particular device with pre-installed software. For example, you have a nice app up and running when you receive a device-specific notifications, a call, an SMS or both. What happens next and how does the app react to these multiple actions? Emulators/simulators are just not equipped to alert you to the errors that might occur during phone interruptions like notifications, app switching, or changing of the screen layout.
Testing in a real environment gives you an opportunity to walk through real time scenarios and evaluate the overall user experience. You can test how convenient and eye friendly the landing page, color and resolution of the screen are; whether the picture is bright and visible under both day and night conditions; how the app behaves while walking down the street or under poor network conditions, and performance lags or in response to human touch. Pinching, zooming and scrolling create a completely different experience that just can’t be captured via a mouse + keyboard.
What This Means for Developers
To sum up, virtual environments can save you lots of time and money by expanding your testing coverage to multiple devices. (Read how you can set up and benefit the most from testing on an Android emulator or an iOS simulator.) However, there are some bugs that can’t be caught with simulator/emulator testing and require tests on a physical device. Of course, not everyone can enjoy access to an in-house device lab such as the one Facebook recently revealed. If you are a young company or are just price and time sensitive but still care about the quality of your apps, the answer for you could be a cloud based device lab like Amazon Web Services (AWS) Device Farm or Testdroid. With these hosted solutions, you can run tests and interact with a large selection of physical devices without actually having to invest in their ownership or setup. Only by integrating both virtual and real world testing can you get definitive results that will let you maintain high standards of quality and avoid the costly scenario of releasing a business-critical application with defects. Until your company is ready to build an in-house device lab, we strongly recommend you automate your testing process and set up testing in AWS Device Farm or Testdroid.