Shifting From Manual Mobile Testing to Automation

Below is an exclusive look at the 6th chapter of my new book, The Digital Quality Handbook, brought to you in partnership with Perfecto:

Mobile testing in general and mobile test automation is not easy or simple to get to and start with. It is even more complex to sustain, grow, and evolve a testing solution so that it would be executed in unattended mode, as part of CI workflows, for example. The key to this kind of transformation is starting with short steps having a certain positive impact on the way organizations operate today, thus increasing their maturity and readiness to embrace and accept such changes. It is a matter of building up trust resulting in strong management support.

The transition also requires an evolution of the existing skillset to face a new reality. Most of all, the success would be very much dependent on the desire and core understanding of the QA engineers involved as to why these would be steps in the right direction. This means that they need to understand “what’s in” for them.

We all know that running hundreds of manual test cases on real devices or emulators is not a fun task — not at all. On the contrary, it is time-consuming, error-prone, lacks coverage, and what not. Often being performed regularly makes it even more so. This is where mobile test automation helps practitioners to invest their time and efforts in a smarter way.

BDD and Open-Source Catalysts — Setting It All Up

Do you know that there is a new open-source behavior-driven development (BDD) test framework that could ease the pain of shifting from manual (or low skill-set based) test tools to an IDE-based approach? This framework was developed by InfoStretch. It was adapted and tuned to the mobile Cloud by Perfecto.

Think about shifting away from Microsoft Excel/ Word or another manual test management tool towards an IDE, such as Eclipse or IntelliJ. There QA engineers can write human readable test scenarios in English, record application flows and objects in a simplified manner, create and populate their own object repository. In addition, it would take advanced manual testers just a few extra steps to regularly run these BDD (Cucumber) tests from within CI, such as Jenkins.

Let us iterate over the basics and get to know how this would look like, work, and do well for us.

1st Step — Setup of Test Development Environment

• Download the Java Standard Edition (Java SE) Java Development Kit (JDK) from Oracle’s website and install it.
• Download and install Eclipse IDE for Java Developers or IntelliJ IDEA and the following plugins: Cucumber, Maven, TestNG.
• For Cloud testing, download the Perfecto plugin.

2nd Step — Execution Framework Installation

• Clone the GIT repository of Quantum to obtain the framework in a local folder on your computer.
• Modify the file named application.properties located in the resources folder as part of your local copy of the Quantum framework in accordance with the guidelines of the project documentation, i.e:
• Choose type of the tested app (Web, Native).
• Set device properties and/ or environment parameters for the Cloud.

The First Automated Scenario to Ease Up the Transition

After going through these steps, the environment will be ready for test case development and execution. As it could be observed on the architecture diagram in Figure 1, Quantum offers to script in BDD terms or Java/ Appium. One could execute the developed tests using TestNG and Maven. From there on, Jenkins CI is a no-brainer. After a test run is complete, results could be explored either by using the built-in reporting features of the TestNG framework or as part of a custom-tailored test reporting solution.

This is another important point on the diagram in Figure 29. It illustrates the unique reporting capabilities of the Perfecto technology stack integrated within the architecture of a Quantum-based automated test solution.

Figure 29: Quantum - An Example of BDD Open-Source Test Automation Framework Architecture.

Going through the mapping example in Figure 30, you could understand how the identification of UI objects could be parameterized through the use of object locator files (in a format pretty much defined by the framework).

Figure 30: An Example of an Object Locator File.

The sole purpose of having an object locator file is to fully separate the implementation of test scenarios from their configuration which naturally results in automated test scenarios that are easier to maintain. They are also much more resilient to changes introduced by software development units. As part of this concrete example, a set of objects (via key-value pairs) is defined. Whenever the native Android app is under test, UI objects could be looked up and identified by the underlying framework either by XPath expressions or alternatively by their unique Object identifier (ID).

Figure 31: BDD Test Scenario Example for an Android Native Application.

Now, having defined the way relevant test objects are about to be looked up, it is prime time to quickly develop a behavior-driven test scenario, similar to the one in Figure 31. Later on, our BDD scenario will be tagged as “Android-TestApp” in the TestNG configuration file, so that the test could be executed by the framework.

Figure 32: TesNG Sample File Used for Execution of the BDD Test on Physical Devices.

Eventually, TestNG needs to be configured. This is done as part of the TestNG.XML file, as illustrated by the example in Figure 32. Here QA engineers would typically specify the following parameters:

• DUT (Device Under Test) — it could be specified to run the test in parallel on multiple devices.
• Path to the files containing BDD test scenarios to be executed (similar to the one just created).
• Define tags such as the “AndroidTestApp.”

This is what it takes for any QA engineer to walk the baby steps towards mobile test automation. It is just a matter of editing a small number of configuration files with valuable support built into your IDE. As you could see, even a single line of source code is not necessary to get the automation scripts ready for execution. Instead, we managed to elegantly hook into the underlying frameworks and harness their special powers through configuration. Thus, we ended up with a fully functional and robust mobile automation test solution that naturally plugs and plays with the common mobile development environments.

This said, benefitting from this approach shall not be so much of an investment for most mobile development organizations, at least to the point of running a quick proof of concept project to explore the opportunities first hand.

Continuous Integration (CI) and Jenkins as a CI Platform

Could the sample above be easily extended, so that Jenkins would automatically run the produced test solution? This is the next step on the road to CI/ CD, remember? Fortunately, it is just a matter of configuring a local Jenkins server to run the ready-made POM.xml file (bundled with the framework) as part of a build job upon some triggers.

For example, a build job could be run automatically after code changes for the app pushed by software developers to the source control system, thus running also our test solution. Some may need to establish scheduled regular test execution for the sake of automating transparent nightly builds. This is all easily done with Jenkins. Of course, to quickly get to execution results the new job will be manually triggered and will only point to the location of the test solution POM.xml file on the filesystem, similar to the example configuration in Figure 33.

Figure 33: Configure Jenkins CI to Run the POM.XML File of Quantum.

From here on, the newly created build job just needs to be triggered. In Figure 34, the sound execution log of this first automated test solution for mobile apps could be reviewed.

Figure 34: An Example Console Log Within Jenkins CI After Test Execution.

Note that only the sky (or the Cloud?!) could be the limit for evolving such automated test suites.

Jenkins is an incredibly powerful and robust CI platform. Its huge community keeps contributing to the availability of an astonishing number of easily installable plugins that makes it a solid foundation for mobile test automation against every major mobile platform.

The whole domain of mobile test automation could hardly be presented within a single chapter. Many complex questions that you could come up with were probably not considered or mentioned at all. Keep in mind, though, that this is just the beginning and by no means the end of the journey towards robust and scalable mobile test automation. Moreover, it could all start with a practical and easy to follow approach that you could try out and, if satisfied with the results, share as pure knowledge with others.