Adding Sauce to Xcode UI Test: XCUITest + Sauce Labs

Sauce Labs has just announced support for running XCUITests on their own device farm on their official blog. As Apple's own Xcode UI Testing framework gets popular, cloud testing service vendors like Sauce Labs, BrowserStack, and Perfecto cannot ignore this fact. In the previous post, we saw the various cloud-based device testing options for XCUITests. Sauce Labs has been helping the community by supporting and maintaining Appium for a long time. Since native mobile test automation frameworks like XCUITest and Expresso started getting popular, they have had to switch gears and support them as well. Apple launched the Xcode UI Testing framework in 2015, and finally, Sauce Labs supports execution of those UI tests on their real device cloud. Sauce Labs has broadcasted a webinar on XCUITest and Real Device Cloud to demonstrate how we can set up XCUITest on their cloud devices. Unfortunately, I couldn't attend the live webinar to ask questions, but I watched the entire video later. In this post, I will share my thoughts on this process and probably some questions for Sauce Labs to answer.

The Process

After watching the entire webinar on XCUITest and Real Device Cloud , I understood the process of running XCUITests on their Real Device Cloud. The webinar is available to watch on Sauce Labs's Youtube channel, or simply watch it below:

I will leave it for you to watch the webinar's live demonstration and understand the process of executing XCUITests on Sauce Labs's real device cloud.

In a summary, the process involves the following steps:

• Write XCUITests for iOS apps using Swift or Objective-C.
• Prepare the two .ipa files, one for your main app and other for the XCUITest runner app.
• Upload those files using Sauce Labs's web interface so that Sauce Labs can see those apps
• Select the devices that you want to run the tests on from the web interface using device identifiers.
• Download the XCUITest Runner JAVA application runner.jar, which runs the tests by passing your credentials.
• A Java app takes your credentials, the path to your both IPA files, devices, and datacenter as the parameter and triggers the tests on specified devices.
• You can then see test results, screenshots, and videos of the tests run in the device cloud.

You can see all these working in the webinar shown above, so it's worth checking it out.

Benefits***

There are certainly some clear benefits of running the XCUITest on the real devices. As iOS simulators don't involve any hardware interfaces. Some of the benefits are

• Feedback from the real device is valuable as these are the devices that real users use.
• There is the possibility of selecting the most used devices and run the tests in parallel on them
• Get access to newly launched devices within a week without the need to buy.
• We can trigger the tests from any Linux or macOS Continuous Integration server which has .ipa assets located.
• Sharing Test Results seems very easy.

Sauce Labs unlike other vendors like Perfecto, AWS Device Farm, Bitbar did the excellent job to get things working, however, there are few questions always there when companies do excessive UI testing. It doesn't always require if an iOS app is architected properly.

Questions

All of us aware of the fact that, UI testing is very slow, brittle, expensive and unmaintainable thing. Many testing pyramids are shown that we should keep UI test count very low. In the iOS automated app testing, there is rarely anything that we can find running tests on devices Vs simulators as long it doesn't involve hardware or pixel perfect tests. Anyway, automation against real devices vs simulators is the completely different topic. I got some questions about cloud device testing in general

CI/CD and DevOps

Its fairly common question that where does real device UI testing fits in the DevOps or CI/CD pipeline. The mobile DevOps and CI/CD has unique challenges and uses different tools than web applications. Especially for iOS CI/CD, we have to use macOS servers either in-house or in the Cloud. Most of the companies outsource the Continuous Integration to some of the widely used vendors like TravisCI, CircleCI, BuddyBuild. Another option is to manage macOS servers in-house which is very challenging. The cloud device testing services always need to connect either from another cloud or companies in-house infrastructure. This process of making connections add another set of challenges for managing infrastructure and makes entire pipeline slow. The teams having more UI tests makes pipelines ever slower and brittle.

In case of the Sauce Labs approach, we have to prepare two IPA files sounds too lengthy. It takes a lot of time to archive an iOS app and prepare IPA files. It involves code signing and compiling all the iOS app assets if we want to run those on Sauce Labs it would be the time-consuming and frustrating process. It will also require a lot of configuration and scripting to make the connection to service to and fro.

Parallelizing Devices

Apple's xcodebuild tool has made parallel testing much easier since Xcode 9. We just have to pass different destination parameters and XCUITest will in the parallel on those destinations.

Sauce Labs has plenty of iOS devices, however, the way XCUITest runner Java app works id we can only pass one device at a time. It would be interesting to know if we can pass an array of devices so that executing the command once triggers tests on multiple devices.

Subset of Tests and Xcode Schemes

One of the question from webinar audience was to if its possible to run the subset of XCUITests on device cloud. The answer was yes but we have to create another test bundle and IPA file which has those selected tests. XCUITests works on the basis of application targets and schemes. In order to create the subset, we have to create the separate scheme and select tests for that scheme. It means this requires another test bundle and another IPA file. However, it's not possible to dynamically pick the tests to run on Sauce Labs.

Mock Data with Private Network

XCUITest has the ability to use mocked data hosted on the private network. It's would be very tricky to give access to companies private network to third-party services which involves several approvals. Its worth knowing how the Sauce Labs will handle this scenario using Sauce Connect.

Code Signing

In order to deploy the iOS app into devices, it has to be code signed using valid Apple developer certificate. We have to code sign iOS app first to prepare an IPA to upload to Sauce Labs. I assume that Sauce Labs then tear down all your signing and re-sign app to deploy on the devices to execute XCUITests. The code signing process doesn't really matter for the testing but its worth noting that code signing identities used for the testing will be different than original code signing used to distribute the app to App Store.

Java in iOS

Sauce Labs has introduced XCUITest runner Java application to trigger XCUITest, However, this language has nothing to do in iOS app development. There are few things to note, Java is not a scripting language and not pre-installed on any macOS server or Linux server which usually used for Continuous Integration. I don't know why Java has been chosen by Sauce Labs to build Test runner app. It can be easily done using bash or similar scripting languages.

Security

In order to execute XCUITests on Sauce Labs, we have to prepare an executable binary file of our application. This file has all the blood, sweat and tears of iOS developers who were working on it for many months. It is basically the app ready to submit to App Store. It's very important to know the security of the iOS app so that your IPA file won't be leaked or misused. Sauce Labs should add some security or encryption mechanism so that users will feel safe uploading IPA to the Sauce Labs portal.

Simulator

As mentioned in the webinar, Sauce Labs are looking to add support for the Simulator in future. With Xcode 9, XCUITests can run headless inside any macOS server when run using the command line. There is hardly need to launch simulators. Just wondering what would be the benefits of using Simulators in the cloud when most of the cloud-based CI services like TravisCI, CircleCI offers it inbuilt.

BDD

While answering one of the questions about BDD style support, its said that BDD style testing is not supported by XCUITest, however, there are many tools available in the market which can achieve this. You can read more about BDD framework for Swift article here.

Conclusion

Sauce Labs has done a great job by providing the support for XCUITests in the real device lab. It clearly shows that XCUITest framework is becoming very popular these days and cloud testing companies like Sauce Labs started supporting it. You can take advantages of Sauce Labs devices by configuring XCUITest to run on them and see whether it fits your requirement. It would be great if Sauce Labs can answer some of my question mentioned above.