Stabilizing the CI By Re-Running Flaky iOS XCUI Tests

Apple released Xcode UI testing, which enables UI testing of iOS application straight from Xcode without any third-party tools like Appium, Calabash, or KIF.  These tools call themselves mobile testing frameworks but they're actually little more than wrappers to UI Automation or Instruments.  

The release of iOS 10 broke all of these open-source mobile test automation frameworks, as Apple stopped supporting Instruments technology. The only options remained was to use XCTest framework from Apple or wait for those tools to build a wrapper around XCUITest.  XCTest is not a new framework but it has evolved quite well with Xcode releases. You can read more about Pros and Cons of using XCTest for iOS app testing here.

The XCUITest and Continuous Integration Problem

The XCUITest framework is still new and some users already complained it's very fragile, especially when we run against Continuous Integration (CI) server. XCUI tests seem to be passing locally, but those tests are nondeterministic on CI. You can find lots of questions about XCUITest on StackOverflow. As a result of this, the CI pipeline has become unreliable and untrusted.

The teams ignore failing UI tests and everybody loses trust in those UI tests, which can become the reason for Continuous Integration failures. The broken UI tests are always ignored by the development team. However, Continuous Delivery won't allow broken builds to be deployed to production. Now, everyone has to pay attention to broken UI tests.

In order to fix the problems, the team might do the following to ensure your UI tests are as resilient as possible:

• Followed a ports and adapter pattern to cover all business logic at domain layer without writing any UI tests. The best example of this would be writing faster iOS acceptance tests or contract tests with Fitnesse.
• Considered testing UI in isolation using stub backends or mock web server to guard against unexpected data to replace real word UI tests.
• Added XCUI computed retry to find UI elements on the application or loop until the XCUIElement was found.
• Reset simulators on each CI build or each test suite. 
• Added static wait to XCUITests to make tests reliable.

After doing all this effort, perhaps few a tests are still non-deterministic and fail randomly. This is the general behavior of the UI tests: they are fragile, brittle, and hard to maintain, no matter how much effort you put in to fix them or how much you defend them. Peculiarities of the runtime environment can conspire against you.

Now, your release to production is blocked because of the flaky UI tests. What do you do?

Solution

In the scenario mentioned above, we can try one sensible thing. Let's take just the failing XCUITests and try to re-run them. But wait — there is another problem. Apple's command line tool xcodebuild still does not have an easy way to run a single XCTest from the command line. It can take Xcode schemes to test, but it's a bit tricky to pass individual tests to xcodebuild.  

The new version of xcodebuild has an only-testing option, but it's a bit hard to implement with the script. Fortunately, there is a fast lane plugin called fastlane-plugin-setup_fragile_tests_for_rescan to rescue us from the problem of flaky XCUI Tests. The full credit goes to the author of the plugin, lyndsey-ferguson. This plugin does the following things:

• Takes failed tests from generated JUnit reports from the Fastlane scan.
• Modifies specified XCUITest schemes to skip passed tests and failed tests to the scheme.
• Re-runs the scan three times to see the results of the test.

It makes sense to fix the current CI and get going, but ideally, the non-deterministic tests should either be fixed for good or deleted. However, sometimes, the world is just not that perfect, Maybe, for example, you aren’t able to improve underlying infrastructure issues. Hence, this may still be a valid course of action.

Let's See It in Action

Just to demo re-running failed XCUI tests, I have created a demo iOS app called CoinTossing. This application has a "Toss" button and the textField shows the result of the toss — either "heads" or "tails." We also have a couple of XCUI tests to check the results. The probability of a failing test is 50%.

Create XCUITest Scheme and Share it

Let's create a new scheme for XCUITest and call it "CoinTossingUITests." Let's make it "shared" so that the CI can see it and we can pass to the Fastlane scan.

Set Up Fastlane With Plugin

Now, we will set up Fastlane from Gemfile and bundler. We need to have the bundler gem installed.

Add "fastlane" to the Gemfile:

source "https://rubygems.org"
gem "fastlane"

Now, we can install bundle with the following command:

$ bundle install

This will download all the required Fastlane tools. We will create a Fastlane directory manually and add an empty fastfile.

$ mkdire Fastlane
$ touch Fastlane/Fastfile

Now we need to add the Fastlane plugin to the project using the following command:

$ fastlane add_plugin setup_fragile_tests_for_rescan

This will ask permission to change the Gemfile type "y." This will create another file in the plugin file inside the Fastlane directory.

Now, we have set up our plugin and it's ready to use. Let's add the following code to Fastfile.

lane :test do |options|
  scheme_to_test = "CoinTossingUITests"

  scan_options = {
    scheme: scheme_to_test,
    device: 'iPhone 6',
    output_directory: 'artifacts/tests',
    custom_report_file_name: 'report.xml'
  }
  begin
    retry_count ||= 0
    scan(scan_options)
  rescue => e
    UI.important("failure noted: #{e}")
    report_filepath = File.absolute_path('../artifacts/tests/report.xml')
    unless File.exist?(report_filepath)
      raise e
    end
    if (retry_count += 1) < 3
      setup_fragile_tests_for_rescan(
        project_path: File.absolute_path('../CoinTossing.xcodeproj'),
        scheme: scheme_to_test,
        report_filepath: report_filepath
      )
      clear_derived_data
      reset_simulator_contents if retry_count > 1
      retry
    end
  end
end

Note that we are passing the CoinTossingUITests scheme to the Fastlane scan and running the scan three times if the test failed. Now, we should able to run the test lane.

$ bundle exec fastlane test

We should be able to see the tests running, and only failed tests will be run again. This will modify out original CoinTossingUITests scheme.

Set Up CI Server

The Fastlane plugin fastlane-plugin-setup_fragile_tests_for_rescan modifies the scheme and adds only failed tests for re-running. It uses a xcodeproj gem to modify the scheme settings, which is the same library that Cocoapods use to change Xcode settings and files. This plugin is not designed to run locally. The idea is to run them on CI server where we don't care if the scheme is getting modified, as we check out some fresh source code for every build. Let's add a TravisCI config to the project so that we can run it on the CI server.  Now, enable the repository to run on TravisCI and create a .travis.yml file with following code:

os:
  - osx
osx_image: xcode8.2
xcode_sdk: iPhone 7 (10.2)
language: objective-c
before_install:
 - rvm install 2.3.3
 - rvm use 2.3.3
 - gem install bundler
 - bundle install
# cache: cocoapods
script:
 - pwd
 - xcrun simctl list
 - bundle exec fastlane test

Now, we should be able to run those tests on TravisCI. Check out the latest output from TravisCI here.

Try It Yourself

The source code for this demo is available on the GitHub repo CoinTossing. In order to quickly check how the plugin works, clone the project and try running the test. Assuming you have a Ruby environment setup,  you can run the following commands to test this:

$ git clone https://github.com/Shashikant86/CoinTossing
$ cd CoinTossing
$ gem install bundler 
$ bundle install 
$ bundle exec fastlane test

Now, you can see the tests start running in the simulator. Observe the test output. If one the test fails, it should see that plugin will re-run that test a couple more times, as shown in the above GIF.

Things to Remember

We should be able to re-run failed XCUI tests in order to make the CI happy and carry on. However, there are a couple of things we should remember before we do this.

• Do not run tests with the Fastlane plugin locally. This plugin modifies the scheme and we have to make sure we revert the changes made by the plugin. This plugin should only be used on the CI server.
• The flaky and non-deterministic test should either be fixed for good or deleted.
• Keep the retry scan count at three. If the test fails after a couple of times, then there is a real issue with the test.
• Currently, we are clearing simulators if the retry count is more than one. It's up to you if really want to clear the simulators.

Conclusion

Apple's new Xcode UI testing framework might be a little flaky on the CI server, but we can re-run the flaky tests using the Fastlane plugin to make the XCUITests more robust and reliable. Hopefully, this will help reduce the pain caused by flaky XCUI tests on the CI server.