One of the goals of having a Continuous Integration setup is to find the issues or bugs early in the application development lifecycle. The sooner we detect bugs, the cheaper it is to fix them. The earliest stage that developer can find the issues in the apps is while writing the code or running the code inside the Xcode. At WWDC 2017, Apple announced Xcode 9 with loads of enhancements for debugging and detecting issues at runtimes. Xcode debugger already highlights most of the issues like compiler errors, compiler warnings, and test failures while running the application or tests. However, Apple has introduced another category of tools called Xcode Runtime tools to find issues while running the code, which can be enabled from local Xcode to help engineers to fix issues quickly. However, there are still some lazy engineers in the world who might check-in the source code ignoring these issues. In order to catch those issues, we can enable Xcode runtime tools on CI servers. In this post, we will find out what those tools are and how to activate them on an iOS CI server.
Xcode Runtime Tools
Xcode has four different types of runtime tools also known as code diagnostic tools to find the issues. These tools are
- Address Sanitizer
- Thread Sanitizer
- Main Thread Checker
- Undefined Behaviour Sanitizer
Apple has brief documentation of all these tools on the official documentation page of code diagnostics. For this post, I have created a demo single view iOS app called Code-Diagnostics which has default unit and UI tests. Now, let's have a quick look at each tool and how to enable them locally as well on a CI server.
Address Sanitizer, a.k.a ASan, reports memory related issues like memory corruption and other memory-related security vulnerability issues. You can read more about the Address Sanitizer here. We can enable the address sanitizer from local Xcode by editing the scheme--> Select Action --> Diagnostics Tab and tick Address Sanitizer.
Enable ASan on CI
In order to, enable Asan on Continuous Integration server, we have the parameter that can be passed to xcodebuild command. The parameter is -enableAddressSanitizer YES that can be used for the build, test actions. We can run our tests using the following command on CI server.
$ xcodebuild test -scheme Code-Diagnostics -destination 'platform=iOS Simulator,name=iPhone 6,OS=11.0' -enableAddressSanitizer YES
Thread Sanitizer, a.k.a TSan, detects data races problems. It also detects other thread related issues like thread leaks. There is detailed documentation about Thread Sanitizer on Apple's official documentation here. We can enable the thread sanitizer same way as of address sanitizer but the important things to note is that Address Sanitizer and Thread Sanitizer cannot be turned ON at the same time.
Enabling TSan on a CI Server
In order to enable TSan on a Continuous Integration server, we have a parameter that can be passed to the xcodebuild command. The parameter is -enableThreadSanitizer YES that can be used for the build and test actions. We can run our tests using the following command on a CI server.
$ xcodebuild test -scheme Code-Diagnostics -destination 'platform=iOS Simulator,name=iPhone 6,OS=11.0' -enableThreadSanitizer YES
Undefined Behaviour Sanitizer
Undefined Behaviour Sanitizer, a.k.a UBSan, detects undefined behaviors in the code at runtime- behaviors like dividing by zero, loading memory from a misaligned pointer, or dereferencing a null pointer. You can read more about UBSan here. We can enable UBSan in Xcode same way we have done with ASan, but select the 'Undefined Behaviour Sanitizer' checkbox.
Enabling UBSan on a CI Server
In order to enable UBSan on a Continuous Integration server, we have a parameter that can be passed to the xcodebuild command. The parameter is -enableUndefinedBehaviorSanitizer YES that can be used for the build and test actions. We can run our tests using the following command on a CI server.
$ xcodebuild test -scheme Code-Diagnostics -destination 'platform=iOS Simulator,name=iPhone 6,OS=11.0' -enableUndefinedBehaviorSanitizer YES
Main Thread Checker
The Main Thread Checker is a new tool launched with Xcode 9 which detects the invalid use of Apple's frameworks like UIKit, AppKit, etc that are supposed to be used from main thread but are accidentally used in the background thread. The effect of the invalid usage can result in missed UI updates, visual defects, data corruption, and crashes. You can read more about the Main Thread Checker here.
Enabling Main Thread Checker on a CI Server
The Main Thread Checker is automatically enabled with Xcode debugger so we don't need to explicitly enable Main Thread Checker. However, disabling it on a CI server is unknown at the moment. If anyone knows how to do this, please, let me know.
Set Up Continuous Integration
Now that we have seen how to enable Xcode runtime tools for a CI server, we will set up CI with Travis for our demo app, Code-Diagnostics. We will add configuration file.travis.yml with the following content:
language: objective-c osx_image: xcode9 install: true script: - echo "Enabling Address Sanitizer to detect" - xcodebuild test -scheme Code-Diagnostics -destination 'platform=iOS Simulator,name=iPhone 6,OS=11.0' -enableAddressSanitizer YES - echo "Enabling Thread Sanitizer, Main Thread Checker and undefined behavior sanitizer" - xcodebuild test -scheme Code-Diagnostics -destination 'platform=iOS Simulator,name=iPhone 6,OS=11.0' -enableThreadSanitizer YES -enableUndefinedBehaviorSanitizer YES
On TravisCI, I have enabled the project for building here.
We can see in the Travis log that project is building with Xcode runtime tools.
You can get more information about them on
- WWDC 2017 Talk - Finding Bugs Using Xcode Runtime Tools
- Apple's Official Documentation - Code Diagnostics
Xcode runtime tools are very useful to detect bugs early in the development lifecycle. It would be great practice to use them all either locally or on a CI server to find bugs.