Top Tips for DevOps Friendly XCTest in the iOS Pipeline
Top Tips for DevOps Friendly XCTest in the iOS Pipeline
These ten best practices for continuous integration for iOS apps will help your DevOps processes run smoother and more effectively.
Join the DZone community and get the full member experience.Join For Free
Open source vulnerabilities are on the rise. Read here how to tackle them effectively.
Apple introduced the XCTest framework to create unit, integration, performance, and UI tests for various Apple platforms. XCUITest is a sleek extension on top of XCTest, which allows developers to write UI tests for iOS and macOS apps. Since launching at WWDC 2015, XCUITest got a lot of attention and enhancements. It uses Swift to create tests, which makes it easy for iOS developers to add UI tests for iOS applications using the same programming language. On the other hand, DevOps and continuous delivery practices entice businesses to release new features as soon as they are developed. Unit and integration tests are lightweight and faster. However, UI tests are very slow, brittle, and unmaintainable. Many mobile testing pyramids show that we should do minimum UI tests and cover the business logic with lower-level lightweight tests. However, UI tests are irreplaceable and give much more value at a higher level if done smartly. A single UI test can beat thousands of unit tests in terms of confidence of releasing the app. In order to mix DevOps and XCTests practices, we have to make some smart decisions on how to write XCTests without disturbing CI/CD pipelines. In this post, we will see some techniques for DevOps-friendly XCTests.
iOS CI/CD Pipelines
In order to enable frequent releases of iOS apps, it's super important to set up and maintain CI/CD pipelines. One of the more painful things in iOS development is to release the app. It involves many activities that can be error-prone, boring, repetitive, and time-consuming. The role of CI/CD pipelines is to automate all the boring tasks. Typical iOS release pipelines consist of
- Checking out the source code
- Static analysis of the Swift code (e.g. SwiftLint).
- Compiling and building the iOS app.
- Running different kinds of tests, e.g. unit, integration, API, UI, performance, and security.
- Code signing and archiving the iOS app.
- Distributing the iOS app to iTunes Connect or third-party testing services.
- Creating tags or releasing on source Control, e.g. GitHub releases.
- Last but not least, notifying people about the new build with release notes.
There might be more things in the pipeline, but some of the very common things are listed above. It's obvious that CI/CD pipelines won't be complete without automated tests.
Top Ten Tips for DevOps Friendly XCTests
Automated tests provide value and confidence to release an iOS app without fear. Now, we will see how to write XCTests smartly to speed up pipelines without compromising the quality of your iOS app.
1. More Unit and Integration Tests
It's always a good idea to cover the production code with unit tests so that continuous integration can detect bugs earlier in the lifecycle. Unit tests are usually lightning-fast and trivial to write. With XCTest, we can cover most of the code by writing unit tests. In some cases, we have to write integration tests where unit testing is not enough to cover the business logic. The benefit of having lightweight and faster tests is that they produce faster builds, which provide early feedback. Apple has great documentation on writing unit tests with the XCTest framework here.
2. Contract Tests for Backend APIs
Its very common practice for iOS apps to use backend or server-side APIs. It's very important to ensure that any APIs used by iOS apps are following the consumer-driven contracts, otherwise there is a chance that they can break the app completely. This necessitates contract testing, which ensures communication between the API provider and client without brittle integration tests. With the XCTest framework, we can draw the contracts and automate network tests. We can also use Pact testing tools in Swift to perform contract tests. There is a library on GitHub for contract testing with Pact using Swift here.
3. API Performance Tests
The speed of an iOS app is key to success in the current competitive market. XCTest has measure blocks which can be applied to any code inside the test method to check the performance of the code. However, backend APIs should be performance/load tested separately by the teams responsible for building those APIs. You can read this article for a detailed guide on performance testing with XCTest. Adding performance tests for the iOS release pipeline makes it more stable and we can detect performance issues earlier before we upload the app to iTunes Connect. However, performance tests are very, very slow and can be only run when needed, e.g. before release or as part of the nightly builds.
4. Accessibility Identifiers for UI Tests
Apple has invested heavily in accessibility and it has been baked into the UIKit framework. Accessibility makes iOS apps usable for everyone, including people with disabilities; it also helps in UI tests using XCTest. You can read about adding accessibility for UI tests here. Adding accessibility to UI elements makes UI tests much faster, which reduces the overall build time.
5. Limit UI Tests
XCTest can be extended to user interface tests, a.k.a XCUITests. We can add as many UI tests as we want, but we should keep in mind that UI tests are slow, brittle, and time-consuming. We should try to test most of the business logic at the integration and API level and keep critical user journeys for UI tests. It's possible to run XCUITests in headless mode on any macOS server machine, but still, they are slow and need to be kept to a minimum.
6. Smart Use of Devices Tests
When the XCTest framework is used for testing iOS apps, it launches simulators to execute tests. However, since Xcode 9, XCTest and XCUITests can run in headless mode when run from the command line. Any macOS server can run XCTests, but it's not enough to run tests inside a simulator — we need them to run on real devices, as well. Before releasing an iOS app, we should get feedback from real devices. On a CI server, you can attach some devices if you have self-hosted CI, or you can use cloud testing services like SauceLabs, Perfecto, or AWS Device Farm.
7. Avoid Third-Party Test Tools
Apple's XCTest tool or other developer tools are enough to perform the majority of quality assurance checks. There are are some other third-party Swift libraries like Quick for doing spec-level unit testing, which adds unnecessary dependencies in the iOS app. There are also third-party UI testing tools like Appium or Calabash which allow you to write tests in other languages like Java, Ruby, etc. These tools are more dangerous to build collaboration between developers and QA engineers and are hard to plug into CI/CD pipelines as it requires unnecessary technologies to be installed and additional brittle configuration to be done on the CI server. Using native tools like XCTest makes the CI/CD pipeline much easier to configure and maintain. If your project is native (Swift), keep it native — don't introduce any dependencies unless absolutely needed. Using irrelevant technologies increases build time and complexity of the projects — introducing Java or Ruby inside iOS projects usually doesn't work out well.
8. Separate Build Configuration
An iOS app has Debug and Release build configurations by default. It's always a good idea to create another build configuration for testing purposes, which we can call "Test" or something like that. With that flexibility, we free up the debug configuration for internal user settings. We can still add some test-related configuration for the "Test" configuration. This also reduces the risk of leaking test code to the production app. Using different build configurations allows us to add test specific code, which makes test execution much faster and covers most of the business logic with tests.
9. Don't Build Twice
Compiling and building an iOS app takes lot of time. Some teams use different schemes to build an app for unit and UI testing. This adds unnecessary build time in the process of continuous integration. It's a good idea to use xcodebuild features like
test-without-building. We can also use the
xctestrun file for a distributed CI build machine in order to save build time.
10. Use the Right CI Servers
In iOS development, it's very important to use the right continuous integration server to run all our XCTests that we wrote for our iOS app. It could be a self-hosted or cloud-based CI system. There are some reasons mentioned here that CI can fail completely if not used or selected properly.
Fortunately, the XCTest framework is a multi-purpose tool and it can be used for unit, UI, integration, API, and performance testing. This reduces the burden of adding third-party tools to cover different types of testing. In short, XCTest is here to rule them all, making iOS testing more and more DevOps friendly.
Published at DZone with permission of Shashikant Jagtap , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.