Continuous Performance Testing of an iOS App Using XCTest

the speed of an ios app is a key to success in the current competitive market. users hate slow-running and unresponsive apps so it has become very important to check the performance of an ios app before it goes to the end users.

apple announced performance testing support for the xctest framework at wwdc 2014. xctest frameworks can measure the performance of the block of code inside the unit or ui tests. in this post, we will see how to perform automated performance testing using the xctest framework.

historical performance testing in ios

apple has various developer tools to check the performance of ios apps. the most common are instruments, leaks, profiler, and allocations. these tools can help to detect performance issues, memory issues, and other bottlenecks. traditionally, an engineer has to babysit and perform all these checks manually over a period of time. this approach is not acceptable in the modern ios development as a small code change can make an app unresponsive, unstable, and unusable. we can't manually check performance issues all the time. there has to be an automated way to check the performance of an app on regular basis. fortunately, there are a few ways we can achieve automated performance testing of ios apps:

• running an instruments check on a periodic basis.
• automated performance tests using the xctest framework.

in this post, we will cover automated performance tests using the xctest framework as instrument checks on a regular basis aren't appropriate, as that still involves repeatable tasks.

performance testing with xctest

the xctest framework has measure blocks which can be applied to any code inside the test method to check the performance of the code.

• the measure block executes the code multiple times depending on the number of invocations set. the default value is ten, which means the block will run ten times.
• at the end of the execution, it calculates the average value that can be used as a baseline for future test execution. we can also set the baseline value manually.
• the performance test will fail if an average time goes higher than the baseline.

applying performace tests at the ui level

selecting the candidates for performance testing is the decision of the team, but as per the xctest documentation, performance tests can be applied to any xctest regardless the level of the test. it means we can apply measure blocks to unit test as well as ui tests. as per most online articles, performance testing is only applied to unit tests. we can also apply performance tests to the xcuitest so that it can communicate to all the application endpoints. this will also give us the opportunity to explore which service/api is responding faster or slower. applying performance tests to ui tests has its own pros and cons. xcuitests will cover different areas, but the performance test might become incredibly slow. if we apply performance tests at the unit level, we can get quick results. when and where to apply performance tests is outside the scope of this article, but for this demo, we will apply it for xcuitests.

performance tests with measure blocks

in order to see how it works in practice, let's create a demo app, xcuitest-performance, with the ui test target. just add a button to the main storyboard with the accessibility identifier "press" without any ibaction. at this stage, we will have a demo app with a button. in the template xcuitest, create a test for the performance test which has the measure block, and inside the block, we can launch the app and press the button. the typical test looks like this:

func testperformanceofapp() {
    self.measure {
     xcuiapplication().launch()
     xcuiapplication().buttons["press"].tap()
}

we can run the test from xcode and see that the test will run the code inside the block ten times. when the test runs the first time, it will fail and prompt the developer to set the appropriate baseline value for future test execution. the test will pass or fail according to the result of the average value and baseline value.

watch this in the gif below:

now that we have set the baseline value for our future performance tests, the performance tests will run as our normal unit tests all the time but the test will fail if the average value increases above the baseline. it means we can automatically test the performance of our code along with unit test execution.

granular performance tests with measure metrics

in the above test, we have calculated the average time from the launch of the app until a user presses the button. however, we can go granular and set the point from where we have to start and stop the measurement of performance. this means we can apply performance tests to the specific area of the test code within the unit test by using the start/stop method. we can achieve this using the measure metric method.

let's apply performance tests only when the button is pressed. we can launch an app in the normal way and then apply the measure metric method to start and stop measuring the performance. we can add the test like this:

func testperformancemeasuremetrics() {
   xcuiapplication().launch()
   self.measuremetrics([xctperformancemetric.wallclocktime], automaticallystartmeasuring: false) {
      startmeasuring()
      xcuiapplication().buttons["press"].tap()
      stopmeasuring()

   }
}

now, if we run this test, we can see that the application is launched only once, but we can calculate the performance by multiple invocations.

the measure metrics method is a great way to optionally set up start and end points for performance measurement during the test execution.

result analysis of performance tests

the results of the performance tests are represented graphically in xcode. the typical performance test result will look like this:

the dialogue reports average, baseline, and max stddev values. in the resulting dialogue, we can see that the performance test is 0.779% worse than the previous execution. those results can be also seen in the console output if we are running the performance tests from a continuous integration server.

pros and cons of xctest as a performance test tool

xctest is a unit and ui level functional testing tool, however, it is extended to do performance testing of ios modules. there are some pros and cons of using xctest as performance testing tool.

pros

there are certain benefits to developers for using xctest for performance testing:

• the xctest framework is a core apple framework, so there is no need to add third party dependencies for performance testing.
• the usual unit tests written using xctest can be easily extended to performance tests by adding measure blocks around the code snippets. no need to learn an additional performance test tools like gatling or jmeter , etc.
• it's easy to write performance tests using swift. no need to learn additional languages like scala or java.
• easily pluggable to any continuous integration server as they can be executed using xcodebuild or fastlane scan .
• you have the ability to match the baseline using device types, so no need to worry about comparing the results with valid data sets.
• you have the ability to use the xcode ide for performance testing.

cons

along with some benefits, there are certain pitfalls using xctest for performance testing:

• the performance test results generated by xctest aren't helpful for a non-technical person. xctest performance test reports can not be exported to a readable format.
• the performance test results can not be shared or hosted easily, unlike gatling or other performance test tools.
• as the number of performance tests increases, the test suite becomes slower, so the tests have to be run separately from the development workflow.
• xctest can not simulate large number of users to check the performance of an ios app.
• xctest can not alter the hardware device setting like cpu or memory to cover complex behaviors for performance tests.

tips for writing accurate performance tests

in xcode, performance tests aren't treated separately. the xctest framework allows us to run performance tests along with our unit or ui tests. it becomes the programmer's responsibility to make sure that the test itself is setup correctly and working as expected. there are a few things we can keep in mind while setting up performance tests:

know what you are testing

before diving into performance testing, we need to understand what and how we are going to test the specific code for performance. it would be a good idea to identify the level at which performance tests can be executed- it might be unit level, integration level, or ui level. we can execute the test at any level as long as the test uses the xctest framework.

avoid conflicts between invocations

the performance test executes multiple innovations of the same code block. we have to make sure that each invocation is atomic and independent from another without having a high standard deviation. anyway, xcode will report an error if there are any deviations.

control the test data

the predefined test data has a higher chance of producing reliable test results than random data while running a performance test. the random data will produce flaky results, which aren't useful to anyone.

run performance tests separately in a different scheme

as the number of performance tests increases, the time taken by test suites increases significantly. it would be a great idea to configure a separate scheme to run only the performance tests without affecting the normal flow of unit and ui tests.

run performance tests continuously on a continuous integration server

as we have created a separate scheme for performance tests, it would be fairly easy to configure with any continuous integration server. it would be super easy if you were using the xcode server. we can create a separate ci job to run the performance tests on a regular basis, e.g. daily, overnight, etc.

source code for this demo

the source code of the performance test discussed in this article is available on github. you can clone the source code and try to run performance tests by yourself.

github repo: xctest-performance

conclusion

apple has extended the core xctest framework for performance testing, which can help us to detect the performance bottlenecks in our ios apps as early as possible in the development cycle. it would be fairly easy for any ios developer to get started with performance testing without the need to learn a new language or framework. with latest xcode, it becomes even smoother. hope you will try this yourself and plug some performance tests into your ios apps soon!