Olympics of the Top Cloud iOS Continuous Integration Servers

Continuous Integration, a.k.a. CI, for iOS apps is one of my favorite topics. Continuous Integration is essential for every iOS project now, as the number of developers is growing in iOS teams. As of now, I have deep experience with self-hosted CI servers like Xcode Server, TeamCity, and Jenkins for iOS projects. However, there is still an array of cloud-based CI services available on the market which have become very popular in recent days. There are a lot of articles and tweets going on for those cloud-based CI services. I am keen to try those services and get some exposure by using them with my favorite personal project, XCFit. In order to do this, I have decided to play an Olympics game involving the best of the cloud-based iOS Continuous Integration services candidates.

Purpose

Most of the services taking part in this competition, I haven’t used before, apart from TravisCI. It will be a good opportunity for me to make myself familiar with those CI services and compare them using my personal project, XCFit. The purpose of this Olympics competition is to share my experiences and first impressions on these cloud-based iOS CI services. Please note that the purpose of this competition is not to recommend a particular service that you should use for your iOS project.

The whole purpose of this competition is to try and share first impressions of these cloud-based iOS CI services.

Disclaimer: Opinions expressed are solely my own and do not express the views or opinions of anyone else in this world. This evaluation is only performed against the XCFit project and it may vary from project to project.

Olympic Players

The players involved in these Olympic games are the most used cloud-based CI servers which support iOS and macOS platforms. The current list has these players:

• BuddyBuild
• Bitris
• TravisCI
• Nevercode
• CircleCI

XCFit: The Game

This Olympic competition will be played against the project XCFit, the library we are going to use to try out the features of CI services mentioned above. This is one of my favorite pet projects, a small library that allows us to write XCUITests in a Behaviour Driven Development, a.k.a. BDD, fashion. This project is an interesting candidate for CI server test, as it consists of

• Swift Framework distributed with CocoaPods and Carthage.
• Swift Package that can be distributed with Swift Package Manager.
• An example app using the XCFit framework built with Carthage.
• Ruby package distributed using RubyGems to download Xcode templates.

As there is a lot involved in this project, it will be interesting to see how all those cloud-based CI servers treat this project to run the builds on respective platforms.

Olympic Events

The Olympic events we will be using for this competition are

• Sign Up & Project Setup
• Default Build Configuration
• Build Execution
• Build Artefacts & Test Reporting
• Real Device Testing
• Build Customization
• Parallelisation & Concurrent Builds
• Build Pipelining
• Infrastructure as Code
• Directory Caching: CocoaPods & Carthage
• Code Signing & Distribution
• Third Party Integrations
• Speed
• Secret Store
• Xcode & Apple Developer Tools Upgrades
• Open-Source Project Support
• Learning Curve
• Documentation & Support
• Clean UI/UX
• Price

The players will be evaluated on the basis of these games. Each of the events will award a medal that will be gold, silver, or bronze. At the end, there will be pros and cons section where we will discuss the outcome of this evaluation. This will give a brief idea about the functionality of the cloud-based iOS CI servers and their features related to iOS development.

20 Olympic Medals

Each of the game mentioned above will award a medal. The medals we are going to offer are Gold, Silver, and Bronze.

• Gold Medal: The player will get a gold medal for the criteria if it is strong to satisfy users and meets requirements clearly as well as exceptionally.

• Silver Medal: The player will get the Silver medal for the specific criteria if it is performed alright and satisfies users up to a certain extent.

• Bronze Medal: The player will get a Bronze medal if it’s met the minimum requirements but doesn’t satisfy users that much. This is the lowest medal that every event will give for performance.

Each player will get 20 medals and the winner will be decided on the basis of the number of Gold medals. There are no good or bad CI services, but the winner will be announced based on how each cloud-based CI server performs against the XCFit project only. There might be different winners for different projects.

5 Magic Medals: At the End

There are 20 different games in the Olympics and each player will get a medal for each game. This means every player will get 20 medals for sure but there will be additional 5 magic medal that will be offered to the player while the evaluation is going on and announced once all players are evaluated. The 5 medals will be based on following criteria

• Social Media support during evaluation
• Correcting my observations during evaluation
• Accepting facts and respecting competitors
• Showing courage to improve services
• User Feedback on Twitter & Public API

BuddyBuild: Player One

In the iOS Continuous Integration Server Olympics competition, the first candidate selected is BuddyBuild, the most heard name on Twitter and other tech blogging sites because of its simplicity and ease of use. BuddyBuild is designed specifically for mobile continuous integration and supports iOS and Android platforms. The clients of BuddyBuild include some big brands like The Washington Post, FireFox, Slack, and Meetup.

Olympic Performance of BuddyBuild

I have done the critical evaluation of BuddyBuild for each of the 20 criteria and the results are shared in the blog post here.

At the end of 20 games, BuddyBuild scored:

13 GOLD

6 SILVER

1 BRONZE

BuddyBuild— XCFit Build Logs

The build logs executed for the XCFit project are available here. We can navigate to details, logs, and test tables to get the more information about the build.

Bitrise: Player Two

In the iOS Continuous Service Olympics competition, the second candidate selected is Bitrise. There are many articles on Medium and a lot of good words on Twitter about Bitrise. Like BuddyBuild, Bitrise is also designed for the mobile continuous integration and supports iOS and Android platforms. Bitrise also has some good clients, like ABB, Foursquare, FOX, and TNT.

Olympic Performance of Bitrise

I have done the critical evaluation of Bitrise for each of the 20 criteria and the results are shared in the blog post here.

At the end of 20 games, Bitrise scored:

13 GOLD

7 SILVER

0 BRONZE

Bitrise — XCFit Build Logs

The build logs executed for the XCFit project are available here but you need to have an account to access those logs.

TravisCI: Player Three

In the iOS Continuous Integration Server Olympics competition, the third candidate selected is TravisCI, the most used cloud-based continuous integration for open-source and private projects. It’s truly the home of open-source testing. As mentioned on the TravisCI page there are about 900K open-source projects and 600K users running the continuous integration on TravisCI. It’s deeply integrated with GitHub and setting up an open-source project on TravisCI is a minute away from developers.

Olympic Performance of TravisCI

I have done the critical evaluation of TravisCI for each of the 20 criteria and the results are shared in the blog post here.

At the end of 20 Games TravisCI Scored :

12 GOLD

5 SILVER

3 BRONZE

TravisCI — XCFit Build Logs

The Travis build logs executed for the XCFit project are available here.

Nevercode: Player Four

In the iOS Continuous Integration Server Olympics competition, the fourth candidate selected is Nevercode. It is designed specifically for the mobile continuous integration which supports iOS and Android platforms. The clients of Nevercode include some big brands like Sainsbury’s, Toyota, and Eurostar.

Olympic Performance of Nevercode

I have done the critical evaluation of Nevercode for each of the 20 criteria and the results are shared in the blog post here.

At the end of 20 games, Nevercode scored:

12 GOLD

5 SILVER

3 BRONZE

Nevercode— XCFit Build Logs

The build logs executed for XCFit project are available here but you need to be logged into the Nevercode portal.

CircleCI: Player Five

In the iOS Continuous Integration Server Olympics competition, the third candidate selected is CircleCI, the continuous integration service that is being used by the most popular tech firms like Facebook, Spotify, Percolate, etc. The CircleCI and TravisCI have most things in common but some features are unique to each platform.

Olympic Performance of CircleCI

I have done the critical evaluation of CircleCI for each of the 20 criteria and the results are shared in the blog post here.

At the end of 20 games CircleCI scored:

12 GOLD

7 SILVER

1 BRONZE

CircleCI — XCFit Build Logs

The CircleCI build logs executed for the XCFit project are available here.

Olympic Medal Table for 20 Games

So far we have done with 20 games and medal table looks like this:

Magic Medal Stage

As of now, we can see that there is a tough fight between Bitrise and BuddyBuild at this moment but other players like TravisCI, CircleCI, and Nevercode are too close. However, the game hasn’t finished yet. We still have 5 magic medals to be awarded to each player according to the following criteria:

• Social Media support during evaluation.
• Correcting my observations during evaluation.
• Accepting facts and respecting competitors.
• Showing courage to improve services.
• User Feedback on Twitter & Public API.

A medal will be awarded for each of the criteria. This means every player will get 25 medals and winners will be the player who won maximum GOLD medals. If there are equal GOLD medal then we will look for the number of SILVER medals.

Magic Medals Awards

Now we need to present magic medals for each of the criteria mentioned above.

BB = BuddyBuild
BR = Bitrise
TC = TravisCI
NC = Nevercode
CC = CircleCI

Social Media Support (SMS)

There was huge support on Social Media during the evaluation process, especially on Twitter. The tweets were liked, replied, and retweeted during the evaluation. The example tweet can be found here. BuddyBuild and Nevercode were on top of Social Media support. CircleCI joined late but there wasn’t any interest shown by Bitrise and TravisCI. The medals for SMS:

Correcting Observations (CO)

There were some features that I hadn't discovered in the first place, but the players corrected me with the right features by communicating with me on Twitter, email, and blog comments. Nevercode, BuddyBuild, and Bitrise contacted me and corrected most of my observations. The medals for correcting observations are:

Accepting Facts and Respecting Competitor

There are some conversations with Twitter and emails on accepting the facts and respecting the competitors work. BuddyBuild and Nevercode are good at it. The medals for the accepting facts are

Showing Courage to Improve Services

Nevercode and BuddyBuild have shown a strong desire to improve their services through email and tweets. The medals for showing courage to improve services are:

User Feedback on Twitter & Public API

I have collected some user feedback on Twitter and public APIs, like the status badge and public logs for the open source projects. BuddyBuild, TravisCI, and CircleCI have both good feedback on Twiter and public logs and APIs. Bitrise and Nevercode lack public logs. Some Twitter users reported issues with the Xcode 9 upgrade for Bitrise, and Nevercode is still failing to build for me with Xcode 9. The medals for user feedback and public APIs are:

Final Medal Table & Winner

Now that we have awarded all the magic medals to all players, it’s time to declare the winner of the competition.

The final medal table:

The winner is...

Congratulations, BuddyBuild!

I would like to thank all the other players: Bitrise, Nevercode, CircleCI, and TravisCI. As you can see from the medal table, it was a tough competition and wasn’t an easy decision to make.

Our Runner-up Is Bitrise

As you can see, CircleCI and Nevercode were very close. Anyways, it was BuddyBuild who was selected as the winner of this Olympics game. Let’s celebrate together for BuddyBuild!

This is the end of Cloud iOS CI Olympics Ceremony 2017! Be ready for next year's Olympics.

Hope you like this post. Please let me know if you think I missed/misunderstood something. I appreciate your feedback.