Just after Apple announced Xcode Sever a.k.a. XCS inbuilt with Xcode 9, I wrote a detailed post on how to setup Xcode Server for iOS continuous integration with Xcode 9. The post also has example project to demonstrate new features of Xcode Server. The post mentioned some of the key features including inbuilt server, headless and parallel testing, automatic code signing, and device provisioning. The post is originally published on my personal blog and reposted on Medium and DZone as well. However, sometime after, I started receiving comments about the limitations of Xcode Server to achieve continuous delivery of iOS apps. I also experienced some issues while using Xcode Server for CI. Some of them are known and some of them were unknown. I thought it was worth sharing; in this post, we will discuss the top limitations of Xcode Server.
Current Limitations of Xcode Server
There is a list of various issues related to Xcode Server here but we will cover the top 10 issues with Xcode 9 and Xcode Server. Note that all these issues are at the time of writing this post.
1. Pull Request Testing
Currently, there is no way to test GitHub pull requests. Many CI servers, like TravisCI, can build the iOS project as soon as pull requests are created so that developers can get early feedback. However, with Xcode Server, this isn't possible at the moment. I was under the impression that Xcode Server isn't built just for GitHub, but looking at the fact that Xcode 9 has very tight integration with GitHub, Apple should consider this problem as well. This is a badly missed feature in Xcode Server. This product won't be complete without this feature.
Previously, there were some tools to provide a workaround. Tools like Buildasaur or XBot Builder can be used to watch GitHub repositories for pull requests. As soon as the pull request is created, these tools can be used to create Xcode Bots and update GitHub with results. However, things have been changed a lot. Those tools no longer work with the latest version of Xcode Server.
Another workaround would be to create a GitHub webhook to watch pull request, create Xcode Bots using Xcode Server API and update GitHub pull request with the result of integration. This involves heavy scripting. The most frustrating part is users can't create Xcode Bots with Xcode Server APIs at the moment which we will discuss later. So currently there is no workaround to test GitHub pull requests
2. Bot Creation With Xcode Server API
Currently, Xcode Server is throwing an internal server error while creating bots using a POST request to the Xcode Server API. I, as well as some other users, are also facing the same issue. If that's the genuine issue, then I would say this is the most critical issue. The use of the Xcode Server API becomes pointless when we cannot create bots using POST requests. This also means we can't test pull requests using custom scripts.
One of the major problems of using Xcode Server at the enterprise level is its scalability and distribution. Xcode Server seems to be a good option for a small team with a dedicated single macOS server or a few servers. However, managing multiple macOS servers with Xcode Server would be a bit tricky. Xcode Server has a few limitations:
- Xcode Bots cannot be assigned to the server dynamically.
- Managing multiple macOS servers requires configuration management; the manual setup would be a nightmare.
- With Xcode Server, we can not utilize idle servers and share loads from busy servers.
If Xcode Server can give us the flexibility of dynamically assigning servers for Bots then it would be the very scalable approach.
4. Lack of Support to Upload IPA to iTunes Connect
At the moment, when the archive option is selected for an Xcode Bot, Xcode Server creates an archive and IPA file that can be installed on the device or uploaded to iTunes Connect. However, the process stops there. There is no inbuilt option to upload an IPA to iTunes Connect for TestFlight testing. There seems to be an option that allows us to submit an archive to the App Store, but we are missing a TestFlight step there.
As a workaround, we have to download the IPA file and manually upload it to iTunes Connect using Application Loader, local Xcode, or the agvtool command line utility. After that, the build will be available for testing on TestFlight.
5. Xcode Bot Backup
Xcode Server doesn't have any clean mechanism to maintain Xcode Bots in case of resetting the CI server. Personally, I would love to tear down and rebuild Mac minis at regular intervals so that we avoid the mess getting accumulated on the server machine. Also, in case of an Xcode upgrade, we need to back up the bots for the newer version. Xcode Server also has an option to reset all the data, but backing up the bots isn't straightforward.
Ideally, we can write a script or use configuration management tools like Ansible or Chef to recreate the entire environment, including bot creation, of a recent configuration. However, this can't be achieved at the moment, as we are blocked by the Bot creation issue. Some people mentioned that we can copy the entire /Library/Developer/XcodeServer directory and paste it into the new version. I'm not entirely sure how clean that approach that would be, as there might be some changes in the database, users, or something else. The related thread is here.
6. Keeping the Last Few Integrations
Xcode Bots, which build, test, and archive an iOS app, eat a lot of disk space. In my experience, every integration takes at least 500MB. This means there is a chance that we will run out of disk space very fast. The related thread is here.
Ideally, there should be a mechanism to keep the last X number of integrations and delete all others, like other CI servers, which do the same thing. They keep the last few builds and pinned/favorite builds and clean up the others. It would be great if Xcode Server had a similar kind of feature.
As a workaround, we have to manually delete the integrations using the Xcode Server API or delete the integration assets on the Xcode Server.
7. Lack of Third-Party Integrations
Xcode Server has the feature of adding pre-integration and post-integration scripts. By using this feature, we can integrate with any possible third-party services like Slack, HipChat, GitHub, Fabric, etc. This approach is very time-consuming and most iOS developers don't have the scripting skills to write those integrations. It involves analyzing the third-party API and writing scripts in another language like Bash, Ruby, or Python. I am not sure about other third-parties, but Apple should definitely consider integration with GitHub.
8. Tagging Builds in GitHub
One of the comments on my blog post stated that Xcode Server lacks the ability to tag builds on GitHub. As Xcode Server doesn't have any third-party integrations, it's not possible to tag builds on GitHub, but it can be easily achieved with a little bit of scripting.
9. Sharing Provisioning Profiles
When we have multiple Xcode Server machines, sometimes it's hard to keep track of the provisioning profiles, and Xcode Server fails to create an archive. Some users are already experiencing problems and are not entirely sure what's missing when they allowed Xcode Server to manage all the certificates and provisioning profiles. Xcode Server can automatically handle the code signing by clicking "Allow Xcode Server to manage my certificates and profiles," but sometime you may get an Archive Failed error. It may be because Xcode Server doesn't have enough details about provisioning profiles needed to code sign iOS app. There is a hack mentioned here where we need to turn off automatic code signing and download all the provisioning profiles using Fastlane sigh in the pre-integration script.
10. Parallel Testing Issue With Multiple Simulators
There were various issues reported on the parallel testing feature of Xcode Server. While selecting multiple simulators for testing on Xcode Server, the build fails but the test passes locally. This tweet shows the error, which says "unknown state for the simulator." I have also faced a similar error and I solved that by sticking to only one simulator.
Although Xcode Server is now a great solution for getting Continuous Integration up and running for iOS apps, there are still some limitations that are painful to Xcode Server users which need to be considered as well. There are workarounds for those limitations, but it would be great if it would be fixed by Apple. What are your experiences? Have you faced any pain from Xcode Server that I have missed in this post? Share in the comments below.