Comparing 11 Mobile Beta Testing Tools
Comparing 11 Mobile Beta Testing Tools
If you're making a mobile app, you probably want to test it. This overview of 11 testing tools measures their integrations, pricing, use cases, ease of use, and more.
Join the DZone community and get the full member experience.Join For Free
As a new year begins, we hope more mobile dev teams will distribute beta builds before releasing to the public. If you’re reading this article, chances are you’re looking for the best beta distribution tools and services. This article covers tools for both iOS and Android, but focuses on iOS (and TestFlight alternatives) as it’s a pickier platform for beta testing with more complexity.
This article compares each tool regarding multi-platform support, integrations and APIs offered, ease of use, popularity, setup time, miscellaneous pros and cons, and, of course, pricing. I discuss and compare continuous integration and continuous development adoption and integrations, as well as user feedback and usability. I’ll also cover ease of use and setup time, analyzing the steps necessary to get everything set up and running. Additionally, I’ll mention any well-known customers for each service, as well as any publically available metrics when calculating the popularity factor.
Dealing with UDIDs (Unique Device Identifiers) is one of the things iOS developers struggle with when distributing beta builds. These are part of the provisioning profiles needed to release beta versions. You’ll notice that in most tools presented in this article, iOS developers will have to handle the provisioning profiles manually and submit a new build with the newly generated provisioning profile, but some tools can automate this task. Unless stated otherwise, assume that the service does not handle the provisioning profiles for you, so you’ll have to deal with them manually.
I compare a total of 11 tools:
- Crashlytics Beta (Fabric)
Popularity: If you’re into the Apple development world, you have probably heard of TestFlight. There are many opinions for and against the use of TestFlight since Apple acquired it back in early 2014, but during its first 3 years of existence before its acquisition, it had already gained much popularity and adoption.
Platforms: Since its acquisition, Apple stopped supporting Android distributions, and it now allows distribution to Apple devices only – iOS, watchOS, and tvOS.
API & Integrations: Fastlane, more specifically its
pilot lane, will be your best friend when delivering your app to TestFlight’s beta testers. With a single command line, you’re able to get your app from development to your beta testers’ hands.
Ease Of Use: If you’re not using an automation tool like Fastlane in order to automate your workspace, it’s still pretty straightforward to get your app into beta testing. The whole process is done through the iTunes Connect portal, and you can add either Internal or External Testers. Internal testers are “part” of your development team, thus only the ones that are currently assigned as developers or have developer roles in your iTunes Connect team (like admins). The app distribution sent to internal testers are the fastest since they don’t require a Beta App Review nor comply with the App Store Review Guidelines, like the distribution sent to external testers.
On the other hand, to invite external testers, who is anyone that’s not in your development team, you only need their email address and they’re good to go once they’ve accepted the invitation. Only TestFlight has this Internal and External Tester concept.
TestFlight’s advantage is that it doesn’t require the developer to handle the provisioning profiles for new devices manually, so that improves its ease of use considerably.
Setup Time & Effort: The setup time is mostly creating the app in the iTunes Connect and submitting the app via Xcode, which could take up to a few minutes to get your app approved through Apple’s automated app revision system. You need to explicitly declare some information about the test build you’re submitting, like a description and what needs to be tested, so that might take a little longer than if it didn’t require that info.
Misc Pros: Every tester needs to download Apple’s official TestFlight app in the App Store in order to be able to download and test app distributions. Another great feature is that you can assign notes and details to the app being distributed so the testers can see what needs to be tested.
Basic information about crashes and sessions are stored in TestFlight Data so the developers may track what’s going on with each of the versions that are being tested. Extra feedback may be sent to the developers via email (within TestFlight app) alongside a log with information of the current device.
Misc Cons: What bothers the developers the most is the Beta App Review that is required in order to invite external testers (most times the most important testers are outside of our development team). So that, and the fact that it doesn’t support non-Apple platforms, are the reasons why many people complain about TestFlight. Other than that, it’s a great tool, it was already one of the most popular (if not the most) on the market before Apple bought it, and it definitely became even more popular after its acquisition.
Pricing: Apple’s official solution to beta distribution is completely free to use.
Popularity And Misc Pros: More complete than TestFlight, and definitely among the most popular beta distribution services out there, Microsoft’s HockeyApp solution to beta distributions also offers user feedback right from the app, insights of how users are using the app, sophisticated team management features, and is well-suited for large companies that develop many different apps.
Platforms: It also supports a greater range of platforms, from Android, iOS, macOS, and Windows, to Cordova, React Native, Unity, and Xamarin.
Ease Of Use And Setup Time & Effort: The setup is pretty much the same as Fabric, which I’ll be covering later on. There are numerous ways to add the HockeyApp SDK to your project, including CocoaPods, Carthage, and manually. Uploading a new build may be done via the web dashboard or via HockeyApp for Mac, and is really straightforward.
Integrations & APIs: It features a huge range of integrations and bug trackers, and provide handy tutorials for the most popular ones like JIRA, Slack, GitHub, Status Board iPad app, Microsoft’s Visual Studio Team Services, and also webhooks. Additionally, HockeyApp also supports various build servers and tools: Fastlane, Jenkins, Travis CI, and Visual Studio Team Services.
Pricing: All this is available in a free tier for up to two apps, and the other plans can be found here. For those looking to use more than two apps and wouldn’t like to pay more for it, HockeyKit open source might be an alternative.
Platforms: According to the HockeyKit GitHub Wiki, HockeyKit is a framework to easily distribute applications with maximum ease of use for end-user and great flexibility for developers, which currently supports iOS and Android platforms.
Integrations & APIs: It’s an open source framework that preceded HockeyApp. It’s an alternative for those who have their own servers and would like to host their own beta distribution system, but it hasn’t been updated for over 2 years and is provided as is. For further support and features, they recommend using HockeyApp, as it’s the business that proceeded this product.
Ease Of Use & Setup Time & Effort: Since it’s an open source framework that must be fully installed on your own server and is provided as is, it isn’t easy to use at all, or at least takes a certain time to have everything set up. It has a good documentation, though, and since it’s a self-hosted solution, you can integrate pretty much anything you need or want.
Popularity & Pricing: It isn’t as popular as HockeyApp, but then again, it’s completely free to use, which makes it the best solution if your company doesn’t want to rely on 3rd party software to deliver beta builds.
Crashlytics Beta (Fabric)
Ah… Finally, Fabric. By far my personal favorite suite of mobile developer tools. Owned by Twitter, this suite of tools include (but is definitely not limited to) Crashlytics, Answers (intelligent mobile app analytics), Beta (mobile app distribution), Digits (mobile app identity and authentication services), MoPub (mobile ad network, and other SDKs, into a single, modular SDK, allowing developers to choose which tools they need for each of their projects, and easily integrate them into each of them.
Setup Time & Effort And Ease Of Use: Focusing on Beta (which is why you’re here), the setup works just like HockeyApp, as I have mentioned before, so it’s pretty straightforward, and everything can be done either via your favorite dependency manager.
Integrations & APIs: I strongly encourage using Crashlytics alongside Answers and Beta, and, of course, why not integrate beta to Fastlane? With the press of a button, your builds will be available to all of your beta testers, simple as that. And since Fastlane development is now operating under Twitter’s HQ, they put up some serious effort into making it the most seamless beta distribution tool on the market.
You can also add Crashlytics to your team’s Slack channel. It’s not really the same as integrating with Beta, but whenever you have important errors and crashes (including in your beta distributions), you’ll have them notified in your Slack (or in your email too, if you’re old-fashioned).
Platforms: Beta supports Android and iOS.
Misc Pros: Fabric also provides an amazing real-time heartbeat-like chart where you can see how your production or beta app is doing.
Popularity: If you’re still not convinced, Fabric was serving over 1 billion active devices after only 9 months of its launch, and by the end of 2016, Fabric was serving over 2 billion active devices (virtually all the devices on the Earth), while landing top spots for app analytics, stability, and monetization of 2016.
Pricing: Lastly, best of all: 100% free.
Another great solution that stands out is Applivery, mainly because of its customizability when distributing your builds, which makes it a great solution if for enterprise distributions (those that are not meant to be distributed in the public app stores).
Platforms: It supports both native and hybrid apps on both Android and iOS platforms, under any programming language supported by those platforms.
Integrations & APIs: It also provides one of the most integrated solutions, offering plug-and-play integrations with Slack, JIRA and Google Analytics. It also allows uploading apps automatically, directly from your build server, being prepared to use Bitrise, Jenkins, Travis-CI or any other continuous integration, deployment and delivery platforms. To top on that, it even provides a Fastlane plugin that will deploy everything with the touch of a button, just like its competitors. And if you want even more direct access to its features and functionalities, it offers a huge REST API covering everything.
Setup Time & Effort And Ease Of Use: If you’re not used to Carthage, you’ll find yourself starting to dig into it, unless you’d like to install it manually, since CocoaPods isn’t an option with Applivery. Sending feedback or bug report is as easy as taking a screenshot when using Applivery, so it adds on great usability.
Popularity: Since it’s a recently founded startup in Spain, it isn’t as popular as the previously mentioned solutions, but it’s definitely a strong competition to the market, due to its enterprise-focused solutions.
Pricing: Due to its B2B nature, it’s not completely free for all of their features, but the free version currently includes two apps, 10 builds/app, five distribution sites, two different teams, and 500 installs. For more information on the other plans, see their pricing table.
Popularity: Outstanding integrations support and a huge set of big companies using it are what stands out in TestFairy. Trusted by companies like Samsung Pay, Groupon, GoPro, Citi Bank, Adobe, Johnson & Johnson, Century Link, Gameloft, Pizza Hut, and Burger King, but of course, the list doesn’t stop there.
Integrations & APIs: As mentioned the list of integrations is outstanding, so bug tracking integrations include JIRA, GitHub, Trello, Bugzilla, Asana, Pivotal, and others. And if you’re doing Continuous Integration, TestFairy’s upload API backs you up with Jenkins, Fastlane, Travis-CI, Circle-CI, Bitrise, or any other CI platform.
It wouldn’t let you down with webhooks either, so you can easily plug it into Slack, Hipchat, or any communication platform, and if your organization is using a Single Sign On platform, TestFairy integrates with Okta, OneLogin or Google Apps.
Platforms: It’s available to use on basically any platform, from native Android and iOS, to hybrid solutions like Xamarin, Adobe Air, PhoneGap, Appcelerator, or Unity.
Setup Time & Effort And Ease Of Use: You don’t even have to add an SDK to your project if you’re only looking for the app distribution feature, so it’s one of the easiest tools to setup and start using. As a plus, you can also count on in-app bug reporting.
Pricing: Their basic package is free, offering a pretty decent amount of perks, but if you’re interested in the Startup, Business or Enterprise versions, you can check their pricing here.
Popularity: As stated in their website: “Some of the world’s most respected brands trust buddybuild” and I wouldn’t change a single word. Slack, Arc’teryx, Firefox, Meetup, NBC, and Monzo are some of the notorious names, which makes it pretty impressive.
Integrations & APIs: Just like TestFairy, buddybuild’s integrations cover pretty much everything you can imagine, from building phase, to deployment and user feedback. Integrations include any git based source code repository, iTunes Connect and Google Play (to automatically deploy TestFlight betas, or put your app into production into the app stores), and most communication and bug reporting tools like Slack, HipChat, Trello, JIRA, GitHub, Pivotal, and more.
Platforms: Android, iOS, and surprisingly tvOS and watchOS are part of the platforms that buddybuild supports. If you’re developing hybrid solutions, they also support any hybrid development framework like Ionic, React Native, PhoneGap, and more.
Setup Time & Effort: Feel free to use any dependency manager you’re comfortable with, like CocoaPods, Carthage, npm or Maven, so all dependencies are automatically detected and installed. It doesn’t require any configuration setup, scripts or files upon setup, so everything is as smooth as possible when getting everything ready to go.
Ease Of Use: When it comes to ease of use, definitely this is one of the most outstanding solutions on the market. Again, with their words: “buddybuild ties together continuous integration, continuous delivery and an iterative feedback solution into a single, seamless platform” and works absolutely beautifully. With every
git push to your git server, buddybuild will trigger a build of your app, run any unit or UI tests (even on physical devices, if needed), and prepare your app to be deployed to App Store or Google Play. And remember the UDID handling mentioned in the beginning of this article? Buddybuild handles everything for you, so you don’t have to worry about it.
When it comes to the feedback phase, your beta testers can report bugs and give feedback simply by pressing the screenshot shortcut on the device, and even highlight an area in the screenshot if he wants to. Additionally, with every bug or crash report, an instant replay is available for your to analyze, giving you an image on how to reproduce that specific bug.
Misc Pros: What is even cooler about them is that they can send you a quick demo so you can see how easy it is to get everything working. They email you a beta version of a 2048 demo app so you can play with it, find out bugs or crashes and report feedbacks. The only bad thing about it is that I personally spent 25 minutes playing 2048 when I first received the demo app. ¯\_(ツ)_/¯
Pricing: Their free tier offers 1 concurrent build, auto provisioning of 5 new devices, 10 screenshots/month and 5 instant replays/month. Higher plans can be found in their pricing page.
Other Companies Worth Mentioning
Highlight: AppBlade is a mobile device management tool, thus it’s a beta distribution tool with a focus on security. You can remotely install, update and uninstall apps into your devices, as well as secure them, add a layer of encryption, and wipe info from the devices, everything remotely via its dashboard.
Platforms: AppBlade is available for the iOS, macOS, Android, and BlackBerry.
Pricing: Free for the first 25 devices, and as cheap as $1/device/month afterward.
DeployGate and Installr
Highlight: DeployGate and Installr register your testing devices connecting to Apple Developer Portal, so they auto-manage your UDID, handle your provisioning profiles and update your IPA automatically. If your business constantly uses new testing devices or testers, this is definitely the feature you were looking for.
Pricing: DeployGate’s has fair pricing for indie developers (from $0 to $15/month), but quite aggressive pricing for enterprises, but that’s up to each business to decide. You can find their pricing table here.
Installr, on the other hand has a really limited free tier and charges “per app.”
Highlight: In case you’d like your app tested but don’t have a QA team, Ubertesters offers exactly what it sounds like: a platform with distributed independent testers to test your app. The testers are divided into three different categories: functional testing, usability testing, and localization testing, so you can hire the testers that will better test your app for your needs.
Alongside with the crowd testing platform, it also offers a QA management tool so you can monitor and manage all processes and your own teams in real time.
Pricing: The on-demand crowd testers service starts at $25/hour. Ubertesters do offer a free tier of services, but their full pricing table can be found here.
Click to enlarge
Each of the 11 tools analyzed here was worth mentioning since each has its own niche, specialty, and highlights, so, depending on your application, you might want to use different beta distribution tools.
I highly recommend start getting used with Fastlane and its amazing toolkit. It integrates with most of the services listed in this article, and that’s not just coincidence.
If your app is restricted to the App Store (and you don’t have any plans on developing an Android version in the future), my choice would be Apple’s TestFlight due to the ease of setup, auto provisioning profile management, and the same build that you use on it can go live on the App Store with the press of a button.
Considering all the pros and cons, it wouldn’t be fair to choose once single winner due to the difference on pricing.
My personal winner for best FREE tool is by far Fabric. With support for Android and iOS platforms, Fabric’s suite of SDKs offer the best solutions for all your mobile needs. Featuring 14 SDKs to the date, they consist of the most useful SDKs to support any development need your team might face. Additionally, Twitter crosses Fabric’s Answers Analytics information with Twitter’s analytics, so you get absolutely amazing audience information without paying any extra for that – a unique feature that only Twitter is capable of offering to its developers.
My personal winner for best PAID tool is buddybuild. If you have some budget to invest on your project, the winner here is definitely buddybuild. With an outstanding range of integrations, auto provisioning profiles management, huge platforms support, and a full continuous integration and deployment support, it just makes sense why all those top tier companies are using it.
Let 2017 be the year of continuous integration, delivery, and deployment, better app testing, and more integrated workflows. Let’s get together with the QA team – they are our friends!
Get new posts directly to your inbox.
Get articles like this about iOS development and mobile dev tools.
Published at DZone with permission of Roger Oba , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.