Maximize Your Cloud Investments with the Right Cloud Service Provider
Maximize Your Cloud Investments with the Right Cloud Service Provider
This cloud services provider comparison can help you make the right choice between Amazon's, Sauce Labs's, and Xamarin's offerings.
Join the DZone community and get the full member experience.Join For Free
Insight into the right steps to take for migrating workloads to public cloud and successfully reducing cost as a result. Read the Guide.
More and more organizations across the globe are in search of the right cloud providers in this cloud-first age owing to the key benefits like better scalability, flexibility, mobility, and security, to provide the cloud services that best fit their organization. A report from Datapipe stating, “External cloud adoption will increase by nearly 46% over the next two years” corroborates the above statement. While there are a number of cloud service providers for testing, the need to answer the question of how to choose the right cloud service provider becomes indispensable. There is no instant definite answer to this question as it depends on a number of factors. Perhaps a study on the comparison of the cloud providers could paint a clear picture. This blog attempts to compare, differentiate, and explain the various functions of the 3 most illustrious cloud-based service providers on mobile testing platforms – AWS Device Farm, Sauce Labs, and Xamarin Test Cloud.
The “usage” translates to how the testing is done – manually or automated. The key benefit of automated testing is that it is more reliable than manual testing since it eliminates the human-error factor that might creep in inadvertently. Manual testing, however, is more cost-effective. The AWS Device Farm not only enables automated testing but also permits the remote access to the devices on which the real-time interaction with apps can be done using gestures or swipes. Sauce Labs provides both the manual and automated means of testing. Xamarin Test Cloud, on the other hand, supports only the automated mode of testing of applications.
The more OS platforms a service provider can work on, the higher its versatility. While all the 3 service providers under study are feasible on both Android and iOS, AWS additionally supports Fire OS – an Android-based mobile OS developed by Amazon.
Testing is done to assess a software and to verify if the given input and the expected output are in compliance. There are different test types, each specific to various OSs. Appium, an open-source test automation tool for native, hybrid, and mobile web applications, is one of the most predominantly deployed tools.
AWS supports various custom test types like Appium Java JUnit, Appium Java TestNG, and Appium Python that are used commonly for Android, iOS and web apps. Additionally, for Android, Calabash, Instrumentation, UI Automator and Explorer are used. Besides the 3 common test types mentioned above, Calabash, UI Automation, XCTest (including KIF) and XCTest UI are used for iOS. All of these custom tests enable the testing of specific flows and business logic within a given app.
Sauce Labs provides Appium, independent of language or frameworks, as a common test type for Android, iOS and Web Apps. The other test types used are Espresso and Robotium for Android and Robotium for iOS. Xamarin Test Cloud supports Xamarin Ul Test, Calabash, and Appium-Java-Junit, which are deployed commonly for Android and iOS. Additionally, Espresso and XCUltest test types are used for Android and iOS respectively. There are no test types that support web apps in Xamarin Test Cloud as of now.
In AWS, the upload limit for file size is 4 GB. Although there is no limit to the number of devices that can be included in a test run, 5 is the general limit for simultaneous testing (which is flexible for some cases). For automated testing, there is a 60-minute limit which can be extended up to 150 mins on special request. In Sauce Labs, the app will be deleted automatically if it is kept idle for more than 7 days. The limit for a test run in Xamarin Test Cloud is 90 minutes with a trial account and 360 minutes with a paid subscription account. The test run period for a Xamarin University account is 30 minutes. The individual Xamarin UI Test and the individual Calabash test should not exceed 30 and 10 minutes each respectively. Trial and University account holders can run a maximum of 10 tests per day.
Tools and Plugins
AWS uses Jenkins CI Plugin – a major open-source continuous integration server built with Java and Gradle Build System which is a tool that focuses on build automation and supporting multi-lingual development. Other than its compatibility with Jenkins CI Plugin, Sauce Labs also enables direct connection with HockeyApp, a service which can manage the iOS, Android, Windows, and OS X Apps. Xamarin Test Cloud is viable with Jenkins CI Plugin and Maven Dependency. The latter is mainly used for Java projects and is a build automation tool.
AWS charges in 3 different categories: a "pay-as-you-go” type wherein the client pays according to his/her usage, a private devices testing type, and an unlimited testing type. Sauce Labs’s tariff is based on the number of concurrent sessions with the number of minutes of usage per month. For Xamarin Test Cloud, the charges are calculated according to the number of concurrent devices for the number of device hours per day. More detailed information on how the 3 cloud providers charge can be found in this comparison of the cloud providers.
An emulator replicates the functions of a host system to a guest system and is developed in machine-level assembly languages. While AWS and Sauce Labs support emulation, Xamarin Test Cloud is yet to have emulator support.
“Real devices” refer to the original physical handsets. More than 350 devices are feasible with AWS. Sauce Labs supports more, with 367 devices. Xamarin, topping both, is compatible with as many as 1320 Android devices and 1317 iOS devices.
AWS stores reports such as the metadata, performance data, pass/fail information, crash reports, and logs for 400 days. Moreover, reports such as the frequency of occurrence of a particular problem, performance samples every second are also captured. Logcat captures and Device Console Logs are used for Android and iOS tests respectively.
The test results of Sauce Labs are archived regularly for reference. It also enables report generation from a third party and Log4j logs – A Java-based logging utility. It provisions log issues directly into JIRA or GitHub. Xamarin Test Cloud’s report includes ingredients like pass/fail status, screenshots of the last step, CPU and memory usages, total time taken for test execution, and more. Paid subscribers can also claim a recorded video of the test execution.
As can be inferred from the comparison analysis, an element of ambivalence comes in when selecting the right cloud service provider for mobile testing. According to a research from RightScale, “Inefficient use of cloud results in 35% waste in cloud spending." So it is evident that, by selecting the right cloud provider, cost savings can be significantly high. Organizations can use this cloud service providers’ comparison study as a handbook to take an informed decision on their cloud investments and reap the maximum benefits.
Opinions expressed by DZone contributors are their own.