Can Artificial Intelligence or Machine Learning Test iOS Apps?
Can Artificial Intelligence or Machine Learning Test iOS Apps?
Third-party tools for testing applications with AI and machine learning are starting to arise - what's the chance one will appear for iOS apps soon?
Join the DZone community and get the full member experience.Join For Free
Read why times series is the fastest growing database category.
The IT market has been shaken by the wave of Artificial Intelligence, Machine Learning (AI/ML) and Continuous Integration, Continuous Delivery (CI/CD) practices. The DevOps and CI/CD movement made the drastic change in the roles of software engineers and it highly impacted QA engineers. Testing at the pace of DevOps is impossible using the traditional QA methodologies and QA engineers are struggling to find their role within the DevOps workflow. As a result of this, confused QA engineers ended up buying the fully theoretical book on "Practical Testing in DevOps" to get some idea of how to survive. However, reading the book won't make them survive in this fast-paced DevOps world. They need to acquire hands-on skills and learn modern technologies to survive in the market. This gives rise to AI/ML-based testing tools like Mabl hits the market. Mabl currently tests the websites using Machine Learning, but it cannot test native mobile applications yet. In this blog post, we will try to find out if any third party AI/ML can test iOS apps in the future.
Before we dive into testing iOS apps with machine learning, let's briefly see what Mabl is and how it tests websites using AI/ML techniques. Basically, a couple of engineers who worked for Google, Dan Belcher and Izzy Azeri, found the problem that testing at the pace of the DevOps is really hard, and the traditional approach of testing software isn't enough at the pace of CI/CD. In order to speed up testing to the DevOps and CI/CD pace, they found the AI/ML-driven tool Mabl that will generate the test scripts and run them on Continuous Integration servers and report the results. Mabl is a scriptless, infrastructureless, and virtual QA engineer that tests our web application without the need for managing anything in-house. We can record test scenarios by teaching the bot about your application like you train a new QA engineer using the Chrome extension. Once we train the bot, the bot will know more about the application and create and maintain test scripts in the cloud. Mabl can be plugged with many Continuous Integration servers like Jenkins and TravisCI. It also sends test reports to Slack or other services. Mabl also has insights that come out of the machine learning pipeline. It's basically a machine learning technique that interacts with a web element on the page - that's where Mabl differs from Selenium WebDriver. Selenium has issues locating an element on the screen which causes a lot of flakiness in the tests and Mabl will supposedly solve that problem. Mabl is in public beta at the moment, so you can give it a try. There is a brief article on Mabl here. In summary, Mabl works like this:
- You provide the URL of your web application to Mabl with login credentials, if applicable.
- Mabl analyzes your app and crawls for JS errors, broken links, etc.
- You train Mabl using teh Chrome extension and record journeys that will be stored in the cloud. Mabl uses machine learning to analyze your app.
- Mabl will execute those journeys on a CI server and report issues.
- There is no need to manage infrastructure or code for testing.
Currently, Mabl looks like a Record and Playback tool hosted in the cloud, but as per Mabl, it uses a lot of AI and machine learning under the hood. As of now, Mabl can only test websites and doesn't have support for testing mobile apps. As of now, there is no plan to add mobile testing support on the third roadmap.
iOS Challenges for Third-Party AI/ML Tools
Now that we have seen that Mabl can't test iOS apps at the moment but let's figure out what are challenges testing iOS apps and will third-party AL/MI can use current Apple technologies to test iOS apps. There might be a few challenges dealing with the Apple ecosystem. Let's cover some of them here.
iOS Build and Deploy Process
In the web world, it's fairly easy to deploy the app to a specific test server and use the URL of that server for testing. However, in the iOS world, we have to consider so many things while we prepare to test an app. In order to prepare a debug or ad-hoc version of the app, we need an app to be signed with a specific developer certificate and have a provisioning profile with a list of devices. It's questionable whether the AI/ML tools that will be used have provisioned devices to test an app. Another challenger: prebuilt binary needs to deployed on the devices before actual testing happens. Are AI/ML able to analyze and build an app by themselves if we teach them how to do that? There are various moving parts that need to be considered before we get an app into a testable state.
Apple has its own tools and frameworks, and they are fairly solid in nature. Apple wants developers to use their tools and technology. It's possible to reverse-engineer Apple frameworks, but it's not worth the time and money to do this, as Apple can break that in the next release. Apple has realized the potential in machine learning and released its own machine learning framework (CoreML) that can be used in iOS apps. Apple might be also investing in strategies to test machine learning features effectively. It's a good idea to leave it to them for now and see what they can offer for us to test machine learning stuff. Efforts made by third-parties to test iOS apps with machine learning won't be sustainable for a long time.
Training Machine Learning Model
In Mabl, they have built a Chrome extension to train a Mabl about your application journeys. It looks like a recorder at the moment. Building recorder for the web apps can be easy but building recorder for iOS apps can be tricky. Apple has its UI testing framework (XCUITest) tightly coupled with Xcode. XCUITest has a recording feature that can record application journeys using the accessibility API of iOS. Can any third-party tools enter the iOS ecosystem and create a recorder, something like Appium did? How will AI/ML handle the communication between app and trainer without entering into the Apple ecosystem? Will it be easy for an AI/ML model to record tests by itself?
There are many questions about training ML-driven tools to teach all things about your iOS apps.
iPhone versus iPad Support
Mabl just announced a product update on Twitter that it supports cross-browser testing. In the web world, once the journeys are recorded, it is not difficult to run them on a different browser using techniques like user agents or engines of other browsers. However, iOS has the iPhone and iPad, with various combinations of screen size with different hardware structures, e.g. iPhone X, iPhone 7+, iPhone 9, iPad Pro, and iPad Mini. The question is, once you've trained an AI/ML model for once device, can it be applied to all the device variants? Will the journey recorded on iPhone work on iPad, or do we need to train the ML-based model differently for iPhone and iPad? How will AI/ML deal with elements on the screen? Will they use Accessibility APIs or some other techniques to identify elements uniquely without causing flakiness?
Testing iOS App Extensions
It's very common to have an extension to iOS apps like Watch Extension, iMessage extension, etc. It would be interesting to see how AI/ML-based tools will deal with iOS app extensions. If the AI/ML-based tool lives in the cloud, will it codesign app extensions or pair with iPhone? Will there be real devices for AI/ML-tools or would it be just simulators?
What’s the Conclusion?
It would be very interesting to see how AI/ML will handle these situations.
Now we have seen a few of the challenges of building AI/ML-driven testing solution for iOS apps. Considering these challenges, we can say that its possible for a third-party to build an AI/ML solution like Mabl, but personally, I think it won't be sustainable within Apple's ecosystem for the long term. Here are things that can be built easily:
- It's possible to ask users to upload a signed
.ipafile to start testing. Once the binary is uploaded, an AI/ML-based tool will be able to install that in the simulators or devices if they are OK to be provisioned.
- AI/ML-driven tests will then launch an app, record some journeys, and store those journeys in an analysis pipeline. It might be challenging to deal with elements of the app unless they have an access to Accessibility APIs similar to XCUITest.
- Integration with Continuous Integration services and third-party services like Slack can be easily achieved with AI/ML.
- Hosting the data in the cloud can also be possible.
These things look achievable. However, it would be hard to implement by reverse-engineering Apple tools and technologies, so it is not going to happen anytime soon. Mabl for iOS still has a long way to go.
Until you are in the Apple bubble, there won't be any Mabl-like third-party tool that comes in and starts testing iOS apps straight away. Apple has released the CoreML framework and will definitely release a testing strategy for machine learning-based iOS apps. Just wait and watch the space for what Apple will bring for QA engineers to test iOS apps. Don't worry about learning Python yet - the Apple ecosystem won't easily allow third-party AI/ML testing tools to enter their space. Always keep your eyes open to see what's happening.
Published at DZone with permission of Shashikant Jagtap , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.