Machine Learning for Automation Testing
Machine Learning for Automation Testing
The goals we are trying to achieve here by using Machine Learning for automation testing are to dynamically write new test cases based on user interactions.
Join the DZone community and get the full member experience.Join For Free
The goals we are trying to achieve here by using Machine Learning for automation testing are to dynamically write new test cases based on user interactions by data-mining their logs and their behavior on the application/service for which tests are to be written and live validation so that in case an object is modified or removed or some other change occurs, like “modification in spelling,” such is done by most of the IDE’s in the form of Intelli-sense like Visual Studio or Eclipse.
Machine Learning in “Test Automation” can help prevent some of the following but not limited to cases:
- Saving on Manual Labor of writing test cases
- Test cases are brittle, so when something goes wrong, a framework is most likely to either drop the testing at that point or to skip some steps, which may result in a wrong/failed result.
- Tests are not validated until and unless that test is run. So, if a script is written to check for an “OK” button, then we wouldn’t know about its existence until we run the test.
The machine can help recover tests on the fly by applying fuzzy matching, which means if an object gets modified or removed, then the program and the script must be able to find the closest object to the one it was looking for and then continue the test. For example, if a web service has the options, “small, medium, large” at first, and the script was written according to that, and if another choice, i.e. “extra-large,” is added, then the script must be able to adapt to that and anticipate that change so that the test run can continue running without fail.
HP Unified Function Testing is one of the well-known tools available in the market used for automated testing. It has a GUI interface for the same, and other than that, there are other tools like selenium libraries (implemented in several languages like Java, C++, C#, Python etc.) and Cucumber. Tools like these allow one to write their scripts and let the scripts take the job of testing from there by running through several cases.
Ideas for Implementation
Before starting the tests, the system needs to learn the cases; we need to give it something to begin with. Before triggering the training part, we had to set up the system where some ads were shown to the normal users in a customized window of time, and during that time, logs were collected and recorded, which helped us in generating a gender ratio and age groups of people who looked at specific ads. The goal was to find out which age group and gender of users were stimulated to purchase something from the website after looking at certain ads presented to them. The results were saved as training data so that tests could be performed on them.
The machine was trained to write test cases based on the collected information. Training data was updated every once in a while so that tests could be run on the latest data based on different demographics and relevant ads could be delivered to potential customers. Had this been done manually, one might have to make modifications in the script or add test cases manually each time trends changes or the website changes.
ML Algorithms for Automated Testing
SVM: It belongs to the “linear classifier” family of ML algorithms that attempt to find a (linear) hyperplane that separates examples from different classes. In the learning phase, SVM treats the training data as a vector of k-1 dimensions. The goal is to find maximum margin (distance). SVM technique is mostly used for the binary classification. Other than this, we have MartiRank, a ranking algorithm, in the learning phase. It takes a number of rounds and during each round/iteration, data is broken down into N sub-lists, each sub-list containing 1/n of the total number of device/app failures.
In case of Regression testing, a suite of test cases needs to be developed. Development of MartiRank is an ongoing process and has been used to detect new bugs. Ex: A dev might have refactored some code in the application and put into a new function. Regression testing showed us that the resulting models were different from the previous ones.
When we write test cases, we test how the software is supposed to behave theoretically, and there’s no real data with us, so some of the test cases might never be used in real life and some that missed the test cases might be the most important ones. That is why data-mining the logs and letting the machine write test cases according to those logs automatically saves a lot of man-hours and helps in practical testing. Services like HockeyApp and TestFlight are providing automated mobile app testing as a service.
As for GUI tests, there are some research papers out there that talk about deep learning and reinforcement learning for automation of the test. The systems that were being tested were first data-mined to get the meaningful clicks, texts, and button pushes on the GUI interface, which generated a good amount of training data. That data was then used to perform tests on the software for a few hours. The best part was that there was no need for models or test cases to be written and the bugs were being found as the time passed by, but some of the cases were not being tested which can be due to lack of training data. The reinforcement approach improved the testing as they were running through multiple iterations.
Intel and Nvidia have been investing heavily in hardware solutions that can aid Deep Learning and related algorithms to achieve results more quickly. Moving from a mobile-first world to an AI first world. We know that for testing a certain product, there can never be enough of the right test cases and that is why developers and testers are encouraged to write more and more test cases in order to make their product more stable. Paul Graham once suggested the used of Bayesian Filter for filtering out spam emails. Thousands of emails were fed to the system, and it was made to learn and then tests were performed on that training data to make sure that the filter was fool-proof. Web crawlers move through different websites looking for 404 or other errors all the time, updating their indexes and updating their test cases in real time.
Published at DZone with permission of Saif Sadiq , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.