Defining the Primary Problems That Test Automation Frameworks Solve

Defining the Main Problem That Test Automation Solves

As we talked in the last article about Generations of Test Automation Frameworks, the need for test tools and frameworks came from the change how people build software. In the beginning, most companies followed Waterfall methodologies releasing once per year or even on longer intervals. But with the evolution of technologies and the business competition, this was not enough, so the Agile development started to be adopted. The release cycles for most apps got shorter and shorter, from 1-2 weeks to a couple of times per day.

At the beginning the manual testing was OK. But with shortening the cycles over and over again, the humans couldn't keep up. This is because with time old applications got bigger and bigger. The companies needed more testers or had to release with an increased risk of regression bugs or not-working new features.

For me, this is the primary problem that the test automation as a whole solves. However, people realized that they need help with many sub-problems.

Sub Problem 1 — Repetition and Boredom, Less Reliable

If you have to execute the same test once per month, it is okay. But if you have to do it 3 times per day, we as humans tend to get bored quite fast. Another natural thing that happens is that we begin to ignore the little details. They, in most cases, hide the bugs. Even the most conscientious tester will make mistakes during monotonous manual testing.

Automation Solution Hypothesis

ASH-1: Execute the Same Tests Over and Over Again

When you have a way to execute all tests the same way over and over again, you have more time to concentrate on writing new tests. Also, you can have more time to learn new tools, programming languages, and test techniques.

Subsequent Problems

SP-1: Automated Tests Are Not Stable

Many teams face this issue. Their tests are not correctly written, and because of that, they fail randomly. Or even worse, providing false-positive or false-negative results.

SP-2: Initial Investment Developing These Tests

The process of creation of stable automated tests that verify the right things is an expensive undertaking. You need to invest in tools, frameworks, training the staff, test data preparation, the creation of test environments, test execution integration in CI/CD, and test result visualization.

Sub-Problem 2 — People Are Not So Good With Numbers/Accuracy

If you have to test a user interface in most cases is a relatively doable job. Comparing labels, messages, clicking buttons, filling up forms, eventually check if something is populated in the DB. However, with time the systems got much more complicated. How do you compare without tools- huge XML/JSON messages or files? It is quite hard to manually check if 100000 orders are upgraded successfully with all calculations. There are many cases where automated scripts can do a much better job than us.

Automation Solution Hypothesis

ASH-2: Reuse Automatic Coded Checks

Once you write logic for verifying that a single order is well migrated, you can create data-driven tests and execute the tests an unlimited number of times. The automated test for the scenario 15670 will be with the same precision and accuracy as test number 1400.

Subsequent Problems

SP-3: Initial Investment Creating Shared Test Libraries

Sometimes to create a generic solution that can be generalized for testing many scenarios and many test cases take lots of effort. Sometimes you may not have the right people to create the library, since significant coding and design skills may be required. Or even if you have these people, they may need to work on something else. It may be a significant time and money investment to code it right.

SP-4: Are Automated Tests Trustworthy? Test the Tests?

Even if you have the time to create these libraries. How do you make sure that they work correctly? It is code so it may contain bugs. Do you spare time to develop tests for the tests or accept the risk- the checks not working correctly?

SP-5: Library Modifications

Creating the initial version of the library may be easy. But it is a different thing whether it is designed in a way to support future modifications. Changes in the requirements and applications are something natural, and with them, our tests should evolve. Many people forgot that, and in many cases, it is easier for them to rewrite the tests instead of making little tuning because the code may not support that.

SP-6: Library Customization- Another Team's Context?

The whole point of creating a shared library is to be used by multiple teams across the company. However, the different teams work in a different context. They may have to test a little bit different things. So, the library code as is may not be working out of the box for them. How easy will be for them to extend or customize it? Is it even possible to do it? Will they need to invest time to create the same test feature again with the required modifications?

SP-7: Knowledge Transfer

Shared libraries are great but do the people across all teams know whether they have a particular feature of your framework or not? Do they know that you already created some automated check?

If they don't know that something exists, they will create it again. This is especially true for new team members. If you were a part of the team building the tests features, then you know most of them. However, when you have just joined the team if you don't have a way where to find this information, someone has to teach you. Otherwise, you will face the problems, we mentioned.

SP-8: Test API Usability

Is it easy for the users to understand how to use the library? Even if people know that something is there but don't know how to use or make it work, will result in creating the same thing again.

If the API is not concise and it is confusing, then you will need again to invest time to teach the new team members separately how to utilize it.

Sub-Problem 3 — Time for Feedback- Release

When you need to release a security hotfix immediately, you don't want to release the new version untested. However, if you have 5000 test cases and your 10 QAs need 5 days to execute them by hand- this is not so fast right?

Automation Solution Hypothesis

ASH-1: Execute Same Tests Over and Over Again

ASH-3: Utilize the Speed of Automated Tests

Let's face it, computers are much faster than us. Even if you drink 5 coffees and click very fast, you will be much slower than a well written automated test. Even if you can match its speed, you will eventually get tired or will start missing some crucial details. Instead of waiting 1 week for all 5000 test cases to be executed by 5 QAs, you can run all of your automated tests distributed on multiple machines for a couple of hours.

Subsequent Problems

SP-9: Test Speed

As mentioned the automated tests are fast. However, this is not always the case. If the tests are not written properly, they can get quite slow. Some people tend to use big pauses in their automation to handle various challenges in automating web or mobile apps. Another possible scenario where tests can get quite slow is when you put retry logic for various failing statements until the test succeeds. Or just retry all failing tests multiple times.

Sub-Problem 4 — Regression Issue Prevention

Regression Testing: "Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made."

Often because of the time pressure, "less important" tests are not executed manually. This happens not only because of tight deadlines but when some of your people are not available- holidays, sickness, or just your team is short staffed.

Automation Solution Hypothesis

ASH-4: Execute All Tests Before Deploying App

You can execute all of your tests before deploying. Only if all tests pass then the app is deployed. Since the automated tests are quite fast, you can deploy the tested version after 30-60 minutes.

Also, since the setup of tests is quite easy, the developers can execute some of the most important tests on their machines and fix the regression problems even before submitting the new code.

Subsequent Problems

SP-9: Test Speed

SP-10: CI and CD Readiness

Some older automation solutions are tough to execute from the command line interface, meaning it is hard to integrate them into your continuous integration or continuous deployment tools. Whatever, tool or framework you use, it now should allow being integrated relatively easy in CI systems. The same is valid for produced test results by the tool or framework. Usually, after CI execution, you want to publish them somewhere.

Sub-Problem 5 — Skipping Part of the Scope

After you test something, you need to show your managers what you have tested. If you don't have a test case management system, this can be hard. Some people cannot be trusted. They can lie that they executed something or just forgot to do it. This is not the case with automated solutions.

Automation Solution Hypothesis

ASH-4: Execute All Tests Before Deploying App

Subsequent Problems

SP-9: Test Speed

SP-10: CI and CD Readiness

Sub-Problem 6 — Ping-Pong

With a dedicated QA team, there was the so-called ping-pong game of throwing bugs. How does this happen?

1. The developer produces a new version of the app and deploys it without any testing.

2. The QA starts testing when he gets free (this may not happen immediately). Testing begins 4 hours after the new version.

3. The QA logs a bug 2 hours later. However, the developer may head home.

4. The developer fixes the bug early in the morning and deploys a new version without testing it. This takes 1 hour.

5. The QA starts testing 3 hours later and finds that the first bug got fixed, but another regression one appeared.

6. The QA logs the regression bug 30 minutes later.

7. The developer is still there but started to work on a new feature, so promises to fix the new bug tomorrow.

Automation Solution Hypothesis

ASH-4: Execute All Tests before Deploying App

ASH-5: Developers Execute Automated Tests Locally

So, for a single feature to be tested and released you needed from 2-3 days. If you had automated tests, the developer could execute them on his/her machine or at least the most important ones. Even if this is not possible, nowadays the most critical tests are performed right after the code is deployed even if we are talking about UI/System tests. Which means that if any regression bugs appear, you are going to catch them within 1 hour after the code's submission.

Subsequent Problems

SP-9: Test Speed

SP-10: CI and CD Readiness

SP-11: Ease of Local Test Solution Setup

How easy is a new colleague to set up everything needed to run the automated tests locally? Do you have instructions? How much time will he/she need? There are some cases where a man needs half a day to install and configure everything required. Which is a bummer since we programmers tend to be lazy most of the time or have so many other things to code. So, many people will prefer to skip the process and just not follow it because there are too many steps.

SP-12: Ease of Local Tests Execution

If you have lots of tests most probably the developer won't be able to execute all of the tests before check-in his/her code.

How easy is to locate which tests he/she needs to execute?

When the tests are executed, is it possible to continue working or should go to drink a coffee? Some automation frameworks make it impossible to work during the test run. Browsers appear on top of all other programs, tests get the mouse focus, move your mouse or make real typing with your keyboard.

Is it possible for the tests to be executed locally but in an isolated environment like Docker?

Can the tests be executed smoothly against the developer's local environment instead of against the shared test environment?

Sub-Problem 7 — Money, Time, Effort for Maintaining Test Artifacts

What are test artifacts? For example, these are all manual test cases you have. When a specific part of the application changes, you need to analyze and update your test cases. Otherwise, you have to rewrite them each time. Even with the usage of more sophisticated test case management system. This may be a task. This is why some more "agile" companies decide that the QAs won't use detailed test cases which of course leads to untested or not-well tested areas of the app. This is especially true if you have to test part of the application which was developed and tested by people that are not part of the company anymore.

Automation Solution Hypothesis

ASH-6: Easier Test Refactoring and Modification

Automated tests are testing artifacts too. But if you have a proper test automation framework and test structure, a single change can update multiple test cases at once, instead of editing each one by itself. Most modern IDEs provide tons of refactoring support features.

In most cases, once a manual test case is automated, you don't need it anymore so you can archive it.

Subsequent Problems

SP-5: Library Modifications

Sub-Problem 8 — Various App Configurations

If you have to test a modern website- you need to make sure it is working on Edge, Firefox, Chrome, Opera, Internet Explorer and Safari. If your software is related to banking or similar service- you need to check if the app is working the same way on the latest 5-10 versions of these browsers. But wait, you have to check the responsiveness of the website on various web browsers and mobile resolutions. It is doable to make all these tests manually, but they require lots of effort and sometimes money to buy and configure all these devices. And let's face it- layout testing is hard for humans. It is a considerable challenge to verify "pixel perfect" designs without tools or code.

Automation Solution Hypothesis

AHS-7: Reuse Tests for Different Platform Setups

Through code, you can easily reuse the same tests for different browsers or mobile devices with various resolutions. Also, the code can enable you to perform pixel perfect layout testing if needed.

Subsequent Problems

SP-13: Cross-Technology and Cross-Platform Readiness

Is it possible to execute the tests with no code modifications on various browsers? Are the tests behaving the same way in different browsers? For example, if you use a pure WebDriver code, it will be almost impossible. How much code do you have to write to change the size of the browser or change the mobile device on which the tests will be executed? In most cases, it is better to skip the large configuration boilerplate code.

SP-14: Cloud Readiness

Some companies have their own- farms of devices and computers with various browser configurations. However, nowadays the cloud providers such as SauceLabs, BrowserStack, or CrossBrowserTesting are a reasonable solution to the problem. Is it possible for your tests to be executed there? How much code is needed to switch from cloud to local tests execution?

SP-15: Docker Support

Another alternative of the cloud providers is to run your tests in Docker containers. However, the setup can be hard. Is it possible your automation solution to be integrated easily with Docker (Selenoid, Selenium Docker)? Does it provide pre-created configured images with all browser versions or mobile emulators?

SP-16: Complex Code Configuration

It is one thing your tool or framework to support various browsers, devices, cloud or Docker integrations. But it is entirely another how much code you need to write to make it happen. It is not only about the initial creation but also if the requirements change, how easy is it to reconfigure the tests to use another configuration?

Sub-Problem 10 — Low QA Team Morale

When you execute the same tests over and over again, people got bored as mentioned. Most of us want to get fun and do something new once in a while. So, we usually get demotivated doing the same things and not improving ourselves. This is one of the primary reasons why people want to change their teams or quit their jobs.

Automation Solution Hypothesis

ASH-8: Test Automation Novelty

Thinking how to automate particular app's functionality, prepare the test data or integrate your tests in CI is an exciting and challenging job. Most people feel challenged working on test automation and had to learn continuously new things.

Subsequent Problems

SP-1: Automated Tests Are Not Stable

SP-4: Are Automated Tests Trustworthy? Test the Tests?

SP-17: Tests Speed Creation

Time spent on maintaining the existing tests is essential. It can take significant time from the capacity of the QA team. However, for people to be motivated, it should be relatively easy to create new tests. For sure, QA engineers will be frustrated if for the whole sprint can produce only a few tests. It shouldn't be a rocket science to create a simple test. Or even if the scenario is more complicated, it shouldn't be so hard to understand what needs to be done to automate it.

SP-18: Troubleshooting, Debuggability, Fixing Failing Tests Easiness

As we mentioned, a big part of maintainability is troubleshooting existing tests. Most in-house solutions or open-source ones don't provide lots of features to make your life easier. This can be one of the most time-consuming tasks. Having 100 failing tests and find out whether there is a problem with the test or a bug in the application. If you use plugins or complicated design patterns the debugging of the tests will be much harder, requiring lots of resources and expertise. Even if you spot the problem, how easy is to fix it? Do you fix the code only in one place to fix multiple tests? In case the library didn't reuse most of the logic but for example copy-paste it, then the fixing will take much more time. If you use a 3rd party framework (open-source or commercial one), is its support fast and helpful?

Sub-Problem 11 — Questioned Professionalism

Every QA wants to be a professional and be recognized for a job done well. However, as mentioned, if you have tight deadlines, management pressure to test and release the app as soon as possible, you cut out of the scope of your testing. Executing only the most important test cases. Anyhow, most of the time when a new bug is found on production, QAs feel responsible that they didn't catch it. In some, cases developers are not aware that this is not our fault and we had to cut out of the scope. So, they start to question how good we are. Even in some edge cases, people get fired or don't get bonuses.

Automation Solution Hypothesis

ASH-4: Execute All Tests Before Deploying App

ASH-5: Developers Execute Automated Tests Locally

ASH-8: Test Automation Novelty

ASH-9: More Time Writing New Tests — Exploratory Testing

With the time you will have more and more automated tests, checking for regression bugs. This will reduce the time for performing the same tests over and over again. Also, the locating-fixing bug cycle will be drastically shortened since the developers will be able to execute the tests locally or all tests will run before deploying the app. The QA team will be more motivated since it will execute more exciting tasks — thinking about how to automate more complicated and challenging scenarios. Moreover, will be able to spend more time experimenting- using manual testing techniques such as exploratory testing to locate new bugs.

Subsequent Problems

SP-1: Automated Tests Are Not Stable

SP-4: Are Automated Tests Trustworthy? Test the Tests?

SP-17: Tests Speed Creation

SP-19: Upgradability

Fixing unstable tests is not the only time-consuming task of having an in-house automation framework. Every two weeks, new browser versions are released. With each of them, a new version of the low-level automation libraries is released — WebDriver, Appium, WinAppDriver. However, since these libraries and tools are open-source, nobody can guarantee that they will be bug-free or backward compatible with everything you have. From my own experience, I can assure you that this task takes at least 3-4 hours per week if no significant problems appear. If a problem occurs, it can take much more time. This is especially true if you need to support all browsers and a couple of their versions (not only the last one). The same is even more valid for mobile automation low-level libraries since there is an unlimited number of devices and configurations.

Because of these problems, many teams don't upgrade so often to spare some time. However, failing to test on the latest versions of the browsers hide many risks of not locating regression bugs.

For example, in the Bellatrix Test Automation Framework, we support all 8 browsers and 10 older versions. The same is true for Appium and WinAppDriver. We upgrade all libraries as soon as the new version of the driver or browser is released. To make sure that everything is working as expected, we maintain 6,000+ system tests and much more unit tests. To be able to receive feedback multiple times a day, we created a distributed test runner Meissa, which enables us to execute all these tests within an hour on multiple test agent machines in the Azure cloud. But even with all these tools and tests, it can take more than a day to make sure that everything is working and is backward compatible. Almost always, it is necessary to make fixes since something in the lower-level libraries stopped working.

Sub-Problem 12 — Provide Evidence of What You Did

When you manually test something, it is a little bit hard to prove what you did. Sometimes it got unnoticed by developers and other colleagues because it is something "invisible." Of course, in more mature test processes you have detailed manual test cases, test reports, etc. But in general, let's be honest when you write code it is much more noticeable and measurable what you did. Because of this issue, I have heard many people complaining that they cannot get some promotion or a bonus. For me, this is one of the reasons why QA people want better reporting tools. To show developers and management what they did. (Software won't get any better because you have a fancy dashboard showing how many test cases you have executed.)

Automation Solution Hypothesis

ASH-10: More Visible Results

Since most of the test cases will be translated into coded tests, it will be more visible what the tests are. The code is something that can be measured and be seen. If needed the test results can be displayed in beautiful dashboards or send via emails.

Subsequent Problems

SP-20: Tests Code Readability

Sometimes, your developer fellows or your managers may want to check what you did. If you name your tests and methods right, you use page objects and another design patterns. It will be much easier for other people to understand what your tests do. If this is true, it will be questionable whether you need more detailed reports.

(You can check out these articles about Naming Methods Right or Naming Classes, Interfaces, Enumerations).

SP-21: Test Results, Reports, Logs, Portal Integrations

If you use an in-house framework, does it produce good-looking test result files? Can these files compatible with the most popular extensions for CI tools? How much effort is required to visualize them with these tools? Is it even possible? There are at least 2 popular open-source solutions for visualizing test results (ReportPortal, Allure); is it possible your test results to be integrated with these tools?

Sub-Problem 13 — Holidays and Sickness, 24 Hours

If you have 1 QA per app area and he/she goes on holiday or gets sick. How do you release your application? Do developers start testing or? Most people can work up to 8 hours. Of course, there are situations where your managers tell you that you need to come on Saturday to test the new version because you need to release it on Monday. But you won't often hear- "let's stay up 2 nights in a row to finish everything". People need to sleep and rest. (automated tests don't care whether it is 3 in the morning, Sunday or New Year's Eve)

There is another aspect of this problem. When someone decides to leave your team or the company, all of his/her knowledge is lost.

Automation Solution Hypothesis

ASH-1: Execute Same Tests Over and Over Again

ASH-4: Execute All Tests before Deploying App

ASH-5: Developers Execute Automated Tests Locally

ASH-11: Tests as Documentation

If the automated tests are readable enough and the test scenario is visible through reading the code, even if you are missing, your colleagues will know what the test does. If they can quickly orient themselves to what is automated and what is not, they will know what they need to check manually till your return.

Subsequent Problems

SP-7: Knowledge Transfer

SP-8: Test API Usability

SP-20: Test Code Readability

SP-21: Test Results, Report, Logs

Sub-Problem 14 — Some Things Cannot Be Tested Manually

How do you perform a load or performance test manually? I can say the same about testing "pixel perfect" designs. Probably you can use some non-automated way using tools, but it is hard.

Automation Solution Hypothesis

ASH-12: Reuse Automated Tests for Complex Test Scenarios

You can reuse some of your existing automated tests and reconfigure them to perform performance and load test. Also, you can use a different API from the framework for pixel-perfect layout testing.

Subsequent Problems

SP-5: Library Modifications

SP-17: Tests Speed Creation

SP-18: Troubleshooting, Debuggability, Easiness Fixing Failing Tests

SP-22: Learning Curve

Is it easy to figure out how to create these complex- performance or load tests? Do you need to read huge documentations (if they even exist) or you can figure out everything from the demo example or the public API comments while typing in the IDE? How much time does a new team member need to learn to write and maintain these tests? Do you need to spend countless hours passing your knowledge to him or the authors of the framework give you better alternatives?

Sub-Problem 15 — Consistency

What happens if you don't have detailed test cases. QAs will be free to experiment more while testing the app. Which in some cases has some serious benefits; this is why exploratory and mutation testing practices exist. However, in some cases, if there are ten ways to accomplish something in the app and you need to test exactly the 9th way. But because of the lack of detail or how the QA perceived what he/she needs to do, performs the operation following path 8. However, if we test something important, we want the tests to be consistent.

Automation Solution Hypothesis

ASH-13: Unified Coding Standards

A significant part of what the test automation frameworks are is that they give all people involved unified ways of writing tests. Unified team standards make the tests look identical and much easier to write. The shared libraries of the framework provide an API that makes completely clear for the user which scenario will be executed if the particular method is called.

Subsequent Problems

SP-7: Knowledge Transfer

SP-8: Test API Usability

SP-20: Tests Code Readability

SP-22: Learning Curve

SP-23: Unified Coding Standards

Coding standards are good. However, you need an automated way to enforce them. Does the framework you use to give you a default set of coding standards? Does it give you an automated way to apply and follow them? Is it easy for people to see what they did wrong? Is it easy to customize the default set of rules? (Some popular solution for .NET are EditorConfig, StyleCop and ReSharper).

Sub-Problem 16 — Faster Scale-Up and Scale-Down

Let's say that you have lots of things to be tested for the new release. Right now you have 2 QAs in the team but to finish on time the manager has to bring one more. But these transitions don't happen for a day. With longer release cycles, this is doable and maybe the right decision. But when you release each week, and you need more man-power- this approach cannot scale. Imagine that you scale-up for the next release after two months, now you have 4 QAs but after that is summer, and the releases will stop for a while. Two QAs will be enough to test all new features. What happens with the 2 additional QAs you brought to the project? Do you move them again? It is not only a problem how you scale-up but also how you scale-down.

Automation Solution Hypothesis

ASH-14: Cloud Test Agents and Docker

Living in the era of public clouds, you can have additional test agent machines for a couple of minutes. The same is valid if you use Docker to executed your automated tests. If you use the appropriate runner, the tests can be distributed across remote machines and performed there in parallel.

Subsequent Problems

SP-14: Cloud Readiness

SP-15: Docker Support

SP-24: Parallel and Distributed Tests Execution

Does your framework support your automated tests to be executed in parallel? Is there a runner for your tests that can run them in parallel or distributed?

You can check out our open-source distributed test runner Meissa that can run tests on multiple machines in parallel.

Summary

As you can see, there are lots of reasons why test automation is necessary. However, like all things that evolve, first generations of test automation tools were not so good as promised. This is why test automation frameworks started to be a much more popular solution. However, to create stable, fast and easily maintainable tests with them require lots of upfront planning and thought. Many more subsequent problems occur that tool and framework vendors don't mention on their marketing pages. I believe that if you are aware of all these problems you will be able to make much more educated choices about where to invest or not.

In the next articles from the series, we will look into the must-haves and nice-to-haves features of a test automation framework.