Why You Need To Care About Automated Functional Testing In 2019
Why You Need To Care About Automated Functional Testing In 2019
Take a look at this deep dive into all the reasons that you need to make sure you included automated functional tests in your process.
Join the DZone community and get the full member experience.Join For Free
Open source vulnerabilities are on the rise. Read here how to tackle them effectively.
How many times have you come across products that have good UI but really bad functionality such as severe lagging experience and an ample number of bugs? There could be multiple reasons for the product to go live, but it definitely gives an indication that thorough testing was not performed. There could be scenarios where a minor software update which was not tested for all the corner scenarios could break the existing functionalities in a software product.
In today's digital economy, testing a software update or complete software package before going live is primarily important so the users of your product do not get a buggy product. This can harm the reputation of your organization and could also result in losses, since a fix has to be provided in order to solve the problem.
Rather than performing the test during the final stages of product development, it is highly recommended to perform continuous testing in DevOps which emphasizes that testing is conducted at every critical phase of software development. With methodologies such as Shift-Left testing, organizations have now started incorporating testing even before the development part is completed. Functional testing is one category of testing that should be done in order to make sure that all the functionalities are working as per requirements documented in Software Requirements Specification (SRS), Business Requirements Specification (BRS), and other Requirements Specification documents.
What is Functional Testing?
Functional Testing is a form of testing performed for estimating the quality of a software product by verifying the features provided by the application with respect to client-requirement documentation. Before the start of software development, the requirement gathering team comes up with different sets of documents like Software Requirements Specification (SRS), Business Requirements Specification (BRS), Customer Requirements Specification (CRS), and Functional Testing is one measure to make sure whether all the documented requirements have been met and implemented as a part of the final product.
While doing functional testing, the tester should do black box testing and should not bother about the implementation details of the functionality. Depending on the nature of the product, you could use a mix of manual and automation testing methodologies to get maximum test coverage. Databases, Application Programming Interfaces (APIs and browser compatibility), API performance testing, client/server access, user interface interactions, security testing, and cross-browser testing are some of the things that can be performed as a part of Functional Testing. The scenarios will definitely differ depending on the type and functionalities of the product being developed by your team.
The test results are compared with the expected results and based on the output/result, a particular functional testing is considered as Pass/Fail. Since it is a part of black-box testing, a test engineer well-versed with black-box testing methodologies can perform functional testing as well.
Importance of Functional Testing
As discussed earlier, a tester acquainted with functional testing has to validate the software product from all angles and make sure that the requirements are well-documented. Also, different requirement specification documents should be met in the product/application under test. Hence, functional testing is performed by the tester where the product is tested more from a customer/consumer point of view i.e. customer requirements, unlike non-functional testing which is performed by giving priority to customer expectations. We would look into the difference between Functional testing and Non-functional testing in subsequent sections.
Unlike other forms of testing like unit testing where tests are performed on particular blocks or pieces, functional testing is done on the integrated product. The kind of issues that are discovered post-integration testing may be quite different from other forms of testing. This is why the bug fixes pushed to solve those issues, could cause possible side-effects. This side effect is a regression defect. A regression defect is a consequence bug that is caused because of a recent bug fix in the application. Understanding regression defects are important for every release process.
As a part of Functional testing, the test team can execute scripts to verify the data-flow, security aspects, UI/UX flow, other features mentioned in the CRS. A Functional tester should have good know-how about the product and should possess good domain competency as well.
The tester can do manual testing or automated functional testing or a combination of both. More testing means better test coverage and better product testing. Along with Functional testing, it is recommended that your test team also looks into Stress testing, Non-functional testing, Regression testing, System testing, etc. depending on the timelines and the nature of software product under development.
Understanding The Difference Between Functional Testing and Non-Functional Testing
Before we look into the difference between the testing methodologies, it is important to understand the main objectives of Functional testing. Some of the core objectives of Functional testing are mentioned below:
- Usability testing: Using manual testing/automation testing/mix of both, testing is done to make sure that there are no usability issues with the product. The user should be able to seamlessly navigate between different screens.
- Error Checking: In order to increase test coverage, all the boundary and error scenarios are tested thoroughly as a part of Functional testing.
- Compare the output with expected outcomes: By supplying relevant test data, all the major functions in the product/application are tested.
Now, let's have a look at the top level differences between Functional and Non-Functional testing
- Objectives: Functional testing is performed to check whether the requirements mentioned in SRS, BRS, CRS, etc. are translated into equivalent features/functionalities. On the other hand, Non-functional testing looks into other parameters like scalability, speed, reliability, and other non-functional aspects in the software product/application.
- Test Methodology: Functional testing can be performed using manual testing and automated functional testing approach or a mix of both, but manual testing is a must since testers have to make sure that all the necessary functionalities are implemented. On the other hand, Non-functional testing should be performed using automation tools as you need to test the system/application under different scenarios that could equate to real-world scenario. For example, increasing the load on all the CPU cores on which the application is running and checking the performance of the application under such a scenario.
- Which comes first?: Functional testing is normally performed before Non-functional tests are executed as necessary features should be ready before non-functional tests can be performed. Once the basic feature is implemented, developers have to make sure that the feature is scalable and would perform well under different scenarios (like varying CPU loads, varying Thermal conditions, and installation of the application in non-secure environment).
Some of the common Functional testing methodologies are: White-box testing, Black-box testing, Unit testing, Smoke testing, andCross-browser testing. Some of the common Non-Functional testing methodologies are Stress testing, Load testing, Reliability testing, and Security testing.
Functional Testing With Example
Imagine that you are building a B2C e-commerce application that does the following
- User Sign-up and sign-in
- List the products (in different categories) from the back-end inventory
- Facilitates buying of products via different online payment options
Now, for instance, we will consider some test factors to keep in mind while performing functional testing with example cases mentioned above.
- Screen testing: Allow a new user to sign-up using the sign-up page. A returning user should be able to log-in using existing credentials and Forgot Password feature should be in place in case the user does not remember the password. Performing the functional testing of an example scenario, the team should test the Sign-up and Sign-In page with different user/password combinations and record the output. They need to make sure that the navigation between screens is fast and features (listed in SRS) for this page are implemented.
- Feature testing: Based on the target market, the product team and development team would have identified the potential browsers and devices on which the application should be fully-functional with respect to cross-browser compatibility. To validate this scenario we perform browser compatibility testing where the application is tested on a combination of different devices (desktops, mobiles, tablets, etc.), Operating Systems (Windows, Linux, Android, iOS, KaiOS, etc.), and Browsers (Firefox, Chrome, Opera, etc.).
- Testing fallback mechanisms: Let us understand this from cross-browser testing point of perspective. In case your application caters to the target market where lot of people use older versions of browsers, your development team should make sure all the requirements are translated into implementation. Once they detect that the web-application is accessed using an unsupported or old version of a particular browser, there should be a fallback mechanism.
Here is a functional testing example, If the browser does not support Adobe Flash, the team should make sure that an alternate link to the video is provided so that the video can be played in that version of the browser. The test team that is doing manual/ automated functional testing for such a scenario should document the browser types and versions and test the features in those versions of browsers (and different device combinations).
By using a combination of both manual and automated functional testing, the team should test the application screens on different combinations or screen resolutions to deliver cross device accessibility. Based on inputs from the product team, you should prioritize testing on combinations which might be widely used by the target customer-segment. Cross-browser testing using manual approach is not highly scalable since it requires procurement of the necessary infrastructure required for testing and the test team cannot achieve good test coverage using that approach. There still are ways to speed up manual cross-browser testing.
Role of Automation in Functional Testing
As Functional testing involves verification of the functionalities implemented in the product, the test team has to make sure that every feature is tested. This also includes covering error scenarios and boundary conditions. Manual testing has its own set of limitations since you may need a bigger test team if your product has a huge number of features. This will increase the overall expenses. You may not be able to achieve best test coverage if you only use manual testing and your QA might reject your product if it does not meet the necessary coverage guidelines.
This is where Automated Functional testing can play a significant role as tests can be automated by writing scripts. These test automation scripts can be scaled to cover different test scenarios and improve overall test coverage and efficiency.
Automated Functional testing can come in handy in scenarios where your project is under extremely tight deadlines. It improves the overall productivity of the employees since they are engaged in more interesting work and the organization can save time and cost. Manual testing is done by testers who are experts in that domain and once they leave the organization, a new person has to shoulder his responsibilities and build the same level of competency.
Points to Consider Before Venturing Into Automated Functional Testing
Though automation testing has a huge number of advantages, the product team should make a decision on features that should be tested using manual testing and automation functional testing. There are certain scenarios where manual testing will have an upper hand, so your test team needs to figure out the role of automation in verification of product features and functionalities. In any case, you can never replace manual testing with automated functional testing since all test scenarios can never be automated (especially those where manual intervention is required).
A team working on automated functional testing should not work in a silo. In fact, the team should work in close coordination with the manual testing team and product team so that they can understand the scenarios where automated functional testing could be more instrumental. This is necessary to gain domain knowledge and also to build a more robust (and scalable) test suite by writing effective test cases.
In order to get the best out of automated functional testing, you should choose the right automation tools. The automation tool should be selected based on its scalability, past working experience within the team, cost, and ease of maintenance. Test reports provide important information about pass and fail scenarios and powerful bug reporting (along with customer support) should be a must-have while shortlisting the automation tool. Selecting the wrong automation tool can result in a loss of time which can affect the project timelines and unnecessary re-work could demotivate your test team. Hence, selecting the right and best test automation tool is vital for success of your test automation strategy.
Myths Associated With Functional testing
There is a sizeable audience that interchangeably addresses different testing mechanisms without knowing the difference between those testing methodologies. There is a lot of confusion around this topic and it is necessary that we break the myths associated with Functional testing.
Functional Testing Is the Same As System Testing
Functional testing is one form of system testing. Though functional testing is done to test the basic functionalities in the product/application, stress testing is more rigorous in nature. As a part of System testing, the tester does the following:
- Testing of the completely integrated product.
- Test cases focus not only on the design, but also the functionalities of the product/application.
- Functional Requirements Specification (FRS) and Software Requirements Specification (SRS) are considered the backbone of system testing.
- Test cases are also focused on security, performance, boundary conditions, power management, etc. The coverage of test cases heavily depends on the product under testing.
If you are building a hardware product, it also involves certification testing, compliance testing, and more. These tests are not included as a part of Functional testing. System testing also includes Stress testing and Load testing where the AUT (Application under Test) is tested against different system scenarios and its performance is measured using tools. Since Functional testing is only a subset of System testing, it is always performed before System testing.
Ideally, a product/web application that does not clear Functional test scenarios should not be validated for system testing.
End-To-End Testing Is a Part of Functional Testing
End-to-end testing is a form of testing where the product or application under test is tested from all angles from start to finish. In simple words, it is testing approach performed from a users perspective. Referring to the earlier example of B2C e-commerce application, while browsing the products on the site/app, there is a possibility that user may face throttling internet connectivity. What should the app display to the user or what happens when there is no internet connectivity?
End-to-end testing would take such an extreme scenario (including third-party dependencies) into consideration as it has a direct impact on the usability factor of the product.
On the other hand, if the product passes the Functional test, it means that the basic flow of the product is working fine and the development team can work on the further improvements of the product.
Always Aim For A 100% Automated Functional Testing
Automated Functional testing definitely has la ot of advantages over manual testing, the primary advantage being better test coverage and savings in terms of time and cost. However, there are many challenges for automated website testing. In test scenarios that cannot be automated or where automation might not be cost-effective, these test scenarios normally require manual intervention or require manual observation.
Referring to the earlier example of B2C e-commerce application, assume that the app is being tested on an Android Phone (that has Android P) and the device switches from Portrait mode to Landscape mode. There is a minor issue with the graphics where a certain part of the graphics does not appear properly. This cannot be automated since only a person who is involved in manual testing can observe such an issue.
It is important that the test team (involved with manual and automated functional testing) sit together and work on test scenarios that can be automated. The whole intention of automated functional testing is more test coverage and building a robust test suite. In ideal cases, a stable combination of manual and automated functional testing should be used to test the product against varied scenarios.
To summarize, the question is never about what test cases can be automated; but it is about how well you can fit automated functional testing in the entire test plan to ship a more stable end-product.
Functional testing should always be performed in the product development cycle to ensure that all the necessary/basic functionalities are working fine in the product. It should be done at an early stage of development so that bugs are logged at the right time, before they snowball into release blockers.
Depending on the nature of the software product, other types of testing – stress testing, cross browser compatibility testing, Integration testing, Compliance testing, White box testing, and more – aneed to be performed as well. With the growing number of devices, cross browser testing has become more important and it is a type of test that can no longer be ignored as it is co-related to the usability aspect of the product.
Published at DZone with permission of Himanshu Seth . See the original article here.
Opinions expressed by DZone contributors are their own.