Testing for Developers: Testing Types and Definitions

In the series, we will define the basic terms that every developer needs to know about testing. The purpose is to give all team members a shared understanding of the fundamental terminology of quality assurance and all related processes. Later, this will improve communication and reviews quality. It will further increase the testing capabilities of each member. In this part, we will talk about the different testing types.

As part of the professional services we provide at Bellatrix, we consult companies and help them to improve their QA process and set up an automated testing infrastructure. After the initial review process and giving improvement recommendations, for some companies we need to hire new talents that can help the company to scale-up the solutions we provided. This was part of the training we did for a company we consulted so that we educate all of their developers.

Definitions

System Testing: the process of testing an integrated system to verify that it meets specified requirements.

Component Testing: the testing of individual software components.

Component: a minimal software item that can be tested in isolation.

Integration Testing: testing performed to expose defects in the interfaces and in the interactions between integrated components or systems.

Functional Testing: testing based on an analysis of the specification of the functionality of a component or system.

How Well Does the Product Behave?

Non-functional Testing: testing the attributes of a component or system that do not relate to functionality, e.g. reliability, efficiency, usability, maintainability and portability.

Acceptance Testing: formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system.

Acceptance Criteria: the exit criteria that a component or system must satisfy in order to be accepted by a user, customer, or other authorized entity.

Off-the-shelf Software: a software product that is developed for the general market, i.e. for a large number of customers, and that is delivered to many customers in identical format.

Acceptance Testing

The focus of acceptance testing is on the customer's perspective and judgment. The customer is actually involved, and it's performed in a customer's environment. Finding defects is not the main focus.

For the integration of a commercial off-the-shelf (COTS) software product into a system, a purchaser may perform only integration testing at the system level and at a later stage acceptance testing.

Operational Testing

• Disaster recovery
• Testing of backup/restore
• User management
• Maintenance tasks
• Security vulnerabilities

Alpha Testing

Takes place at the developer's site as a cross-section of potential users and members of the developer's organization are invited. Developers observe the users and note problems.

Beta Testing

Beta testing send the system to a cross-section of users who install it and use it under real-world working conditions. The users send records of incidents with the system to the development organization.

Functional Testing

The function of a system (or component) is what it does.

• Also referred to as black-box testing
• May be performed at all levels
• Can be done from two perspectives:
  • Requirements-based
    • Uses a specification of the functional requirements
  • Business-process-based
    • Uses knowledge of the business processes
• Use cases are a very useful basis for test cases

Non-Functional Testing

How well or with what quality the system should carry out its function.

• Performed at all test levels
• Includes, but is not limited to:
  • Performance testing
  • Load testing
  • Stress testing
  • Usability testing
  • Maintainability testing
  • Reliability testing
  • Portability testing

Non-Functional Quality Characteristics

Reliability: the ability of the software product to perform its required functions under stated conditions for a specified period of time, or for a specified number of operations.

Reliability Testing: the process of testing to determine the reliability of a software product.

Usability: divided into the sub-characteristics understandability, learnability, operability, attractiveness and compliance.

Usability Testing: testing to determine the extent to which the software product is understood, easy to learn, easy to operate, and attractive to the users under specified conditions.

Efficiency: the capability of the software product to provide appropriate performance, relative to the amount of resources used under stated conditions.

Efficiency Testing: the process of testing to determine the efficiency of a software product.

Maintainability: the ease with which a software product can be modified to correct defects, modified to meet new requirements, modified to make future maintenance easier, or adapted to a changed environment.

Maintainability Testing: the process of testing to determine the maintainability of a software product.

Portability: the ease with which the software product can be transferred from one hardware or software environment to another.

Portability Testing: the process of testing to determine the portability of a software product.

Re-testing (Confirmation Testing)

After a defect is detected and fixed, the software should be re-tested. To confirm that the original defect has been successfully removed. This is called confirmation.

Regression Testing

• Adaptive maintenance (Planned)
• Product is adapted to new operational conditions
• Corrective maintenance (Ad-hoc)
• Defects being eliminated

• Anything new or changed should be tested
• Testing is needed even if only the environment is changed
• Regression testing is required
• The rest of the software should be tested for side effects
• The scope is related to the risk of the change, the size of the existing system and to the size of the change
• May be done at any or all test levels and for any or all test types
• Key activity is impact analysis

Maintenance Testing

Maintenance Testing: testing the changes to an operational system or the impact of a changed environment to an operational system. The process of testing to determine the maintainability of a software product.

• Planned modifications
• Enhancement modifications (adapting software to the user's wishes, for instance by supplying new functions or enhancing performance)
• Changes of the environment (planned operating system or database upgrades)
• Corrective and emergency changes
• Migration (from one platform to another)
• Should include the operational test of the new environment
• Retirement of a system
• May include the testing of data migration or archiving if long data-retention periods are required

Main Тypes

• Adaptive maintenance (Planned)
• Product is adapted to new operational conditions
• Corrective maintenance (Ad-hoc)
• Defects being eliminated

Testing After Maintenance

Impact Analysis: the assessment of change to the layers of development documentation, test documentation and components, in order to implement a given change to specified requirements.

Triggers for Maintenance Testing

Static Testing

Static Testing: testing of a component or system at specification or implementation level without execution of that software, e.g. reviews or static code analysis.

• Can only be applied to software code
• Detect defects and to determine quality attributes of the code
• Dynamic testing — testing that involves the execution of the software of a component or system.

Static Analysis: analysis of software artifacts, e.g. requirements or code, carried out without execution of these software artifacts.

Early detection of defects prior to test execution. Early warning about suspicious aspects of the code or design by the calculation of metrics, such as high complexity measure. Identification of defects not easily found by dynamic testing. Improved maintainability of code and design. Prevention of defects, as lessons learned in development.

Dynamic Тesting

Software is executed using a set of input values and its output is then examined and compared to what is expected.

Dynamic testing and static testing are complementary methods, as they tend to find different types of defects effectively and efficiently.