Cognitive Bias in Tests: The Most Human Side of Testing

People make a huge number of decisions every day, from the most trivial and mechanical, such as the amount of milk that we put in our breakfast coffee or what shoes we wear, to the most complex, like the way to write an email to our superior or the best way to program the method of a class.

Obviously, we cannot spend the same amount of time solving every uncertainty that we encounter, or we would live in a permanent mental block that would prevent us from taking any action. Clearly, this circumstance would have led us to extinct as a species many thousands of years ago.

To avoid this "analysis paralysis" situation, our brain tries to be as efficient as possible every time we face a decision. For this reason, instead of carrying out an in-depth analysis and evaluating all possibilities, it is based on elements such as memory, lived experiences, what we have been told, what we have read, where we were born, education, our physical characteristics, gender, etc.

That is to say, in the majority of the occasions, our brain makes a decision almost instantaneously being based on the points previously mentioned and in a way practically imperceptible for us. This is what we might know as "impulse", "intuition", "first impression" or even "prejudice".

On the other hand, both the study of the brain and the way in which we behave as a society have been documented more than 100 cognitive biases, which describe a series of patterns that occur in people when making decisions.

Therefore, it is obvious that these "shortcuts" that our brain follows to evaluate situations are affected by these biases to a greater or lesser extent. In fact, denying that our own decisions may be biased is, in itself, a documented bias.

And How Does This Affect Testing?

The work of a tester is not based, or should not, solely on reporting the bugs of an application. The greatest contribution of value in the testing phases arises from the detection of "problems" in general and giving early feedback on the use of a system. That is, in addition to reporting errors, should be reported any case that may pose a problem or annoyance to users, such as issues arising from the interface, user experience, visualization of elements, layout, flow of actions, ease of use, cultural differences, problem derived from the location of users (time zones, formats, currencies) and accessibility among many others.

As we can see, many subjective, emotional, empathic and perception related factors come into play in the daily work of a tester. That is why denying that the result of their work will be affected by cognitive biases would be very naive on our part.

To give examples, surely we do not get the same errors and feedback when an application is evaluated by different testers such as: an adolescent girl, a person born in India, a pregnant woman, people sensitized with LGBT collectives, someone suffering from color blindness or a man of middle class with higher education.

The cultural environment, mentality, lived experiences, vital moment and physical characteristics of each one of those people will cause that some biases may have more weight than others and that therefore, their perception of the system they are testing is different and even not evaluate in the same way the importance of the same mistakes.

Next, I would like to list a series of biases that I have been encountering that may affect the testing phases and ways to mitigate each of them

Negativity Bias

Past negative experiences have too much weight when making decisions.

How It Manifests

Having detected a large number of bugs in previous phases or phases that have been put into production with many problems can make our vision of the current system unrealistic.

This pessimistic approach can make us place too much importance on less relevant bugs or unjustifiably lengthen the duration of the testing phase trying to cover the most unlikely cases.

Ways to Mitigate It

We must always keep in mind both our accumulated general experience as testers as well as the particular problems detected in the project on which we perform the tests. Obviously, we cannot forget everything we've learned in each testing phase, but we must be aware that the systems are modified and evolve as well as the needs of the users who use it.

In short, as testers, we should not be stoppers or bottlenecks in each deployment of a new version.

Strategies

• Risk analysis should be updated in each new release
• Each bug must be categorized with its severity and degree of impact
• Evolution/trend analysis of bugs reported and solved over time by severity.
• Having a stable test environment will increase the degree of confidence in the testing phases, which will most likely favor the stability of the production system.
• Developing a suite of automatic checks, at least for smoke testing, will make it possible to verify the functioning of the core functionality at all times.
• Implementing a system of feature flags in our system will allow deactivating functionalities in case they are causing serious problems in production and we increase the flexibility of the deployments and the response to failures.

Confirmation Bias

We tend to look for information or cases that reinforce our point of view

How It Manifests

This is, without a doubt, the most human bias of all. The possibility of being wrong gives us insecurity. Having positive reinforcements that reinforce our beliefs makes us feel better.

It is possible that due to our past experiences in other similar projects, personal tastes about interface design or the personal relationships that we have established within the work team, we can approach the conditioned testing tasks. This predisposition could give situations such as:

• "I do not like the design chosen for the portal so I tend to report a greater number of bugs related to the user experience instead of focusing on functional aspects."
• "I know the development team and I have more confidence in the performance of some people than in others. This leads me to focus on certain parts of the system while ignoring others."
• "I do not agree with a technical solution taken by what I do an overstrain in a type of tests leaving aside other relevant aspects in the system."

These examples illustrate how the search for points that reinforce our opinion can significantly affect the reliability or suitability of the test set executed.

Ways to Mitigate It

We could summarize it with the phrase of Edwards Deming: "Without data, you're just another person with an opinion." That is, during the testing phase, we should try to follow a focus as objectively as possible and try to base our decisions on measurable values and not on personal opinions.

Strategies

• By means of A/B testing, we can measure the response of the users to change the design and compare it with other versions.
• Add phases of beta testing that are performed by people outside the project would help the tests are not "tainted" by cases such as those described above.
• Establishing cross-tests between team members can cause a greater part of functionality to be covered
• How others are doing it? It is always interesting to have other reference applications have resolved certain technical or user experience issues.

Anchoring Effect

Relying too much on the first information we receive regarding a topic.

How It Manifests

This type of human behavior is used daily by media and opinion groups. The phenomenon of "misinformation" also takes advantage of this aspect, since we tend to consider the first information that comes to us with respect to a subject of which we have no prior knowledge as more certain.

In the field of testing software projects, if the information does not flow properly to the testers, it is very possible that they give more weight to the information received in initial iterations.

It could be the case that a tester arrives at a document that specifies the full functionality of a system and months later receives another updated with important changes. In this case, even if the tester discards the first document it is very possible that doubts arise about points that were described in it and that would seem appropriate and now appear modified without further clarification. It is likely that you will ask the reasons that have led to these modifications and if the new changes are aligned with the real needs of the users/customers. In short, it is something that can affect the tests.

Ways to Mitigate It

It is convenient that the people in charge of the tests are part of the development team or at least, are informed of the decisions that are taken that may affect the functionality.

For this, approximations could be followed, such as:

• A shift-left testing approach will make the testers involved from the early phases of the project, be aware of the changes carried out in the project and report problems as soon as possible.
• No silos: Having a testing department or QA disconnected from the projects means that the knowledge of the context of the project on the part of the people who carry out the tests is less.
• The advisable that testers have the opportunity to have direct communication with the client or decision maker to share doubts and have first-hand information.

Bandwagon Effect

Give too much weight and get carried away by the opinion of the people around us

How It Manifests

Typically, the number of testers in a project is much lower than the number of people who develop. The fact of being a profile that is in a minority can cause group pressure to affect their decisions and ways of testing.

This could occur in situations like:

• The whole group considers valid a technical solution or design on which the tester raises certain doubts. There is a risk that these will not be treated correctly by the group when considering them as a minority opinion
• The team assumes that part of the application has a high degree of robustness and that therefore it is not necessary to perform tests on that part. The tester can be influenced by this majority opinion and not complete tests of the system.
• The team opts because the tester only performs manual tests and does not assume automation work or code revision.

Ways to Mitigate It

The main strategies to minimize the effects of this bias could be the following:

• The tester must be one more component of the team. Establishing classes and considering them as a second-class member will negatively impact their work and the results of the tests.
• The tester must always have a critical thinking.
• Ask constant questions, do not take anything for granted.
• As noted above, that the tester has direct communication with client or decision maker will minimize the effect of group pressure since it will always have the option to contrast opinions and points of view with the main stakeholders.

Sunk Cost Fallacy

We do not address new actions or changes in order not to lose what has already been invested in a certain way of doing things.

How It Manifests

In certain situations, people present many difficulties when evaluating the cost/benefit of a specific action. In many occasions, we give too much importance to the current way of doing things and its "accumulated cost." The response of "we have always done it this way, it would cost us too much to change now" is a clear indicator that we may be falling into this kind of bias.

In the case of testing, there could be cases like:

• Not changing or updating the tools used for the control and monitoring of test cases and bugs with more efficient ones
• Not addressing changes in the tools or libraries used in the automation of tests for the cost of complete refactoring
• Failing to analyze the effectiveness of the testing process within the SDLC.
• Failing to evaluate the suitability of continuous integration tools or strategies that favor a DevOps approach.

Ways to Mitigate It

In this, we should also make sure our decisions are based on objective data.

We could carry out strategies such as:

• Objectively analyze of the ROI of the solution implemented today and compare it with possible alternatives.
• Establish improvement hypotheses, test them through controlled concept tests and evaluate the results obtained objectively.
• Establish frameworks of good practices that favor the maintainability of current systems.
• Certainly, it is not always possible to implement complete solutions from scratch, however recommendable they may be. Therefore, continuous refactoring of current systems will favor that they do not become obsolete and provide value for a longer time.

Inattentional Blindness

Obvious information, at first glance, goes unnoticed.

How It Manifests

At the time of testing, this type of "blindness" can occur in two different situations: fatigue by repetition or tunnel vision when putting too much focus on very specific aspects of the system.

An example of the first could be the following: after performing several cycles of tests on the same functionality or part of the system, our attention level decreases and we involuntarily begin to overlook details that would attract the attention of any other person. This effect usually occurs when carrying out regression tests of the system since they focus on executing the same tests periodically.

The second case is something more dangerous from the point of view of testing since it is not derived from fatigue but from the method of executing the tests. That is to say, the person is putting all his attention on carrying out the tests but focusing too much on very specific aspects, which can make him unaware of other important faults that are obvious.

This situation usually occurs when we start from an excessively exhaustive and detailed test plan in which you have to worry only about executing exactly what is specified there.

Ways to Mitigate It

It is not easy to maintain the same degree of attention to a repetitive task, so we must choose to apply some extra focus when performing the tests

• Performing peer reviews or pair testing on the functionality developed or even on the implemented code can cause many errors to be detected in early phases and during a peer review new scenarios and casuistics not initially planned arise
• Implementing automated checks that verify the recurring core functionality will free the tester of this task allowing you to focus on an exploratory approach
• A cross testing and rotating within the team will reduce the monotony of a person always test the same parts of the system.
• Avoid script testing in favor of the exploratory: Make the test cases do not establish too detailed steps to prevent the person executing them from focusing too much on the concrete steps and not see beyond.

Conclusion

Biases are part of our human nature so we must learn to live with them both in personal life and at a professional level.

As testers, we must assume that the greatest contribution of value to our work is given when we are involved in the projects from the beginning, we are part of the team, and we have direct communication with the main stakeholders and we can express our concerns freely and without pressure.

On the other hand, we have to be clear that many of our day-to-day tasks, such as assigning the severity of a bug or evaluating the suitability of a certain interface, have a clear subjective character. That is why we must use all the tools at our disposal to minimize the effect of biases and that the quality of our work is not affected.