When a QA Met Reality Technology: A Strategy to Test a Reality-Based Application

Reality technologies, like augmented reality, virtual reality, and mixed reality technologies are amongst the top technologies that business is looking forward; and it is a phenomenal task to find an expert to test app developed based on these technologies.

There are a lot of platforms and tools available for reality-based application development, but what we lack is a standardized reality-based application testing strategy. Testing is performed on reality-based applications which differs a lot from the “traditional” testing which is a health hazard due to multi-dimensional environments and simulations that use vision, hearing and touch to interact with the artificial world.

As we know with the given immersive component of these technologies, lab testing and automation are completely ineffective. Therefore, a robust testing strategy is required to understand the current parameters for testing; a shift from traditional testing to immersive testing; and best practices to test are the need of the hour.

You may also like: Why Is Software Testing and QA Important for Any Business?

After testing numerous clients’ reality-based applications and our own internal applications by the CoE (center of excellence), we observed that it is impossible to anticipate and cover all the situations that can be experienced by end-users with a device and an application.

Traditional testing strategies are not working well for reality-based application – testers face new challenges in their daily work, while their experience and structured methods are becoming (to a certain degree) ineffective.

The world around us leveraging the reality-based technology

Different Types of Reality Technology

Augmented Reality — Environment where the user is made to interact with the objects living in the real world. It is greatly supported by the capability of a computer to generate information across multiple sensors including visual, auditory taking the effects to sound text, etc. to greatly enhance the user real-world experience

Virtual Reality — To create an identical simulated environment with the assistance of a computer for 3d effects. Users are made to submerge in the 3d environment. Senses such as the vision of the eye, smell, touch, etc are imitated to create the artificial world

Mixed Reality — It is the combination of both Augmented and Virtual Reality where both physical and artificial objects co-exist simultaneously rather existing individually Physical objects are made to communicate with the objects of artificial world with the help of immersive technology.

Different Types of Augmented Reality

Markerless AR — The user does not require an anchor to the real world. Suppose you want to place virtual utensils and household equipment’s into your kitchen. Users can try different combinations of objects, styles, locations. User needs to decide the location of placing the virtual objects. This means all objects will appear to float in the air

Marker AR — The user requires an anchor to the real world. Suppose a user is reading a book and wants to go to the page. At first, the device requires to know the number of pages from the camera view. Based on this, information is taken, and animation is started.

Location-based AR — The virtual imaginative world is in a physical location. It combines augmented reality in a physical location. Suppose you are on an unknown road in any location and want to go to a friend’s house. You are using a physical camera to get the house number on the road. This will help to reach a friend’s house

Different Type of Virtual Reality

• Fully immersive — Most realistic virtual environment experience, user totally immersed incomplete visual, sound and high-resolution content.
• Semi-immersive — Users with a partially virtual environment to interact with the application. This type of VR mainly used for tutorial/educational purposes.
• Non-immersive — The Best example is video games for non-immersive VR to detect user motions and respond to screen.

Challenges of Testing for a Reality-Based Application

• Expensive testing.
• Require more space to test.
• No standard interfaces.
• Compatibility issues.
• Motion testing.
• Test automation.
• Testing time is higher than traditional testing.
• Non-availability of performing accessibility testing for the reality-based application.
• Most important it is difficult to pull off AR/VR based applications in CI/CD.
• No or minimal simulator, hardware and software support for testing.

Testing of Reality-Based Application

The complexity of development, the involvement of different technology in developing reality-based applications makes it infeasible to get the right quality of an application. Instead, applications are developed using iterative processes and tested parallelly and after development more as user experience and specification, testing is taking more time for immersive testing for these technologies.

To ensure the quality of product, software or hardware one needs to have proper testing plans and strategies with a focused milestone to achieve the acceptance criteria and key performance indicators.

The first phase starts with the unit testing of the VR application. Unit testing is testing pieces of code in order to test whether code meets expected results of intended functionality. Take the example of any method UpdateNameAllowsLettersToBeAddedToName.

It makes it clear to what is being tested. A test suite is prepared to contain all unit tests. If any test fails, then the whole suite fails. Unit Testing utilizes the NUnit framework with Test Runner. A unit test is written in Test RUNNER and made to run on that.

Integration Testing of how modules of code work together. Example-You have a game that kills Sea Monsters while you are traveling on the ship. It was likely to involve a physics engine, event tracker which keep track of all events. Testing multiple modules of code working together.

Usability Testing — This is a testing in which the user's compatibility and easiness of interaction with VR are measured. These two devices are requires-Headset and mobile phone. So, a short session is arranged for a sample of the audience. They interact with the VR app in a predefined environment. Their reactions are examined by experts.

Heuristic testing — It is a most economical methodology to test VR application because it follows a standard design. It is mostly performed by one expert only. It traps a high ratio of usability issues. It is generally done before usability. In short, there are few principles which are followed-(a)-Synchronization of user interaction with the real world, Interaction, natural actions while exploring, synchronization of action and representation, the output of interactions as according to the law of physics, Navigation support, etc

Performance testing — It is to be done to test device overheating, battery consumption while playing, device charging. Performance testing is done by measuring the non-intrusive comparison of the changes in virtual space using a robot. Some of the emerging tools are OptoFidelity, this tool helps performance measurement with reality-based technologies

Compatibility testing — This testing is done VR application can use different devices like Desktop monitor, Virtual Headset, Telepresence. A desktop monitor is used in the medical industry in order to study the malpractices of industry. Telepresence is used in highly dangerous areas like in Bomb disposal etc where the user works and controls remotely.

Focus groups — This is done to study and interview people from different demography regarding their experience while interacting with VR.

UI testing — Different scenarios are tested like the real world does not disturb users from interacting with VR, broken flows, Moving Images, View in 360 degrees, etc.

Security testing of VR applications-Copywrites issues, privacy, compliances, data issues to be covered.

Immersive testing — This is done to ensure that hardware components are fully integrated and work properly with the software for the intended functionality to be done. It helps in successful visualization. E.g. Display of VR headset to be compatible with the intended functionality.

Accessibility testing — VR can have serious effects on health impacts like headaches, dizziness in the eye, etc. So, it must be properly tested to counter the aftereffect of VR

Crowd testing — Most important testing for reality-based application, as the quality depends on specification and user experience and to get an idea about how the user will experience need to do crowd testing.

Motion sickness testing — This testing is difficult and harmful to health and different from traditional testing.

Methods for Testing Motion Sickness

The frame of reference — It adds a Visual Frame of reference for image. It assists in understanding the impact due to contradictory sensor conflict between visual and vestibular sensations. E.g. Once the research was conducted among some participants to understand the effect of with rest frames and without rest frames, on a roller coaster, a grid of two horizontal and two vertical white lines was used as a rest frame.

Then the subjects were filled with Simulator Sickness Questionnaire. The result was that Rest frame proved enough to counter the effect of motion sickness

Visible Path — It’s a method that is understood as placing waypoints along a specified path in a virtual environment. Thus, the user felt and anticipate movement in virtual reality. In this method, the user is not in control. Rather they are in a passive mode where they can only anticipate the movement and they are not in control.

Field of view — An experiment was conducted once where a group of participants was exposed to a simulation of 50 minutes. There were two kinds of FOV-Internal and External. Here External mean-screen size and distance from it and Internal means-angle of a camera in-game. Motion sickness occurred when Internal and External FOV both agreed.

Regression Analysis — The regression model can be used for testing and predicting motion sickness. The sensorimotor (sensory-motor coupling of sensors to motor) data is collected from the sample for the age of people. It is then exposed to VR in a zero-gravity space simulator. Using the regression model, it shows how much Motion Sickness participant has experienced

Testing Methodologies

With several testing methodologies in the software development cycle, two methodologies will work better for reality-based applications.

Extreme Programming

Development and testing are done in parallel. Test-driven development approach is followed where a test is written before and it is ensured that these tests are get failed and then only write code to pass. This approach is to test the code in a testable manner.

Unified Process Model

Shift left approach testing done beginning test cycle from the architecture design phase to deployment. Here, regression testing done unlike extreme programming as code added in a codebase regression suit executed so that to identify and an additional feature is causing any error in the application.

Scope for usability testing for augmented reality and virtual reality-based applications.

• Visibility of the application.
• Match between the system and the real world.
• User control and freedom.
• Consistency and standard.
• Error preventions.
• Recognition rather than recall.
• Flexibility and efficiency of the use.
• Aesthetic and minimalist design.
• Help users recognize, diagnose and recover from an error.
• Help and documentation.

Key performance indicators for reality-based application.

• Accuracy — How accurate is your application with the positions and virtual content
• Cost-effectiveness — Evaluate overall cost for application development, environment, tools, platform, etc.
• Satisfaction — Interaction is an important aspect of reality-based applications, a measure of interaction score with the user.
• Efficient — How efficient your system is with given content, environment, etc.

Some tools used in reality-based technology.

• AirTest.
• Poco.
• Unity.
• SteamVR Performance tool.
• 360° EYETRACKER.

Conclusion

Although, testing new technologies and applications is always exciting, seeing their own strategies and tools fail in testing can be frustrating even for well-seasoned testers. I aim to provide testers with a better understanding and help them to apply their critical thinking to deal with uncertainty in their test objects.

In addition, understanding the business needs for opportunities to optimize and innovate, and develop a quality product. The fact is that the realm of reality technology is very different, therefore it needs­­­­­­ specialist for testing these technologies. Data needs to be collected to understand the user goals, application goals, number of participants, statistical and graphical types of evaluations.

Further Reading

How to QA Test Software That Uses AI and Machine Learning

Game Testing: Emerging Trends and the QA Role

Top 5 Myths About QA Testing