Navigating the Challenges of Software Automated Testing of a Moving Robot

In my previous article, I explained how to automate the E2E home robot using April Tag technology. There is a big challenge in the automation of a home robot or any robot. How do you set up the device? In the world of devices, mobile phones, and mobile apps, the devices are connected to a Host Machine using USB, and the devices will be consistently connected to a power source. Now, testing a robot introduces a unique challenge of how this can be tested when the robot is moving. It cannot be physically wired to the host machine. How do we validate an out-of-box experience? Don't worry. I will explain this in the current post.

Complexity and Challenges

As I briefed you about the various challenges with respect to the setup phase during the automated testing of robots, let me dive deeper into more challenges. 

• The robot is moving, and physically wiring to a host machine is not possible. It may be possible for some component-level testing but not possible for E2E testing. 
• An out-of-box experience is when the device is not connected to wifi. How can the host machine interact with the device when the device is brand new? This is a very common scenario in day to day testing of a robot.
• Recover the robot in case of any error or exception. The main goal of automated testing is to surface the potential software and hardware issues with the robot. In case of an issue, how do we report and recover the device?
• The robot will drain the battery, and we need a reliable source to power the robot's battery. 
• Scaling the same setup to multiple labs and home environments. This is very critical since we cannot sign off or test the robot in one single environment. 

Let's see how to solve these problems in chronological order. 

Pre-Setup Robot Using Raspberry Pi

This is the most important problem to solve. Consider the robot is in a test automation lab in some location. There is a host machine that is connected to your company's internal network for sending and receiving commands and accessing various source codes and internal tools and infrastructure. We will have a Raspberry Pi connected to the device, and we will have a rest service running on the Raspberry Pi to communicate with the host machine and the device. A pictorial representation of the setup is below.

Solving the Out-Of-Box Experience Use Case

Now we see how we can pre-set up the test environment with the help of Raspberry Pi. Now, let's see how we can solve the out-of-box experience where the device is not connected to wifi. 

This is where the restful service on the Raspberry Pi helps. The restful service should contain certain endpoints, like the following: 

• Flash device
• Complete OOBE
• Set up wifi, etc.
• Get DUT IP address

Now, it becomes simple for us to connect to the device wirelessly, and the device is completely set up for our testing. 

Recover the Robot in Case of Any Error or Exception

This is a very common scenario to happen. Do not be overwhelmed or frustrated by these kinds of issues. This is when we have to use the low-level components of the device to drive the device back to its original position. For example, as I spoke about various software stacks in the previous article, we need to access the platform or the mobility layer to drive the robot back to its original position. This is the trickiest and the most challenging thing to do, so use other techniques like April Tag or other external systems to drive the device back. This leads me to the next problem to get the device back to the charging dock to prepare the device for the next test run.

Scaling to Multiple Test Environments

Use the low-level commands, which will be more reliable to get the device back to the charging dock. This is very helpful to use the low-level layers because using any move or navigation command from the top layer will seem to have not a good effect. Failure rates are higher when we use the top layer vs the efficiency is increased when we drive from the platform or the mobility layer. 

Tips to scale the set-up to multiple test environments.

• Keep your setup simple. Do not overcomplicate or over-engineer the solutions
• Have a deployable Raspberry Pi image so that any new Raspberry Pi can be easily loaded. 
• Keep the test code in remote repositories. Moving them locally cannot scale. 
• Finally, robotic automation is not simple and requires lots of design work and needs to be done by considering other options in the market. There is no one-size-fits-all. I recommend doing a Proof of Concept before jumping on to the final solution.

Alternative Solution

Owing to the nature and complexity of the problem we are trying to solve, the easy question that comes to the top of our mind is, "Are there any alternative solutions?" The short answer is yes. We could effectively use emulator/simulator-based testing to cover most of the use cases we described, but there's no alternative to real device testing.