There seem to be a few public software testing cups and competitions. They all seem like jolly good fun. The closest I’ve come to taking part is when I attend public bug bashes. I’ve been to a few in the past, so I’ll list some pro tips and experiences.
Adopt a Different Belief Model for Testing
One thing I particularly enjoy about the competitions and bug bashes is that testing is very hard to judge as a competitive sport. I love the irony that we fall back on the thing that we are all told not to do in real testing, which is to judge the testing by the bugs that we raise. We all know that testing isn’t all about bugs, that the value of testing is in the information, not the bugs, etc. However, this is not so in a bug bash [insert evil laugh here].
The first set of pro-tips embody the following.
- This is a competition, and that means that there are winners and losers.
- The losers are going to be the people who hang onto beliefs about what testing means.
- “Bug bash” doesn’t imply “testing;” it means, “hit the software hard and fast and find as many bugs as you can.”
- Think like a bargain basement testing warehouse.
- Pile em high, sell 'em cheap.
- The winners will be the people who can:
- Find bugs fast.
- Document bugs into the tracking system (because you’re going to be judged on the bugs you raise in the system, not those in your notebook).
- Have bugs that are well-supported with evidence: steps to re-trigger, screenshots, succinct and clear descriptions.
How to Also Go Deep
Now, you don’t have to embody that. You can go into the competition with an attitude that you're going to learn, that you're going to test properly, and that you're going to value quality over quantity.
I’ve done that. It can be a good personal exercise. I was involved in one bug bash on a test environment where I didn’t race to the bottom and pick up everything I saw. Instead, I scouted around, observed, found some good risks, followed the risks to their functional source, infiltrated the functionality, went into deep cover, and found a massive security issue that gave me access to all the customer details on the production system (and nobody cared [insert laugh of incredulity here]).
I learned a lot doing that, which helped my personal test approach. I also learned a lot about the company in question.
This can be a good personal strategy to adopt in a bug bash. However, when you adopt this attitude, don’t join a team that wants to win.
Often, when I follow this strategy, I end up finding big bugs, so it is generally the strategy I use for private bug bashes when we conduct them internally. The hard part about this strategy is that you often stumble over bugs on the way. In a team, a good strategy can be to pass those on to other teammates while you continue on your deep infiltration mission.
Based on experience, here are a few “don’ts,” assuming you want to win (if you want to learn, then these might be suitable tactics, but you won’t be on the winning team using these).
- Don’t try to use tools you’ve never used before.
- You will lose a lot of time learning and configuring the tools.
- Do use tools you already know how to use.
- Don’t assume you’ll configure your machine when you get there.
- Make sure you know how to configure your machine prior to the event for different WiFi networks, proxies, and VPNs.
- Connect your mobile decide to your laptop via a cable, WiFi, or bluetooth.
- If you normally use your laptop with an external monitor, mouse, and keyboard, make sure it works without those connected.
- Don’t check you’ve got the tools you need when you get there.
- That USB stick with all those portable tools that you used last year? Check that it works and has the tools you need.
All of the above relate to a lack of approach planning, so don’t not plan.
You won’t have all the information that you need to create a detailed plan, but you should have enough information to think through the problem.
- If you’re testing a web app, then:
- Make sure your browsers are up to date.
- Have a range of browsers available to test on.
- Know how to use the dev tools for the browsers that you’re going to use.
- Make sure you can configure your proxies with the browers.
- Make sure you have the proxy https certificates installed on your laptop and browser.
- Make sure you know how to use the proxy.
- Make sure you know how to toggle between proxy and without proxy.
- Have link checkers available.
- Have a mix of local and remote tools.
- For example, if your link checker is cloud-based but the test environment is locked to a local IP range or WiFi network, then you better know how to create a tunnel to allow the remote client to connect or have a local tool that will work so long as your machine can connect.
- Know how to use your proxy functions in unconventional ways (i.e., fuzzers for automating over a massive data set rather than writing code to automate the system).
- If you’re testing a device-based app, then:
- Make sure you know how to toggle developer mode on your device.
- Make sure you know how to sideload apps onto your device (just in case).
- This also allows you to rip the app off the device for local decompilation and binary artifact investigation.
- Make sure you know how to take screenshots.
- If you choose to do all your testing and bug reporting from the device itself, then I recommend being able to hook up a keyboard to it.
- If you can, have the ability to take a movie.
- I recommend using your device in conjunction with a laptop.
- Make sure you can transfer screenshots, files, etc. from the device to the laptop.
- Connect the device to the laptop.
- Show the screen of the device on the laptop.
- Take screenshots on the laptop.
- Capture movies in parallel on the laptop.
- Use the laptop rather than the mobile device to raise defects.
- Have the essentials:
- Screenshot tools.
- Screen movie capture tools.
- Text editors.
- File compare tools.
- Data generators, i.e., counterstrings.
- Learn to touch-type quickly prior to the event.
- Use tools that can statically analyze as you test (i.e., static analysis plugins on the tools to check traffic for issues),
- Scan HTTP traffic logs looking for 4xx and 5xx status codes.
- Use tools that can work in parallel to you, such as:
- Link checkers that spider and scan the site.
- Spiders that pull comments from HTML pages.
- Standard compliance tools that spider sites.
- Binary analysis tools for common vulnerabilities.
Have tools that can work in parallel, alert you to issue that you might not notice, and are a good way to pick up low-hanging bugs for free:
- Spelling errors on pages.
- Missing images.
- Missing includefiles.
- Server errors on ajax requests.
- Broken links.
If you aim to win, then you are going to do more in a short period of time than you have ever done before, and you’re going to use a range of tools. Make sure that you start up all the tools you use before you go and that they are up-to-date and functional.
Work as a Team
This is essential.
It is way too easy for everyone on a team to hit the same area and then follow someone else into that area when they find a problem.
Working in pairs can be useful, particularly if it is a tricky area that neither of you understands fully. Also, your partner might observe something that you’ve missed.
I’ve found that doing the following can be helpful:
- Sending everyone out on their own (to different areas of the app or to the same area, with different observational and attack focuses).
- Concentrating on each person's strengths.
- "You like to do accessibility."
- "You like to get technical."
- "You know how to use tool X, so you can follow behind Bob when he does Y."
A more organized wide dispersal coverage in parallel seems to work better than a self-organizing swarm if you want to win.
Also, the team needs to communicate.
- When you find a problem, communicate it to your team, not so that you can swarm and see it, but more so that you can generalize to a root cause or pathology and see if a similar fault exists in the other areas of the app that other people are looking at.
- Sometimes, it is useful for someone to find a problem, then pass on their notes to another team member to raise the defect while they continue to sweep the area for further issues when they are in the groove.
- Communicate quietly.
- The other teams (for the period of time that the competition is on) are your enemies.
- They can, and will, steal your intel if you shout it out (you might not be able to exploit the intel, but they might).
- They can prioritize. If they know that you are working in an area of the app, they could hit it hard and fast to find low-hanging fruit that you waste time raising when they have already raised it.
- They could steal your defects if you are slow to raise them, but discuss them loudly and carelessly.
- Remember loose lips sink ships.
It depends on the rules for the bug bash, but if the bug bash has a rule that the “first to find the defect claims the defect and duplicates don’t count,” then there are a few more things to consider.
Get Efficient in Raising Defects
This means that you need to be efficient in raising defects. I recommend copying and pasting into a defect tracking system rather than writing it directly into the browser. Why?
- Defect tracking systems can crash.
- You can create your own template to copy and paste commons stuff (i.e., priority, environment, version) into mandatory fields with no default.
- You have a local record that you can scan without having to use their system.
- If it crashes or messes up when you raise the defect, you just create a new defect and paste it from your notes rather than typing again.
Use low-hanging fruit to demoralize the enemy.
Early on in the competition, have at least one person monitoring your parallel tools and reporting spelling errors or broken links and images. Raise these into the bug tracking system quickly and make a big deal of it. Then, it looks like your team is raising bug after bug after bug. This can panic the other teams into losing sight of their strategy and following you into the areas that you have already cleared.
When you do raise bugs, don’t announce the content; just celebrate loudly the raising. You might get lucky and the other team might waste time going into the defect system to read your reports.
If you are lucky, then periodically, the organizers will announce how many bugs have been raised. If you have dominated the bug count, even with low severity issues, you can gain a psychological edge on your opponents, aided for free by the Progress Propaganda Department of the organizers.
A public bug bash is not your normal testing exercise. Particularly if you want to win.
- Target bugs, pile 'em high, and pile 'em fast.
- Get ready before you go, install what you think you’ll need, make sure you know how to use it, and make sure it all works before you go.
- Harness parallelism.
- Demoralize your opponents by announcing defects.
- Organize as a team.
- Hit the system hard and hit it fast.
- Report quickly and well.