Gamifying Product Testing – Learn How Cask Did It
Gamifying Product Testing – Learn How Cask Did It
Fix some bugs, earn some points. This blog post gives you a run-down of Cask's model for their recent hackathon event, and why it was such a success!
Join the DZone community and get the full member experience.Join For Free
Hortonworks Sandbox for HDP and HDF is your chance to get started on learning, developing, testing and trying out new features. Each download comes preconfigured with interactive tutorials, sample data and developments from the Apache community.
Last week at Cask was especially abuzz with activity. Why, you ask?
Over the last couple of months, the engineering team at Cask had been focusing on building a brand new version of our flagship product, the Cask Data Application Platform (CDAP) version 3.5, which will include many new engaging and useful features like enterprise-grade security capabilities, new features in Cask Hydrator, data usage analytics, and much more.
Customarily, we dogfood our own software releases by testing and running internal hackathons for the entire engineering team after feature implementation is complete. This happens before every GA release at Cask, and CDAP 3.5 is no different.
Only this time, we decided to add a twist! This blog post gives you a run-down of our model for the special event, and why it was such a success!
The theme for the 3.5 hackathon was gamifying the process of finding bugs – #BugSmash. Our goal was to find and fix the maximum number of bugs, deliver a strong stable release, and bring out the fun and competitive spirit in everyone! We modeled our #BugSmash after Tour de France:
- Winning team decided per day based on highest points tally as of 5pm
- The overall winner was based on combined points for all the days.
Setting It Up
Every summer at Cask is particularly lively because our engineering interns join us! For this Hackathon, the interns played a huge role.
We starting with planning for a week-long event starting Monday, July 27th. Next, we divided up our engineering team into groups of 4-5 members, each led by a fearless intern!! Each team lead got to pick a name for their team, and we decided to follow the on-going Pokemon Go trend and use Pokemon names!
The next step was to establish some rules for the #BugSmash:
- The entire engineering team at Cask wore their testing hats for the week
- Each competing team was assigned an area in CDAP to focus on breaking
- Team lead was responsible for making sure the documented test plan was executed and bugs were filed and fixed
- The testing ride started Monday 07/25 at 11am (kickoff)
- The daily points tally was conducted at 5pm, with a daily winner announced
- The ride ended on Friday 07/29 at 7pm
- Grand prizes for the teams with the maximum number of cumulative points on the final day!
Our scoring system was designed to incentivize finding and fixing bugs, writing integration tests for corner cases, and to foster collaboration between the teams. At the same time, we also needed to discourage reporting invalid and duplicate issues, using negative points. In addition, each day had surprise rules – like, bonus points for the team with first 5 issues filed, for the highest testing coverage etc. The combination of fixed and flexible runs made each day different and exciting.
The winner of both Day 1 and Day 2 was team Magikarp by quite a small margin. They got bonus points for fixing and resolving the maximum number of JIRAs, as well as overall good test case coverage!
The winner for Day 3 was Team Ditto for the most number of JIRAs reported + high test coverage!
Day 4 and Day 5 were won by Team Lucario and Gengar respectively, and this came as a quite a big surprise with their performance improving dramatically!
Check out what Kashif Nazir, engineering intern from Stanford University and team lead for Magikarp, had to say about the team’s success on Day 1 and Day 2:
— sreevatsan raman (@sreeSF) July 26, 2016
The Results and the Outcome
The overall winner of this Cask #BugSmash was team Ditto, led by Shenggu Lu, intern from Carnegie Mellon – with steady progress on a daily basis and a cumulative total of 158 points!
Team Snorlax led by Riwaz Poudyal, joining us from Williams College, came a close second with 130 cumulative points, and also had the maximum number of JIRAs (55) filed!
Team Magikarp led by Kashif was only team to top daily leaderboard twice!
Team Lucario (led by Chris Lu of Bowdoin College) and Team Gengar (by Iraida Ermakova of UC, Berkeley) displayed great collaboration in running tests on some complex aspects of CDAP and taking wins on day 4 and day 5 respectively.
Now for some statistics. Over a span of five competitive days, we reported 262 issues and resolved 143, pushing our overall manual test coverage to over 65%. This definitely sets new records at Cask for the number of issues filed in 5 days, and gives us a lot of confidence with the stability of this upcoming GA release of CDAP 3.5.
I am very happy to share that BugSmash turned out to be a huge success, bringing the team closer while finding and fixing issues. Working at Cask has always been fun and enjoyable, and events like these add immensely to the quality of work life at Cask. There’s no wonder we’ve been named a Great Place To Work
Published at DZone with permission of Priyanka Nambiar . See the original article here.
Opinions expressed by DZone contributors are their own.