As the anticipation to this year’s FIFA Soccer World Cup begins to mount, and millions of soccer fans around the world anxiously wait for the event of the year, some of us are distracted by another World Cup: the Software Testing World Cup.
A couple weeks ago I had the privilege of being a judge for the Software Testing World Cup. The STWC brings software testers from around the globe together to show off their skills and compete for great prizes (and bragging rights, of course!).
How it all works
The competition is broken down into two phases: preliminaries and finals. Before the preliminary phase starts, teams of up to four people register their interest in participating. During the preliminary phase, teams compete within their own region (Africa, Asia, Europe, North America, Oceania and South America).
Preliminaries are three-hour events. Participants are given software to test and a bug management tool to use. Upon receiving these tools they are given a briefing by the "customer" and have three hours to read through the information, consult with the customer to clarify any questions, decide upon a test strategy that maximizes the limited time available, actually test the software, raise bugs and write a test report. If that sounds like a lot to do in three hours, that’s because it is!
One of the major decisions for each team is to decide what to test, what not to test and how to test it – all based on the information the customer provided. STWC’s lead judge, Matt Heusser, acted as the customer on YouTube LiveStreaming for the duration of the preliminary round.
Teams were evaluated on whether or not they are on mission (context is king!); on the importance, quality and reproducibility of their bugs reports; on the quality, accuracy and relevance of the test report; on whether or not they included non-functional testing; and bonus points were awarded for asking the customer important and interesting questions. A few bonus points were available for how the teams interacted with the judges, and for exceptionally demonstrated teamwork.
STWC North America
I was one of seven judges of the North America preliminaries which happened on April 25. I joined the livecast on YouTube, where Matt was giving the "customer" briefing to the teams. After the briefing was over, we stayed on YouTube answering questions participants posted on the comments section and on Twitter. The first half hour of competition was hectic, with questions being asked by several people at once about the competition, the mission, the software under test and the bug tracking tool.
Around the one hour mark things started quieting down as teams were busy testing and raising bugs. With about 30 minutes remaining until the end of the session we could still see participants raising bugs. Matt reminded them that they still had to write their test reports, which was a very important assessment criteria and could take some time to accomplish. Maik Nogens was monitoring the competition’s inbox for submitted test plans and had a very small number by that time. I started to feel anxious for the participants, thinking about the pressure they were under to get this done. But by the final whistle most teams had their test plans submitted.
For the volunteer judges, the fun started after the competition finished. The first step for us was to get together to review the assessment criteria, where we decided upon judging heuristics to use (we wanted to make sure every judge had guidelines to help ensure fairness and consistency). Once we agreed on the heuristics, each judge had the massive task of reviewing 582 bugs and 36 test reports before we started to grade each one of the teams. We also attempted to reproduce most high and critical bugs to see if they were actual bugs or not.
And the winner is...
It took us over a week of hard work to review the deliverables and evaluate each team. After the judges had rated each team, we collated our individual scores (thanks goodness for Google Docs!). Even with agreed upon guidelines for judging, there were some outliers when we collated all the information together.
Again, with the aim of being as fair as possible, we reviewed each other’s scores, paying particular attention to scores that were either significantly higher or lower than the average of other judges’ scores. We then discussed reasons for scores to ensure no one was overlooking anything. After the review, the scores were averaged and the top nine teams for North America were announced. The winner for North America was Team Quadcore, and they were also awarded the Most Valuable Bug prize.
The winning team of each region’s preliminary round will receive tickets to Agile Testing Days (including airfare, accommodation and entry to the conference for the whole team ), which is where the finals will be held live on November 10, 2014. In the finals, first prize is €3,000, second prize is €2,000 and third prize is €1,000. Not too bad if you ask me!
The STWC experience
Being part of the judging team was more work, but also more fun, than I had anticipated! Something else I did not anticipate was how much I would learn from the experience. Here are some takeaways:
- By reviewing exceptional bug and test reports I picked up a few things I was able to take to my workplace. I was reminded that writing a good test report is an art form - a report that includes just enough relevant information for decision makers to be well informed. They don’t have to be long, or have piles of jargon.
- There is no substitute for good, clear communication. Teams that performed best where those who understood the mission clearly by asking the right questions, thinking critically and challenging assumptions along the way.
- It was also a great example that there is not one "right" way to test. There is not one best practice that applies to everyone and/or every situation. In this case, we had 36 teams testing the same piece of software and every single one of them approached the task differently. And even amongst the top scoring teams, there were not two teams that tested similarly.
There may not be financial remuneration in volunteering, but that doesn’t mean we do not take anything from it. It was a very rewarding experience to be a judge in the Software Testing World Cup. I’m looking forward to watching the epic final online in November, as the best team from each region engage in an epic battle to win the top prize - and the coveted STWC trophy.
See also:
[dfads params='groups=937&limit=1&orderby=random']