Crowd-Sourcing Bug Detection with Betas, Bounties and Bashes
With so many software applications passing data through shared networks and cloud environments, the demand for software quality is more prolific than ever before. With ever-more critical systems being developed in the finance, healthcare, government and defense sectors, businesses are rightly concerned with security and vulnerability issues. Many companies are also faced with the difficulties of uncovering complex issues that would be expensive to set up and test in-house.
So what can you do?
Many organizations have gone the route of crowd-sourcing bug detection. This process isn't as simple as just complaining on Twitter or reporting issues to support. There are actual organized - and sometimes highly paid - methodologies behind this technique. Here are a few of the strategies companies are employing in order to tap into the larger, more varied world that real users represent.
I know, I know this is nothing new. As an industry we’ve been around the beta block a couple thousand times - from managed betas to private betas to public betas… we’ve adopted them all. The benefit of a beta test is that it is typically time-bound and overseen by someone in the company. The downside is that it is time-bound and overseen by someone in the company.
What do I mean by that? Well, we all bring our own biases to the table whether we intend to or not. When companies select beta customers based on some criteria and/or assign them tasks to perform, they often miss crucial issues because they limit customer actions and environments.
This is slowly becoming a lucrative business because many companies see the value of “in the wild” testing, especially in the mobile space. You can only create and re-create so many user scenarios in a lab environment unless you have unlimited time, money and resources. Using a crowd-testing service can provide you with much-needed insight into your application quality and uncover some difficult-to-find bugs in the process. But beware – if you want clean defect reports with steps to reproduce and analysis of user impact, this is not where you’ll get that. I prefer to think of crowd testing as a great ongoing feedback loop outside the normal rigor of your in-house testing.
"Find that potential bug and bring it in – we’ll pay you a bounty."
That’s the basic premise behind bug bounties, which are primarily aimed at researchers rather than testers. Having a security breach can be very costly - just ask any company that has been hacked in the past. But it’s very difficult to unearth security flaws despite code reviews and security testing – so what companies are focusing on are the potential problems in their applications, otherwise known as vulnerabilities. It’s a win-win for both the researcher, who has the potential to make a lot of guaranteed money from finding a security flaw, to the company, which would otherwise face enormous payouts to consumers and government for security breaches. Hiring the right people full-time to look for these flaws is also a costly proposition without a long-term value prop.
I love bug bashes. Sounds like a party, doesn’t it? And in a way, that’s the idea.
A bug bash is a time-boxed testing frenzy where testers are often rewarded with prizes and recognition. Some companies still keep their bug bashes to themselves, turning the whole company into an extended testing organization for a few days in an effort to flush out any bugs that evaded the test team. Others get more courageous and open the bash to their external users, simulating the power of crowd testing but under more controlled circumstances by setting specific challenge guidelines and timeframes. The idea, of course, is to get as much brain power and ‘randomness’ into the mix as possible but within constraints and under the oversight of your own testing organization.
What does this mean to the future of testing?
I don’t see any inherent danger to the future of software testing from these types of strategies. In fact, I think it’s refreshing to see the renewed vigor behind software quality that is prompting people to find more ways of finding and wrangling bugs than they have traditionally had at their disposal. In the testing world, we’ve always known there is a limit to the types and quantity of bugs we can find within a project’s scope – often we’re constrained by the availability of test environments and the limitations of project schedules. Being able to tap into the larger user pool and their real-world environments can only be a boon to the software industry, and helps to alleviate the strain on most testing organizations that are fully aware that they can’t get to every bug.