Test Monkeys and a Method to Madness
Back in my days as a manual tester, I always had fun finding new ways to break software. I could get very creative with testing various limits or simply using random inputs at random time intervals to see how the application reacted. These random exploratory tests drove the developers nuts but proved over time to improve the quality of the software.
At the StarEast conference last week, I was introduced to a concept by John Fodeh that really intrigued me – it was a way to give random tests a uniform structure and add it to our automated test scripts. Those random tests are called test monkeys and the structured randomness being intelligent test monkeys. (What’s with the obsession with monkeys? Chaos monkey, test monkeys, coding monkeys, intelligent test monkeys…)
What I found ironic about test monkeys was that there was a method to this type of chaotic testing. By creating a test script that automates random input into an application, we can find bugs that would otherwise never be found with our regression tests. The problem with regression tests is that testing the same functionality with each new build, while important, we are simply iterating over a path we have taken before. While this helps us ensure that the functionality does not de-stabilize with each release, it doesn’t result in any new findings. This is why we have exploratory testing.
But test monkeys take this a step further because they inject random operations that we would not be able to replicate with manual testing. For example, we usually wait for an operation to load before we press a button, but what if we kept pressing keys on a keyboard while the application was loading and the app crashed? Is this not a defect? Defects surely can be introduced with or without proper user interaction.
This is what intrigues me about using test monkeys. But to get real value from them, we should incorporate a formula to prove whether or not software is stable. For instance, if an application were to crash within 20000 random operations we could consider the application unstable. If the application could pass 80000+ random operations before crashing, we could assume the application is stable.
To take this a step further we can introduce test scripts that follow a standardized form of chaos by quantifying this chaos. We could use a table or model to say that for every hundred random operations the script will press random keys 10% of the time, click on elements of an application 50% of the time and change these ratios and time intervals accordingly. These are called intelligent test monkeys.
If you want to add some test monkeys to your test automation, here are some useful tips:
- Log your random data. It's great if you find a problem thanks to randomized input, but it's even better if you know what random data was entered.
- Make sure your random data covers all the possible inputs, including invalid ones. For example, if you have a field that only accepts numbers, make sure you generate digits and dates too (because you know some of your users will do that)
- Always have a way to reset the system back to a base state. After you've entered your random data or clicked a random control, you'll need a way to get your application back to the point where you can try again.
Test monkeys as a form of orderly randomized testing can help us find defects that would usually be found after a product release. Let’s admit it, as testers we tend to use the same patterns of testing and this can be a way to break the mold just once in a while and I don’t know about you but I think it is also a way of making testing fun.