When the Chicago Board Options Exchange trading system was knocked out a few weeks ago, investors and customers wanted answers, and they wanted them fast. The CBOE answered, but the outage wasn’t a hack as many had guessed – it was a software defect that, by all accounts, the development team was fully aware of before the release.
The cause of this defect was based on poor tracking of requirements that were not met. The CBOE itself stated as the cause, “preliminary staging work related to the planned reconfiguration of our systems in preparation for extended trading hours on CBOE Futures Exchange (CFE) and eventually CBOE options.”
Basically, they didn’t meet a necessary requirement that their competitors had met, so they scrambled and ended up releasing a buggy version.
Many development teams can relate to what happened at the CBOE and, with it, the intense pressure to get software out as quickly as possible, which often results in bypassing critical test steps. Of course, the CBOE is liable, and will lose credibility and market position due to this serious technology flaw. But most development teams have done the same thing, albeit on a smaller scale – release it quickly, keep your fingers crossed, and fix the issues later.
That should scare you.
The question “Can we test this?” should never be asked. Yes, you can test it. From the CBOE PDF: “Unfortunately, the nature of a software bug is sometimes only identifiable once the system is operationally ready…” which is something you never want to have to say. Here are three ways you can safeguard your company from falling into this unhealthy routine.
1. Understand Requirements
If you are changing code, understand why. What requirement does this fulfill, what user story, what test case? What is being changed and why?
If it’s a re-configuration of your system, make sure you have it covered. Repeat your regression tests against the new configuration. Repeat your regression tests against a new build.
2. Understand Tests
Are they complete? What changes are being made that need to be tested? What isn’t being changed that needs to be re-tested?
New builds, along with test scripts, will need to be tweaked and all changes captured via source control and test versioning.
3. Understand Ratio of Company Liability to Time Spent on Software Testing
A simple Android app may not introduce the same security flaws or liability as the CBOE defect. Think about how many people will use the product and whether this signifies a need for a higher degree of test maintenance. What are the users using the product for? Each user story constitutes a different approach on testing, whether a user is using an application for financials, entertainment, data transfer, etc.
This is why requirements tracking and traceability need to be used as safeguards in projects. With the help of a test management tool – development teams can link their tests to each iteration and then, most importantly, to their requirements. This, in turn, leads to full transparency and leverages your test coverage and defects to the requirements that need to be met.
Another important benefit to traceability with the help of source code control is that it allows software teams to pinpoint defects from a specific build and put out fires quickly. This is also an important way for development teams to improve because they can analyze problem areas that will be apparent after a release.
The risk of releasing defective software is never 0% and, as the CBOE has shown, we are all running too fast sometimes to hit the brakes. Development teams need to spend more time analyzing their requirements and use effective traceability mechanisms in order to increase quality - increasing test coverage is great but will never solve everything. Testers also need to use requirements tracking and traceability to present liability to the whole development team and truly determine if more time needs to be spent on testing.
We must be able to hit the emergency brake before we crash into a wall.