Case Study: Scottish Water

SmartBear QAComplete Delivers Same Robust Testing and Defect Management Capabilities as HP Quality Center and IBM Rational TestManager—at 10X Less the Cost.

About Scottish Water

Scottish Water, a publicly-owned company answerable to the Scottish government, provides 1.34 billion litres of drinking water to the people of Scotland every day while also removing 847 million litres of waste water. The company has been recognised as the best large employer in Scotland and attained its highest water quality level ever in 2015—with a rating above 99.9%.

Business Need

As Technology Build and Test Manager Peter McInally came onboard at Scottish Water, one of his first missions was to ensure that QAComplete, which had been purchased for a single project, was actively used across the upcoming delivery portfolio. It was important to identify and deploy a centralized tool for software testing and defect management so that the team could get better visibility into the software development process and ensure the delivery of quality software for Scottish Water. Other than the one project, teams at Scottish Water were either using no tools or a variety of tools such as Excel or Bugzilla for test and defect management. Furthermore, working with outsourced offshore development and test teams in the form of consulting companies, who are present in different parts of the world, further necessitated having a centralized test management system. 

“We did not have a documented process and relied primarily on spreadsheets for testing and defect management,” McInally says. “We, therefore, were in dire need of a better way to track the progress of tracking tests and defects. Having a centralised process would come in handy while establishing clear traceability across user stories, tests and defects.”

Deploying a test management tool would also generate additional benefits for Scottish Water. For example, using the variance on the planned vs. actual test efforts would enable McInally to accurately predict how many testing resources were needed for future projects. Over time, using the data pulled through an advanced testing tool would also enable McInally to build highly-accurate estimation models for resource allocation during the test creation phases. 

“We wanted to start documenting how long it takes teams to not only execute tests, but also build those tests,” McInally explains. “Tracking the amount of time it took the team to write tests would help us to build highly-accurate resource estimation models.”

As a result, Scottish Water hoped to introduce the right resources from consulting firms at the right stage of the testing and development phase. “In fact, over time we intend to take the process to the next level and build a predictive analysis model for resource allocation using that data,” added McInally.

“The main drivers for us are the extensive functionalities QAComplete offers at a low cost,” says McInally. “This makes it much easier for me to prove a high return on the investment that we can expect by implementing QAComplete. Both HP Quality Center and IBM Rational TestManager are very expensive and do not justify the ROI in comparison to QAComplete. Add to that the customisation capacities of QAComplete, and we can easily modify the tool as per the processes that our offshore development and testing teams follow. This further simplified our decision as the post rollout of the tool is much easier.”
Peter McInally, Technology Build and Test Manager

Targeted Solution

While looking for options to deliver a significant portfolio of work, regarding test management tools, McInally realised that one of the smaller teams within Scottish Water was using SmartBear QAComplete. “One of our teams had previously purchased QAComplete to capture defects and create test scripts for a project,” McInally points out. “For that one project, they used the tool right out of the box with very little configuration, and we needed a tool for the next regularity period.”

As McInally began to assess QAComplete and other tools, QAComplete impressed him with its functionality that was as equally robust as the tools he had used previously (HP Quality Center and IBM Rational TestManager). The SmartBear solution also came at a fraction of the cost of those other tools—approximately 10X lower. In comparison to free tools, such as Bugzilla and Excel, QAComplete offered more robust test and defect management features.

Furthermore, since QAComplete is a Software-as-a-Service (SaaS) tool, it comes with the added benefit of scaling per user needs and allowing multiple offshore teams to access the tool as and when needed—without any maintenance and installation hurdles.

“The main drivers for us are the extensive functionalities QAComplete offers at a low cost,” says McInally. “This makes it much easier for me to prove a high return on the investment that we can expect by implementing QAComplete. Both HP Quality Center and IBM Rational TestManager are very expensive and do not justify the ROI in comparison to QAComplete. Add to that the customisation capacities of QAComplete, and we can easily modify the tool as per the processes that our offshore development and testing teams follow. This further simplified our decision as the post rollout of the tool is much easier.”

While deciding on the post roll out process, there were additional features that prompted McInally to select QAComplete over HP Quality Center and IBM Rational TestManager. For instance, given QAComplete’s intuitive interface, learning the tool for new users is really easy.

Unlike HP Quality Center and IBM Rational TestManager—where consulting services are needed to ramp up and learn the tool—QAComplete enables Scottish Water to rely on internal resources to train software developers and testers as well as third-party consultants on how to use the tool. Additionally, when using QAComplete, the team can get to the point where they roll out testing templates for each new project without any further configuration.

Benefits and Results

“We now use QA Complete to coordinate defect management and test management,” McInally says. “QAComplete is also being used by the outsourced offshore development and test teams. This in turn helps us get a complete view of the requirements that come in, the tests that are written for user stories, and the defects that are raised in the process.”

Scottish Water uploads software requirements into QAComplete from spreadsheets and Orbus iServer so the requirements can be viewed and traced in QA Complete as the testing and defect management processes take place. The software development teams also use the 40+ reports available in QAComplete to demonstrate test coverage, traceability, defect trending, etc.

“QAComplete is proving to be a big benefit to our software developers in helping them to mitigate defects and speed up the overall development process as much as possible,” says McInally. “This approach works much more efficiently than before when the testing activities were stored in isolated spreadsheets. We can now more accurately measure the quality of our development and test work.”

Using reports available in QAComplete, Scottish Water has also improved upon estimating the time it takes for testing teams to create test scripts. McInally emphasises that while testing and development teams at many companies focus primarily on test execution results, it’s important to look at the time teams spend on preparing test scripts.

“This way, you can find out the variance between the planned and actual test efforts,” McInally explains. “This in turn feeds into future resource allocation models. If you bring the testing team in too early, you pay for time you don’t need; if you bring them in too late, you slow down the development process.”

Scottish Water uses QAComplete to see how well the development teams manage not only the test management process, but also evaluate how they perform on the test creation process—especially if they spend the estimated time on writing tests. This factors into the overall project development timelines.

By leveraging QAComplete, Scottish Water has created a comprehensive software development estimation model that the team continues to hone over time. If the team has 1,000 requirements to test, for example, managers can use data provided by QAComplete to estimate how long it will take to write all the test cases for those requirements.

“A complicated requirement might require two hours for creating the test script,” says McInally. “We then know how many resources and how much time we need. If the delivery time is six months, we can bring in enough QA analysts or QA Managers to meet that timeline. We then track the activity so that at the end of the process, we can evaluate our time estimate. From that perspective, QAComplete serves as a very valuable tool within our overall development process.”

Business Challenges

  • Deploy centralised tool for streamlining software testing and defect management.
  • Create documented and automated processes for software testing and defect management.
  • Enhance tracking the progress of software defect fixes.
  • Document how long it takes to build test scripts to more accurately forecast development timelines.

SmartBear QAComplete Solution

  • Provides end-to-end software testing and defect management traceability.
  • Delivers functionality equal to but at approximately 10X lower the cost of other mainstream providers.
  • Features comprehensive software test management capabilities with centralised reporting.
  • Enables software development teams to manage, plan, organise and execute all test cases and defects associated with software releases in one single place.
  • Offers online access to facilitate collaboration between onsite and offsite development teams.

Measurable Results

  • Compares the number of test cases created per day to forecasted projections.
  • Coordinates defect management, test management, scripting and user-acceptance testing.
  • Generates complete views into scripts created and defects identified.
  • Creates reports to evaluate how many test scripts pass/fail and whether defects have been fixed or remain open.
  • Evaluates quality of test scripts from a central place where the data is easily accessed.
  • Determines how well the test management process is managed and how much time is needed to write test cases.

All company names and/or products referenced herein are either registered trademarks or trademarks of their respective trademark holders.