MicroSystems Automation Group

Improving Product Quality and Testing

"Prior systems suffered from poor quality and insufficient testing before their scheduled releases. TestComplete and automated testing has increased our initial product quality and helped us eliminate re-introduction of past software problems. We now see a noticeable increase in our product's quality at delivery. We also have a vast increase in testing volume and coverage. Before we had limited use in regression testing, but we are rolling out a full testing discipline now," said Ted Campbell from MSAG.

Choosing TestComplete

Ted explained how MSAG chose TestComplete: "TestComplete was chosen after an extensive review of available test products in 2003. The analysis of Software Testing Products has been reviewed three additional times, by committees, to ensure suitability. Official adoption of TestComplete by the Bureau of Educational and Cultural Affairs came at the end of FY 2006, after three years of review and completing evaluations."

"I implemented limited demonstration test plans on our five major projects: Exchanges Information System, Exchange Visitors Database – Enhanced, Citizen Exchanges Database rewrite (Eureka), ESMS Data Transfer and electronic National Program Association and Council of International Visitors management application. TestComplete performed very well with these limited test plans and we decided it was the right choice for us."

Eliminating Bugs in Releases

"TestComplete has basically eliminated the release of products with show-stopping behaviors. Testing of basic menu, UI, process, reporting and regression functions has prevented us from releasing project builds with flaws. By using TestComplete, it has reduced the need for point releases on projects where before, it was used for each release. Besides eliminating bugs in our releases, we have also experienced a better audit trail and better process compliance. All of these benefits are due to a more complete test coverage on our Main UI, Help, Menu, Reporting and Configuration options," said Ted.

TestComplete Saves Time and Money

Ted described how TestComplete has saved them time and money: "Automation of general test scripts removes up to 90% of the labor associated with executing a test specification. Complex and infrequently used program features are tested manually by Subject Matter Experts or specific test scripts. It has also saved money on projects by reducing testing time, code rework and by significantly diminishing the interval between introduction of a programming flaw and its discovery."

Improved Time-to-Market

"Our market is career US Government Staff, Foreign Service Officers and political appointees. In addition, all products produced for public diplomacy have significant Section 508, privacy and security compliance requirements. By reducing test times and increasing product quality with TestComplete, product release cycles have been reduced significantly," said Ted.

Testing Different Release Candidates

Ted explained that MSAG has hundreds of tests for several different projects: "We have hundreds of tests that test for show-stoppers every automated build. We also have automated test scripts for all alpha release candidates, manual, one off, complex and essential new functionality test scripts for all beta release candidates and User community Acceptance and Ad Hoc testing for every final release candidate. Our general goal is to automate any repeatable or data-driven test script after it has been performed more than two times."

MSAG TestComplete Test Plan

"TestComplete’s scripting language support and object orientation allow for the use of structured test development that mirrors code development," said Ted.

"All of our TestComplete projects are structured in the same way.
A project suite will contain the following functional test projects:

• Main: start and stop, query test run defaults, test all help
• MainMenu: worker scripts open and close all main menu choices
• MainUI: worker scripts to open and close all main window dialogs and sub systems
• Regression: worker scripts to implement pass/fail tests for all past reported error conditions
• Process: test scripts to demonstrate, validate and log discrete program requirements from Technical Design
• Configuration: worker scripts to test all remembered settings and configuration options
• Reporting: scripts to test and output auditable results demonstrating program reporting requirements from Technical Design

"In addition, a switch to allow batch automated testing is included so top level testing can be performed during an automated build. This provides a pass/fail condition for the program delivery system."

MSAG on TestComplete

"After a brief training and startup time, TestComplete will reduce the time needed to produce and run test scripts manually the second or third time a test script is run. Avoiding large changes in UI control naming and functionality is key to avoiding rework in test scripts," said Ted.

"Once a body of test code and regression tests exist, you will see a reduction in point releases and reported bugs that is proportional to the number of test case failures disclosed by testing."

"By using TestComplete in your efforts you will increase the quality of your product and reduce the test time."