"What surprised me about TestComplete was some of the unexpected advantages we never thought about before, but became apparent when using the software." Dana expressed that one of the primary examples is how their software development's lifecycle has changed as it relates to defects:
- New bugs immediately get assigned to QA when reported. QA then builds an automated test script to reproduce the bug, attaching it to the Case and reassigning it to the developer who needs to work on it. This test is then used by the developer to not only reproduce the issue, but to ensure that the fix passes the test. Only after it passes the test can the code be checked into the source control server, and the case be resolved.
- The tests are then immediately added to their automated testing framework once the Case is resolved. In this way this bug should NEVER reappear in the future. If it does, they will immediately catch it before we ship it out. This is regression testing at its finest.
- All new features must have a set of tests completed before it can be added to the production codebase. This new workflow process helps them to think more objectively of what the feature does, and how it will interact with the rest of the system.
- Some of these tests are what they call "public facing" tests. In other words, in the future they can ship an executable test harness to customers to run specific tests on their own systems. This will allow people to not only evaluate their products, but expose problems that may exist on the specific platform being tested. This will let their Customer Service reps get an immediate indication of what is going on without burdening the end user with tons of questions.
"Other benefits include the fact that we are now able to add a lot of security testing without having to invest in new tools and technologies, and the education/learning curve that comes with it. We also have a better understanding of the quality of our software at any given time due to the confidence we have in our tests."
"It’s weird, but we routinely find new ways to use TestComplete to automate some new task as part of our development process."
Dana states that, "primarily TestComplete has freed up staff to work on more important things that require their critical thinking, rather than focusing on the tedious repetition of mundane tasks. I enjoy watching staff spending hours developing a new automated test script that will literally save them days or months of man hours that can be performed on a daily basis."
"TestComplete has caught critical defects before they got into the mainstream production codebase. Traditional defect cost analysis shows the difference in monetary value in fixing bugs before they are shipped verses when you fix them in the field. On top of that, the money saved in staffing requirements has been very beneficial to the company. Although our pool of tests continues to grow along side of our products, we haven’t had to hire additional staff to work in QA yet. We can do more with less! That’s something extremely critical for many small software businesses like my own."
Higher Confidence and Reduced Defects
Dana explained that they have increased the time it takes to release a product because they have mandated new workflow processes that take more time to build tests. "However, in doing so, we have benefited from a higher level of confidence in the product at time of release than ever before. And it has reduced the defects that we are seeing in the production software. In the end, the extra time spent SAVES the company a lot in future customer service and support costs."
While Dana stated that he has lost count of how many tests Scorpion Software has, he stated; "We literally have hundreds of test conditions that are tested in various TestComplete projects. As we learn how to optimize tests more we routinely deprecate old tests in favor of new streamlined tests. We don’t follow a traditional documented test case process in favor of automated scripts being tagged to functionality within the product. However, I can tell you that in some cases, some tests take over 24 hours to run on a P4 3GHz machine with 2GB of ram. That’s a lot of different scenarios being tested!"
"We run regression testing daily at noon with TestExecute on a dedicated build and test system. This allows developers to check in work before noon and know before the end of the day if their submissions for fixed defects or new feature have been accepted against the entire product codebase. We also run fault injection, fuzz testing and security testing once a week over the weekend, where the test system can run for 24 to 48 hours straight without worrying about new test cases being introduced."
"Before TestComplete, we would run through all tests manually, if we were lucky, before shipping a new version, which would take weeks of man hours to complete. As such, we only did it on a limited basis, as we simply couldn’t afford the investment in time to perform all manual testing."
Dana explains, "TestComplete's best quality is the simple recording of pre-defined actions that can be augmented with simple but powerful scripts. The test accuracy that is afforded to my company through the use of TestComplete has eliminated manual testing that was prone to human failure and which was expensive to the business."
"Further more, It is my opinion that quality assurance test specialists have traditionally had to be better programmers than the developers of the original codebase to properly build the test harnesses needed to build quality software. With TestComplete, this no longer has to be the case. However, if they are proficient in programming, you benefit immensely with in depth testing that can rival any testing team from big software companies."
All company names and/or products referenced herein are either registered trademarks or trademarks of their respective trademark holders.