Automation Framework - A New Table Driven Technique Using TestComplete
Note: The concepts and structures described in this article are geared towards advanced users of TestComplete and test automation in general. It does require some intermediate knowledge of Jscript coding practices and advanced knowledge of TestComplete’s programming objects. ENTJG72Q9RYC
First, a little history:
When I first started using TestComplete, 9 years ago (back then it was called AQTest and before their new automated testing framework TestLeft), my first automation effort was simply exercising the application code using what amounted to record and playback procedural scripts. I would sit down at my workstation, click the record button, and then manually execute my test steps from a documented manual test plan. I would repeat this for each set of steps and then call each recorded test in sequence to execute the larger test suite.
This actually worked pretty well in that it gave us some excellent coverage of a set of repetitive tasks and caught a lot of bugs. In fact, to this day, that project is still in use at that company and is catching bugs even to this day. There were some modifications made to these recorded scripts to take certain steps and make them procedures of their own. I could then construct a new step by calling these libraried procedures in order without having to record a new task.
A lot of the code in these procedures had some hard coded values for entering into text fields or selecting particular buttons and/or keystrokes. This was fine so long as the data configurations remained static and that all previous tests executed cleanly and successfully as data generated by a previous test would affect the data utilized by the next test. I started, then, experimenting around with creating data driven tests where the data to be used by my test steps were queried from a set of CSV files and then called within the code This was before the creation of the DDT objects in TestComplete so these CSV file calls were all manually coded using ADO SQL Query objects. This worked very well so long as each test included the same set of steps. If a step needed skipped or a new step needed to be added in, I would have to add logic into my procedural loop and controlling data in the data record to determine the order of steps, etc. This created very large, complex procedural loops and extremely long data records.
I found an article, then, about 6 years ago written by Cem Kaner (http://www.kaner.com/pdfs/autosqa.pdf) talking about the need to improve the maintainability of automated tests. Additionally, I found another article from James Bach critiquing some of the “snake oil” that traps many organizations when it comes to automated tests using GUI driven tools (http://www.satisfice.com/articles/test_automation_snake_oil.pdf). The combination of these two articles and, more recently, reading “The Automated Testing Handbook” written by Linda Hayes (http://www.softwaretestpro.com/Item/4772/The-Automated-Testing-Handbook/Automation-Quality-Assurance-Test-and-QA-Technology-Testing) inspired me to work on the concept of a table-driven testing framework within the TestComplete environment. One earlier version of this framework came out of and adapted a lot of the code from my previous efforts at creating an automated testing environment, coded specifically with that application in mind.
Recently, though, I’ve created an initial version of a universal framework structure to be used in TestComplete that can be adapted for most any application where there is a need for functional regression tests to be added and maintained. This article describes the various components of the framework and how to adapt those components and build an automated regression project.
The framework I’ll be describing, as with any automation, is not a silver bullet that will cure all your problems when it comes to developing well-formed, maintainable regression automation. There are some parts of the end product that are not included in this framework and there is some effort that will be required in building the particular code for your application under test (AUT).
In Linda Hayes book, she describes a framework as having several sections that are standard within the environment. You may read her article to find out the specific descriptions of these various sections of the framework. There are many areas of the framework that are going to be specific to the application being tested (e.g., the MONITOR is going to need to be coded to know what is expected/unexpected, SETUP is going to be unique for each application and environment, etc) and so, for a packaged distributable framework, they don’t apply.
The framework I developed focuses on the DRIVER portion of the framework and, to some extent, the TEST portion as applies to how the DRIVER calls the tests and interacts with them. I anticipate that, as I work with my framework, I’ll be able to expand on it to include at least a few generic routines to handle the LOGERROR and RECOVER sections but it remains for the end user to customize those routines that would apply to the specific AUT.
Script Code Language
My background is mostly in DelphiScript and Pascal structured language. However, considering the spread of languages structured more like C++ and with the rise of C# as a code base, the framework is written in Jscript. If you prefer a different language like DelphiScript, VBScript, or something else, the main parts of the framework can be converted to that code base but I’m going to personally maintain it as a Jscript environment.
The framework makes use of two main features of the TestComplete IDE environment.
- DDT – To read through, parse, and process the tables used for driving the tests, the TestComplete Data-Driven Testing object is utilized extensively. The file formats used by the DDT drivers are CSV files as they are editable without any special software and are able to be versioned and compared through most source control applications.
- ODT – While many script languages include their own ability to create and manipulate custom objects, the ability to create dynamic objects in code, instantiate them using specific code routines, and have them dynamically link to other code units and routines requires some of the special “tricks” of the ODT (Object-Driven GUI Testing) programming object.
These two features of TestComplete are available in both the Standard and Enterprise editions of TestComplete. Everything else needed for the framework is standard Jscript code and other more standard functions and features of the TestComplete automated testing tool.
At the core of the framework are three CSV files that contain the majority of the information needed to execute the tests. These files “live” in the Script folder of the framework project.
The TestCases.CSV file contains a listing of the different test cases to be executed, uniquely numbered and with a description that details what the case does. A sample file would look like:
“1”, “Execute test case 1”
; “2”, “Execute test case 2”
“3”, “Execute test case 3”
A semi-colon at the beginning of any line in this file indicates to the framework to skip that test case.
The TestSteps.CSV file contains, for each test case in the TestCases.CSV file, the list of ordered steps to be executed for that test case. This is the “meat” of the framework as this file determines the order of the steps, which application to perform the step again, the specific step to perform, and any supporting data needed by that step.
“TestStepNo”, “TestNo”, “AUTName”, “StepAction”, “SupportingData”, “Comments”
“1”, “1”, “MyApp”, “ACTIONTYPE1”, “Field1|1|Field2|2”, “Execute Step 1”
“2”, “1”, “MyApp2”, “ACTIONTYPE1”, “Field1|test”, “Execute Step 2”
“3”, “1”, “MyApp”, “ACTIONTYPE2”, “”, “Execute Step 3”
The link between the two files is by way of the “TestNo” column. As the framework goes through to execute the test cases, it reads in each step for each case and then parses out the “SupportingData” and uses that information in the execution of the step.
Because every application of this framework could potentially have a different set of applications involved, the framework dynamically creates testing step classes based upon the applications being tested. For this reason, a CSV file needs to be supplied to the framework listing these applications.
“MyApp”, “My main AUT”
“MyApp2”, “A utility program for processing information into the main AUT”
The important data in this file is the AUTName although a comments field is provided to describe the application itself.
These three files feed into the code of the framework and, when you need to add a new test, are all that need to be updated to create a new test case.
The three CSV files described above are read into the code of the framework. This code parses out the data and then drives and creates the tests. The framework consists of three main code units.
Code Unit: Initialization
The Initialization.sj unit contains code to declare ODT classes that will be used for executing the different steps. This unit reads from the AUTList.CSV file and declares the classes that will be used for executing the steps. The InitualizeAUTClasses function should be called once before the rest of the driver of the framework is executed. In order, though, for the classes created to actually be able to execute any code, for every AUT in the AUTList.CSV, there should be a code unit named <AUTName>TestStep.js where <AUTName> is the name in the AUTList file. That code unit needs at minimum a routine called Execute<AUTName>TestStep that contains the necessary code for processing the test step as indicated in the TestSteps.CSV file.
Code Unit: TestDriver
The TestDriver code unit contains two code routines that go through the TestCases and TestSteps CSV files and uses the information found there to execute the indicated tests. ProcessTests goes through the TestCases file using a DDT driver and, for every row in the TestCases, queries for rows in TestSteps. The resulting DDT driver is used for creating an ODT.Data group containing variables corresponding to each test step. ExecuteAllSteps is then called which goes through that data group one variable at a time and executes the step based upon the method on the object for that particular step class as determined in the Initialization routine.
Code Unit: TableDrivenTestLibrary
In this first version of the framework, there is one additional library file included that all the <AUTName>TestStep.js classes utilizes for populating the properties of instantiated objects. This library unit needs to be included in the project as well.
Two script extensions are included in the framework as well. I added these to make calling certain common code routines easier than having to always include a new unit of code or adding the USEUNIT call at the top of other existing units. The Test Log Utilities consists of a runtime object that is used for building a log structure that is descriptive of what actually is being executed. This was adapted from Aaron Frost’s excellent guest blog http://blog.smartbear.com/post/11-02-08/how-to-guide-to-readable-test-logs-using-testcomplete/ and encapsulated into the script extensions. The Data Driven Helpers currently consists only of a routine to detect if a DDT Driver is open and, if so, close it. I found myself repeating this code throughout my work and, rather than having to constantly re-write it, I turned it into a script extension.
Without going into extreme code details, here’s how you would use the framework to test a simple application.
- Create MyAppTestSteps.sj – First, I need to create a code unit for the testing steps for the calculator application. The AUTList.CSV file would contain a row that would have, as the “AUTName”, the value of “MyApp”. The MyAppTestSteps.sj file would then consist of a single routine for executing the test steps:
Self.AddProperty(aqString.GetListItem(Self.SupportingData, DataIndex), aqString.GetListItem(Self.SupportingData, DataIndex + 1))
if (!DoActionType1(Self)) throw new Error(1, “Failed to execute Step type 1”)
if (!DoActionType2(Self)) throw new Error(1, “Failed to execute step type 2”)
default: throw new Error(1, "Invalid Step action: " + aqString.ToUpper(Self.StepAction))
Self.IsStepSuccessful = true
Log.Warning("Could not execute step " + exception.description)
Self.IsStepSuccessful = false
Note that the unit has a USEUNIT call. This points to the library of code that contains the actual executing code for DoActionType1, DoActionType2, etc. This could be contained within this unit but I prefer to separate execution code from driver related code.
Also, notice that the first item in the routine is a for loop that uses the SupportingData property to create additional properties on the objects. That way, while each object is of the same class, these additional properties are unique to each instance and give a particular distinction as to what kind of step it is executing.
The switch statement then executes the different steps based upon the value of step action.
- Creating the CSV files – The three CSV files then need to be populated with the different tests to be executed, the applications to be tested, and the steps per test. The examples given further up in the document give the proper structure and layout for using the code given above.
- Create the Supporting Code Libraries – The final part, although it may be the first thing you do, is to create the code that is executed for each step. This is what would be called for DoActionType1 and would consist of the button clicks, keystrokes, and other manipulations done to test the application. Contained within that could would be the various tests, log actions, and other measurement tasks needed for that particular step. Those routines should then return either a true or a false as to whether or not the step was successful. These code libraries could be actual Jscript code or they could be Keyword tests and either of them could either be hand coded routines or recorded tests. They could even be a data driven loop themselves using a keyword test and the Data Driven Loop operator.
As stated above, the purpose of such a framework is to improve the maintainability of your test automation. This kind of framework lends itself to good modular code practices and separates the creation of tests from the creation of code. Additionally, if the application under test changes so that it breaks one or more steps, rather than having to rewrite the entire project to accommodate those changes, all that needs to be updated is the supporting code library. If additional steps need to be created and inserted into a test cases, it’s a matter of adding to that code library, adding a step to the Execute method, and then updating the CSV files with new data. All existing steps and data can stay as is and you have less code to update and maintain in the long run.
One final advantage to such a framework is that it does not require testers to have any knowledge of coding practices to create new test cases. A dedicated automation person or team could maintain the majority of the code and then, when a tester wants to add a new test case, all they’d need to do is update the CSV files and run the tests. This could even lend itself to an Agile environment where testers in an Agile team could use record and playback to create the supporting code for the test steps and then automation people can take that code and adapt it into the framework along with CSV file updates for the new test cases. That way, for each iteration, new regression is added to make sure that any new code does not break existing already tested code.
This framework is only in its infancy as version 1.0 so there are definitely places of growth and improvement. I welcome all input towards improving this and adding capabilities for other areas of the framework. You can contact me at:
I’m also present frequently on the TestComplete forums as a regular contributor and all-around helpful guy.
Join the TestComplete Users Group on Linkedin. Over 500 members and growing stronger every week!