Running TestComplete Tests with Multi-Configuration Jenkins Projects

  June 30, 2011

Customer Guest Blog

Łukasz Morawski, QA Engineer



In my previous article, Cross Browser Web Testing – a TestComplete and Jenkins Framework I showed a way how to schedule a multi-browser test in Jenkins. It required us to create a separate job for every browser version we want to test. This solution requires a lot of effort and in fact there is a simpler way to do it. Jenkins provides multi-configuration project. With this option we can create only one job with many configurations. Each configuration will be executed as a separate job. This is exactly what we need to simplify our scheduled tests, which can be used in conjunction with TestComplete or TestExecute.

Creating a new job

Create a new job, but this time use the Build multi-configuration project option. Now, on the job's configuration there is a new section only available for this type of job - Configuration Matrix. Press button Add Axis. For our purpose User-defined Axis is just fine. There can be other axis types available according to other Jenkins’ configurations.

[caption id="attachment_6552" align="aligncenter" width="724"] Figure 1: New job[/caption]

For instance, if there are many JDK's configured in Jenkins, there would be a JDK axis available, letting tests run against different JDK configurations. If you do distributed testing with slave nodes, there would be a Slave axis – run tests on particular slaves, a group of slaves or nodes with particular labels.

The axis configuration

Provide variable Name and in Values enter values of our configuration. For our purpose values would be a browser version: FIREFOX30, FIREFOX35, FIREFOX36, IE60. Figure 2 shows the configuration of the user-defined axis.

[caption id="attachment_6576" align="aligncenter" width="760"]figure2.userdefinedaxis Figure 2: User-defined Axis configuration[/caption]

The build step

Now, we also have to change the build step. The name of our axis is in fact a name of a variable. The variable is used with leading and trailing “%” character. So we have to use it instead of the browser's configuration string. Figure 3 shows how the build step could be configured. Please note the BROWSER variable usage.

[caption id="attachment_6582" align="aligncenter" width="760"]figure3.executewindowsbatchcommand Figure 3: Execute windows batch command changes[/caption]

After configuring our axis and build step, save configuration. So, how does it work? When you open the job's page you will see each configuration according to the values you've entered – FIREFOX30, FIREFOX36 and so on (Figure 4). Now, when the job is executed, all of the configurations will be triggered. Also take a look at Console output and you will notice that the %BROWSER% string is resolved to a particular value.

[caption id="attachment_6592" align="aligncenter" width="760"]figure4.jobconfiguration Figure 4: Multi configuration project page[/caption]



About the author:

Lukasz Morawski has six years of experience in the area of automated testing. He is currently employed as QA Engineer at Cognifide ( – a digital technology agency that specializes in planning, building and supporting medium- and large-scale digital marketing technology platforms.

See also:

Avoiding the “Unable to Find Object” Error in TestComplete

Exploratory vs. Scripted Testing: One or the Other, or Both?

Bias and the Human Side of Software Testing