Customer Guest Blog
Lukasz Morawski, QA Engineer
Cognifide
Introduction
One of the most important problems faced while implementing automated tests for Web applications is providing cross browser independence. When you write a test, you don't want to write versions for every browser. You would like to have a generic test which works in every environment you require. Another problem is scheduling a test run – how to provide a reliable and flexible way to execute your test? I would like to introduce one possible solution to this problem – a simple, neat framework with TestComplete and Jenkins for continuous integration. TestComplete and Jenkins make continuous integration when cross-browser testing extremely easy.
Utilu tool
Utilu is a set of two tools: Firefox Collection and IE Collection. These tools let you install different versions of Firefox and Internet Explorer on a single machine. These are standalone versions and work exactly the same as native versions. This means – no emulation and no virtual machines (at least so far). Download the tools and install the versions you require. Pay attention to browser compatibility with operating system, for e.g., IE 9 can be installed only on Windows Vista and later. Utilu also lets you install some add-ons like Firebug and Developer Toolbar.
Figure 1: Utilu Firefox Collection installation dialog
TestComplete
TestComplete is a tool which can automate functional web tests. I am familiar with Jscript, so examples will be provided in this scripting language. Launch TestComplete and create a new project with default content. First step is to add all required browsers to the TestedApps collection (Utilu installs browsers to C:Program FilesUtiluMFC by default). Set appropriate names for each item, so you will have no problems identifying particular browser versions. Also, set command line parameters and working directories (you can use the browser's shortcut properties as a reference).
Figure 2: TestComplete TestedApps collection
Figure 3: TestComplete Firefox 3.6 run parameters dialog
Let's start implementing our framework. So, what do we need? What are the ingredients?
- a runner, for each browser version to start it,
- a function choosing desired runner,
- a function processing command line parameters,
- a test to run in different browsers,
- a couple of global variables to fill up our framework.
A runner
The idea behind a runner is to have a separate function for every browser version. If you want to add a new browser later you have to implement an additional runner. Here is an example of RunFireFox36() function which runs Firefox 3.6:
{
try
{
var ff = TestedApps.firefox36.Run();
while(! ff.Exists)
{
aqUtils.Delay(1000);
}
PROCESS = "firefox";
}
catch(e)
{
Log.Error("RunFireFox36", e.description);
}
}
Let's take a quick look at this code snippet. We’ll run firefox36, as configured in the TestedApps collection. Then we wait for the process to exist and then set the PROCESS variable to “firefox.” This variable corresponds to the browser's as seen by Windows, so for all versions of Firefox it will be “firefox” and for all of IE it will be “iexplore”.
A runner chooser
Next step is to implement a function which will choose and run the appropriate browser's runner. Function would take one parameter – browser name:
function RunBrowser(browser)
{
try
{
Log.Message("Running: " + browser);
switch(browser.toUpperCase())
{
case "FIREFOX36": RunFireFox36(); break;
case "FIREFOX35": RunFireFox35(); break;
case "FIREFOX30": RunFireFox30(); break;
case "IE60": RunIE60(); break;
default: Log.Warning("Running default browser: FireFox"); RunFireFox35(); break;
}
}
catch(e)
{
Log.Error("RunBrowser", e.description);
}
}
Upper casing a browser name will make it case insensitive. As default, Firefox 3.5 will be run.
A command line processing
How do we tell a test for which browser it should run? We could have some variable and then change its value. We could…but if we want to have our tests flexible, let's pass it as a parameter into TestComplete and control it from the outside world. To achieve this task we need to process command line parameters. The second parameter is our browser version, that's why we use ParamStr() with argument "2".
function getParams()
{
try
{
var param = BuiltIn.ParamStr(2);
Log.Message("Parameter: " + param);
if(param === "")
{
BROWSER = DEFAULT_BROWSER;
}
else
{
BROWSER = param;
}
}
catch(e)
{
Log.Error("getParams", e.description);
}
}
A test
And now our simple test. It opens (string stored in variable “URL”), enters string „smartbear”, submits a search and then clicks a link with inner text "Tools for Software Testing and Development - SmartBear Software". We need to find this link first – that’s why we use „FindAllChildren” method. This test shows the usage of “PROCESS” variable.
function Test1()
{
try
{
var browser;
var page;
var all;
var item;
browser = Sys.Process(PROCESS);
browser.ToUrl(URL);
page = browser.Page(URL);
all = page.document.all;
item = all.q;
item.Click(37, 14);
item.SetText("smartbear");
item.Keys("[Enter]");
page.Wait();
var link = all.FindChild("innerText", "Tools for Software Testing and Development - SmartBear Software");
link.Click();
page.Wait();
}
catch(e)
{
Log.Error("Test1", e.description);
}
}
The Main
And finally, let's get it all together in the main function. This function should be set as a project's TestItem run function. Processing command line parameters, then running the appropriate version of the browser based on the passed parameter, testing and closing the opened browser:
function Main()
{
try
{
getParams();
RunBrowser(BROWSER);
Test1();
CloseBrowser();
}
catch(e)
{
Log.Error("Main", e.description);
}
}
CloseBrowser() function closes all opened browsers using CloseAll() method of TestedApp object.
Jenkins CI
The TestComplete part is done. Now let's do the Jenkins. Jenkins is a continuous integration tool which is used to build software but also to test software - we can use it to schedule our test runs. All you have to do is download the latest version of Jenkins and run it: “java -jar jenkins.war “ (Jenkins is written in Java). After a while Jenkins will be available at the “http://localhost:8080/” address. Jenkins executes its jobs on nodes and by default there will be only one node – Master. If you only want to execute tests on this node, then you need to change the node's configuration “# of executors ” and set it to “1” (“Manage Jenkins->Configure System”). It will tell Jenkins to run only one instance of a test at a time – TestComplete and TestExecute cannot run multiple instances or in parallel. Otherwise your tests would fail if you scheduled more than one test. That's all the required configuration for now – for our purpose we don't need to change anything else.
We need to create a new job for each browser version we want to test. To do so, choose New job, then Build a free-style software project and set the name of your job. Later, if you want to save some time, you can copy from the existing job. In the job's configuration Add build step and select Execute Windows batch command. Now you need to specify a command line to execute your tests. Figure 4 shows the command line for Firefox 3.0. Note that the browser version string has to be the same as the one used in the RunBrowser() function. In this example TestComplete will open without a splash screen (/ns option) in silent mode (/SilentMode) and run CrossBrowserTest.pjs project suite. A parameter “FIREFOX30” will be passed into TestComplete and after test execution – will exit (/e option). For another command line option, please refer to “TestComplete Command Line“ help topic. If you plan to use TestExecute, you need to use the appropriate command line. Now save changes to your configuration. Repeat these steps and create a job for every browser version you want to use.
Figure 4: Jenkins job configuration
Now we will create a main job, which will execute all jobs we created earlier. Once again, create a new job using a free-style project, but this time in the configuration page in Post-build Actions check the Build other projects check box. In Projects to build edit, name all the projects you want to run. Also select Trigger even if the build fails radio (see Figure 5).
Figure 5: Run All Tests job configuration.
When it's done, Jenkins' main page should look like Figure 6. The only thing to do is start the “Test Run All” job - to do so, press the clock icon. You will notice that all your jobs are added to Build Queue and will be executed one after another.
Figure 6: Jenkins jobs
OK, but we don't want to execute tests manually every time. We need to schedule our jobs. Jenkins has three possibilities here and it's configurable in the Build Triggers section of the job's configuration. For our purpose the Build periodically option suits best – this lets us schedule the date and time of the job's execution. Jenkins uses cron style syntax consisting of five fields separated by white space. A scheduled run shown in Figure 7 will be executed 5 minutes after midnight every day.
Figure 7: Jenkins Build triggers
What's next?
What can we do now? This example shows how to run tests on a single testbed. For more advanced solutions – this isn’t enough. One of Jenkins' features is distributed testing, where a test runs on multiple hosts. In our example, after running “Run All Tests” every test would run on a different, dedicated node. This would speed up testing and reduce the time of execution. Of course you have to provide resources, meaning physical or virtual machines on which tests would run.
Our example uses scheduled runs – we set a date and time of execution. Next step would be continuous testing where a software build triggers a test build. If your company already uses Jenkins for software building, it can be easily extended to run a test just after successful build (or after unit and/or integration tests). Of course it's not a simple task. Functional tests tend to run much longer than unit tests or integration tests but with a combination of distributed testing it can create a very interesting and nice testing framework.
References
Includes
About the author
Lukasz Morawski has six years of experience in the area of automated testing. He is currently employed as QA Engineer at Cognifide (www.cognifide.com) – a digital technology agency that specializes in planning, building and supporting medium- and large-scale digital marketing technology platforms.