Integrating Automated Testing with Jenkins

As teams strive to be truly agile and deliver high quality software quickly, Continuous Integration (CI) has evolved to become an integral part of the software ecosystem. When the discussion pivots to CI, the first tool that comes to mind is Jenkins.

Jenkins is the most popular of any of the CI tools used today by software teams and is playing a significant role in the accelerating the ‘Dev’ in DevOps. It is an open source CI platform built in Java and has thousands of native integrations with useful tools. Jenkins enables developers to build, deploy, and automate projects; thereby, improving time to market and product quality.

As automation testing continues to help organizations scale their software quality efforts, it is imperative that automated tests are also part of the CI pipeline to achieve true continuous delivery. Most test types - including unit, integration, functional, and regression tests - are run via CI depending on the size of the test suite and the type of application under test.

TestComplete has a native Jenkins plugin that enables you to easily run tests in a Jenkins Freestyle or Jenkins Pipeline CI project. In addition, the plugin stores the history of test runs and enables you view test results directly from Jenkins.

Getting Started with Jenkins and TestComplete

Let’s walk through a quick tutorial on how to quickly set up a Jenkins Job  that automatically kick starts and executes your automated functional tests in TestComplete and provides report logs for instant feedback.

1. To get started, ensure that the following is installed on your test computer
   1. Jenkins version 2.121.1 or later
   2. TestComplete (or TestExecute)
   3. The application under test. In case of web applications, install all the required browsers
2. Navigate to the Jenkins Plugin Manager and install the ‘TestComplete Support’ plugin.
   The plugin provides an easy-to-use and flexible UI to configure your TestComplete project to run in your CI process. Once this is done, restart the Jenkins server for the changes to come into effect.
   
3. Prepare the testing environment.
   For tests to be able to simulate user actions on the UI, an interactive session must be opened on the test machine. See Enable Remote Desktop Connections, Configure Group Policies and Disable Secure Sign-In.
4. Configure your Jenkins agents.
   Ensure that your Jenkins agent is set up to run in interactive mode. If the agent is running as a Windows application under a user account, you are good. However, if your Jenkins agent is running as a service, you need to configure it to run under the Local System User account and then open a user session via the plugin.
5. Copy the TestComplete project to the agent. Before we run our TestComplete tests, we need to place the TestComplete project to a location where Jenkins agents can access it. The easiest way to achieve this is to store your TestComplete projects in a source control repository. It is also possible to place TestComplete projects in a shared network location and copy them to your agents.

Creating a Jenkins Freestyle project

1. Create a new item and select ‘Freestyle project’.
   


2. Enter a label and check the ‘Restrict where this project can be run’ box.
   


3. You will notice that the TestComplete plugin adds a ‘TestComplete Test’ open under the Build step.


4. Specify the TestComplete project suite associated with the tests you wish to run. Note: The location here is relative to the Jenkins agent’s working folder.


5. Next, you can specify what you wish to run - an entire project suite, a specific project, or a specific Keyword, Script, or CrossBrowserTesting test. There are a few additional settings that may be useful to you.
   


6. Specify how the Jenkins tool should behave if your test reports any warnings or errors.


7. Allow opening an interactive user session before the testing starts.If you are unable to use this approach for some reason, then an alternative to run an interactive user session is to launch the test machine and control it via Java Web Start.
   


8. Select the type of test logs after the test is complete, Junit style report or TestComplete’s native MHT log file.

Jenkins Plugins Settings

Here is a list of all the configurable settings available with the TestComplete Jenkins plugin:

Project Suite file:

A project suite holds one or more related TestComplete projects and stores information that lets you define the suite as a whole. For instance, if your application includes several executables, you can create an individual project to test each executable and then unite these projects into a project suite.

Choose from Entire suite, Project test, Script test, Keyword test, CrossBrowserTesting test or Other (Low-Level Procedure, Network Suite, etc.)

Optional settings:

1. Test runner: TestComplete or TestExecute; Specifies whether the step will use TestComplete or TestExecute for the test run. If both are installed and (Any) is selected, TestExecute is run
2. Versions: 9, 10, 11, 12 or 14; Specifies the TestComplete or TestExecute version to be used in case you have several versions of these products installed on the node. Jenkins uses the latest installed version by default
3. Action on warnings: Mark as Unstable or Mark as Fail; Specifies whether Jenkins should mark the entire build as failed or unstable when the TestComplete test log contains warnings
4. Action on errors: Mark as Unstable or Mark as Fail; Specifies whether Jenkins should mark the entire build as failed or unstable when the TestComplete test log contains errors
5. Additional command line arguments: Specifies arbitrary command-line arguments to be passed to TestComplete. Note: Arguments "/run", "/SilentMode", "/ForceConversion", "/ns", "/exit" are added by default
6. Use test timeout:Specifies the maximum test execution time in seconds. If the test is not finished until the specified period elapses, Jenkins will mark the entire build as failed
7. Run interactive user session
8. Generate MHT log file
9. Generate JUnit-style report

Run Your Jenkins Job

Post-build actions are useful depending on your ecosystem setup. Publish a JUnit test result report, e-mail notifications, or set GitHub commit statuses are popular ones.

7. Start the build: After you have finished setting up and configuring your Jenkins build, click ‘Build Now’ to run your tests.
   
   
8. Review results. After the job run is over, you can view test results in one of three ways:
   1. JUnit-Style Test Reports: On the project status page, Jenkins displays a graph with test results for the builds that created JUnit-style reports. This graph shows the ratio of failed tests to the total number of tests.
      
   
   
   2. On the Test Result page, you can examine the results of a complete test run, along with summary information about the executed tests and their total execution time.
      
      
   3. On the History page, you can view execution time of the tests and the ratio between the failed and passed tests. Use show count to view the number of executed tests rather than time.
      
      
   
   
   4. TestComplete Test Results – The page displays a summary of TestComplete test runs, information on what machines tests ran, their run time and the number of logged errors and warnings.
      
      
      
      
   5. Accessing TestComplete Test Results Through APIs.To get the results of TestComplete tests via an API, send GET requests like the one below to your Jenkins server. You can also view all the finished builds in the ‘Build History’ section.
        http://localhost:8080/job/My%20Job/241/TestComplete/api/xml​

In addition to Freestyle project, a Jenkins job can also be run as Pipeline. See Configuring Jenkins Pipelines to Run TestComplete Tests.

Conclusion

Continuous Integration and Continuous Delivery (CI/CD) are no longer a nice-to-have for organizations building software. In order to compete in the market, your ecosystem must be integrated seamlessly and TestComplete fits into any CI environment. The steps above showcase how to integrate automated testing with Jenkins, enabling you to create great software, fast.

Download TestComplete