Guide to UI Testing with Continuous Integration

Functional user interface tests, or UI tests, are one component of the testing process, and the testing process is just one component of the software development process. Before diving into the details of UI testing, it's important to understand how it fits into the bigger picture of Agile development and the DevOps process.

Let's take a look at Agile development, testing pyramids and how DevOps plays a role in both.

An Agile Development Primer

Agile development is an iterative approach to software delivery focused on building incrementally from the beginning rather than delivering it all at the end. By breaking projects down into small pieces, prioritizing them, and continuously delivering them in short cycles, the process helps keep software projects on time and on budget.

For example, an Agile team may come up with user stories to describe certain pieces of functionality, prioritize them in the order of importance, and deploy new functionality in two-week cycles. Stakeholders can collect user or customer feedback from these smaller iterations and use it to inform future development work to avoid working on the wrong things.

UI Tests & the Testing Pyramid

Test-driven development, or TDD, has become one of the most important components of Agile. According to a growing body of research, these practices greatly reduce the time it takes to deliver a software project and enhance product quality by reducing the number of bugs that reach production and ensuring the business requirements are met on a continuous basis.

Source: Software Testing Help

There are many different types of tests:

• Unit tests ensure that individual methods or functions return the correct results. These tests are usually written by developers before writing any code, tested in isolation (e.g. without dependencies), and constitute the majority of the tests (e.g. the base of the testing pyramid).
• Integration tests ensure that different parts of an application work together. These tests are usually written by test engineers before merging any code and constitute the middle portion of the testing pyramid.
• Functional UI tests are similar to integration tests in that they test different parts of an application, but they ensure that the high-level functionality described in the system specification passes. These tests are usually written by test engineers prior to deployment and constitute the top portion of the testing pyramid.

The idea behind the testing pyramid is that the majority of the tests are unit tests, while the fewest tests are functional UI tests. In most cases, unit tests are quick and inexpensive to produce and maintain, but provide a high return on investment by ensuring that key functions work. While there are fewer functional UI tests than unit tests, they are critical for ensuring that an application works correctly and delivers business value.

How DevOps Can Streamline It

Most Quality Assurance (QA) teams would agree that unit and integration tests should be automated with continuous integration (CI) build server. With each commit and merge, the CI server builds the application and runs the associated tests. Any failures prevent the merge from occurring and generate relevant error reports for follow-up by the team.

For example, a developer might write a unit test for a given piece of functionality and won't commit it until they write the code to make it pass. Test engineers might run integration tests based on business requirements and run them with every merge. If something doesn't pass, they create a bug report and send it back to developers to address.

Functional UI tests are unique in that they may be manual or automated. Manual UI testers go through key user workflows to determine if an error occurs before each deploy. While manual testing is ideal for exploratory tests, automated tests are a great way to cut down on the time it takes for regression testing of UI workflows.

Looking Ahead

UI tests can be challenging to automate without the right tools and processes in place. A failure to overcome these challenges may lead some organizations to abandon UI testing altogether, which could compromise product quality by removing key safeguards.

In the next chapter, we take a look at how to write more robust functional UI tests before moving on to CI/CD integrations and other more advanced topics.

UI tests can be challenging to create and automate. For example, suppose that your application relies on a third-party API that experiences a temporary outage — your tests may fail even though there's nothing wrong with the underlying application code!

Let's take a look at how to write more robust UI tests and how TestComplete can help simplify the process and enable more complete test coverage.

How to Improve Your UI Tests

Functional UI tests are challenging because the user interfaces tend to evolve more often than the rest of the codebase. For example, a small change to the class or ID of an element on a web page could cause an entire test to fail if the script used it as an identifier. It's hard for developers to understand the ripple effect of these small changes.

There are some cases where tests can be nearly impossible to run in an automated fashion. For instance, you may want to run tests in a virtual desktop (e.g. Citrix) environment with a single large image. Automating these tests may require using XY coordinates to identify and interact with user interface elements — a time-consuming and error-prone approach.

Unreliable UI tests are a significant obstacle to Agile development because it blocks high-speed continuous delivery. Since there are more people committing code than maintaining tests, it's impossible to fix each flaky test one-by-one. These flaky tests would be worse than no tests at all since at least the lack of tests doesn't hold up the development process.

How to Make UI Tests Scale

There are several steps that you can take to make functional UI tests more robust and scalable. The key is isolating them from dependencies, accurately recognizing elements, and taking steps to improve maintainability over time. By investing more time at the start and using the right tools, you can save a lot of time over the long-term and avoid brittle tests.

Some important steps to consider include:

• Mocking backend servers to isolate front-end code and eliminate any issues with unreliable API calls or other backend requests that can throw errors.
• Avoiding any trivial UI tests that aren't important. Every test should be either an important user workflow for the business or a highly trafficked user workflow.
• Building tests that are easy to refactor by using don't-repeat-yourself (DRY) principals, sensible variable names, and other techniques.
• Including any edge cases and negative cases for each test to ensure complete coverage and avoid unexpected errors despite passing tests.
• Improving reliability by adding wait times for async processes, data loading, or other cases where you're asserting a state change.

It's also important to choose a good testing framework. While many UI testing frameworks are accurate, there's a learning curve involved with writing the tests for non-developers. GUI-based UI testing tools are easier to use, but they may be less flexible and adaptable to changing user interface elements over time.

TestComplete Simplifies UI Tests

TestComplete is an automated UI testing tool that makes it fast and easy to create, maintain and execute functional tests across desktop, web and mobile. With an AI-powered hybrid object recognition engine, the platform accurately finds dynamic user interface elements — even if they are user interface elements appearing on a virtual desktop.

TestComplete's GUI Object Recognition

You can store object information separate from automated UI tests in one central object repository that can be stored locally or shared across teams. If any property changes, the platform's 'Intelligent Fix' enables you to update them dynamically during runtime or with a single click, which makes maintenance a lot easier over time.

Finally, TestComplete record and replay capabilities mean that you can create new UI tests in minutes rather than hours with minimal scripting. If there’s a test that needs rewriting, it’s a relatively simple task that doesn’t have to go on the back burner until a test engineer has the time to follow-up and rewrite an entire script.

Looking Ahead

The biggest challenge after creating functional tests is automating them as part of a continuous integration and deployment pipeline. For example, how often should the test run and what should trigger them? If there are any issues, who is responsible for maintaining them? Without the answers to these questions, UI tests can quickly become unmaintainable.

In the next chapter, we take a look at these issues in-depth before diving into generating bug reports and addressing cross-platform challenges.

Most Agile development teams use continuous integration and deployment (CI/CD) processes. For example, they may have a Jenkins build server that automatically compiles an application after each commit and run the associated tests to check for errors. If there is an error, the server may reject the commit until the code passes the test or the test is removed.

Let's take a look at how UI tests fit into these CI/CD workflows and some best practices to keep in mind when implementing them in practice.

When to Run Functional UI Tests

Functional UI tests take longer and consume more resources than unit tests. For example, testers may need to spin up a headless browser and execute a series of actions to test a user interface. These requirements mean that it's limited in terms of how many tests that it can run at once and how fast it can run them based on how many hub/node configurations exist.

Given the greater overhead, functional UI tests are run less frequently than unit tests that are run with each merge. Most quality assurance teams run them over the weekend every week to provide enough run time and a nice break in-between to fix any bugs and improve the automation. Other teams run UI tests before each deploy to a new environment.

The good news is that new technologies are making it easier and faster to run functional UI tests. For example, TestComplete dramatically cuts down on the time that it takes to create end-to-end tests, while distributed testing capabilities make the tests a lot more scalable than other testing options on the market. This means that you can run them more frequently.

Example of Integration

There are many different functional UI test frameworks, but the most popular continuous integration framework is Jenkins. It's a widely used and well-documented tool that is free, open-source, and available across many major operating systems. It also integrates well with a wide range of different tools and technologies. 

Assuming that Jenkins is already configured and you have a UI test written, you can add the test to Jenkins by converting the test into a batch file and adding it as a new job in Jenkins. You can then create a schedule for the job (e.g. weekends or based on a trigger), as well as configure an email or other notifications with the results.

In addition to the learning curve for setting up this integration, it can be challenging to manage a large suite of UI tests that have been integrated into a single build server. Many organizations use external test management suites, such as Jira, to manage complexity, streamline reporting, and ensure that everyone is on the same page.

TestComplete's Integration with Jenkins

TestComplete simplifies the process via integrations with Jenkins and other DevOps tools. Using a lightweight execution engine and distributed testing, the framework makes it easy to automate tests on virtual computers and integrate with top CI systems, like Jenkins, and SCMs, like Git, with minimal configuration — you just have to install the plugin.

How to Address Any Issues

A continuous integration server records any errors or failures based on its configuration. For example, a UI test could be configured to take a screenshot every time a failure or error occurs to help diagnose the problem. These screenshots can be generated and saved by the CI server or even automatically posted into bug trackers.

The biggest difference between unit tests and UI tests is that there are more false positives. For instance, a slow third-party API could lead to a timeout, which isn't necessarily caused by a problem with the application code. It's important to weed out these false positives before classifying them as bugs and placing them in front of developers to fix.

Valid bugs are typically evaluated by a test engineer, augmented with screenshots or videos, and then placed in a bug tracking system for developers to fix. After the bugs are fixed by developers, the functional UI test is re-run to confirm that there are no remaining issues before the changes are deployed to a test, staging, or production server. 

Looking Ahead

A key challenge with addressing these issues is clearly communicating any bugs with developers. Without the right information, developers may be unable to replicate the error and ultimately find a solution. Test and quality assurance engineers should take measures to ensure that they properly communicate this information to everyone.

In the next chapter, we look at how to generate helpful bug reports from functional UI tests before getting into cross-platform testing strategies.

Bug reports can be challenging for functional UI tests since there are many different factors involved. For example, developers might need to know the browser version, screen resolution, or other factors in order to troubleshoot the problem. This means that testers must carefully document issues with screenshots and ensure it's reproducible.

Let's take a look at some best practices for developing helpful bug reports for UI tests.

Define & Stick to a Process

The process of creating a bug report may seem straightforward, but it can quickly grow complex in a fast-paced team environment. After all, the number of bugs almost always exceeds the resources available to fix them, making prioritization of bugs a challenge, and duplicate bug reports are a frequent problem in large projects. 

A typical bug reporting process involves several steps:

1. A new bug is discovered by a user or tester and the tester writes up a bug report.
2. The bug is assigned to the appropriate developer by the project manager with an appropriate priority.
3. Once the developer fixes the bug, it's reassigned to a tester for verification that it's squashed.
4. The tester verifies that the bug is fixed or re-opens the bug if it still exists and the process repeats.

The key to success is writing clear bug reports that are well-organized in a bug tracker. That way, developers can immediately see what bugs they're responsible for, reproduce them with minimal effort, and quickly dive into the problem to find a solution without wasting time. The result is more bugs closed per unit of time and higher quality software.

Essential Bug Report Components 

Bug reports should follow a standardized format to streamline the report creation process for test engineers and make it easy for developers to find information. While the specific requirements may vary between companies, there are common components that most UI-focused bug reports contain, which are a good starting point for developing a template.

The most important components include:

• Title: A concise summary of the bug and its category.
• Summary: A short summary of when and how the bug occurred.
• Visual Proof: A screenshot or video to help developers understand the problem faster.
• Expected vs. Actual Results: A clear summary of what result was expected and how what happened differed.
• Steps to Reproduce: A set of well-defined steps to reproduce the bug with no ambiguity.
• Environment: The browser, operating system, screen size, zoom level, pixel ratio, and other information.
• Console Logs: All of the web page's errors in the console, which can help identify the root problem.
• Source URL: The URL for the bug.
• Severity & Priority: The impact on the user experience and how quickly it should be investigated.

TestComplete makes it easy to collect many of these elements without having to go through a manual process. Test engineers can easily share images, videos, or detailed logs captured during automated UI test execution in various formats — even if they don't have TestComplete installed. You can even access video recordings to find issues visually without readings logs.

Best Practices to Remember

There are a handful of best practices that test engineers should keep in mind when writing bug reports. While these best practices may not apply to every situation, there represent some common practices that your organization may want to consider when developing your bug reporting processes.

The most important best practices include:

• Only include one bug per issue. Don't combine multiple related bugs into a single issue for developers to solve since they may be unrelated under the surface.
• Search to make sure that the bug hasn't already been reported. If it has been, check the existing bug report to see if you can add any new information based on your findings.
• Take the time to reproduce the bug before creating a report — don't take a user's word for it that a bug exists. You can then show the developer how to reproduce the error.
• Keep feature requests separate from bug reports. Since feature requests are more desirable for developers, keeping them in the same place could lead to ignored bugs.

Looking Ahead

Clear bug reports and a well-defined bug reporting process goes a long way in improving functional UI tests. Using modern tools like TestComplete, you can quickly collect relevant screenshots, videos, and console data to share with developers to help them reproduce and quickly diagnose bugs without having to spend as much time collecting the material by hand.

In the next chapter, we take a look at cross-platform testing strategies for applications that need to work across mobile, web and desktop.

Maintaining functional UI tests across different platforms can be a challenge. For example, you may need to create tests that ensure a web application works across various desktop and mobile browsers with different screen resolutions. With so many different combinations, it's hard to know where to start and how to maintain such a test suite.

Let's take a look at how cross-platform UI tests are normally handled and how TestComplete makes the process easier.

What Platforms to Test

The ideal functional UI test suite would test every combination of browser, operating system and device, but with tens of thousands of combinations, that's impossible. In addition, it doesn't make much sense to spend hours building and maintaining a test suite for — and updating an application to work with — an old browser version with a handful of users.

On the other hand, you cannot necessarily only support the latest browsers. For instance, a software-as-a-service application targeting enterprise customers may have a lot of users with older browsers and operating systems since enterprise upgrade cycles tend to be longer. Other companies may run their own proprietary versions of certain browsers.

Google Analytics offers a precise way to discover the browsers, operating systems and devices that your existing visitors use. After setting up Google Analytics, you can find this data by going to the "Audience" section. The "Mobile" subsection shows mobile device usage data, while the "Technology" subsection shows you browser and operating system breakdowns. You should consider testing any configurations with significant usage.

In addition to browsers, you may have companion mobile and desktop applications that require UI tests. The good news is that it's easier to determine the devices and operating systems to test (in most cases), but the bad news is that these tests typically run using different platforms than web-based user interface tests, making it challenging to integrate.

Building Cross-Platform Tests with TestComplete

TestComplete simplifies the cross-platform testing process with a robust set of capabilities. With minimal scripting, you can easily create cross-platform tests that leverage artificial intelligence to recognize objects on a screen rather than searching for them in code. You can even customize mobile gestures and other advanced UI interactions. The platform works with nearly any popular software or device, eliminating the need for patchwork solutions.

TestComplete supports most major platforms:

• Desktop: You can automate UI tests for desktop applications across a wide range of tools and controls, including .NET, Windows 10 and Java.
• Web: You can create reusable cross-browser tests for over 1,500 real browsers, operating systems, devices and resolution configurations.
• Mobile: You can create and automate functional UI tests on physical for virtual iOS and Android devices without jailbreaking your phone.

By supporting multiple platforms, TestComplete enables you to keep all of your functional UI tests under the same roof, making it easy to standardize testing and reporting. The ease of use also means that you can create more test coverage across all of these platforms with the same amount of resources to devote to testing efforts.

Looking Ahead

Cross-platform UI tests can be a challenge due to the wide range of potential configurations. While many quality assurance teams use a patchwork of solutions, TestComplete makes it easy to keep everything under one roof and integrate with external testing tools, including Jenkins and Git. It’s a great way to ensure maximum coverage with minimal investment.

In the next chapter, we take a look at why you should try TestComplete and answer some common questions.

Functional UI tests can be challenging for a variety of different reasons. Since user interfaces frequently change, UI tests tend to be brittle and throw a lot of false positives. There are also a large number of potential browsers, operating systems, devices and platforms to test, which makes it challenging to reach complete test coverage. 

TestComplete alleviates these issues by handling every aspect of end-to-end functional UI tests within a single platform. Using AI-based object recognition, the platform can easily recognize UI elements based on their appearance rather than looking at classes or IDs that might change. The platform also includes support for 1,500 different combinations of browsers and operating systems, native desktop applications, and native mobile applications.

“TestComplete improved application testing time between 40-50%. It has helped us reduce resource costs by as much as $350K.” - Sandor Albert, Senior Software Engineer, Philips Healthcare.

“We've been using TestComplete heavily for the last 3 years. Since that time we've been catching defects earlier in the release cycle, releasing quicker and the number of defects found post-release has dramatically dropped.” - G2 Reviewer

Download a free trial to learn how TestComplete can help you today!

Which browsers does TestComplete support for testing web pages?

TestComplete supports all major browsers, including Chrome, Firefox, Edge and Internet Explorer. Beginning in version 14.4, TestComplete also supports Safari on MacOS and browsers on iOS and Android devices.

The platform also provides access to web pages and their elements displayed in WebBrowser and Chromium Embedded Framework controls that reside in any application. In other words, TestComplete supports any browser created based on the WebBrowser and Chromium Embedded Framework controls.

How can I take test results from TestComplete and send them to developers or to my boss?

• Select the desired log node in the tree on the left of the test log page (if you select the root node, TestComplete sends the results of the whole run, if you select a subnode, the item contains results shown by this subnode and its child nodes). 
• Right-click the selected node and choose "Send via E-mail" from the context menu.

TestComplete packs the results to a Multipart Hypertext Storage file (.mht), attach it to the email message and initiate the sending process. In addition you can also export TestComplete results as JUnit XML that can be imported into any reporting tool of your choice.

Are there checkpoints available in TestComplete?

Currently, you can use the following checkpoint types:

• Property checkpoints - verify an object's property value. 
• Table checkpoints - verify data in tabular controls (grids, list views, list boxes, etc.) 
• File checkpoints - compare files. 
• Mobile checkpoints - verifies that the desired image is displayed on the screen of a mobile device.
• XML checkpoints - compare XML documents. 
• Database checkpoints - verify data stored in a database. 
• Region checkpoints - compare images, or searches for an image within another image. 
• Web audit checkpoints - perform a quick performance, accessibility and SEO audit of your web application or website
• Web accessibility checkpoints - perform various checks of web pages (for example, checks for broken links). 
• Web comparison checkpoints - compare web pages.

Does TestComplete allow multiple users to work on a test project simultaneously?

There are several ways to allow multiple users to work on a test project simultaneously:

• Work with test projects in shared mode - Store your TestComplete project in a network drive, and give all users access to it so that they are able to open, edit, and run these test projects on their computers.
• Bind test projects to a source control - Add your TestComplete test projects to your source control system such as Git so that all the needed users can access the repository, get the needed test project files, commit changes they made to the project files to the repository, and so on. In addition, TestComplete provides advanced support for a number of source control systems allowing you to work with them directly from TestComplete IDE.

Does TestComplete integrate with Continuous Integration (CI) systems?

Yes, TestComplete integrates with all common Continuous Integration systems today, including Jenkins. 

Does TestComplete support BDD and Gherkin-style syntax?

Yes, TestComplete has a native Gherkin IDE and Cucumber engine to help you accelerate your Behavior-Driven Development initiatives. 

Does TestComplete integrate with Jira?

Yes, TestComplete integrates with Jira natively or via our test management solution called Zephyr for Jira.