Why You Should Automate UI Tests
Functional user interface tests, or UI tests, are one component of the testing process, and the testing process is just one component of the software development process. Before diving into the details of UI testing, it’s important to understand how it fits into the bigger picture of Agile development and the DevOps process.
Let’s take a look at Agile development, testing pyramids and how DevOps plays a role in both.
An Agile Development Primer
Agile development is an iterative approach to software delivery focused on building incrementally from the beginning rather than delivering it all at the end. By breaking projects down into small pieces, prioritizing them, and continuously delivering them in short cycles, the process helps keep software projects on time and on budget.
For example, an Agile team may come up with user stories to describe certain pieces of functionality, prioritize them in the order of importance, and deploy new functionality in two-week cycles. Stakeholders can collect user or customer feedback from these smaller iterations and use it to inform future development work to avoid working on the wrong things.
UI Tests & the Testing Pyramid
Test-driven development, or TDD, has become one of the most important components of Agile. According to a growing body of research, these practices greatly reduce the time it takes to deliver a software project and enhance product quality by reducing the number of bugs that reach production and ensuring the business requirements are met on a continuous basis.
Source: Software Testing Help
There are many different types of tests:
The idea behind the testing pyramid is that the majority of the tests are unit tests, while the fewest tests are functional UI tests. In most cases, unit tests are quick and inexpensive to produce and maintain, but provide a high return on investment by ensuring that key functions work. While there are fewer functional UI tests than unit tests, they are critical for ensuring that an application works correctly and delivers business value.
How DevOps Can Streamline It
Most Quality Assurance (QA) teams would agree that unit and integration tests should be automated with continuous integration (CI) build server. With each commit and merge, the CI server builds the application and runs the associated tests. Any failures prevent the merge from occurring and generate relevant error reports for follow-up by the team.
For example, a developer might write a unit test for a given piece of functionality and won’t commit it until they write the code to make it pass. Test engineers might run integration tests based on business requirements and run them with every merge. If something doesn’t pass, they create a bug report and send it back to developers to address.
Functional UI tests are unique in that they may be manual or automated. Manual UI testers go through key user workflows to determine if an error occurs before each deploy. While manual testing is ideal for exploratory tests, automated tests are a great way to cut down on the time it takes for regression testing of UI workflows.
Looking Ahead
UI tests can be challenging to automate without the right tools and processes in place. A failure to overcome these challenges may lead some organizations to abandon UI testing altogether, which could compromise product quality by removing key safeguards.
In the next chapter, we take a look at how to write more robust functional UI tests before moving on to CI/CD integrations and other more advanced topics.
How to Write Robust UI Tests
UI tests can be challenging to create and automate. For example, suppose that your application relies on a third-party API that experiences a temporary outage — your tests may fail even though there’s nothing wrong with the underlying application code!
Let’s take a look at how to write more robust UI tests and how TestComplete can help simplify the process and enable more complete test coverage.
How to Improve Your UI Tests
Functional UI tests are challenging because the user interfaces tend to evolve more often than the rest of the codebase. For example, a small change to the class or ID of an element on a web page could cause an entire test to fail if the script used it as an identifier. It’s hard for developers to understand the ripple effect of these small changes.
There are some cases where tests can be nearly impossible to run in an automated fashion. For instance, you may want to run tests in a virtual desktop (e.g. Citrix) environment with a single large image. Automating these tests may require using XY coordinates to identify and interact with user interface elements — a time-consuming and error-prone approach.
Unreliable UI tests are a significant obstacle to Agile development because it blocks high-speed continuous delivery. Since there are more people committing code than maintaining tests, it’s impossible to fix each flaky test one-by-one. These flaky tests would be worse than no tests at all since at least the lack of tests doesn’t hold up the development process.
How to Make UI Tests Scale
There are several steps that you can take to make functional UI tests more robust and scalable. The key is isolating them from dependencies, accurately recognizing elements, and taking steps to improve maintainability over time. By investing more time at the start and using the right tools, you can save a lot of time over the long-term and avoid brittle tests.
Some important steps to consider include:
It’s also important to choose a good testing framework. While many UI testing frameworks are accurate, there’s a learning curve involved with writing the tests for non-developers. GUI-based UI testing tools are easier to use, but they may be less flexible and adaptable to changing user interface elements over time.
TestComplete Simplifies UI Tests
TestComplete is an automated UI testing tool that makes it fast and easy to create, maintain and execute functional tests across desktop, web and mobile. With an AI-powered hybrid object recognition engine, the platform accurately finds dynamic user interface elements — even if they are user interface elements appearing on a virtual desktop.
TestComplete’s GUI Object Recognition
You can store object information separate from automated UI tests in one central object repository that can be stored locally or shared across teams. If any property changes, the platform’s ‘Intelligent Fix’ enables you to update them dynamically during runtime or with a single click, which makes maintenance a lot easier over time.
Finally, TestComplete record and replay capabilities mean that you can create new UI tests in minutes rather than hours with minimal scripting. If there’s a test that needs rewriting, it’s a relatively simple task that doesn’t have to go on the back burner until a test engineer has the time to follow-up and rewrite an entire script.
Looking Ahead
The biggest challenge after creating functional tests is automating them as part of a continuous integration and deployment pipeline. For example, how often should the test run and what should trigger them? If there are any issues, who is responsible for maintaining them? Without the answers to these questions, UI tests can quickly become unmaintainable.
In the next chapter, we take a look at these issues in-depth before diving into generating bug reports and addressing cross-platform challenges.
How UI Tests Fit into CI/CD Pipelines
Most Agile development teams use continuous integration and deployment (CI/CD) processes. For example, they may have a Jenkins build server that automatically compiles an application after each commit and run the associated tests to check for errors. If there is an error, the server may reject the commit until the code passes the test or the test is removed.
Let’s take a look at how UI tests fit into these CI/CD workflows and some best practices to keep in mind when implementing them in practice.
When to Run Functional UI Tests
Functional UI tests take longer and consume more resources than unit tests. For example, testers may need to spin up a headless browser and execute a series of actions to test a user interface. These requirements mean that it’s limited in terms of how many tests that it can run at once and how fast it can run them based on how many hub/node configurations exist.
Given the greater overhead, functional UI tests are run less frequently than unit tests that are run with each merge. Most quality assurance teams run them over the weekend every week to provide enough run time and a nice break in-between to fix any bugs and improve the automation. Other teams run UI tests before each deploy to a new environment.
The good news is that new technologies are making it easier and faster to run functional UI tests. For example, TestComplete dramatically cuts down on the time that it takes to create end-to-end tests, while distributed testing capabilities make the tests a lot more scalable than other testing options on the market. This means that you can run them more frequently.
Example of Integration
There are many different functional UI test frameworks, but the most popular continuous integration framework is Jenkins. It’s a widely used and well-documented tool that is free, open-source, and available across many major operating systems. It also integrates well with a wide range of different tools and technologies.
Assuming that Jenkins is already configured and you have a UI test written, you can add the test to Jenkins by converting the test into a batch file and adding it as a new job in Jenkins. You can then create a schedule for the job (e.g. weekends or based on a trigger), as well as configure an email or other notifications with the results.
In addition to the learning curve for setting up this integration, it can be challenging to manage a large suite of UI tests that have been integrated into a single build server. Many organizations use external test management suites, such as Jira, to manage complexity, streamline reporting, and ensure that everyone is on the same page.
TestComplete’s Integration with Jenkins
TestComplete simplifies the process via integrations with Jenkins and other DevOps tools. Using a lightweight execution engine and distributed testing, the framework makes it easy to automate tests on virtual computers and integrate with top CI systems, like Jenkins, and SCMs, like Git, with minimal configuration — you just have to install the plugin.
How to Address Any Issues
A continuous integration server records any errors or failures based on its configuration. For example, a UI test could be configured to take a screenshot every time a failure or error occurs to help diagnose the problem. These screenshots can be generated and saved by the CI server or even automatically posted into bug trackers.
The biggest difference between unit tests and UI tests is that there are more false positives. For instance, a slow third-party API could lead to a timeout, which isn’t necessarily caused by a problem with the application code. It’s important to weed out these false positives before classifying them as bugs and placing them in front of developers to fix.
Valid bugs are typically evaluated by a test engineer, augmented with screenshots or videos, and then placed in a bug tracking system for developers to fix. After the bugs are fixed by developers, the functional UI test is re-run to confirm that there are no remaining issues before the changes are deployed to a test, staging, or production server.
Looking Ahead
A key challenge with addressing these issues is clearly communicating any bugs with developers. Without the right information, developers may be unable to replicate the error and ultimately find a solution. Test and quality assurance engineers should take measures to ensure that they properly communicate this information to everyone.
In the next chapter, we look at how to generate helpful bug reports from functional UI tests before getting into cross-platform testing strategies.
How to Generate Helpful Bug Reports
Bug reports can be challenging for functional UI tests since there are many different factors involved. For example, developers might need to know the browser version, screen resolution, or other factors in order to troubleshoot the problem. This means that testers must carefully document issues with screenshots and ensure it’s reproducible.
Let’s take a look at some best practices for developing helpful bug reports for UI tests.
Define & Stick to a Process
The process of creating a bug report may seem straightforward, but it can quickly grow complex in a fast-paced team environment. After all, the number of bugs almost always exceeds the resources available to fix them, making prioritization of bugs a challenge, and duplicate bug reports are a frequent problem in large projects.
A typical bug reporting process involves several steps:
- A new bug is discovered by a user or tester and the tester writes up a bug report.
- The bug is assigned to the appropriate developer by the project manager with an appropriate priority.
- Once the developer fixes the bug, it’s reassigned to a tester for verification that it’s squashed.
- The tester verifies that the bug is fixed or re-opens the bug if it still exists and the process repeats.
The key to success is writing clear bug reports that are well-organized in a bug tracker. That way, developers can immediately see what bugs they’re responsible for, reproduce them with minimal effort, and quickly dive into the problem to find a solution without wasting time. The result is more bugs closed per unit of time and higher quality software.
Essential Bug Report Components
Bug reports should follow a standardized format to streamline the report creation process for test engineers and make it easy for developers to find information. While the specific requirements may vary between companies, there are common components that most UI-focused bug reports contain, which are a good starting point for developing a template.
The most important components include:
TestComplete makes it easy to collect many of these elements without having to go through a manual process. Test engineers can easily share images, videos, or detailed logs captured during automated UI test execution in various formats — even if they don’t have TestComplete installed. You can even access video recordings to find issues visually without readings logs.
Best Practices to Remember
There are a handful of best practices that test engineers should keep in mind when writing bug reports. While these best practices may not apply to every situation, there represent some common practices that your organization may want to consider when developing your bug reporting processes.
The most important best practices include:
Looking Ahead
Clear bug reports and a well-defined bug reporting process goes a long way in improving functional UI tests. Using modern tools like TestComplete, you can quickly collect relevant screenshots, videos, and console data to share with developers to help them reproduce and quickly diagnose bugs without having to spend as much time collecting the material by hand.
In the next chapter, we take a look at cross-platform testing strategies for applications that need to work across mobile, web and desktop.