Six Stages from Manual to Autonomous Testing

A decade ago, artificial intelligence (AI), machine learning (ML), and even automation were barely fathomable concepts in the software testing industry. Automated testing has since become the cornerstone practice for teams looking to implement modern development approaches like Agile and DevOps, or to simply keep up with the increasing market demand for high-quality software.

Teams can now tactically implement automated testing to reap the benefits of reduced testing cycles, faster times to market, improved test coverage and accuracy, and enhanced team efficiency. AI and ML have even begun to weave their way into testing and development, with automated testing tools incorporating intelligent algorithms into product features that can be leveraged by QA teams to enhance testing.

Despite the acknowledged benefits of automation, less than half of QA professionals are automating more than 50% of their tests. Graphical user interface (GUI) tests are notoriously difficult to automate, and as a result, the majority of GUI test cases are still conducted manually. For QA teams responsible for GUI testing, 42% will run their tests via a combination of both manual and automated testing, with 31% running their tests solely via manual execution.

Moving from manual testing to automation can be daunting for many software teams. The transition between the two approaches requires not only a change in regulations and infrastructure, but also mindset - especially when it comes to anything that is process-related. Most importantly, incorporating automation into testing cycles demands a guarantee that it is just as reliable and accurate as manual testing.

The Society of Automotive Engineers (SAE) describes the transition to fully autonomous vehicles in six stages split between two phases of driving: monitored driving and non-monitored driving.

Monitored vs. Non-monitored Driving

The first three levels focus on the driver being responsible for monitoring the car and making decisions. In the latter three levels, the driver starts to relinquish controls to not being needed at all (i.e., self-driving cars). The biggest transition for the driver is between monitored and non-monitored testing.

Advanced Testing vs Autonomous Testing

Transitioning from driver to driver-less capabilities not only applies to the autorotative industry, but also software development and testing. With each stage, tool capabilities in software testing provide QA engineers and testers with the ability to rely on automation to speed up software development.

The first four stages in the transformation process from manual to autonomous testing focus on building advanced automation capabilities. The tester, or team, moves from manually building, executing, and maintaining tests to leveraging an automation tool to assist with those aspects of their testing cycle. 

In the latter two levels, teams will start to accelerate their testing processes and relinquish control to the tool, and in an idealistic near future, reach the final stages where tests can be automatically created and executed without human intervention.

Deep-Dive into the Six Stages

Walking through the six stages will help you understand how to move from manual to autonomous testing. At each stage, we will cover:

• Definitions to describe what each stage entails
• Features in TestComplete to help optimize UI testing during each stage
• Best Practices to help progress to the next stage

Stage definition: Manual testing is the process in which a tester acts as the end user to validate that a piece of software works and behaves as expected. The goal of the tester is to find defects within the application by physically clicking or interacting with the product.

In manual testing, the tester is in complete control of the test scripts at all times. They are responsible for test creation, execution, maintenance, monitoring, and fallback scenarios for when tests fail. More than half of all testing today is done manually, most of it on the UI layer. While the use of API and automated API testing grows, there seems to be a larger struggle for teams transitioning to test automation on the UI layer.

Stage 1 Best Practices:

• Be an Application Expert: The number one best practice that should occur throughout your transformation and especially when conducting manual tests is to understand the intricacies of your application.
• This will help ensure a quality user experience and instill confidence in your application’s behavior after each build. The top two ways to do this is to read documents about the application beyond just the requirements and to conduct exploratory testing to understand the most critical user scenarios that are important to your customers.  

Stage Definition: The first step to ensuring a complete and successful transformation is with assisted automation. In this stage of automated testing, a quality assurance (QA) team or individual tester will leverage an automation tool to assist them with test creation and execution. However, at the end of the day, the tester is still required to manage and maintain the test scripts manually to ensure quality.

This is a critical stage for moving from manual to automated testing because it required you to define your automated testing framework. The more organized and structured an automated testing framework is, the more efficient, stable, and scalable your automated testing efforts will become.

The Importance of Object Identification in Automated UI Testing

Most tools on the market today have features that will address your needs at this stage. For UI testing specifically, it’s vital to the success of any automated process that the tool has robust object recognition and object storage capabilities (i.e. an object repository).

Automated UI testing tools, including TestComplete, work by finding objects through the Document Object Model (DOM) layer. The tool will capture elements that exist on the user interface. Each object captured has properties or attributes that can be used to describe what the user is seeing in the application and where it lives, such as number of pixels, colors, handler ID, and text.

These objects are stored in an Object Repository and can then referenced in automated tests via script. Having the ability to accurately locate, identify, and test objects on the screen is central to ensuring your UI tests function properly and provide enough test coverage to instill confidence in your application’s behavior.

Automated Testing Tool Features to Leverage in TestComplete

Popular tool features that can assist with testing at this stage, especially for those new to automation, are those that can accurately identify objects and enable testers to create and run tests without having to write scripts.

Object Spy: It’s essential to your success that your automated UI testing tool is able to accurately identify and store objects on your user interface. In TestComplete, the Object Spy will enable you to explore the properties and methods of any UI element in your application, which can be added to its object repository and hierarchy referred to as Name Mapping.

Record and Replay: Record and replay features allow testers to build UI tests by recording on-screen user actions such as keystrokes, typing, and clicks, and then playing them back with different scenarios to compare results against the expected behavior. The benefit of this technique is that it requires little to no scripting skill, meaning anyone can create and run automated tests. In TestComplete, getting started with test automation is as simple as clicking Record on the Record and Replay toolbar.

Keyword-Driven Testing: In keyword-driven testing, automated UI tests consist of a myriad of words (keywords) that define an action to be performed in that test. These can be actions mentioned previously, such as mouse strokes, clicks, but also dragging and dropping items or opening and closing application windows.  For those just getting started with automation, the easiest way to create keyword-driven tests is by recording them with Record and Replay.

In TestComplete, you can combine a series of keywords to create an operation, or the tests. In the image below you can see the operations, or test actions, on the left, and the test case on the right. TestComplete makes it easy for teams or individuals to create complex automated test scenarios with simple point-and-click actions without any expertise in scripting languages.

Data-Driven Testing & Data Generator: Data that is hardcoded into a test inherently means that test can only be used for testing one scenario. In order to test a different data set, you would need a separate test.

Testers regularly need to re-run tests which would make this process not only time-consuming, but also a headache for maintenance. For anyone moving to automation, this simply isn’t practical. Creating a test for every scenario will become the bottleneck in your journey before you’ve even truly gotten started. TestComplete will enable you to separate test data from test commands, meaning you can run UI tests with different sets of input data.

The data you need can be stored externally in excel or csv files and called upon in a test command. Manually generating test data however, can take an exorbitant amount of time. TestComplete comes with a built-in Data Generator, which will enable you to generate data sets such as email addresses or social security numbers. As shown in the picture below, all you have to do is specify the parameters you are looking for. 

The benefit of data-driven testing with TestComplete is that it will improve team efficiency by cutting down on test times, increasing coverage by enabling you to cover more scenarios, and reducing maintenance efforts. All critical components in your journey to an autonomous process.

Stage 3 Best Practices

• Pick one language for your test scripts - When you’re just getting started and building out your automated testing framework, you should pick one language to build your tests in. Whether in a large or small company, having a standardized language for tests will make sharing code and reusing code easier, will help with training employees, and will facilitate better communication between testers and developers.
• Create test tags for each test - Outline your test types and define what each test is in your framework. By tagging each test appropriately, you set up your framework for reuse.
• Link each test with a requirement number - Linking each test with a requirement number helps monitor your test coverage and testing workflows. This will also make it easier to analyze test results and load those results inside your test management tool.

Stage Definition: In this next stage, the test automation tool will take control of various aspects of test creation, maintenance, or execution, but the tester is yet again still responsible for monitoring the tests and the application.

Automated Testing Tool Features to Leverage in TestComplete

Browser & Device Loops: In web and mobile testing, it’s essential to validate that your application works across the various and numerous types of browsers and devices available to consumers today. Having a separate test for each configuration is not scalable for teams that are looking to move quickly.

In TestComplete, you can leverage the Browser Loop operation to launch multiple web browsers and have your testing tool execute each test case in each browser. In a similar fashion for testing native mobile applications, you can set up a Device Loop operation in TestComplete to execute tests across the mobile devices of your choice.

In both scenarios, the tool itself is taking on responsibility for executing the same web or mobile test across multiple browsers and devices, speeding up testing cycles and reducing duplicate test steps for the team. Once those tests are run, the tester is required to analyze the results and verify tests run successfully to ensure application quality.

Data-Driven Loops: In Stage 2 we discussed what data-driven testing is, why it’s important to leverage for those just getting started with automated testing, and how the Data Generator in TestComplete can save time. Setting up a Data-driven loop in TestComplete is the operation you would use in the tool to run multiple data sets through a single test. All you have to do is select where you would like the loop to pull data from.

Intelligent fixes & updates: One of the most useful features in TestComplete is the tool’s ability to provide in-depth insights into failed tests and provide recommendations on how to fix common UI testing errors.

After running a test in TestComplete, you can analyze the results by viewing the test log and clicking on any error to identify which test step caused the test case to file.

One of the most common errors testers will come across in UI testing today is, ”Object Not Found.” When updates are made to an application’s underlying code, object properties can change, and GUI elements can move, making UI tests extremely brittle.

In TestComplete, when an “Object Not Found” error occurs, the tool’s Intelligent Recommendation System will notify you that this happened, and will provide suggestions for how to fix the issue.

When development makes changes and object properties are adjusted, the tool will also compare the updates with your existing Object Repository.  If there are any discrepancies, TestComplete will notify you of the update. The tool will also notify you if any objects or their properties are no longer being used at all.

With the Intelligent Recommendation System, maintaining automated UI tests has never been easier. For those just starting out with automation, finding out why tests failed and understanding how to fix them is critical.

Stage 3 Best Practices:

• Set up shared utilities - create shared utilities for common user interface (UI) actions such as menu navigation and text input fields. These are the basic building blocks of your tests, which you can then piece together to form test logic and reuse in test automation. This will also help anyone looking at your test to determine if failures are an asset problem or an actual defect in the application.
• Create verification points -  Verification points help with test maintenance. For example, if you’re verifying that the text field only accepts numeric characters and not text, you only have to update that in one place and not every single test when you build a verification point.
• Your verification data should also be shared. Different input utilities should be able to accept data objects created in previous steps so you can chain items together. 

• Take notes on your automation efforts: As you start to create automated test, it is easy to forget why or how portions of your application testing were automated.
• Throughout the entire testing process, you should be recording and taking notes on all your automation effort: data preparation, and environment and verification building. By creating a good mechanism here, you are able to create an outlet for feedback and continuous learning to then be leveraged at the start of the next test cycle.

Stage Definition: In Stage 4, the test automation tool takes control of different aspects of test creation, execution, or maintenance to accelerate automation, but the tester is responsible for monitoring the application to ensure quality as long as the conditions do not change.

Automated Testing Tool Features to Leverage in TestComplete

Distributed Testing & Parallel Testing: Both parallel and distributed testing will enable your team to run UI tests simultaneously across different machines, meaning you can increase test coverage and get more done in less time. While very similar, parallel and distributed testing do have distinct use cases.

In distributed testing, you typically have one test that has several components or steps which are dispersed across different machines. These will coordinate with one another during a test run to ensure tests are run in a sequential manner.  Take a chat platform like Slack for example. To test that the chat is working, you would need to run a single test that ensures messages can be received and responded to on two separate computers. The step on one machine would have to listen for a passed step on the other machine to continue running the test. In TestComplete, this can be done with the tool’s Network Suite.

In parallel testing, functional tests are run across multiple machines simultaneously, but also individually in that the tests do not interact with each other. In TestComplete, parallel testing can be accomplished through the tool’s add-on, TestExecute.

In each scenario, TestComplete is taking control of test execution across predefined browsers, devices, or virtual or real machines. As long as the initial test script and environments do not change, the tool will automatically leverage the same test case to run concurrently. This will cut down on testing cycles and allow your team members to turn their attention to more important tasks than recreating duplicate tests and test steps.

CI/CD Tool Integrations: Continuous integration and testing are the building blocks that make a continuous delivery engine. Continuous integration is developing your build in smaller packages and continuous testing is the ability to automate your tests to allow teams to detect issues earlier and obtain immediate feedback on business risks.

To successfully implement continuous integration, tools such as Jenkins, Bamboo, Travis CI, can be used to run automated tests after each build on one or more computers to build, test, and deploy your software regularly.

Many software teams use source control repositories such as Github to help developers store and manage their code. CI tools such as Jenkins continuously listens to any changes coming from Github. Once it sees a change, it pulls in the new source code and will run unit tests and further analysis against it. If something fails, you can catch defects quickly and if it all passes, any CI system can be triggered from the command line in tools like TestComplete to automatically take control and run functional UI tests against the application.

TestComplete integrates with Jenkins, Bamboo, and Git, along with other CI/CD tools and source control management systems, making it a powerful automation tool that fits seamlessly into any continuous pipeline.

Stage 4 Best Practices:

• Keep your workspace clean – Wipe workspaces before or after jobs to keep a “clean” environment at the start of build or test phase. Things can get messy and disorganized fast with each build. For example, if there is a Java Version upgrade, a build tools needs to be replaced, projects are refactored and renamed in Git, or if new projects are created or are deprecated, then your workspace will instantly become obsolete. Keeping a clean workspace will help better manage your continuous pipeline from the start of each build phase.


• Use Parallel Builds – Parallel builds speed your development and test execution time. Jenkins has this capability allowing you to get through testing your build quickly on one or more test computers.


• Use a Source Control Management – Source Control Management tool to store and manage code, scripts, and configurations you have inside your delivery pipeline. Jenkins and Git work together well to capture changes and provide statistics such as build times to help speed up delivery times without sacrificing quality metrics.

At every step in the QA cycle, we see AI infusing itself to mitigate key software development issues and accelerate test creation, maintenance, and execution -- making it the most important technology for software teams to understand today. Stages 5 and 6, advanced unmonitored driving, represent the future of software quality in which tools extensively infuse AI and ML capabilities and testers abdicate more control to the tool to manage key day-to-day responsibilities for a QA engineer.  

Stage Definition: In Stage 5, the automation tool takes control of different aspects of test creation, execution, or maintenance to accelerate automation and the tester becomes less responsible for monitoring the application to ensure quality when conditions change.

Automated Testing Tool Features to Leverage in TestComplete

AI-powered Visual Recognition in TestComplete is test automation’s first step to relinquish its object recognition capabilities to the tool. AI-powered visual recognition uses Optical Character Recognition (OCR) to extract and validate text found embedded in dynamic images, charts, mainframes, and PDFs.

Even when the UI layer changes or an image is updated, the tool does not depend on any identifier or property hard-coded in a test script, but can take control to identify what is available on-screen. By applying AI to OCR, libraries are able to stay up-to-date, allowing the tool to take control of maintenance and be more reliable when testing new parts of an application.

Compared to the other technologies, AI-powered visual recognition in TestComplete is twice as accurate, making it possible for UI automation engineers to detect and test against application components not previously recognizable and abdicate more control to the tool.

Stage Definition: In Stage 6, the automation tool will be redefined as an autonomous tool and will be able to take control of test creation, maintenance, and execution. It will be able to learn from failed tests and make decisions on how new tests should be created and executed even when conditions change.

Currently, autonomous solutions in software testing are still in infancy. Functional testing tools have adopted various forms of autonomous capabilities from discovering an application structure to predictive self-healing to intelligent bug hunting. However, end-to-end autonomous testing solutions have yet to be widely adopted by large enterprises.

Not every organization is ready for truly autonomous testing. Embracing total autonomy requires the combination of human and machine to ensure scalable, stable, and secure AI-powered systems. For teams to successfully transition from manual to advanced automation to autonomous testing, hybrid engines must exist that marry humans and machine capabilities.

Teams can benefit from modular AI investments in software testing through hybrid testing solutions to address key issues and lessen the burden from time-consuming tasks. For example, software teams that leverage hybrid automation testing tools that combine property-based and AI-powered visual recognition will realize high levels of accuracy from directly creating tests based on during test creation and ease of test maintenance.

Property-based recognition relies on creating test script hard-coded against identified application properties, while visual recognition depends on the tool’s ability to capture UI elements dynamically. Leveraging both will allow software teams to benefit from each approaches’ strengths -- stability and accuracy from property-based recognition and testing of the former, and the speed and scalability from AI-powered visual testing of the latter.

The advancement of technology is gaining speed, and software teams in every industry are inquiring about its implications to produce highly stable, scalable, and secure applications. Teams face a spectrum of alternative approaches and tools to tackle development and quality goals -- sometimes inadequate and often hard to use.

Infusing AI with traditional automated testing tools creates a hybrid engine to help many teams with their transformational journey from manual to advanced test automation.