Artificial Intelligence for Faster and Smarter UI Testing

While artificial intelligence is the next step in empowering teams to deliver higher-quality software faster, it’s important to understand how this works today. User interface (UI) testing is notoriously difficult due to the underlying code constantly being changed during development. These updates make objects hard to find, test design brittle, and ultimately, will cause tests to break.

Top challenges with UI testing that AI can address, include:

Intelligent Test Design

According to the world quality report, 48% of respondents believe that automating the design process of test cases is the next frontier. Intelligent test design will speed up delivery cycles and team efficiency by reducing much of the manual effort needed to update the Document Object Model (DOM), evaluate and build a framework, and by helping testers prioritize their time through automatic risk profiling.

Object recognition: One of the top issues quality assurance professionals face in UI testing is not only recognizing objects, but also recognizing the application itself. Whether they are a programmer writing test software, an open-source test software user, or a commercial test automation framework, reading and interpreting the DOM is critical, so that each object can be described and used in an automated test.

In order to automate UI testing, the most basic functionality a tool needs is the ability to navigate the software and verify elements. Using AI to continuously gather and analyze data on the DOM will enable teams and the systems they use, to tackle object recognition challenges automatically.

Risk Profiling: As testing cycles become shorter, it’s inevitable that teams will forgo testing extraneous aspects of their application. It’s essential that the risks of doing so are profiled adequately, so teams can ensure they’re testing the most important features first. With AI, systems can identify what should to be tested or prioritized based on prior test history or risks that are ranked by how critical they are, their complexity, and performance.

Identifying and classifying updates, risks, and defects requires technology that can create AI classification algorithms. Classification, a data mining technique in machine learning, assigns items to a category so that its algorithms can successfully predict where a set of data belongs. Bayesian networks, the most common type of classification algorithm, can help define steps an ideal customer journey embarks on.

As AI-powered test automation explores these paths, defects from application updates and enhancements can be identified. An AI classification will then learn from correlations in past results to refine application test coverage through recommended test scenarios based on risk and identified root causes to quickly resolve defects.

Framework Generation: Building a UI automation framework is complex, time-consuming, and arguably one of the most challenging parts of the SDLC. With a well-built framework however, teams can optimize their automated testing processes and improve efficiency.

Having frameworks autonomously generated with artificial intelligence can ensure that all critical workflows of the application and the DOM are accounted for and tested against, can ensure you are applying automation to the most repetitive tasks to optimize your automation, and can help define your test strategy based off of data-driven insights from user behavior versus assumptions made by development teams.

At their core, AI-powered systems are based on a principle of inputs and outputs. In order for intelligent test design to work in software testing, users will need to upload existing tests for the system to analyze.

Based on the data provided, the intelligent system will be able to learn which application controls or properties are required and provide recommended outputs for new test cases and new test designs.

As each change to the underlying code of the application is made during development, the system will be able to update object definitions and properties to provide recommendations on what needs to be fixed. The more tests that are created or uploaded, the more data the system gathers about the application under test and will be able to continuously learn and optimize.

Artificial Intelligence (AI) is the capacity of a computer has the ability to learn and adapt like a human being. There are many perceptions of what AI is, driven by pop culture and represented in movies and TV shows like Westworld, ExMachina, and Altered Carbon. This fantastical portrayal of AI that looks like a human and can formulate ideas and express emotion is a long way from reality. What then are we referring to when we say ‘artificial intelligence?’

A common misconception of AI is that it is used interchangeably with the terms ‘machine learning’ and ‘deep learning.’ To understand how AI today works, it’s important to understand each, and how they tie together:

• Artificial Intelligence (AI): Any technique that enables machines to imitate human behavior.
• Machine learning (ML): A subfield of artificial intelligence that gives computers the ability to learn without being explicitly programmed. At a more basic level, it is the computer’s use of statistical methods and algorithms to learn and improve from experience.
• Deep Learning (DL): A subset of machine learning that layers ML algorithms to extract data from massive data-sets.

Artificial intelligence refers to the scenarios in which machines mimic the cognitive functions associated with human minds such as learning and problem-solving. The key difference between the definitions is that AI is often viewed as an umbrella term for all three and will encompass both ML and DL. Many AI-based technologies today leverage machine learning or deep learning algorithms to perform necessary functions.

These intelligent algorithms have weaved their way into much of the technology industry, and are being used by companies like Facebook, Apple, and Amazon. While some use cases may be more obvious than others, AI is already impacting our day-to-day lives across industries

Examples of Artificial Intelligence in Modern Technology

Consumer Electronics: Digital Personal Assistants:

Siri, Apple’s voice recognition software on its phones and Alexa, Amazon’s smart speaker for households are the epitome of AI being used in modern consumer technology. People can use Siri or Alexa to search and the browse the web, look up directions, book appointments, and manage many other daily activities.

Both assistants use machine learning algorithms and data collected from consistent use to understand our habits, better predict our requests, and make recommendations. Alexa now also connects with Nest, the smart thermostat, which uses behavioral data to adjust a room’s temperature automatically based on home owners’ personal needs.

Automotive Industry: Self-Driving Cars

Google’s self-driving car project, Waymo, is another notable example of how AI is being used today. Engineers developing autonomous vehicles worked in conjunction with Google’s Brain team to apply deep learning algorithms to the car’s detection system for pedestrians. In just a few months, the team was able to drop the error rate by 100 times, making the cars on the road safer and more reliable.

Other automotive makers have also included ‘auto-pilot’ features. Tesla recently released ‘hands-free driving’ capabilities, which uses advanced algorithms combined with mapping and sensory data collected by the cars to continuously learn and improve its driving capabilities.

Technology Software: Smart Conversations:

Chat bots have cropped up across every major industry and are commonly used for answering customer service requests. While responses can be pre-programmed, companies can’t predict every question they will receive. Some questions require a massive amount of data to be analyzed, and chat bots will need to rely on machine learning algorithms to provide a detailed answer.

Another example is Google’s recent ‘Smart Reply’ feature. Google now provides sample replies for users to leverage when responding to emails. These pre-determined responses are tailored to the user. Google’s underlying machine learning algorithms have evaluated your prior conversations over the past few months to a year to suggest an email response that matches your communication style.

Media & Entertainment: Movie, Music, and Purchasing Recommendation Engines

Two of the biggest multi-media platform examples that leverage artificial intelligence are Netflix and Pandora. Both use machine learning algorithms to track and analyze data from consumer viewing history or listening habits and provide recommendations based on similar movie or song attributes.

Amazon it also notorious for making product suggestions based on anything you’ve previously bought or viewed. What each of these examples require is the ability to access, process, and analyze vast amounts of data. Without it, machine learning will lead to unreliable and unscalable results. As we move towards an increasingly interconnected world from smart phones to smart fridges accessing the same networks and data storage increasingly more robust, we are paving the way for artificial intelligence-based technologies; And the more data we accumulate, the more accurate machine learning models will be.

UI testing tools today such as TestComplete, have already started to incorporate intelligence in testing to alleviate many of the challenges testers face, by infusing object recognition capabilities with AI.

TestComplete is an automated testing tool that enables teams of all skill sets, to build, run, and manage UI tests for desktop, mobile, and web applications. The tool, now infused with AI, will help teams run smarter tests faster than ever with three key features:

• Hybrid Object recognition with AI: When application properties are not detected, the Object Recognition Engine will automatically leverage artificial intelligence to find objects visually. This will help teams capture UI elements at a textual level, regardless of the individual’s technical capabilities, to broaden test coverage.
• Intelligent Recommendation System: When objects can’t be found, the Intelligent Recommendation System in TestComplete will step in and recommend a fix. It will also automatically acknowledge when application changes have been made, and prompt the tester to make the necessary updates, or to delete unused application components.
• Support for Modern Web Frameworks: TestComplete supports dynamic, custom attributes for modern JavaScript frameworks. The tool easily enables teams to test popular applications based on React and Angular Web frameworks.

With the tool’s AI-powered visual recognition feature, teams can test notoriously difficult UI application components such as charts and PDFS, as well as enable SAP and Mainframe testing.

Organizations are excited about the potential impact artificial intelligence and machine learning could have on a businesses’ bottom line.  As AI becomes more advanced, companies will be able to process larger quantities of data to uncover trends and insights in customers’ behavior not previously possible.

Alternatively, mundane activities in the day-to-day lives of consumers will become easier and more automated. Imagine having your groceries delivered to your door by self-driving cars, from an autonomous servie triggered when your smart fridge notices you’re running low on certain items. AI will not only revolutionize how we live our personal lives, but also transform business processes globally. The software industry is no exception.

From a business perspective, AI promises to:

• Boost efficiency by enabling the workforce, regardless of industry, to get more done in less time, and in a more consistent and accurate manner. Artificial intelligence will be able to eliminate the mundane and repetitive tasks people currently do, allowing them to focus on more complex responsibilities.
• Speed up data mining and analysis for organizations everywhere. AI and machine learning algorithims can more quickly process complex data sets and provide invauable insights into consumer behavior and purchasing patterns.
• Improve consumer experiences, and ultimately boosting a company’s public perception, by providing 24-hour access to their services and support through technologies like chat bots.
• For the healthcare industry, AI could help doctors diagnose patients faster and more accurately with access to vast amounts of information on all sorts of diseases, viruses, and a patient’s medical records and family history. AI could help researchers identify patterns in various infections and pick out at-risk patients before they show any symptoms.

Take Autodesk Dreamcatcher AI as an example. Dreamcatcher AI is a 3-D Computer-Aided Design software tool that can auto-generate designs based on pre-defined goals and constraints.  A designer can provide a list of requirements about their desired product (i.e., functional needs, material types, cost restrictions etc.) and the tool will generate thousands of design options. Imagine the designer is developing a bike will the following requirements:

• Can hold up to 300 pounds
• The seat is 18 inches off the ground
• Materials cost less than $75 per unit

The machine learning algorithm in Dreamcatcher AI will take all of the data inputs into consideration and pull specs from thousands of available bike seats on the market to provide product design suggestions. This process done manually by the designer can take hours, if not days or weeks to research, develop, and produce the same quantity and quality of options. Saving upfront time during the design phase opens a door to creative concepting not previously possible as the designer can now view thousands of iterations and optimize, as opposed to developing two or three.

As a software professional, imagine being able to lay out the functional requirements of your product, and the environments you need and have an intelligent system automatically create a framework based on your design specifications. Or what if the system could automatically generate GUI tests based on historical test data and consumer behavioral data collected on how your customers interact with your product.

The benefits for AI in software testing are astronomical. While still in its infancy, artificial intelligence is starting to impact software development and quality assurance and seems to be the natural progression in the race to seamless and fast delivery cycles.

The demand for getting the latest and most up-to-date products into the hands of consumers quickly has forced the software industry to evolve and adopt new practices and develop new tools to increase the speed of the software delivery lifecycle (SDLC). Trends around robotic process automation (RPA) and open source platforms are continuing to shape business priorities to reach new levels of speed, quality, and costs – and it all starts with automation.

Automated testing across the board is an underutilized practice, with teams automating less than 50% of their tests . Only teams leveraging automation can take full advantage of Agile and continuous testing practices and eventually, autonomous testing.

Maintaining applications is a painstaking task due to the complex architectures and workflows of the software, which are intimately intertwined with critical business processes. While traditional automated testing tools and cycles still require some manual intervention by skilled technical engineers, artificial intelligence will be able to mitigate the risks associated and help teams successfully scale their automated testing efforts.

Predictive Self-healing: Application changes are a regular and needed occurrence, but updates made will typically result in UI tests breaking because objects can’t be found.  This causes major headaches for testers responsible for executing tests, who need to also maintain those tests suites and object repositories. With an AI-infused tool, test suites could dynamically be updated when application changes are made, cutting down on the number of flaky tests and time spent on maintenance.

Intelligent Bug Hunting: Exploratory testing is a manual practice that is vital to the success of organizations today. Testers are required to act as the end user and manually click or interact with the application’s UI, effectively exploring the software, to catch hard-to-find bugs.

Many of these are difficult to find with average automated processes because of dynamic properties.  While there are some aspects of software, like application aesthetics, that still require a human eye, artificial intelligence will be able to automatically run ad-hoc or exploratory tests and discover bugs quickly to provide rapid feedback.

By relying on AI to conduct exploratory testing, features can be tested as soon as they are shipped, and critical bugs can be found. This is ideal in agile environment, where exploratory testing is conducted as a first pass.

Cloud-based virtualization: Managing test environments is another time-consuming task that is often reduced to the bare-minimum needed to complete testing. Over the last few years, the number of mobile phones, tablets, computers, browsers, operating systems, screen sizes, and resolution has increased exponentially, introducing thousands of environment configurations for teams to filter and choose from to run their UI tests on. To add to the complexity, every year, new versions of devices are released, and some are made obsolete.

Having to continuously maintain real devices is expensive. While some tools will provide cloud access to all available environments, testers still need to configure their tests and identify which environments are needed. With artificial intelligence, historical test results, and user data can be analyzed to help teams automatically identify the required environments and have them spun up.

Process Automation: End-to-end testing is a process in which teams will validate an entire application workflow, from start to finish.

The root goal of running end-to-end tests is to ensure critical business processes function properly, across an entire application’s architecture and user scenarios.

Automating end-to-end testing is possible, but is extremely hard to maintain and scale. With an artificial intelligence, automation tools can easily access and validate both front-end and back-end systems and data. They won’t be limited to the application at hand, but can spread their coverage across system integrations.