How is End-to-End Testing Performed?
Without conducting end-to-end testing, you risk your entire application collapsing in the event one of those sub systems fails. For example, imagine you are purchasing a software product via an ecommerce platform. From your perspective, as the end user, your process may resemble the following:
- Go to the website
- Navigate through to the pricing page (perhaps you stopped to read about the features, or product specifications along the way)
- Add desired items to your cart
- Click ‘check-out’
- Proceed to a payment page where you input your name, credit card, address, and contact information
- Click ‘Submit’
- Finalize purchase
Let’s also say that in this scenario, you receive a ‘thank-you’ email from the vendor, as well as a follow up email with the product that is ready to be downloaded. From start to finish, you would expect this process to flow seamlessly, from the website loading properly, to being able to navigate through the ecommerce process, to receiving the email and being able to download and use your new product.
An end user will rarely think about the back-end systems that are triggered each step along the way. For the quality assurance (QA) team looking to validate the ecommerce platform, the number of steps can make testing the platform complex and time consuming, but the array of back-end systems needed to make the process seem flawless can make it nightmarish.
The underlying pieces needed to make sure the website doesn’t 404, that the user interface (UI) behaves and looks as expected (i.e, buttons work, input fields are accessible), that the private information you insert is passed on, stored, encrypted, and secured, and that the emails and product are delivered, could consist of a variety of different external software products. Let’s say the company uses a cloud-based customer relationship management (CRM) platform such as Salesforce.com to manage their business data, a marketing platform such as HubSpot or Marketo to manage their outreach, and a separate system for ecommerce. They will also need a firewall and a network in place to connect each of these pieces while protecting the user’s information.
End-to-end testing in this example, will not only test the workflow from the user’s perspective and validate the various user interfaces, but also ensure the flow of information between the CRM, the marketing platform, network, and the firewall all work as expected.
Key Steps in Setting up End-to-End Tests:
As a testing method, end-to-end testing consists of many different parts, but the most common steps you’ll take in setting up your end-to-end process are as follows:
- Review the requirements you’ll be using end-to-end testing to validate
- Set up the test environments and outline the hardware /software requirements
- Define all the processes of your systems and its integrated subsystems
- Describe the roles and responsibilities for each system
- Outline the testing methodologies you plan to use and any testing standards (i.e., language, tools)
- Create the requirements for tracking and designing your test cases
- List the input and output data for each system
End-to-end testing is essential for those looking to scale their testing efforts. As your application, or software system grows, you’ll need to not only ensure it works across a vast array of devices, environments, and operating systems, but also validate that your APIs, database interactions, and network communications function and perform as intended.
The thoroughness of the method will not only increase the productivity of the application, but also your confidence in the product prior to pushing it live.
Vertical vs. Horizontal End-to-End Testing
There are two types, or two ways to perform end-to-end testing – vertically and horizontally. The type most commonly associated with end-to-end testing, and used most frequently, is the horizontal variation in which testing occurs across multiple applications. Take a look at the diagram below.
In our previous scenario, the ecommerce platform needed to not only work from a UI standpoint (i.e., navigation, payment), but also needed to connect with a CRM, marketing platform, and network. A horizontal end-to-end test would entail verifying each workflow or transaction through each individual application from start to finish to ensure that each related process occurs correctly. While this type of test could occur within a single application or interface, many software systems today interact and connect with external applications as we saw previously, both from third-party vendors or those developed internally.
A vertical end-to-end test entails testing each layer of a single application’s architecture from top to bottom, meaning you’ll work your way through each sub system independent of one another from testing at the granular level with unit tests then continuing on through tests at both the UI and API layers.
Both types of end-to-end testing, horizontal and vertical, will help you accomplish the same goal, but each have a separate set of advantages and prerequisites that may make it easier for some teams than others to implement.
Prerequisites & Advantages for Horizontal Tests
With horizontal tests, because you are verifying an entire workflow across multiple applications, it’s vital the test environments for your system and all of its sub-systems are set up in advance.
Advantages
- Entails focusing tests on the user perspective
- Prevents issues from reaching production
- Ensures coverage for business logic requirements
Prerequisites & Advantages for Vertical Tests
For vertical tests, since you’re focusing your tests on the architecture of a single application, you’ll want the support of a testing or development strategy such as behavior-driven development, test-driven development, or continuous testing. You’ll want to ensure that each major stakeholder, such as the developers, tester, product owners, are involved in the project.
Advantages
- High code coverage
- Faster test execution
- More focused tests
- Useful for safety critical software
4 Key Metrics for Measuring End-to-End Test Success:
As with any process, you’ll want to ensure your time and efforts haven’t been wasted, and the best way to do so, is metrics. While there are a vast number of pieces you need to keep track of, the following four measurements are the best way to gauge how successful your end-to-end tests really are:
- Test Case Preparation Status: This can be tracked via graphs visualizing the progress of your planned test cases that are being prepared against the number of ones that were originally planned.
- Weekly Test Progress: The goal with this measurement is to analyze the progress and details of your completed tests week-by-week. Ideally, you’ll want to track the percentage of total tests completed, as well as the number of ones that passed and failed.
- Defects Status & Details: You’ll also want to track these measurements weekly and focus on the percentage of defects that were opened and closed, as well as the distribution of each based on their severity and priority.
- Test Environment Availability: Here you’ll be tracking two key measurements, the amount of time allotted for running tests in a specific environment as well the amount of time actually used while conducting your end-to-end tests in those environments. For example, you’ll want to count the total number of hours each environment is "up" and the total number of hours scheduled per day for testing.