Tackling the Hardest Part of Functional Testing: Test Maintenance
  May 08, 2019

Recently, we hosted the webinar “Test Maintenance: Tackling The Hardest Part of Functional Testing Like a Pro” where we discussed test case and test step checklists, test case clean up, and test reviews.

Here are the top takeaways for creating an effective test maintenance strategy that will benefit your team and your test suite.

Easier Maintenance Starts at Creation

Good maintenance always starts with a good test plan, and easier maintenance starts at creation. You can take a look at your test plan like a filing system and search based on naming convention. 

The test plan should ideally include test cases and test cycles. Test cycles should include build information, environment information, and start and end date. 

Meanwhile, test cases should have a checklist of items including:

  • Defined test naming conventions - Including test case number is a must when it comes to test naming conventions because you can have the unique identifier linked to your requirement or issue. Additionally, the component, or the area of the app or website to be tested, should be included as well. 
  • Test attribute - These are aspects that describe the test case’s purpose and goals. They’re basically adjectives that describe what we want to accomplish with the test to help create streamlined test suites later on.
  • Priority - Priority should be seen as business criticality -- how important is the test to the dependencies that you have to critical workflows in your application? It’s easy to say categorize test cases as “low,” “medium,” or “high,” but justifying it by the number of users it impacts will help you understand what the priority truly is.
  • Description - The descriptor you add at the end should verify the functionality and purpose of the test. 
  • Test data - If the test data can be reused over time, mention the test data being used in the test case so you can create modular tests down the road. Additionally, include the type of test data, which is required to run the test. 
  • Assumptions - Assumptions, or preconditions of the application, are not always understood by teams, but they can be crucial in the backend debugging process.

Going deeper into the test case, a test step checklist can help outline a few guidelines to follow. Not only should the test case be numbered, so should the test step, which should have description, expected result, and actual result. 

The No.1 Rule for Test Maintenance

While test maintenance may seem straightforward, it can be much harder to implement due to the time, effort, and teamwork required. 

This is why it’s so important to keep tests granular and independent, which will help you run tests more frequently. Many times, problems with test maintenance arise when you attempt to add tests into the original test workflow, making it more complex. 

If you keep following this pattern as you add more features, it will quickly become brittle and hard to maintain. This is because the only way to understand the starting point for one test is to understand the test steps that came before it -- if any fail, every test run after it can fail as well. 

Coupling test and creating complex scenarios can not only create brittle tests, it also prevents you from running tests individually. Not to mention, automated UI tests are often slower, which can make this an especially challenging roadblock if you want to run a small subset of tests to shorten your feedback cycle.

Additionally, the following techniques have proven beneficial to test maintenance:

  1. Design from the end user perspective and take it into account before drafting the test case. Think about who the app is being used by and the requirements the functionality should address. 
  2. Follow test creation best practices for effective test writing, such as naming convention. Make sure that there’s a buy-in from stakeholders because there’s no point if everyone’s following a different process.
  3. Give details to the steps or required test data -- they shouldn’t just occur on the test case level. Specify the data the test should include to make it relevant.
  4. Even though we want to focus on granular tests, creating a test that is reusable is key, especially in automation so you don’t have to recreate the same steps in a new test.

Test Case Clean Up

Addressing what tests should be deleted will help you decipher which should also be created.

The following tests should be candidates for deletion:

  • Duplicate Tests - Sometimes we might copy and paste or test the same thing just to be safe, but duplicate tests are not something you want in your test suite.
  • Uninformative Tests - If the test fails and is not providing enough information, or if it’s checking too many operations, it’s not a valuable to the test suite.
  • Unreliable Tests - Flaky test, unstable platforms, or tests that can’t run everywhere at anytime do not belong in your suite.
  • Misleading Tests - Tests that always pass, have changing data sets, incorrect checks, or unclear naming conventions will not provide accurate information.
  • Heavy Maintenance Tests - Tests that have a lengthy setup, are hard to learn, and are verbose with long run times should be recreated to be modular.

Tackling Test Reviews

While we often talk about the benefit of code review, test reviews have also been beneficial to ensure that a test case is accurate, easy to understand and execute, and repeatable and reusable, while helping the team saving time and money.

Test reviews allow teams to find problems earlier and help teams understand if it addresses the requirements and looks at the application from and end user perspective.

They also enforce modularity and best practices, while encouraging knowledge sharing across the team by exposing people to different approaches.

Like code review, test review allows for a variety of roles to get involved such as arcitects, developers, and peers, and they can take place at anytime -- whether it’s before committing or after merging. 

When deciding what to review, you want to consider readability, performance, dependency on other systems, impact on other systems, and compliance. Having a checklist to help guide what to review can be extremely beneficial here.

Further Learning

Want to take a deeper dive into test maintenance? Watch the full webinar on-demand.