Introduction
The process of writing, executing and managing test cases forms the foundation of ensuring software quality. But what does it take to write a high-quality test case? Let’s take a deep dive into an actual, tangible example of what a good test case looks like. How might we construct a test case for an actual website, and what might managing that, and other similar test cases, look like?
Test Cases and Test Case Management
To recap briefly, test scenarios describe a general user's objective when using an application. Test cases serve as the building blocks of test scenarios. They get more specific about, exactly, what we should test. And finally, test scripts exist as a component of the test case, supplying one or more exact descriptions of how to test the test case. You may execute these scripts manually or in automated fashion.
The process of keeping all of that straight is known as test case management. And test case management is difficult. It's so difficult, in fact, that you'd be quickly lost without some kind of system or tool to keep all of it straight.
That's all review from the previous post, but it's worth repeating for context. I mention it here because, in this post, we're going to take a look at specifics. What are some examples of these things?
Car Loan Payments: Let's Create a Test Scenario
For example, let's imagine that we're in charge of testing and general quality assurance for this site. The site advertises itself as a car payment calculator, but if you look at the main menu, you'll see it offers a lot of other calculations as well. But let's limit our interest in this to the car payment calculator functionality.
In case you aren't familiar with this sort of thing, I'll explain briefly. When you go to buy a new car, you generally see the car's price listed as a lump sum. For instance, at the time of writing, Honda lists its Civic model's suggested price at $18,740. But most people don't walk in with a stack of bills and purchase the car like a pack of gum. Instead, they finance it with an auto loan and then make monthly payments. And those monthly payments are really what interest most buyers. But a number of different factors and variables affect the amount of the monthly payment, so handy calculators exist to turn sticker prices into monthly payment figures for prospective buyers.
First up, we need to create a test scenario. Recall that we reason about test scenarios as user objectives. Why might a user visit a site that advertises itself as a car payment calculator? That seems like a good place to start. For our first scenario, let's keep it nice and simple: confirming that a user can calculate monthly car payments.
Creating Our First Test Case
Alright, let's now create a test case for this scenario. You're going to find that there's a certain art to slicing up your test scenarios. You could leave the test cases pretty vague or get really granular. To know which route to go, you'll need to get a feel for your organization's particulars, such as willingness to invest in testing and the cost of defects.
For our purposes here, I'll go kind of middle of the road. The payment calculator has 6 variables: price, down payment, trade in value, tax rate, interest rate, and loan term. Every loan needs a price and a loan term, but the rest are optional. So I'd want to create a basic test case, a test case that isolates each of the additional variables, and one that takes everything into account.
Let's create the following test case: Basic Loan with only Price and Loan Term.
Adding Information to the Test Case
We just created the title of the test case. But recall from the last post that the test case actually encapsulates a lot more information. Let's address the fields that I mentioned in the last post.
- Test ID: You should use a test case management system, and it will generate this unique identifier for you. (If, for some reason, you manage this manually, I'd use something simple like a number that you increment for each case.)
- Title: Basic loan with only price and loan term.
- Related requirement/scenario: Confirm that a user can calculate monthly payments (your test case management system will actually use the scenario's unique ID to make this association)
- Remarks/Notes: "This is the simplest scenario where we calculate a monthly payment with just loan term and price, leaving everything else as 0."
- Script: We'll add this shortly.
- Pass/Fail Status: N/A (Your test case management system will fill in something here, probably indicating a state where you have not yet executed this, but perhaps defaulting to fail.)
- History and Audit Trail: Created on today's date (Test case management systems will almost certainly manage this for you, without you entering anything.)
Test Cases in The Wild
As I said in the last post, this is a relatively minimal set of concerns. A test case management system will definitely give you more fields that you can fill out if you choose. For instance, "test priority" is a common one. If we were going to fill that out here, I'd say this priority is "high" since this basic functionality represents the backbone of the application.
But we could go way down a rabbit hole with every possible field, so let's stick to the relative basics. Your tools will offer as wide a range of fields as possible, to account for all of their customers. But you don't need to use all of them, and I would ignore fields for which you don't have immediate use, lest you create make-work data entry.
Adding a Script
At this point, we've defined the scenario and the test case. As you can see, this gives us a pretty good feel for what we're testing, but not exactly how to test it, just yet. We want to verify that users can calculate monthly payments, and we want to do so for the specific case of a loan with just a price and a loan term. But what price? And what loan term? And how, exactly, do we do that?
Well, we do that with a test script. Here's one:
- Navigate to the site's main page.
- For Vehicle Price, enter 30000.
- Down Payment: enter 0.
- Trade-In: enter 0.
- Tax Rate: enter 0.
- Loan Term: enter 5.
- Click Calculate.
- Verify that the monthly payment says $500.00
You'll associate this test script with the test case. Some test case management suites will specify one script per test case, while others may allow you to associate multiple test scripts with the test case. The idea here is that you might want to add an alternate script where you entered $60,000 for the price and then verified the monthly payment as $1,000, for instance.
What About Automation?
You'll notice that the test script above is very human-centric. It gives a tester a play by play of what to do and what to verify. But what if you want to automate this?
Well, you certainly have that option. One possible solution would be to use Selenium, an open-source browser automation tool. You can translate each of those script steps easily into steps in a script for Selenium. Additionally, you could also use a commercial tool like TestComplete to automate these scripts. TestComplete will enable you to not only build tests in your favorite languages, but it will integrate your Selenium tests and allow you to create complex scripts through industry-leading record and replay capabilitites.
Test case management software will generally be able to handle either of these approaches with aplomb, as well as a mixing and matching of manual and automated scripts. After all, some situations lend themselves better to one approach or the other. For automated tests, the test case management software will facilitate their execution and store the results. For manual tests, you'll use the script to test it and then log the results yourself.
The Complexity of Test Cases
So far, we've created one scenario, one test case, and one script (with a second one suggested). Perhaps this concrete example helps you understand how quickly managing test scenarios and test cases can become burdensome.
As a quick thought exercise, think of the other test cases that we can create just for this scenario:
- Loan with a down payment.
- Loan with a trade-in.
- Tax rate included in loan.
- Non-zero interest rate.
- Loan with down payment, trade-in, tax rate, and interest rate.
With those test cases alone, you can create many script permutations using different variables and settings. Oh, and let's not forget the unhappy path scenario. What should the calculator do if I try to type my name in the down payment field? Should it not let me? Should it show an error message? You get the idea.
The Importance of Test Case Management
Calculating car payments is, all things considered, fairly simple. I mean, the calculation itself is bit involved and math-heavy, but it's really just a stock formula. Predictable inputs produce predictable output. So just make sure nobody messes up the input. Many applications are far more complicated.
Yet, just running through this, you can see how involved test cases become for even simple applications. For just the monthly payment calculator scenario, we could easily have 8 test cases with 20 test scripts. In fact, we could easily have more. Now, go explore the menu on that site, and realize that we've just scratched the surface of one test scenario, while at least another 15 wait in the wings.
Some back of the napkin math suggests that this simple tool could easily have 15 scenarios with 120 test cases and 300 scripts. That's a lot of testing context to track, and a lot of pass/fail data to log with each candidate patch or release. And your application is probably WAY more complex.
You need a tool to manage this, and that tool can't realistically be Excel. A good test case management solution will guide you with suggested fields and information about good testing practice. But more importantly than that, it will let you keep up with your application's behavior and it will let you keep your sanity.