Functional testing is great for finding some types of problems. But using that technique alone is a bit like volunteering to put blinders on to other risks.
Let's talk about how to bring on the strong points of functional testing while reducing those other risks.
What Is the Risk of Only Focusing on Functional Testing?
Think about an iPad app that will be used by clinical staff at a hospital to document surgery cases. There are a lot of custom components to help the doctors and nurses enter data faster — data grids, dosage selections for drugs, and vitals entry. Testing this seems dead simple. Download the newest build and get the new vitals entry feature. Use the goldilocks and the three bears heuristic to test heart rate values; this heart rate is too fast, this heart rate is too slow, this heart rate is just right. Make sure that the data saves in a reasonable amount of time and can be reported on, and presto, testing is done.
But, oh no, what is that? A few days after pushing that new app version to the App Store, reports start coming back along with a few angry calls to the CEO. There are two big bugs. If a doctor enters a value and then edits that three times, the app crashes. The other bug is a seemingly unexplainable data loss. A nurse enters a new heart rate for a patient and then poof, a few seconds later it disappears from the screen. Functional testing missed two black swans, problems with big consequences. At a surface level, that heart rate feature is just a simple input mechanism. Tap the tool, type a couple of numbers or use the selector, and tap out of the dialog. For the user though, features are never just about entering data.
What is left out of all this?
Functional testing can be a good strategy to test features one by one, seeing how they work by themselves. There are entire categories of problems that will be missed if that is the only strategy, though. Interactions for example. Functional testing might catch that the heart rate documentation feature fails when a large value is entered. But, that strategy is not likely to help a tester discover that the app crashes when heart rate is documented more than twice in an hour, or that data is reported on incorrectly when heart rate is too high because of how the variable is stored in the database. Every feature or function in a software product interacts with something else. In the words of John Donne, no feature is an island. Other problem categories functional testing misses are how easy the feature is to use and how much value the user gets, how reliable and stable the feature is, and how easy it is to test.
Planning for more is the way to get past risk. When testing is done at the functional level, the most obvious questions are always around how that function behaves. Can it save? Will data display correct? Is the page state reasonable? Adding little reminders in the test documentation is one strategy to move past that. Another light weight strategy is sitting down with the developer and the product manager and asking a few questions that will remind everyone about what this feature will interact with, who the customer is and how they like to work, and why they would value this new function.
What is functional testing good for?
Nearly every release cycle starts with a list of new functionality — add zip code to a demographics report, create new tab to handle discounts for a sales tool, update image storage to improve performance. Each item in that feature list is a new thing the software will do, some piece of value a customer wants that the product doesn't offer right now. Each item in that list also represents a new function, or maybe even a group of functions. Exploring that list, one line item at a time, while ignoring everything else that is happening in that software universe is what some people call functional testing.
The term shift left in testing encourages testers to get involved much earlier on in the development process. Some interpret that as getting testers into planning and design meetings so that they can ask questions that influence feature development before any code has been written. At some point, software has to be tested. Getting involved early on usually means that the testing group will see those changes dribble out, a piece at a time. This increases the risk for regression — that some feature that behaved as expected on the first build fails on the fifth. What it can be a risk to test 'just' the new API, before the other pieces are assembled, that can also be a sweet spot for functional testing.
If only an API is ready, the tester can perform most of the data testing they would normally do through a user interface — discovering the upper and lower boundaries of a field, see what happens with when fields are dependent on each other, test for elegant failures from bad data and special characters. If only the user interface is ready, the tester can explore ideas around input masks on fields, page state when navigation happens, and design aspects.
Each part of a function that is tested can help build a coverage map. Adding a zip code to a report for example seems simple, but there are a few parts to it. First the zip code has to be captured so the API has to be tested, the UI has to be tested, and then the two have to be tested together. After that there is the matter of reporting on the zipcode. Does it show up in the report? How would someone know if the report is correct? Capturing all those smaller sub-functions of the main function can guide future testing and act as an aid for knowing how much testing work is done, and how much more there is to do.
Functional testing can be a good place to begin. This technique can help teach basic lessons on how a new software feature, or even small parts of a feature work. Remembering the customer and what they value can help round that out. Functional testing is the start, not the end.
Automated Functional Testing in TestComplete
Functional Automation tools, like, TestComplete, support functional testing by automating its repetitive aspects and producing results in a flexible, filtered form. In addition, TestComplete enhances its functional testing feature by providing the following opportunities:
Continuity: Even the simplest functional test should be applicable throughout the life of a project, and it should be capable of automatically measuring results against an already-validated standard output. TestComplete’s Test Log is designed to meet these criteria.
Application-grade tests. Functional testing should be kept external to the application as much as possible.
TestComplete provides features that can be made required, so that this criterion is met:
- Visually constructed keyword tests and scripts using any programming constructions in your tests: loops, if-then conditions, try-catch blocks, comments and so on.
- A sophisticated library of functional testing objects with easy and transparent methods and properties that perform any functional testing actions over applications as well as simulate user actions.
- More objects that directly interact with the system or with data base servers (ADO, BDE).
- Advanced properties and methods for functional testing of .NET, Java and web applications.
- Special properties and methods for careful functional testing of applications with third-party controls: Windows Forms, Developer Express, Infragistics, Syncfusion, MFC, Qt, Telerik, and others.