Let’s consider a list of testing services:
- Functional Testing
- Automated Testing
- Security Testing
- Web Performance Testing
- API Performance Testing
- API Functional Testing
- Mobile Testing
These sorts of lists are universal. But rather than just running down a list of tests a team should, or should not be running, why not instead think about the problems your company is trying to solve and how to mitigate them?
Thinking in this way, can help address a common challenge that often comes up when different tests are put in silos.
Imagine, for example, the 'functional API tester' who is not concerned if the data will be fit for use. Or the 'performance API tester' who cares if the data comes back fast enough — but not if it is correct.
Today, we'll talk about the intersection of functionality and API performance — and what to do about it.
Living through it
A few years ago I was testing a simple website; it took data that previously was updated by hand in SQL Server and exposed Create, Read, Update, Delete, and a little automagic to the customer service staff. Once we had the functional tests, management saw no need to performance test. I saw an application that would be used by a hundred people at the same time.
You guessed it. At one of the levels of the application, a programmer had written a singleton pattern, designed so that only one object would be created at a time. The software served up data that looked generally right. No 400 errors, no timeouts, and the speed was right — only it was the page the user sitting next to you was supposed to see.
Performance testing at the API level alone wouldn't catch this problem, and neither would functional testing, not really. What we needed were both. With a small application that was under development for three or four sprints, we could test with a room full of testers quickly. In production, the application would only have a handful of customer service reps using it at one time.
That kind of testing can find the first level of performance problems — problems with multi-threading and memory management will become obvious with two or three simultaneous users. The next level up takes a lot more work. These days it is rare for an application to be that simple or under development for that short a period of time, or consumed by a similar, homogenous group of users. The use pattern for APIs is even less clear.
The common suggested fix is to take the functional tests and run them as performance tests, slowly cranking the load or number of users over time.
Working through the issues
"Take your API functional checks and replay them" to test performance is easy enough to say; it's a bit harder to do.
That's hard enough for regular old end-to-end checks, but APIs lead an entirely new level of challenge. First, they are likely disconnected from the user experience. Flipping ten thousand API calls sideways won't simulate what actually happens in a production environment; the tests won't be similar to how the user works. You'll likely need to simulate a number of users just like real life, which is something API functional tests don't do. The GUI tests might, and it might be possible to capture the API calls in those GUI tests, and replay them, which we've seen often enough.
But, the results will change. Even if we can set up the exact same database and run the exact same traffic and hold the APIs static, so we expect the exact same results, the results are likely to have dates and times in them. Ignore the results, just looking for obvious error codes, like 404 and 500, and we are back to the problem above, where API performance testing loses sight of API quality.
It gets worse.
If the team is pursuing a microservices strategy — a real one, with different data sources, then the problem will be finding which data sources choke first (the bottleneck) and not looking at each as a component. Healthcare.gov, for example, initially kept a web browser session open while calling back-end services, like Medicaid and Medicare history, or trying to coordinate enrollment with a private insurance company. As soon as one of the APIs in the chain failed, the entire chain failed. Worse, some of those services reported generated incorrect results under load.
A needle of an idea ... in a haystack
Here's an idea.
API functional tests demonstrate that the API generates the right answer. If we are really lucky, they take a look at how long it takes to generate the answer.
So generate realistic load on the system some other way, using accounts that do not overlap with yours. Then run the API tests at the same time the system is experiencing stress. This will tell you how the software performs under conditions of load. The ideal tool, of course, will have the ability to mix the two, with some elements of API functional testing and load working together, as the database changes over time.
A tool like LoadUI NG Pro will let you reuse your functional API tests, built in SoapUI, to run load tests on REST and SOAP web services. You can use the tool to run several load test scenarios simultaneously to see how various conditions interact with each other and impact your API's performance.
If you don't know where you stand when it comes to the interplay of functional and API testing, here's one final, easy idea.
Change the underlying data right before you run your API performance suite. If you are testing search, delete a lot of rows of the database for things you know will be searched for, and add others. Then run the performance suite. If it is passes, then you've got to work to do on your API performance strategy.
Of course you might get specific, wonderful, beautiful failure. It could happen. But I'd be reluctant to bet on it.
Ensuring API Speed and Performance
Whether you’re launching an API of your own, or are concerned about the third party APIs that power your applications, you need to understand how your APIs are performing.
In our newest eBook, Ensuring API Speed & Performance, we look at two of the most important processes for ensuring the performance of your API both during development and in production — API load/performance testing and API monitoring.
- The benefits of each of these API performance strategies
- Implementing API load testing and monitoring
- Using testing and monitoring together
- Finding the right tools for ensuring API performance
Get your copy!