Good API testing is part of the fabric of a technology enterprise, whether the company is a lean and mean startup or a well-established Fortune 500 business. Comprehensive API testing does not happen in a vacuum. It’s a result of management decisions. The quality of API testing is only as good as the quality of the thinking and actions of those managers responsible for making APIs that work for their customers. No manager woke up one day and took a pill that provided the vision to create effective API testing. Most had to learn the hard way. These are their stories.
Mervine: The developer is responsible to ensure that his or her code works
Joshua Mervine is seasoned technologist who has held significant executive and technical positions at many big players on the technology landscape, YellowPages.com and EarthLink to name few. Presently Joshua is a Service Reliability Engineer at Heroku, the cloud application provider. Heroku supports mission critical applications for customers such as Citrix, Macy’s and Toyota. Heroku is prime time technology that needs to be up and active, worldwide all the time, 24/7.
When it comes to the management of API Testing, Joshua has fundamental position: The developer is responsible to ensure that his or her code works.
To my thinking when it comes to the Management of API Testing, the first order of business is that the developer is responsible to ensure that his or her code works at the unit testing level and that the code is adequately covered. The developer really owns their code and is responsible for it. … This includes setting everything up, ensuring that it's running and responding quickly when it fails.
Another thing I might add regarding responsibility is ownership, which I mentioned a few times above. The more closely a developer owns their code -- that is to say, designs, PMs, QAs, etc. -- the higher quality code they're likely to produce. When you introduce other owners, you introduce an abstraction of responsibility. That's why Open Source works so well. The developers really and truly own their work.
Roodyn : 3 Tips from Dr. Neil
Dr. Neil Roodyn has been working on the web since there was a web. He worked on the TabletPC with Microsoft and is a Regional Director for the company. His company, nsquared is a leader in making collaborative tools for the enterprise. Affectionately known in the software development world as Dr. Neil, his company has a variety of products in play in which testing in general, an API testing in particular are a critical part of the company’s success. Dr. Neil says the following:
We work with BIG software companies, like Microsoft on helping to define their APIs and we produce some apps and services of our own that have interfaces for developers to plug into. We have a few 'tricks' we use:
- Automate tests, for everything you can think of and then think of some more. This will help you prevent most backwards compete issues
- Have a set of test harness applications that put your API through their paces and make sure they work as expected.
- Use your own APIs in your apps; don't use backdoors or 'hacks' to bypass your own APIs in your own products
Mays: Think ahead and build for test
Steve Mays is CTO at Trizic, a provider for the financial services industry. Steve has an impressive work history that goes way back, to NeXT and Broderbund. When it comes to the management of API testing, he knows his stuff. Steve’s mantra is to think ahead:
Making sure you think ahead and build for test.
It’s a concept I learned at NeXT. Don't hand code/document your API. Don't wait until app layer is done as others will be waiting for you. Don't make a monolithic API head; make a modular/self-registering route service (API can be like SOA). Learn how to use modern frameworks/mix in Node.JS tech to make even your Java projects fly. THEN teach a team to use all these tools. All of this is hard. The last piece is the worst, though.
Neale: APIs should be directly validated as well as included in integration testing
Bennett Neale works as a consulting CTO for many big companies in the Los Angeles area. He was a system architect for Edmunds.com, one of the largest websites in the world for automotives. He’s a roll-your-shirtsleeves-up executive, a down in the trenches guy. His views are real world:
… most of the previous projects I have consulted on required API development. Typically, they are "internal" APIs to enable decoupling between sub systems. E.g., the front-end of a web application written in React or Ember consuming a REST API, as well, in breaking up a monolith system into subsystems for moving to a micro-service architecture. I have a project coming up that will be in the millions per day due to consuming Twitter data but that's a different beast. Sports, Travel, and Gaming
[Management] Policy is that APIs should be directly validated as well as included in integration testing. There are a number of needs that have to be met per project/function, such as SLA and security.
The procedure is typically creating of unit tests, integration test suites, load testing, etc. to generate a baseline and then optimization, whether that be at the software level or in scaling hardware.
When asked about the biggest mistake most companies make when it comes to API testing:
Inability to foresee system failure, scenarios where parts of your architecture will fail. For example, what happens if one of the servers in your app or DB tier cluster fails or what happens if they are only responding 50% of the time?
These are hard to test because they are mostly manual, so a lot of times they are looked over.
So let’s talk about what happens when you don’t test properly.
Clark: You can't test an undiscovered endpoint.
Matthew Clark is a Product Manager at Indeed.com. Prior to work at Indeed.com, Matt was Vice President of Product Development at a major provider of services to the worldwide casting industry. Matt looks back the challenges he faced then.
Most of our web service endpoints were not published in any sort of documentation. They are hardly even documented in the code. There were some attempts to document newer code on the wiki (MediaWiki) manually, but adoption and discovery of the documentation has been low. Most recently some new independent services had generated documentation and that was good to see.
[There was ] no, API testing policy in force. UI testing "proved" the API was at least resolving to expected behavior, but nearly all testing of APIs was through the browser
When asked about how the lack of testing policy affected the company he replied:
We've seen the classic issues like dirty data. But the core problem is a lack of understanding or appreciation for all our existing endpoints, which has led to duplicate code and stifled creativity.
In terms of doing thing differently, Matt says:
I would start a UI Testing policy with a discovery project to identify all end points. Even having a simple list with a yes/no column for "documented" would be a good foundation. The sheer magnitude of all the API or web service endpoints would set a good foundation to get people to pay attention. No free passes for legacy code. I would actually encourage the team to start with the oldest, or least used, software and work their way forward.
You can't test an undiscovered endpoint.
Reselman: Spend the Money
Let me throw my hat in the ring. I have been in the technology game for over 25 years. I have managed teams and departments. I’ve slung a good deal of code. I’ve tested more than my share of APIs. Here is the essential lesson I’ve learned: At the end of the day it all comes to down to budget.
Getting a culture of competent API testing is not magic. You have to spend the money. For me, good API testing is about how much your company is willing to spend on tools and education. If you have technology payroll that is running $2 million a year, which is on average a staff about 20 managers and engineers combined, and you are spending $50,000 a year on tools and education, you’re probably running a bit light. That $50K is only 2.5% of your labor costs. Me? I usually go for about 10%, if not more. True, money cannot buy wisdom, but it can buy a lot of efficiency and labor savings.
The CEO of a company I worked at a while back had a very direct way of stating the obvious. To this day I remember what he said:
I don’t ever want you to come to me and say we failed because we did not spend enough money.
It does not get any simple than that. Spend the money.
Putting It All Together
The experience expressed by the experts cited above comes down to these basic points:
- The developer is responsible for the code
- Automate for everything you can think of
- Use your APIs (a.m., eat your own dog food.)
- Think ahead and build for test
- Test to SLAs and established security policies
- Know all your endpoints
- Spend the money
Yes, there is a lot of common sense in the bullet points above. But more importantly there is wisdom, wisdom that has been acquired over time, by trial and tribulation.
Wisdom lives in that place where lessons have been learned from big mistakes and phone calls at 3 in the morning.
You cannot buy wisdom for yourself directly. You have to acquire it. Of course, you can buy the counsel of wise people. But most times, those that are truly wise give it away for free. All you need to do is pay attention.
Remember, Good API Testing = Good API Management.
Bob Reselman (@reselbob) is a nationally known executive software developer and technology writer. Bob’s work has appeared in ITWorld, CodeGuru, Developer.com, DevX and InformIT. Bob enjoys reading his reader’s opinions and commentary. He can be reached at [email protected].