Lies, Damned Lies and Benchmarks
Benchmarks can reveal the truth – well, some of the truth – about technologies. A well-constructed benchmark can provide a way to compare performance, reliability and other metrics that can make a difference in comparing product quality and effectiveness. But you have to look closely at what a benchmark is measuring and how it was run – and, even then, take its results with a large grain of salt. Here’s how to approach benchmarks and their touted results. Let's start with the fundamentals. We'd all like to have some magic number that reveals "The Truth" about how good any given device or program is. There are no such numbers. Anyone who tells you that there is such a single performance benchmark is either lying or is trying to sell you something. What a good benchmark can do is give you is accurate data about some performance aspects of a given technology for a specific audience. Then, taken in context, the benchmark can help you decide if that technology is what you need. Before you can decide if a benchmark is "good," you need to know the facts about them. There are two general types of benchmarks: component and system level.
Breaking Down Benchmarks
Who's right? Who's wrong?
You can argue about what should and shouldn't be measured in any specific benchmark until the cows come home but Carl Nelson, owner of HardCoreWare, a PC hardware review and benchmarking site, sums matters up nicely when he writes, "You shouldn’t be using one benchmark to determine the performance of a system anyway." He's correct. You should use multiple benchmarks before deciding on what's Gospel truth about any given technology. Benchmarks are limited in other ways. People expect them to produce some perfect, objective truth. They don't. All the best benchmarks really can do is give you an approximation of how well software or hardware does in a specific circumstance. Even with the best benchmarks, bias can appear. "Every benchmark needs to be taken with a grain of salt," says Miller. "The battery life estimates that just about every major notebook vendor uses are perhaps the best example of how even a relatively fair benchmark can be misused. We all know that nearly every notebook gets worse battery life than what the company claims, usually based on a test called MobileMark, which is also created by BAPCo. In part, that's because to get the best score, the vendors turn off or turn down a number of features (wireless, screen brightness, etc.) to get the best MobileMark score. But it's also because, to create a fair test, MobileMark focuses on things that are easily repeatable." You see this kind of concern raised with any benchmark. For example, back in 1999, I ran the first benchmarks showing that Linux, with Samba and Apache, was a faster file and Web server than Windows NT. I often heard from people, both Linux and Windows system administrators, who complained that I hadn't optimized the servers. That was absolutely true. But had I done so, I would have been measuring, in part, how good we were at optimizing server performance, rather than measuring each platform by default in delivering files and Web pages. So, then, as today, you should not merely run a benchmark, or read the results. You need to review how the benchmark is meant to be run and how it actually was run. For example, many Windows PC benchmarks are run with the supplied anti-virus (A/V) software turned off. Does this give a more "objective" view of how Windows works on a specific PC? Perhaps, but it also doesn't reflect the real-world experience of anyone who wants his PC to work well for more than a few hours. Others would argue that to test a PC, all extraneous factors—screen savers, video-acceleration, anti-virus software, disk caching—should be adjusted to common settings. There's merit to both approaches. But to really use the benchmark's results for your own decision-making, you need to know exactly what assumptions were used on the test-bench. In the case cited above, my A/V-equipped PC is very likely to run slower than the same system without anti-virus. It's also far too common for vendors to publish results based on equipment that you won't find in any real server room or data-center. For example, the TPC-H benchmark, which is often used to measure DBMS performance, can be greatly influenced by the hardware it's used on, far more than by the DBMS it's testing. Curt Monash, a well-regarded technology industry analyst, observed that "Most TPC benchmarks are run on absurdly unrealistic hardware configurations." In addition, anytime you see comparisons using benchmarks make sure that they're really comparing apples to apples and oranges to oranges. This is actually much harder than it sounds. With a PC, for example, every component—the processor, the I/O bus, the amount and type of memory, etc.—affects its overall benchmark performance. If any one piece isn't matched properly with the other, its performance will be slower than their simple sum might lead you to believe. Indeed, some systems are almost impossible to benchmark "fairly." Peter Wayner, contributing editor of the InfoWorld Test Center, recently found when he was benchmarking "identical" Amazon EC2 Micro cloud instances with the Da Capo Java benchmarking suite, the "results were all over the map." While "Medium machines [3.75GB of RAM and a promise of one virtual core with two EC2 Compute Units] were much more consistent … even these numbers weren't that close to a Swiss watch.” Why were the results so varied? Because on the cloud, you trade control for flexibility. As Wayner says, "On a bad day, you could end up sharing a CPU with 10 crazy guys trying to get rich by minting Bit-coins by melting the processor; on a good day… you could end up sharing a machine with 10 grandmothers who only update the Web pages with a new copy of the church bulletin each Sunday." I can't blame you at this point if you think all benchmarks are worthless. They're not. You just have to use them carefully; consider precisely what each one tries to do; and look critically at the methodology and the results. That done, benchmarks can help you find the right technology for the right job. Just never think that they're anything so special that they will automagically make your IT decisions for you. They won't.