Dr. Dobb's Journal March 1998
Benchmarking is typically thought of as running a set of standard tests on a system to compare its performance with that of others. Several benchmarks have been proposed to measure hardware and basic software speed:
No matter the kind of benchmark, all good benchmarks use a well-defined testing methodology based on real-world use of a computer system. In addition, they measure performance in a deterministic and reproducible manner, allowing the system administrator to properly judge the performance and capacity of their systems. When used appropriately, benchmarks can provide a means of determining tuning parameters, reliability, bottlenecks, and system capacity -- not to mention providing users with impartial buying information.
Benchmarks are made up of three basic components: a specification, control logic, and implementation. A benchmark specification is typically a document that describes the details, design goals, data points, and execution plan. A specification can become lengthy, depending on the complexity of the benchmark (the TPC-D specification, for example, is about 170 pages long).
The control logic should have a well-defined initialization sequence that is separate from the test and data collection. The control logic is not really part of the benchmark itself, but the benchmark does depended on it for repeatability and accurate statistics. Repeatability means that benchmark results from multiple runs of the same test with the same configuration test environment and parameters are consistent with each other. If a benchmark is not sufficiently repeatable then the results have no meaning.
Implementation of a benchmark specification can vary, depending on the system under test. The benchmark specification should always have audit trails in place to verify the implementation is done in the spirit of the benchmark.
To give you an idea of what's involved in writing a complex benchmark, I'm making available electronically (see "Resource Center," page 3) the source code for a benchmark that consists of a single database table and three transactions. This benchmark demonstrates how to create and load the table, and how a transaction can be coded using ODBC. To build the benchmark, I used the Benchmark Factory, a tool my company (Client/Server Solutions, http://www.csrad.com/) developed. Benchmark Factory includes a MFC-like C++ framework called "Benchmark Foundation Classes," for writing custom benchmarks. Benchmark Factory includes a DLL Wizard used in conjunction with Microsoft's Visual C++ 5.0 which creates the template source files, and gives you a head start in developing the benchmark definition. The source code for the example benchmark is compiled into a Windows DLL, then registered with Benchmark Factory. By coding your benchmark is this manner, Benchmark Factory takes care of all your timing and scaling, and provides the tools required to perform reporting and drill-down data analysis.
DDJ