Benchmark (computing)

This article is about the use of benchmarks in computing, for other uses see benchmark.

In computing, a benchmark is the result of running a computer program, or a set of programs, in order to assess the relative performance of an object, by running a number of standard tests and trials against it. The term, benchmark, is also commonly used for specially-designed benchmarking programs themselves. Benchmarking is usually associated with assessing performance characteristics of computer hardware, for example, the floating point operation performance of a CPU, but there are circumstances when the technique is also applicable to software. Software benchmarks are, for example, run against compilers or database management systems.

Benchmarks provide a method of comparing the performance of various subsystems across different chip/system architectures.

Purpose

As computer architecture advanced, it became more and more difficult to compare the performance of various computer systems simply by looking at their specifications. Therefore, tests were developed that could be performed on different systems, allowing the results from these tests to be compared across different architectures. For example, Intel Pentium 4 processors have a higher hertz rating than AMD Athlon XP processors for the same computational speed, in other words a 'slower' AMD processors could be as fast on benchmark tests as a higher hertz rated Intel processors.

Benchmarks are designed to mimic a particular type of workload on a component or system. "Synthetic" benchmarks do this by specially-created programs that impose the workload on the component. "Application" benchmarks, instead, run actual real-world programs on the system. Whilst application benchmarks usually give a much better measure of real-world performance on a given system, synthetic benchmarks still have their use for testing out individual components, like a hard disk or networking device.

Benchmarks are particularly important in semiconductor microprocessor design, giving processor architects the ability to measure and make tradeoffs in microarchitectural decisions. For example, if a benchmark extracts the key algorithms of an application, it will contain the performance-sensitive aspects of that application. Running this much smaller "snippet" on a cycle-accurate simulator, can give clues on how to improve performance.

Prior to 2000, computer and microprocessor architects used SPEC to do this, although SPEC's Unix-based benchmarks were quite lengthy and thus unwieldy to use intact. Since 1998 and the founding of EEMBC by Markus Levy and 12 semiconductor company representatives Embedded Microprocessor Benchmark Consortium, with its focus on embedded benchmarking, architects can run much smaller and more focused algorithm-based benchmarks. EEMBC is unique in that its benchmark scores and benchmark reports must be certified by the consortium's EEMBC Technology Center before they can be published. EEMBC's next great challenge is benchmarking power consumption and energy utilization by processors as more and more embedded processors go into low-power environments such as cellular phones and other mobile applications.

Computer manufacturers have a long history of trying to set up their systems to give unrealistically high performance on benchmark tests that is not replicated in real usage. For instance, during the 1980s some compilers could detect a specific mathematical operation used in a well-known floating-point benchmark and replace the operation with a mathematically-equivalent operation that was much faster. However, such a transformation was rarely useful outside the benchmark until the mid-1990s, when RISC and VLIW architectures emphasized the importance of compiler technology as it related to performance. Benchmarks are now regularly used by compiler companies to improve not only their own benchmark scores, but real application performance.

Manufacturers commonly report only those benchmarks (or aspects of benchmarks) that show their products in the best light. They also have been known to mis-represent the significance of benchmarks, again to show their products in the best possible light. Taken together, these practices are called bench-marketing. See this article for an excellent example of how application benchmarks can differ significantly from synthetic benchmarks.

Users are recommended to take benchmarks, particularly those provided by manufacturers themselves, with ample quantities of salt unless the benchmarks are certified and relate directly to a recognizable application workload. Ideally benchmarks should only substitute for real applications if the application is unavailable, or too difficult or costly to port, to a specific processor or computer system. If performance is really critical, the only benchmark that matters is the actual workload that the system is to be used for. If that is not possible, benchmarks that resemble real workloads as closely as possible should be used, and even then used with skepticism unless independently certified. It is quite possible for system A to outperform system B when running program "furble" on workload X (the workload in the benchmark), and the order to be reversed with the same program on your own workload.

Challenges

Benchmarking is not easy and often involves several iterative rounds in order to arrive at predictable, useful conclusions. Interpretation of benchmarking data is also extraordinarily difficult. Here is a partial list of common challenges:

Vendors tend to tune their products specifically for industry-standard benchmarks. Use extreme caution in interpreting such results.
Benchmarks generally do not give any credit for any qualities of service aside from raw performance. Examples of unmeasured qualities of service include security, availability, reliability, execution integrity, serviceability, scalability (especially the ability to quickly and nondisruptively add or reallocate capacity), etc. There are often real trade-offs between and among these qualities of service, and all are important in business computing.
Benchmarks do not measure TCO.
Benchmarks seldom measure real world performance of mixed workloads — running multiple applications concurrently in a business context. For example, IBM's mainframe servers (System z9) excel at mixed workload, but industry-standard benchmarks don't tend to measure the strong I/O and large/fast memory design such servers require. (Most other server architectures dictate fixed function/single purpose deployments, e.g. "database servers" and "Web application servers" and "file servers," and measure only that. The better question is, "What more computing infrastructure would I need to fully support all this extra workload?")
Vendor benchmarks tend to ignore requirements for test and QA computing capacity. Vendors only like to report what might be narrowly required for production capacity in order to make their initial acquisition price seem as low as possible.
Benchmarks are having trouble adapting to widely distributed servers, particularly those with extra sensitivity to network topologies. The emergence of grid computing, in particular, is making benchmarking that much more difficult since some workloads are "grid friendly" and others definitely aren't.
Users can have very different perceptions of performance than benchmarks might otherwise suggest. In particular, users appreciate predictability — servers that always meet or exceed SLAs. Benchmarks tend to emphasize mean scores (IT perspective) rather than low standard deviations (user perspective).
Many server architectures degrade dramatically at high (near 100%) levels of utilization — "fall off a cliff" — and benchmarks should (but often don't) take that factor into account. Vendors, in particular, tend to publish server benchmarks at continuous ~80% utilization — a totally unreal situation.

Types of benchmarks

Real program
- word processing software
- tool software of CDA
- user's application software (MIS)
Kernel
- contains key codes
- normally abstracted from actual program
- popular kernel: Livermore loop
- linpack benchmark (contains basic linear algebra subroutine written in FORTRAN language)
- results are represented in MFLOPS
Toy Benchmark
- user can program it and use it to test computer's basic components
Synthetic Benchmark
- Procedure for programming synthetic Bench mark
  - take statistics of all type of operations from plenty of application programs
  - get proportion of each operation
  - write a program based on the proportion above
- Types of Synthetic Benchmark are:
  - Whetstone
  - Dhrystone
- Its results are represented in KWIPS (kilo whetstone instructions per second). It is not suitable for measuring pipeline computers.
I/O benchmarks
Parallel benchmarks: used on machines with multiple processors or systems consisting of multiple machines.

Common benchmarks

Open source benchmarks

Dhrystone: integer arithmetic performance
Whetstone: floating-point arithmetic performance
ApFloat: floating point
Linpack / LAPACK
GliBench: a Gui based benchmarking tool to check CPU and hard disk performance.
BYTEmark benchmark suite
STREAM: memory bandwidth benchmark
MemPerf: memory bandwidth
LLCBench: a group of benchmark for cache, MPI,etc.
LMbench: a suite of simple, portable benchmarks
nbench for Linux/Unix : Memory, integer and floating point comparison with AMD K6 233MHz
Ubench - Unix Benchmark Utility
NAS parallel benchmarks
PovRay: 3D render
Iozone file I/O
Bonnie++: File I/O
netperf : network throughput and latency benchmark
GENESIS distributed memory benchmark suite
Himeno Benchmark

Microsoft Windows benchmarks

Lavalys EVEREST
REALiX HWiFO32
SiSoftware Sandra
DocMemory Diagnostic software
CD Speed 99
CPUmark
3DMark
InfoTool
BAPCo SysMark
WinBench 99
Whetstone
Futuremark PCMark

External links

http://www.netlib.org/ Repository for many open source benchmarks
http://www.benchmarkhq.ru

Computer benchmarks

Gourt Home::Gourt :: Benchmark (computing)