Empowered Networks

A sound benchmark must meet four basic criteria: It must be repeatable, reproducible, stressful and meaningful.

Repeatability: All things equal, multiple iterations of the same test on the same test bed should produce the same, or similar results.  This is table stakes.  If your test produces wildly different results with each iteration, consider new testing gear, or outsourced testing.

Reproducibility: Different teams running the same test on different test beds should be able to produce similar results.  Assuming all use the same test instrument and system under test (SUT) and follow the same procedures.  The difficulties in verifying that all test beds are the same, in terms of software, hardware and configuration for both the test instrument and the SUT, can make reproducibility can be an elusive goal.

Stressful:  A stressful benchmark finds the limits of system performance – where a successful iteration is followed by a failed iteration. As an example, in throughput tests for a network device, the highest rate at which a device forwards traffic without dropping frames is the limit.  Knowing where the limits are, and what happens beyond them, can be essential insights into device performance.

Meaningful: What makes measurements meaningful is how they apply to the use case being tested.  Consider two devices benchmarked for switching latency, where one attains results of 10 microseconds, as compared to 50 microseconds for the second device.

• If the devices connects to WAN link spanning thousands of miles, the difference in switching delay – 40 microseconds – is tiny, compared to propagation delays at 10 milliseconds or more.
• If the devices are within a single rack in a data center carrying financial transaction data worth millions every second, a four-fold increase in switching latency becomes very meaningful.

The measurements themselves have no intrinsic value. What’s more, vast quantities of irrelevant data can make significant findings difficult to see.

Making sound decisions requires using benchmarks that are repeatable, reproducible, stressful and meaningful.  If your benchmarks don’t meet these four basic criteria, we’d love to help you improve your testing.

-Ron Watt

Ron Watt has been with Empowered Networks for over twelve years, holding progressively senior roles within the IT Management Solutions Portfolio. Ron leads Marketing, with primary responsibility for Empowered’s seminars and events, and communicating Empowered’s value – solutions to simplify and optimize testing and management of IT, Networks and Applications.  Ron has played a key role in developing Empowered’s portfolio of services to deliver training, ongoing support and independent assessment services.  Prior to joining Empowered, Ron spent nine years with Foreign Affairs Canada as Manager of Internetworking, overseeing design, implementation and operations of FAC’s global internetwork with 160+ sites in more than 100 countries worldwide.