Benchmarking Series "The Elements of a Benchmark" (2/5)
[Prior Post in Series] [Next Post in Series]
In Part 1 of the Benchmarking Series we discussed the need for a proper benchmark and how a benchmark can play an important role in Release Management, Scalability, and Troubleshooting Performance.
When we talk about the elements, we're talking about the different pieces that you need to bring together in order to create a successful benchmark.
ADDRESS THE NEED
Chances are if you have gone through the process of creating a benchmark or are looking at creating one now you are looking to answer some or all of these questions:
- How has my change to the hardware, software, or application affected the performance of my system?
- What will happen if our user base or load increased 10X, 100X?
- Will I see a performance increase by upgrading?
- The users are complaining that performance has gotten worse over time... how can I validate?
HARDWARE
Regardless of the needs or questions that you are trying to answer having appropriate hardware available to use for your benchmark tests is essential. The hardware that you use for your benchmark tests should be as close to identical as the hardware that you currently have in your production environment.
You want to be able to report with confidence that the changes being made will or will not have a positive effect and not have an outside variable like a hardware difference play a factor into your results.
TESTS... AUTOMATED AND REPRODUCIBLE
Tests are obviously the most important part of your benchmark because if you don't have good tests that represent normal system activity the results of your benchmark will be incomplete and possible irrelevant.
What you want to be able to reproduce with your benchmark tests are activities that best represent the core functionality of your system, from there you can look at the needs you're trying to address (Release Management, Scalability, Troubleshooting, etc) and make changes to compare against your baseline benchmark results.
You should try to isolate and minimize the changes you make so you're not in the situation where you don't know if change A, B, or C is what caused the performance increase or degredation. Example:
- Change A results in a 50% increase in performance
- Change B has no effect to performance
- Change C has a 25% decrease in performance
If you group these changes together all you're going to see is a 25% increase in performance without really understanding which change had what affect.
SOFTWARE
There are many different tools available to assist you in performing benchmark tests. Two of the more popular options out there are:
Both of these applications are considered enterprise-ready tools that deliver functionality that streamline the benchmarking/testing process.
The good thing about Visual Studio Test Professional is that depending on what license you have for Visual Studio, you might already have it.
Visual Studio Comparison Chart
These elements of a benchmark (addressing the need, tests, hardware, and software) are all pieces of a complete benchmark puzzle. If you don't address and utlize all of these key elements the effectiveness and quality of the benchmark results of your will suffer.
In Part 3 of our Benchmarking Series we will discuss the applicability and usage of a benchmark.
_____________________________________________________________________________________________