Common mistakes made when measuring the speed of code

During times I have seen problematic ways how developers measure the speed and performance of their code. There are some easy tricks that help you make your measurements way better and accurate comparing to measurements made not so well. In this posting I will give you some hints how to get more accurate results when measuring the speed of your code.

1. Don’t use Debug configuration

This is one common problems that developers do. They forget to switch configuration to Release and therefore they are measuring the assemblies that are full of debug symbols and debugger interactions.
Code compiled without debug symbols and other debugger stuff works sometimes hundreds of times faster than Debug version of same code. On live environments you must use code compiled with Release configuration. There is no point to measure Debug code because it brings too much noise and you have no idea how debugger and other tools affect the measuring process.

2. Use Stopwatch instead of DateTime.Now

Measuring the speed of code is often done using DateTime.Now values before and after measuring. This is wrong because resulting time contains also time delays when current thread was suspended by operating system by example.
If you want better results then use Stopwatch class. This class is perfect fit for scenario like this. Here is the example.

static void MeasureCalculation()

{

    var watch = new Stopwatch();

    watch.Start();

    // do your calculations here

    watch.Stop();

    Console.WriteLine("Time: " + watch.Elapsed);

}

---

If you want to find out more about internals of Stopwatch class then please read it’s documentation from MSDN Library.

3. Measure only the code you need to measure

To get better results you have to measure only the code you need to measure and nothing else. This is not very uncommon way how some developers write their measuring code.

---

static void MeasureCalculation()

{

    var watch = new Stopwatch();

    watch.Start();

    var inputList = GetInputData();

    RandomizeInputs(inputList);

    // do your calculations here

    inputList.Clear();

    watch.Stop();

    Console.WriteLine("Time: " + watch.Elapsed);

}

---

This is wrong because with calculations we want to measure we also measure how much time it takes to load data, prepare it before calculations and clear the list with input data after measuring is done. These three activities should not be the part of measuring. Let’s write this code the way it should be.

---

static void MeasureCalculation()

{

    var inputList = GetInputData();

    RandomizeInputs(inputList);

    var watch = new Stopwatch();

    watch.Start();

    // do your calculations here

    watch.Stop();

    inputList.Clear();

    Console.WriteLine("Time: " + watch.Elapsed);

}

---

This code measures only the calculation and nothing else.

4. Turn off diagnostic messages while measuring

Before measuring we need to make sure that our code contains only the functionalities that we need to measure. Still developers forget different diagnostic messages there and we cannot be sure in measuring results anymore. Take a look at the following code.

---

static void MeasureCalculation()

{

    var inputList = GetInputData();

    RandomizeInputs(inputList);

    var watch = new Stopwatch();

    watch.Start();

    // Start calculation

    Debug.WriteLine("Initializing data for algorithm");

    // Some more line

    Logger.WriteLine("X(Y, Z)=" + z);

    watch.Stop();

    inputList.Clear();

    Console.WriteLine("Time: " + watch.Elapsed);

}

---

Well… who the hell is Logger and where it writes it’s output? Is it screen, Windows event log, ULS log, some file, database or what? It doesn’t really matter where output goes. The problem is that the output is there and we don’t need it to be there. So, let’s remove all those diagnostic and tracing messages from code we want to measure. And we are back where we were before.

---

static void MeasureCalculation()

{

    var inputList = GetInputData();

    RandomizeInputs(inputList);

    var watch = new Stopwatch();

    watch.Start();

    // Start calculation

    watch.Stop();

    inputList.Clear();

    Console.WriteLine("Time: " + watch.Elapsed);

}

---

Nice and clean code and measuring is focused only to code that does something we need to measure. Besides diagnostic messages you should remove also all other code that is not necessary in the context of code to be measured.

5. Measure more than once and calculate averages

I think you have seen something like this in office or in internet:
Q: “How long it took to run?” 
A: “On my machine it takes about 1.5 seconds to run” 
Q: “1.5 seconds… Is it average or just one run?” 
A: “Just hit F5 and read the result, that’s it”
Well, my question is – what other processes were running on same machine that possibly may affect the results? Okay, there are a lot of processes. Instead of using one measuring round we need to make more runs and take their average. Here is the example.

---

static void MeasureCalculation()

{

    var inputList = GetInputData();

    RandomizeInputs(inputList);

    var watch = new Stopwatch();

    var cycles = Math.Pow(10, 6);

    var times = 0D;

    for (var i = 0; i < cycles; i++)

    {

        watch.Reset();

        watch.Start();

        // Call calculation

        watch.Stop();

        times += watch.ElapsedMilliseconds;

    }

    inputList.Clear();

    Console.WriteLine("Time: " + (times / cycles));

}

---

This code measures quick calculation and we have no problem to make million measuring cycles. When cycles are done we calculate average time that one calculation took. If you want to analyze or process measuring results you can save them after watch is stopped. This way we don’t affect the code to be measured.

Conclusion

Measuring the speed of code is not complex task. There are common mistakes that I listed here and if you avoid these mistakes you get way better and accurate measuring results. It is also possible to use testing frameworks but I focused here on quick and simple measuring method so you get your code measured right there where it is. You don’t have to install any additional tools or spend your time setting up more complex measuring environment. As you saw – measuring the speed of code is not rocket science.




http://weblogs.asp.net/gunnarpeipman/archive/2010/09/09/