C++ optimization - Measure and profile

Although it's good to comply with the regular heuristic rules like pass by reference, using pre-increment instead of post-increment etc, it's a very bad practice to fully rely on them when optimizing C++ programs. It's always recommended to measure and profile your program using a tool or timer.

We can use a profiler software to profile our program. The output of the profiler will contain the time taken by each function, and also the memory they consume. There are two kinds of profiler.

1. Compile the program with the profiler, by using special flags. By doing this, the compiler adds hidden statements in the executable, one before the function and one after the function.
2. We attach the profiler to the program externally.

The output of the profiler contains the function names, along with their run-time duration. Going through this output, we can identify the functions which take the most time (the hot functions), and start optimizing them. As we start optimizing the programs, we need to make sure of recording all the measurements. It can either be written down in a book or in excel sheet. This helps us to pinpoint our good/bad performance to a certain change in the program. Sometimes the performance can also improve due to an external factors like a new library or a new networking equipment. Documentation of results can help identify such external factors and helps to avoid mixing them up with performance characteristics of your own code.

Due to varying external factors like process scheduling and system load which affects how many times memory and cache is swapped, we get different results each time we measure the performance. Contrast the system which is heavily loaded (where you are running a test) and in the background you have web browsers with multiple tabs open, and a music player running, and you are compiling another big project VS the system where you  are only running the test with nothing else running on the system. Given that both the systems have the same specs, the test in the former scenario will definitely take more time. Hence to get proper results, we should keep the conditions consistent when running and measuring performance tests . Also to remove the randomness caused due to these external factors, we should also measure our program multiple times.

Hence when optimizing, a developer, like a good scientist should,

1. Make hypothesis about why the program is running slow, and suggest improvements.
2. Incorporate the code improvements, and document it.
3. Take measurements (a number of times),  and document these results along with it's corresponding code changes.
4. Continue steps 1-3, till satisfied with results.

But care should be taken to not make the program overly complicated or unreadable, while optimizing. Long term maintainability of the code-base is a very important criteria to keep in mind.

Having documentation can also aid developers in convincing managers and other stake holders to incorporate their changes.

We could also time our functions by using the calls to high_resolution_clock::now. Make this call at the beginning of the function and another time at the end of the function, and then take the time difference between the two in-order to find how long the function will take to run. However the reading will never be accurate due to -

• Computers clocks are not built capture time accurately.
• The variability introduced by outside factors.
• Inherent latency and communication delays among the different components of the computer.

Hence it's important to take measure of the function's performance as many times as required and take the average. We can also record the minimum and maximum, each time we measure our optimization effort.

I highly recommend checking out Optimized C++ by Kurt Guntheroth. He discusses much more in details about the history and evolution of computer clocks, how accuracy is different from resolution, profilers among other things