minitrace VS gperftools

I propose you to use something like Google Chrome traces. But you need to annotate the source code. Here is a lib that can be helpful https://github.com/hrydgard/minitrace

For profiling you are right clang has no -pg that works. But there are options, since clang supports PGO the fprofile flags could be what you need. they will generated a profraw file for you. There is also gperf tools which work for more than just linux. https://github.com/gperftools/gperftools

Because we couldn't identify the issue using the results we got from Callgrind, we reached for another profiler, gperftools. It's a sampling profiler and therefor it has a smaller impact on the application's performance in exchange for less accurate call statistics. After filtering out the unimportant parts and visualizing the rest with pprof, it was evident that something strange was happening with the send function. It took only 71 milliseconds with the previous implementation and more than 900 milliseconds with the new implementation of our Bolt server. It was very suspicious, but based on Callgrind, its cost was almost the same as before. We were confused as the two results seemed to conflict with each other.

gprftools (https://github.com/gperftools/gperftools) can be easily plugged in using LD_PRELOAD and signal, and has nice go implemented visualization tool https://github.com/google/pprof.

Google's tcmalloc