gperftools VS pprof

For profiling you are right clang has no -pg that works. But there are options, since clang supports PGO the fprofile flags could be what you need. they will generated a profraw file for you. There is also gperf tools which work for more than just linux. https://github.com/gperftools/gperftools

Because we couldn't identify the issue using the results we got from Callgrind, we reached for another profiler, gperftools. It's a sampling profiler and therefor it has a smaller impact on the application's performance in exchange for less accurate call statistics. After filtering out the unimportant parts and visualizing the rest with pprof, it was evident that something strange was happening with the send function. It took only 71 milliseconds with the previous implementation and more than 900 milliseconds with the new implementation of our Bolt server. It was very suspicious, but based on Callgrind, its cost was almost the same as before. We were confused as the two results seemed to conflict with each other.

gprftools (https://github.com/gperftools/gperftools) can be easily plugged in using LD_PRELOAD and signal, and has nice go implemented visualization tool https://github.com/google/pprof.

Google's tcmalloc

Damn I hate it when you write a whole project and someone comes along and says "this already exists" and you realise how much time you wasted (yeah even if some of it counts towards learning I'd still rather not needlessly repeat other people's work).

Anyway, pprof has a fantastic interactive Flamegraph viewer that lets you narrow down to specific functions. It's really very good, I would use that.

https://github.com/google/pprof

Run `pprof -http=:` on a profile and you get a web interface with the Flamegraph, call graph, line based profiling etc.

It's demonstrated in this video.

https://youtu.be/v6skRrlXsjY

They only show a very simple example and no zooming, but it works very well with huge flamegraphs.

How do flame graphs handle the case where most of the time is spent in some leaf function that is called from all over the program? In this case, each individual stack would not take much time but in aggregate, a lot of time is spent in the function at the top of all of the call stacks. This should not be that uncommon to have hotspots in things like copying routines, compression, encryption etc that are not associated with any particular stack.

pprof from https://github.com/google/pprof can produce a DAG view of a profile where nodes are sized proportional to their cumulative time, e.g.,

The pprof format is not tied to Go. From my understanding, it's used within Google across multiple languages. The format is defined in the pprof repository[0], and the visualization tool is source-language agnostic. I've seen libraries in numerous languages (e.g. Python, Java) to publish profiles in pprof format. This is an indicator the pprof format has become de-facto. Grafana Pyroscope[1] is a tool that's capable of parsing the pprof format, agnostic to the source programming language, and has instructions for Go, Java, Python, Ruby, node.js, Rust, and .NET.

My understanding is that you're searching for a combination of the profiles, metrics, and tracing. Caddy supports all 3.

[0] https://github.com/google/pprof/blob/main/doc/README.md

[1] https://grafana.com/docs/pyroscope/latest/

metrics and tracing need to be manually enabled (for now, perhaps)

Because we couldn't identify the issue using the results we got from Callgrind, we reached for another profiler, gperftools. It's a sampling profiler and therefor it has a smaller impact on the application's performance in exchange for less accurate call statistics. After filtering out the unimportant parts and visualizing the rest with pprof, it was evident that something strange was happening with the send function. It took only 71 milliseconds with the previous implementation and more than 900 milliseconds with the new implementation of our Bolt server. It was very suspicious, but based on Callgrind, its cost was almost the same as before. We were confused as the two results seemed to conflict with each other.

github.com - google/pprof

The application performances in a specific code path (e.g. gdb, pprof, …).

pprof is https://github.com/google/pprof, it's a very useful tool in golang , and really really really convenient

Go also has pretty good out of the box profiling (pprof[0]) and third-party runtime debugging (delv[1]) that can be used both remotely and local.

These tools also have decent editor integration and can be use hand in hand:

https://blog.jetbrains.com/go/2019/04/03/profiling-go-applic...

https://blog.jetbrains.com/go/2020/03/03/how-to-find-gorouti...

[0] https://github.com/google/pprof

[1] https://github.com/go-delve/delve