jemalloc VS gperftools

jemalloc (as opposed to GNU libc and LLVM) sometimes performs better. [1]

[1] https://jemalloc.net/

There are a number of opportunities for optimizing the use of the underlying CPU resources. These include optimizing memory management and thread allocation to the structure of the underlying CPU hardware. Memory management can be improved through the use of advanced memory allocators (such as Jemalloc and TCMalloc) and/or reducing memory accesses that are slower (i.e., across NUMA nodes). Threading allocation can be improved through appropriate configuration of the OpenMP threading library and/or use of Intel's Open MP library.

Certain build processes determine the page size at compile time and assume it's the same at run time, and fail if it is not: https://github.com/jemalloc/jemalloc/issues/467

Some memory-mapped files formats have assumptions about page granularity: https://bugzilla.redhat.com/show_bug.cgi?id=1979804

The file format issue applies to ELF as well. Some people patch their toolchains (or use suitable linker options) to produce slightly smaller binaries that can only be loaded if the page size is 4K, even though the ABI is pretty clear in that you should link for compatibility with up to 64K pages.

jemalloc has its own problem with threads - if you have a multi-threaded application that uses jemalloc on all threads except the main thread, then the cleanup that jemalloc runs on main thread exit will segfault. In $dayjob we use jemalloc as a sub-allocator in specific arenas (the rest of the application uses libc malloc, but for some cases it allocates pages using mmap and then uses jemalloc to partition them). $dayjob's code is written in Rust, whose unit test framework defaults to running tests in one or more threads and the main thread of the test binary just orchestrates them. So the test binary triggers this segfault reliably.

( https://github.com/jemalloc/jemalloc/issues/1317 Unlike what the title says, it's not Windows-specific.)

As a demonstration, I want to measure something that caught my attention months ago—an interesting topic brought up by Jason Pearson: the use of jemalloc as a native memory allocator for Android builds. The initial claim is that this usage brings a reduction in memory usage by optimizing how memory is allocated and deallocated. jemalloc is designed to minimize memory fragmentation and improve performance, particularly in multithreaded applications, making it ideal for resource-intensive builds.

jemalloc as well has some handy leak / memory profiling abilities: https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-P...

The worst memory performance bug I ever saw turned out to be heap fragmentation in a non-GC system. There are memory allocators that solve this like https://github.com/jemalloc/jemalloc/tree/dev but ... they do it by effectively running a GC at the block level

As soon as you use atomic counters in a multi-threaded system you can wave goodbye to your scalability too!

The linked talk video mentioned they're playing with it in jemalloc and tcmalloc.

I found this https://github.com/jemalloc/jemalloc/issues/1440 but couldn't find tcmalloc doing similar.

These guys are aware of mesh and compare against it: https://abelay.github.io/6828seminar/papers/maas:llama.pdf

For profiling you are right clang has no -pg that works. But there are options, since clang supports PGO the fprofile flags could be what you need. they will generated a profraw file for you. There is also gperf tools which work for more than just linux. https://github.com/gperftools/gperftools

Because we couldn't identify the issue using the results we got from Callgrind, we reached for another profiler, gperftools. It's a sampling profiler and therefor it has a smaller impact on the application's performance in exchange for less accurate call statistics. After filtering out the unimportant parts and visualizing the rest with pprof, it was evident that something strange was happening with the send function. It took only 71 milliseconds with the previous implementation and more than 900 milliseconds with the new implementation of our Bolt server. It was very suspicious, but based on Callgrind, its cost was almost the same as before. We were confused as the two results seemed to conflict with each other.

gprftools (https://github.com/gperftools/gperftools) can be easily plugged in using LD_PRELOAD and signal, and has nice go implemented visualization tool https://github.com/google/pprof.

Google's tcmalloc