hotspot VS hashbrown

perf with Hotspot 👌

If you’re able to use perf on Linux, I would recommend hotspot for visualizing the results.

I can recommend you using the https://github.com/KDAB/hotspot/ tool whenever you want to study performance.

Every Linux C/C++/Rust developer should know about https://github.com/KDAB/hotspot. It's convenient and fast. I use it for Rust all the time, and it provides all of these features on the back of regular old `perf`.

Flamegraphs alone aren't a full picture of what your application is doing, but it can give you hints as to where to look. Another tool I often use is Hotspot which can open the perf.data file and provide more options for filtering and digging into the gathered data beyond the single flamegraph.

Ignore the command, it's just a placeholder to get meaningful values. The -d flag adds basic cache events, by adding another -d you also get load and load miss events for the dTLB, iTLB and L1i cache.

But as mentioned, you can instrument any event supported by your system. Including very obscure events such as uops_executed.cycles_ge_2_uops_exec (Cycles where at least 2 uops were executed per-thread) or frontend_retired.latency_ge_2_bubbles_ge_2 (Retired instructions that are fetched after an interval where the front-end had at least 2 bubble-slots for a period of 2 cycles which was not interrupted by a back-end stall).

You can also record data using perf-record(1) and inspect them using perf-report(1) or - my personal favorite - the Hotspot tool (https://github.com/KDAB/hotspot).

Sorry for hijacking the discussion a little, but I think perf is an awesome little tool and not as widely known as it should be. IMO, when using it as a profiler (perf-record), it is vastly superior to any language-specific built-in profiler. Unfortunately some languages (such as Python or Haskell) are not a good fit for profiling using perf instrumentation as their stack frame model does not quite map to the C model.

> [...] how Perf compares to vendor tools like vTune [...] ?

Regarding the hardware events that Perf can capture on x86, it has pretty much all of them. So it should be equivalent to vTune for all practical purposes.

The big difference is in the UI -- or absence thereof. Perf is a low-level tool and its output is mostly text files. There is a curses-based TUI for perf-report (and even gtk version, but it is essentially the same as the TUI, just using GTK2 widgets), but that's about it.

By contrast, vTune comes with a heavy (electron-based?) GUI and is quite helpful in guiding beginners, with many graphs and explanations.

Of course, one can (and is expected to) complement Perf with an assortment of tools that process its output for visualization. For example, the flamegraph [1] and heat map [2] tools described in the article. But also KDAB hotspot [3] or HPerf for a vTune-style perf-report.

[1] https://github.com/brendangregg/FlameGraph

[2] https://github.com/brendangregg/HeatMap

[3] https://github.com/KDAB/hotspot

[4] https://www.poirrier.ca/hperf/

https://github.com/KDAB/hotspot might contain parsing code you could use as an example (other than perf script). It always accepts raw perf.data, and there doesn't seem to be a way to feed it the output of perf script, so it might be parsing it directly instead of calling perf script.

That's because you're looking at a wrapper around the actual implementation (which lives in an external package). Notice "use hashbrown::hash_map as base;" at the top.

There's far more unsafe there: https://github.com/rust-lang/hashbrown/blob/f2e62124cd947b5e...

You might want to check out how popular ecosystem crates do some of these things. Particularly relevant to you are probably crates providing collections, such as smallvec, hashbrown, or indexmap.

https://github.com/topics/swisstable

rust-lang/hashbrown: https://github.com/rust-lang/hashbrown

CuPy has array but not yet hashmaps, or (GPU) SIMD FWICS?

NumPy does SIMD:

Thanks, great point indeed. I am looking into this https://github.com/rust-lang/hashbrown

The way I think about it -- rather naively, I suppose -- is that I care more about the references cells make to each other than the actual grid of cells displayed on a table. The latter feels more like a "view" of the data than an actual data structure?

This also seems to align with the relative priority of (sorted from highest to lowest): figuring out the order of evaluation, calculating those evaluations, and finally displaying the results of the evaluation

updated std's hashbrown dependency to 0.14 which contains some optimizations

The std hashmap is actually very fast and uses state of the art hashmap design, namely because it's implemented by hashbrown

I believe Rust uses hashbrown as the underlying implementation now. This just calculates the number of buckets based on the number of items requested:

https://github.com/rust-lang/hashbrown/blob/009969a860290849...

Is it really the case that rehashing can guarantee that the number of buckets allocated will be sufficient for any given set of keys? In principle you could fail to rehash in a way that reduces collisions after k attempted rehashings.

For example, since the hashbrown crate is marked with #![no_std], it can be used as a dependency for the standard library.

To implement something that cannot be expressed in safe Rust, or at least cannot be expressed succinctly in safe Rust, like fundamental datastructures. The hashbrown crate contains a lot of unsafe code, but it's such high quality that it's now the backing implementation for std::collections::HashMap.