RoaringBitmap VS pyroscope

I was recently reading about Roaring https://roaringbitmap.org/ which is a highly optimized compressed bitset implementation. I reccomend reading about it if you are interested in this sort of thing. The talk at https://roaringbitmap.org/talks/ is especially good.

However, I would love compressed (not just packed) bitsets too, which is something different to me. I would make it another class with a similar interface, based on something like roaring. It doesn't need to be in the standard, but it would be nice if the API was a such that one could easily swap implementations.

As an aside if you find yourself having to compute them on the fly, know that the Roaring Bitmaps libraries is the way to go [1]. The bitmaps are compressed, and can be streamed directly into SIMD computations (batching XORs and popcnts 256 bits wide!). The Jaccard index is just intersection_len / union_len [2] away

[1] https://roaringbitmap.org/

[2] https://roaringbitmap.readthedocs.io/en/latest/#roaringbitma...

Use a two stage approach, with a bloom/cuckoo filter stored as a https://roaringbitmap.org/ in memory. Then a secondary key/value store on disk (bolt or anything else).

As an aside, if you're dealing with large bit sets, you might also want to evaluate Roaring Bitmaps.

Sidetracking a bit the conversation. What a coincidence that the author (Lemire) is also represented on Today's #1 "Ask HN: What are some cool but obscure data structures you know about?" as he is the main contributor of RoaringBitmap https://github.com/RoaringBitmap/RoaringBitmap and one of the main authors of the data structure.

What do you mean? pyroscope.io was slow for you? or the blog?

Tries (or prefix trees).

We use them a lot at Pyroscope for compressing strings that have common prefixes. They are also used in databases (e.g indexes in Mongo) or file formats (e.g debug symbols in macOS/iOS Mach-O format are compressed using tries).

We have an article with some animations that go into details about tries in case anyone's interested [0].

[0] https://github.com/pyroscope-io/pyroscope/blob/main/docs/sto...

You could use continuous profiling with Pyroscope which uses async-profiler under the hood, but with the added functionality that you can add relevant tags to your VMs (example).

Java Flight Recorder (JFR) is a format for collecting diagnostic and profiling data from Java applications. A while back someone created an issue for Pyroscope , an open source continuous profiler written in Go, to support ingesting profiles in JFR format, but there were no existing parsers that were also written in Go.

This cloud version is actually a slimmed-down version of Pyroscope which is open source and so you can run it locally.

At Pyroscope (open source continuous profiling) we use flamegraphs extensively to visualize and analyze profiling data. However, one of the worst parts about using flamegraphs for analysis is that they are kind of annoying to share.