bitmap VS roaring

Good question, I think there's many different lessons. To your point about bitmasks, you can imagine that each component (i.e. column) has an array of data and a large bitmap that identifies whether a component is present or not. Had to build a SIMD implementation so you can do and, and not, or and xor operations on millions of components within reasonable amount of time. Interestingly enough, you still need a hashmap or b+tree in case you want to retrieve a component by it's ID instead of an index, but the rest of things can be modeled with bitmap indexes.

For (3), and if you’re interested in checking if specific bits are set or not, take a look at https://github.com/bits-and-blooms/bitset and https://github.com/RoaringBitmap/roaring.

Each topic has a roaring bitmap. Each bit corresponds to a subscriber in the skip list. For each topic, head and tail cursors are also maintained to ensure that newly added subscriptions are always tested and deleted subscriptions are always evicted. Roaring bitmaps are compressed and discontinuous so memory usage again remains bounded as subscribers come and go.

Definitely take a look at a roaring bitmap. https://github.com/RoaringBitmap/roaring

Roaring bitmaps + btrees are a rock solid indexing approach. Alternatively, bleve has a lot out of the box but that's introducing a new datastore, basically.