RoaringBitmap
pebble
RoaringBitmap | pebble | |
---|---|---|
24 | 11 | |
3,388 | 4,448 | |
0.8% | 1.1% | |
8.5 | 9.8 | |
10 days ago | 6 days ago | |
Java | Go | |
Apache License 2.0 | BSD 3-clause "New" or "Revised" License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
RoaringBitmap
-
Iterating over Bit Sets Quickly
I was recently reading about Roaring https://roaringbitmap.org/ which is a highly optimized compressed bitset implementation. I reccomend reading about it if you are interested in this sort of thing. The talk at https://roaringbitmap.org/talks/ is especially good.
- Roaring Bitmaps
- Roaring bitmaps are compressed bitmaps, can be 100x faster
-
What feature would you like to remove in C++26?
However, I would love compressed (not just packed) bitsets too, which is something different to me. I would make it another class with a similar interface, based on something like roaring. It doesn't need to be in the standard, but it would be nice if the API was a such that one could easily swap implementations.
-
Jaccard Index
As an aside if you find yourself having to compute them on the fly, know that the Roaring Bitmaps libraries is the way to go [1]. The bitmaps are compressed, and can be streamed directly into SIMD computations (batching XORs and popcnts 256 bits wide!). The Jaccard index is just intersection_len / union_len [2] away
[1] https://roaringbitmap.org/
[2] https://roaringbitmap.readthedocs.io/en/latest/#roaringbitma...
-
Looking for fast, space-efficient key-lookup
Use a two stage approach, with a bloom/cuckoo filter stored as a https://roaringbitmap.org/ in memory. Then a secondary key/value store on disk (bolt or anything else).
-
BitSet Vs BigInteger
As an aside, if you're dealing with large bit sets, you might also want to evaluate Roaring Bitmaps.
-
Negative Incentives in Academic Research
Sidetracking a bit the conversation. What a coincidence that the author (Lemire) is also represented on Today's #1 "Ask HN: What are some cool but obscure data structures you know about?" as he is the main contributor of RoaringBitmap https://github.com/RoaringBitmap/RoaringBitmap and one of the main authors of the data structure.
- Ask HN: What are some 'cool' but obscure data structures you know about?
- Roaring bitmaps: A better compressed bitset
pebble
- Fivefold Slower Compared to Go? Optimizing Rust's Protobuf Decoding Performance
-
Hummock: A Storage Engine Designed for Stream Processing
To improve the compacting speed of L0 files, we took inspiration from the design of the CockroachDB storage engine pebble.
- RocksDB/LevelDB inspired key-value database in Go
-
Is there a lightweight, stable and embedded database library?
https://github.com/cockroachdb/pebble ?
-
Understanding Fantoms Underlying Tech & Lachesis Consensus Algorithm + The Most Recent Developments
Pebble is a LevelDB/RocksDB-inspired key-value database focused on performance. PebbleDB is replacing Fantom’s previous RPC software to match Fantom’s sheer throughput. PebbleDB is consistently faster than previous options and it has helped the Fantom network to consistently reach synchronization with the network quicker while reducing the duration of API requests.
-
Looking for fast, space-efficient key-lookup
https://github.com/cockroachdb/pebble Pure go SSD native key-value store. You could think of it as map[[]byte][]byte on persistent storage.
-
Open Source Databases in Go
Genji is a document-oriented, embedded, SQL database. It is build over Pebble which is a port of RocksDB in Go, by the authors of CockroachDB.
-
CockroachDB: The Resilient Geo-Distributed SQL Database
Out of curiosity, why write https://github.com/cockroachdb/pebble and not just use https://github.com/dgraph-io/badger?
- Recommendation for Key/Value storage
-
CGo-free sqlite database/sql driver 1.8.0 for darwin/amd64 released
There's the semi-recent pebble from the folks that make cockroach db. Haven't used it, but seems promising: https://github.com/cockroachdb/pebble
What are some alternatives?
HyperMinHash-java - Union, intersection, and set cardinality in loglog space
bolthold - BoltHold is an embeddable NoSQL store for Go types built on BoltDB
lucene - Apache Lucene open-source search software
LevelDB - LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.
CQEngine - Ultra-fast SQL-like queries on Java collections
badger - Fast key-value DB in Go.
Primes - Prime Number Projects in C#/C++/Python
pogreb - Embedded key-value store for read-heavy workloads written in Go
Feign - Feign makes writing java http clients easier
Redis - Redis Go client
maven-compiler-plugin - Apache Maven Compiler Plugin
goleveldb - LevelDB key/value database in Go.