yottaStore
RoaringBitmap
yottaStore | RoaringBitmap | |
---|---|---|
9 | 24 | |
79 | 3,390 | |
- | 0.9% | |
1.8 | 8.5 | |
about 1 year ago | 15 days ago | |
Java | ||
MIT License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
yottaStore
- Ask HN: Why are there no open source NVMe-native key value stores in 2023?
-
How to deal with overflowing counters?
I'm building a database, and I'm working on the storage format. For each record I have a logical clock which increases by 1 every time there's a write operation on the record, and I would like to be able to compare the clocks of two writes to understand which happened first.
-
Need help porting a wait free trie from C to Rust (and other silly questions)
If you're curious to know more, the tree is used for rendezvous based routing, as used by my datastore. I'm doing machine learning on a 200 TB dataset, using around 200 machines.
-
Looking for fast, space-efficient key-lookup
I copied this approach from several papers, with some improvements, for my datastore.
-
How to handle hundreds of routes?
I have a server with hundreds of routes, representing all the possible operations I can do on a datastore. How can I organize my code better?
-
Async-rdma v0.4.0: A Rust lib for writing high-throughput, low-latency networking apps simply
Yes I'm building a database where storage and compute are decoupled. I use io_uring to do pseudo-RDMA, and I'm looking to add ePBF to make it even more effective.
-
Avoid hash flooding without a secret key?
I'm currently building an implementation of the dynamo paper, yottastore. Imagine it as a huge, distributed, hash map.
-
How to deal with a very big hash table?
I'm building an implementation of the dynamo paper, yottastore. Given a key, I need to find which NVMe block stores the data. To do that I hash the key to find the shard where I have an in memory array in which at position [hash] I can find a struct with:
-
Golang is better than Rust for next generation in-memory database
I have to be honest, I'm very skeptical about your results and your code. I'm building a database, yottastore, both in javascript, golang and rust so I think I can share my opinion:
RoaringBitmap
-
Iterating over Bit Sets Quickly
I was recently reading about Roaring https://roaringbitmap.org/ which is a highly optimized compressed bitset implementation. I reccomend reading about it if you are interested in this sort of thing. The talk at https://roaringbitmap.org/talks/ is especially good.
- Roaring Bitmaps
- Roaring bitmaps are compressed bitmaps, can be 100x faster
-
What feature would you like to remove in C++26?
However, I would love compressed (not just packed) bitsets too, which is something different to me. I would make it another class with a similar interface, based on something like roaring. It doesn't need to be in the standard, but it would be nice if the API was a such that one could easily swap implementations.
-
Jaccard Index
As an aside if you find yourself having to compute them on the fly, know that the Roaring Bitmaps libraries is the way to go [1]. The bitmaps are compressed, and can be streamed directly into SIMD computations (batching XORs and popcnts 256 bits wide!). The Jaccard index is just intersection_len / union_len [2] away
[1] https://roaringbitmap.org/
[2] https://roaringbitmap.readthedocs.io/en/latest/#roaringbitma...
-
Looking for fast, space-efficient key-lookup
Use a two stage approach, with a bloom/cuckoo filter stored as a https://roaringbitmap.org/ in memory. Then a secondary key/value store on disk (bolt or anything else).
-
BitSet Vs BigInteger
As an aside, if you're dealing with large bit sets, you might also want to evaluate Roaring Bitmaps.
-
Negative Incentives in Academic Research
Sidetracking a bit the conversation. What a coincidence that the author (Lemire) is also represented on Today's #1 "Ask HN: What are some cool but obscure data structures you know about?" as he is the main contributor of RoaringBitmap https://github.com/RoaringBitmap/RoaringBitmap and one of the main authors of the data structure.
- Ask HN: What are some 'cool' but obscure data structures you know about?
- Roaring bitmaps: A better compressed bitset
What are some alternatives?
async-rdma - Easy to use RDMA API in Rust async
HyperMinHash-java - Union, intersection, and set cardinality in loglog space
solid_cache - A database-backed ActiveSupport::Cache::Store
lucene - Apache Lucene open-source search software
cdb - A native golang implementation of cdb (http://cr.yp.to/cdb.html)
CQEngine - Ultra-fast SQL-like queries on Java collections
KVRocks - RocksDB compatible key value store and MyRocks compatible storage engine designed for KV SSD
Primes - Prime Number Projects in C#/C++/Python
ssd-nvme-database - Columnar database on SSD NVMe
Feign - Feign makes writing java http clients easier
uNVMe - KV and LBA SSD userspace NVMe driver
maven-compiler-plugin - Apache Maven Compiler Plugin