Our great sponsors
-
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
Hi, you seem to know a bit about Sanakirja!
It stores 4kb blobs, right? Does Pijul first parses the data (copying it to other allocations), or uses the data as is? I mean, there are some libraries like cap'n'proto[0] and rkyv[1] that can directly use the file contents as an in-memory data structure, I was wondering if Pijul did anything like that.
I mean, is this btree page [2] stored exactly like this on disk, and does Pijul exploits that to avoid further copying data?
(I guess there's a trouble with compression there: to decompress you really need to write in another buffer)
Also, is the I/O done with something that prevent userspace copies like mmap or io_uring, or does it eventually calls read() to copy the data to its own buffer?
I want to build something like Sanakirja, but with those features, so I'm wondering if there's any overlap.
[0] https://github.com/capnproto/capnproto-rust
[1] https://github.com/rkyv/rkyv
[2] https://docs.rs/sanakirja-core/latest/sanakirja_core/btree/p...
Hi, you seem to know a bit about Sanakirja!
It stores 4kb blobs, right? Does Pijul first parses the data (copying it to other allocations), or uses the data as is? I mean, there are some libraries like cap'n'proto[0] and rkyv[1] that can directly use the file contents as an in-memory data structure, I was wondering if Pijul did anything like that.
I mean, is this btree page [2] stored exactly like this on disk, and does Pijul exploits that to avoid further copying data?
(I guess there's a trouble with compression there: to decompress you really need to write in another buffer)
Also, is the I/O done with something that prevent userspace copies like mmap or io_uring, or does it eventually calls read() to copy the data to its own buffer?
I want to build something like Sanakirja, but with those features, so I'm wondering if there's any overlap.
[0] https://github.com/capnproto/capnproto-rust
[1] https://github.com/rkyv/rkyv
[2] https://docs.rs/sanakirja-core/latest/sanakirja_core/btree/p...
should bring your machien to a halt!
[0] https://github.com/EpicGames/UnrealEngine
I'd like to think it was my project (https://github.com/martinvonz/jj), but other possibilities include Gitless (https://gitless.com/) or Bazaar (https://bazaar.canonical.com/).
Related posts
- A new major version of jql has been released
- Is there a library like Serde but which makes it easy to mutate serialized data stored in a [u8] or Vec<u8>?
- rkyv 0.7: Endian-agnostic types, `no_std` validation, performance improvements, github sponsors and more!
- rkyv 0.5: Comparison derives, serialize bounds, and the future
- rkyv is faster than {bincode, capnp, cbor, flatbuffers, postcard, prost, serde_json}