Experimental fast implementation of Protobufs in Go
Article author here, good to see it on HN, someone else has submitted it (thanks :-)).
If you are interested in the topic you may be also interested in a research library I wrote recently: https://github.com/splunk/exp-lazyproto, which among other things exploits the partial (de)serialization technique. This is just a prototype for now, one day I may actually do a production quality implementation.
FlatBuffers: Memory Efficient Serialization Library
My go-to for needing to deserialize structured data in a fast way these days is flatbuffers. It compacts nicely and more importantly is zero copy/allocation(within the constraints of your language where possible) in deserialize. Which lets you do neat things like mmap it from disk.
We used to store 20-30mb of animation data with it and we'd just mmap the whole file and let the kernel handle paging it in/out, worked great.
I don't know how up to date their benchmarks are but my experience has been that it beats almost every other off-the-shelf solution(other than maybe capn-proto which has some similar properties).
Download talent.io’s Tech Salary Report. Median salaries, most in-demand technologies, state of the remote work... all you need to know your worth on the market by tech recruitment platform talent.io
OpenTelemetry Enhancement Proposals
This. The statelessness of the OTLP is by design. I did consider stateful designs with e.g. shared state dictionary compression but eventually chose not to, so that the intermediaries can remain stateless.
An extension to OTLP that uses shared state (and columnar encoding) to achieve more compact representation and is suitable for the last network leg in the data delivery path has been proposed and may become a reality in the future: https://github.com/open-telemetry/oteps/pull/171
OK, but I just want readers to be aware that the whole idea that it could take five minutes to parse a million protobufs is completely preposterous. I reimplemented their benchmark just now and it runs at roughly 8 million protos per second, orders of magnitude faster than they state, and I didn't even do anything to optimize it.
sqlite-zstd: Transparent dictionary-based row-level compression for SQLite - An SQLite extension written in Rust to reduce the database size without losing functionality
2 projects | reddit.com/r/rust | 31 Jul 2022
[AskJS] Delimiting array buffers in a single array buffer
Disappointing performance of struct serialization/deserialization to/from byte buffers (memcpy vs member-by-member)
1 project | reddit.com/r/cpp_questions | 6 Jun 2022
Pointer Tagging for x86 Systems
2 projects | news.ycombinator.com | 31 Mar 2022
Next step in zpp::bits serialization - protobuf without code generation - no proto file required- (experimental)
2 projects | reddit.com/r/cpp | 9 Jan 2022