Our great sponsors
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
-
-
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
FlatBuffers does the trick as a better replacement to ProtoBuf when running in some resource-critical devices. We were working on a project that leverage Arrow IPC (internally FlatBuffers) and shared memory to collect metrics on edge devices with limit CPU and memory, hopefully we can open source it soon.
But in this case the format is defined by [Prometheus](https://github.com/prometheus/prometheus/blob/main/prompb/re...) and we just adopt that.
Idiomatic quick-protobuf is much faster than non-idiomatic rust-protobuf: https://github.com/tafia/quick-protobuf/tree/master/perftest
To bring this discussion back to earth, the Go code in question is: https://github.com/VictoriaMetrics/easyproto/blob/master/rea...
I don't see why did you label Aliaksandr Valialkin, the author, an "expert". I mean, he's no dummy but what exactly makes him an expert on optimizing Go code?
As someone who also writes Go, I don't see any "hyper optimizations" in the code. It just decodes the bytes of Protocol Buffer using straightforward code that I would expect a competent developer to write.
It really is just: read bytes from memory ([]byte) and interpret them according to PB spec.
There's only one trick there: unsafeBytesToString() that does no-allocation conversion of []byte to string. This is unsafe in general but safe in their specific case. And I've seen this trick before so it's not some secret, expert-only knowledge.
Most comments here are like bad LLMs: hallucinating opinions without bothering to spent even few minutes acquiring the data to base those opinions on.
That would be true if you used `Vec::clear` too, it doesn't allocate a new vector. My point was that you still end up running Drop implementations with RepeatedField, just not all at once. See https://play.rust-lang.org/?version=stable&mode=debug&editio...
The benchmark is not comparing apples to apples.
prost is the most widely used Protobuf implementation in Rust, maintained by the Tokio organization. prost generates structs and serialization/deserialization code for you.
easyproto according to GitHib Search is used only by two projects. easyproto provides primitives for serializing and deserializing Protobuf, and requires hand writing code to do both.
A fair comparison would be prost vs google.golang.org/protobuf, or easyproto vs parts of quick-protobuf.
In most cases you can make Go as fast as Rust, but from my experience writing performance-sensitive code in Go requires significantly larger time investment and overall requires deeper language expertise. Pebble (RocksDB replacement in Go by CockroachDB) is a good example of this, the codebase is littered with hand-inlined[1] functions, hand-unrolled loops and it's not[2] even using Go memory management for performance critical parts, it's using the C memory allocator and manual memory management.
[prost]: https://github.com/tokio-rs/prost
