pbf VS protoc-gen-validate

it's been about 3 years, but in Grafana at the time we were using something like ArrowJS + Arrow Flight + protobuf.js and then render the datasets into dashboards on Canvas, especially for streaming at ~20hz.

when i benchmarked the fastest lib to simply convert the protobuf decode (https://github.com/mapbox/pbf), it was 5x slower than native JSON parsing in browsers for dataframe-like structures (e.g. a few dozen 2k-long arrays of floats).

does it beat https://github.com/mapbox/pbf ?

at least in the frontend (without WASM), it depends.

i tested https://github.com/mapbox/pbf and while it was faster for deep/complex structs vs an unoptimized/repetative JSON blob, it was slower at shallow structs and flat arrays of stuff. if you spend a bit of time encode stuff as flat arrays to avoid mem alloc, JSON parsing wins by a lot since it goes through highly optimized C or assembly, while decoding protobuf in the JIT does not.

The runtime is taken from mapbox/pbf (with basic type definitions)

My understanding is that the powers that be within Google have decided that validating messages is outside the scope of schemas and serialization. protoc-gen-validate provides a portable way to perform validation: https://github.com/bufbuild/protoc-gen-validate

The problem with required fields is it kicks the can down the road when you want to deprecate a field. Keeping everything optional is much, much better for everyone in the long run.

I built a (go) very rough draft to support 3.1 + a json pointer pkg to support a refactor that was in progress before I bailed on the idea. The amount of effort to get a conventional API was dwarfing that of grpc, especially with protoc-gen-validate.

Yes, I am currently using this. I prefer something like this https://github.com/envoyproxy/protoc-gen-validate

We've been using Rust with protobuf, but had to write lots of conversions from protobuf struct to actual DTO, since prost generates Options for nested types, and does not really support custom types. So I extended prost with the options that we needed. I might aim to also support protoc-gen-validate.