Protobuf VS MessagePack

(((uint32_t)clz + 7) * 9) >> 6

3. Hand roll a jump table taking advantage of arm's fixed instruction width to calculate the jump target with a shift.

https://github.com/protocolbuffers/protobuf/commit/b039dfe26...

This results in one branch for 1 byte varints, plus one additional branch for any larger size, and the branch predictor does not have to track a varying trip count through a loop. This approach resulted in a 2.64% speedup for overall encoding (which includes a lot more than just varints) on mid sized arm cores.

I think it's very difficult to beat a single comparison and branch for the 1 byte case for actual encoding forwards or backwards, unless you know there's going to be long runs of one-byte values.

I wrote a portable abstraction for this that works across Linux, MacOS, and Windows: https://github.com/protocolbuffers/protobuf/blob/4917ec250d3...

I call them "linker arrays". They are great when you need to globally register a set of things and the order between them isn't significant.

This kind of "expected next field" optimization has a long history in protobuf, but results are mixed.

The generated code in C++ used to check for the expected next field before falling back to the switch() (example here: https://github.com/protocolbuffers/protobuf/blob/460e7dd7c47...) but this was removed in 2016 when tests found that it hurt more than it helped.

One tricky part of making this optimization work is making a good guess about what the next field should be. Miguel's article alludes to this:

> Each field specifies which fields to try next. This allows the compiler to perform field scheduling, by carefully deciding which order to try fields in based both on their declaration order and a rough estimation of their “hotness”, much like branch scheduling happens in a program compiler. This avoids almost all of the work of looking up the next field in the common case, because we have already pre-loaded the correct guess.

> I haven’t managed to nail down a good algorithm for this yet, but I am working on a system for implementing a type of “branch prediction” for PGO, that tries to provide better predictions for the next fields to try based on what has been seen before.

One delightful thing about the tail-call parser design is that the CPU's branch predictor effectively takes over the job of guessing. With a tail call parser, the dispatch sequence ends up looking like this:

    cmp    QWORD PTR [rdi+0x8],rsi               # Bounds check

> Would be great if you report such remote code executions to the authors/Google. I am sure they handle CVEs etc.

I wasn't getting paid to fix their code, I have no interest in helping Google for free, and don't want to help Google.

> There has been a security audit like

A checkbox report from six years ago. That's ancient times at the pace that things are added to gRPC.

> Are you making shit up as you go?

No. This [0] repo I used to reproduce a stack smash issue before `main()`. I reported the issue here [1]. I don't get paid to fix Google's things and found a workaround for the purposes I needed.

[0]: https://github.com/keith-bennett-airmap/grpc-stacksmash

[1]: https://github.com/protocolbuffers/protobuf/issues/12732

> I can see various fuzzers in the code base so that claim is also unsubstantiated

Fuzzers are cool, but they don't cover the whole codebase.

This is doable via this trick:

https://github.com/protocolbuffers/protobuf/blob/ae0129fcd01...

Install the protoc compiler manually from here and add it to your PATH.

We wanted to use Ractors in our latest Ruby project (which is Rust based using rb-sys/magnus for some parts), but since our users use Google Protobuf inside the code that may run in Ractors, we could not because the library is not Ractor safe[0] (and it's unreasonable for us to ask our users to not be able to use the official Protobuf library).

0 - https://github.com/protocolbuffers/protobuf/issues/19321

For Java, I think you mean Jackson, not gson, unless something has changed recently. Goes to show that even the behemoths can get this wrong.

https://github.com/protocolbuffers/protobuf/blob/6aefdde9736...

Building Neon requires 3.15+ version of protoc (protobuf-compiler). If your distribution provides an older version, you can install a newer version from here.

I have searched extensively, and have tried a few libraries from msgpack-c (C++), to YAS, and a few more not to name.

how about https://github.com/msgpack/msgpack-c/tree/c_master ?