gogoprotobuf VS cbor

The Go tooling for gRPC is inexplicably bad, both in terms of ergonomics and in terms of performance.

The GoGoProtobuf [1] project was started to improve both. It would generate nice Go types that followed Go's conventions. And it uses fast binary serialization without needing to resort to reflection.

Unfortunately, the gRPC/Protobuf team(s) at Google is famously resistant to changes, and was unwilling to work with the GoGo. As a result, the GoGo project is now dead. [2]

I've never used Buf, but it looks like it might fix most of the issues with the Go support.

[1] https://github.com/gogo/protobuf

[2] https://x.com/awalterschulze/status/1584553056100057088

That probably won't happen either, because the Google gRPC team is also quite resistant to any kind of change (that isn't driven by internal needs at Google). Projects like GoGo Protobuf had to be developed to fill gaps that the Google team refused to fill or accept PRs for.

the gogo protobuf compiler toolchain is looking for a maintainer https://github.com/gogo/protobuf u/MehdiHK

protobuff : Google's binary data transfer protocol

Looks like Crowdstrike just pushed out publicly a replacement for gogo/proto. It looks like the library solves 2 problems:

Seems like we're pretty middle of the road. I can only guess as to why but it probably has to do with heavy usage of pointers and reflection. Gogo/protobuf solved this performance with code generation, but the go protobuf implementation has essentially eschewed it. I do wonder how the benchmark would look using the new vitess proto library for Go (which has many of the benefits of gogo but with active development and an API built on top of the Google one)

Yeah, it's a pretty thick Makefile and set of shell scripts, but it's seemingly using things like: k8s.io/code-generator/cmd/go-to-protobuf and github.com/gogo/protobuf/protoc-gen-gogo and github.com/gogo/protobuf/protoc-gen-gogofast which from what I can tell... [it looks like this can probably help me a bit in understanding how to make custom types](https://cloud.redhat.com/blog/kubernetes-deep-dive-code-generation-customresources).

repositories: # Definition depends on the 27156597fdf4fb77004434d4409154a230dc9a32 version of https://github.com/googleapis/googleapis # and defines its name as GOOGLE_APIS # It can be referenced in importPaths by $GOOGLE_APIS GOOGLE_APIS: https://github.com/googleapis/googleapis@27156597fdf4fb77004434d4409154a230dc9a32 # Definition depends on the 226206f39bd7276e88ec684ea0028c18ec2c91ae version of https://github.com/gogo/protobuf # and defines its name as GOGO_PROTOBUF # It can be referenced in the importPaths by $GOGO_PROTOBUF GOGO_PROTOBUF: https://github.com/gogo/protobuf@226206f39bd7276e88ec684ea0028c18ec2c91ae

В проекті ми використовуємо офіційну бібліотеку Protobuf github.com/protocolbuffers/protobuf, яка підчас серіалізації використовує рефлексію і будує слайс байтів через append. А потім я дізнався про "Protocol Buffers for Go with Gadgets" github.com/gogo/protobuf, бібліотеку-fork яка генерує додатковий код щоб прибрати рефексію підчас серіалізації і вже записує в слайс байтів по індексу бо так швидше. Коли змінював одну бібліотеку на іншу то важливим вважав що стало працювати швидше і написані раніше тести пройшли успішно. І все б було гаразд але в проекті існувала латка яка через пару тижнів після заміни перезапустила мікросервіс через паніку:

Someone made a benchmark of serialization libraries in go [1], and I was surprised to see gobs is one of the slowest ones, specially for decoding. I suspect part of the reason is that the API doesn't not allow reusing decoders [2]. From my explorations it seems like both JSON [3], message-pack [4] and CBOR [5] are better alternatives.

By the way, in Go there are a like a million JSON encoders because a lot of things in the std library are not really coded for maximum performance but more for easy of usage, it seems. Perhaps this is the right balance for certain things (ex: the http library, see [6]).

There are also a bunch of libraries that allow you to modify a JSON file "in place", without having to fully deserialize into structs (ex: GJSON/SJSON [7] [8]). This sounds very convenient and more efficient that fully de/serializing if we just need to change the data a little.

--

1: https://github.com/alecthomas/go_serialization_benchmarks

2: https://github.com/golang/go/issues/29766#issuecomment-45492...

--

3: https://github.com/goccy/go-json

4: https://github.com/vmihailenco/msgpack

5: https://github.com/fxamacker/cbor

--

6: https://github.com/valyala/fasthttp#faq

--

7: https://github.com/tidwall/gjson

8: https://github.com/tidwall/sjson

Ha, no suggestions for cbor https://github.com/fxamacker/cbor

If you need to support non-Go code I suggest you explore CBOR. There are a few implementations in Go but his is my favorite.