jsoniter VS goprotobuf

https://github.com/json-iterator/go an alternative JSON encoding package which allows to stream (flush out) encoded data as soon as it's able to (which is in contrast with the stock package which buffers everything until the encoding is known to be complete and OK).

What to use Easyjson is about the top of the pack and it's straightforward. The downside of efficient tools is that they use code generation to create the code required to turn your structs into json to minimise allocations. This is a manual build step which is annoying. Interestingly json-iterator also uses reflection but it's significantly faster. I suspect black magic.

jsoniter for low level access to JSON encode and decode

I think you have to parse the json, if you dont know exactly what you are looking for and want some validation und prevent manual parsing errors. For parsing big json files it is recommend to read and decode it as stream. Here is an example. If you have serious performance criteria take a look at jsoniter. It can be used as 1 to 1 replacement for standard library.

Takes like 10 minutes to write and parses very efficiently. https://github.com/json-iterator/go looks like it can provide such simple parsing

Need to compare with https://github.com/json-iterator/go too!

The classic answer is protobufs. You can serialize out to binary format.

Seems like go is pretty middle of the road. I can only guess as to why but it probably has to do with heavy usage of pointers and reflection which are much slower than other implementations. Gogo/protobuf (RIP) solved this performance with code generation, but the the official go protobuf implementation has essentially eschewed it. I do wonder how the benchmark would look using the new vitess proto library for Go (which has many of the benefits of gogo but with active development and an API built on top of the Google one)

go get github.com/golang/protobuf/protoc-gen-go

Maybe I'm missing something, but my read of [golang/protobuf#364](https://github.com/golang/protobuf/issues/364) was that the re-organization in protobuf-go v2 was allow for optimizations like gogoprotobuf to be developed without requiring a complete fork. I totally understand that the authors of gogoprotobuf do not have the time to re-architect their library to use these hooks, but best I can figure this generator does not use these hooks either. Instead it defines additional member functions, and wrappers that look for those specialized functions and fallback to the generic ones if not found.

I am thinking about stuff like the [ProtoMethods](https://pkg.go.dev/google.golang.org/[email protected]/reflec...) API.

I wonder why not? Did the authors of the vtprotobuf extension not want to bite off that much work? Is the new API not sufficient to do what they want (thus failing some of the goals expressed in golang/protobuf#364?

In this case try approach with line by line generation. Very much like what protoc-gen-go does for Go code: https://github.com/golang/protobuf/blob/ae97035608a719c7a1c1c41bed0ae0744bdb0c6f/protoc-gen-go/grpc/grpc.go#L142, need to implement this kind of generator yourself.

Something like this: https://github.com/golang/protobuf/blob/master/internal/gengogrpc/grpc.go

The difference to the client-server solution is subtile when it comes to the details in implementation. protoc-gen-go and other plugins in the Protocol Buffers ecosystem use this approach.