Our great sponsors
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
generational-arena
Discontinued A safe arena allocator that allows deletion without suffering from the ABA problem by using generational indices.
-
-
-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
> Arenas are, however, unfeasible to implement in Go because it is a garbage collected language.
If you are willing to use cgo, google already implemented one for gapid.
https://github.com/google/gapid/tree/master/core/memory/aren...
Rust has an arena allocator too[1], but it is implemented with 165(!!!) usages of unsafe. :)
[1] https://github.com/fitzgen/bumpalo
This is far from the only arena allocator written in Rust.
From the same author, a zero-unsafe arena allocator: https://github.com/fitzgen/generational-arena
There are many, many arena implementations available with varying characteristics. It's disingenuous to act like Rust requires the author of an arena library to write "unsafe" everywhere.
Maybe I'm missing something, but my read of [golang/protobuf#364](https://github.com/golang/protobuf/issues/364) was that the re-organization in protobuf-go v2 was allow for optimizations like gogoprotobuf to be developed without requiring a complete fork. I totally understand that the authors of gogoprotobuf do not have the time to re-architect their library to use these hooks, but best I can figure this generator does not use these hooks either. Instead it defines additional member functions, and wrappers that look for those specialized functions and fallback to the generic ones if not found.
I am thinking about stuff like the [ProtoMethods](https://pkg.go.dev/google.golang.org/[email protected]/reflec...) API.
I wonder why not? Did the authors of the vtprotobuf extension not want to bite off that much work? Is the new API not sufficient to do what they want (thus failing some of the goals expressed in golang/protobuf#364?
So, I thought this at one point, too. But it turns out that methods is a type alias to an unnamed type, so there's no package level privacy issues: https://github.com/protocolbuffers/protobuf-go/blob/v1.26.0/...
Funny timing, I've just written most of a TypeScript generator for protobufs. I learned about some fun corners of protobufs I didn't expect trying to pass the protouf conformance tests [1] (which this one passes, that's no mean feat!).
- If you write the same message multiple times, protobuf implementations should merge fields with a last write wins policy (repeated fields are concatenated). This includes messages in oneofs.
- For a boolean array, you're better off using a packed, repeated int64 (if wire size matters a lot). Protobuf bools use varint encoding meaning you need at least 2 bytes for every boolean, 1+ for the tag and type and 1 byte for the 0 or 1 value. With a repeated int64, you'd encode the tag and length in 2 varints, and then you get 64 bools per 8 bytes.
- Fun trivia: Varints take up a max of 10 bytes but could be implemented in 9 bytes. You get 7 bits per varint byte, so 9 bytes gets you 63 bits. Then you could use the most significant bit of the last byte to indicate if the last bit is 0 or 1. Learned by reading the Go varint implementation [2].
- Messages can be recursive. This is easy if you represent messages as pointers since you can use nil. It's a fair bit harder if you want to always use a value object for each nested message since you need to break cycles by marking fields as `T | undefined` to avoid blowing the stack. Figuring out the minimal number of fields to break cycles is an NP hard problem called the minimum feedback arc set[3].
- If you're writing a protobuf implementation, the conformance tests are a really nice way to check that you've done a good job. Be wary of implementations that don't implement the conformance tests.
[1]: https://github.com/protocolbuffers/protobuf/tree/master/conf...
[2]: https://github.com/golang/go/blob/master/src/encoding/binary...
[3]: https://en.wikipedia.org/wiki/Feedback_arc_set#Minimum_feedb...
Funny timing, I've just written most of a TypeScript generator for protobufs. I learned about some fun corners of protobufs I didn't expect trying to pass the protouf conformance tests [1] (which this one passes, that's no mean feat!).
- If you write the same message multiple times, protobuf implementations should merge fields with a last write wins policy (repeated fields are concatenated). This includes messages in oneofs.
- For a boolean array, you're better off using a packed, repeated int64 (if wire size matters a lot). Protobuf bools use varint encoding meaning you need at least 2 bytes for every boolean, 1+ for the tag and type and 1 byte for the 0 or 1 value. With a repeated int64, you'd encode the tag and length in 2 varints, and then you get 64 bools per 8 bytes.
- Fun trivia: Varints take up a max of 10 bytes but could be implemented in 9 bytes. You get 7 bits per varint byte, so 9 bytes gets you 63 bits. Then you could use the most significant bit of the last byte to indicate if the last bit is 0 or 1. Learned by reading the Go varint implementation [2].
- Messages can be recursive. This is easy if you represent messages as pointers since you can use nil. It's a fair bit harder if you want to always use a value object for each nested message since you need to break cycles by marking fields as `T | undefined` to avoid blowing the stack. Figuring out the minimal number of fields to break cycles is an NP hard problem called the minimum feedback arc set[3].
- If you're writing a protobuf implementation, the conformance tests are a really nice way to check that you've done a good job. Be wary of implementations that don't implement the conformance tests.
[1]: https://github.com/protocolbuffers/protobuf/tree/master/conf...
[2]: https://github.com/golang/go/blob/master/src/encoding/binary...
[3]: https://en.wikipedia.org/wiki/Feedback_arc_set#Minimum_feedb...