Add extra stuff to a “standard” encoding? Sure, why not

1. protobuf

   1 19 10,245 8.3 JavaScript

   Protocol Buffers for JavaScript & TypeScript.

   

   Seems like protoduf.js has the exact same methods as Google’s implementations with the same names (encode/decode to not prepend with length, encodeDelimited/decodeDelimited for prepending length). It is hard for me to say they’re adding to the standard when they’re just replicating Google’s libraries.

   https://github.com/protobufjs/protobuf.js#toolset

2. InfluxDB

   www.influxdata.com featured

   InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.

   

3. Protobuf

   2 194 67,752 10.0 C++

   Protocol Buffers - Google's data interchange format

   

   I actually went through all projects listed in [1] because I remember this very quirk. It turns out that there are many such libraries that have two variants of encode/decode functions, where the second variant prepends a varint length. In my brief inspection there do exist a few libraries with only the second variant (e.g. Rust quick-protobuf), which is legitimately problematic [2].

   But if the project in question was indeed protobuf.js (see loeg's comments), it clearly distinguishes encode/decode vs. encodeDelimited/decodeDelimited. So I believe the project should not be blamed, and the better question would be why so many people chose to add this exact helper. Well, because Google itself also had the same helper [3]! So at this point protobuf should just standardize this simple framing format (with an explicitly different name though), instead of claiming that protobuf has no obligation to define one.

   [1] https://github.com/protocolbuffers/protobuf/blob/main/docs/t...

   [2] https://github.com/tafia/quick-protobuf/issues/130

   [3] https://protobuf.dev/reference/java/api-docs/com/google/prot...

   [4] https://github.com/protocolbuffers/protobuf/blob/main/src/go...

4. quick-protobuf

   3 3 462 6.3 Rust

   A rust implementation of protobuf parser

   

   I actually went through all projects listed in [1] because I remember this very quirk. It turns out that there are many such libraries that have two variants of encode/decode functions, where the second variant prepends a varint length. In my brief inspection there do exist a few libraries with only the second variant (e.g. Rust quick-protobuf), which is legitimately problematic [2].

   But if the project in question was indeed protobuf.js (see loeg's comments), it clearly distinguishes encode/decode vs. encodeDelimited/decodeDelimited. So I believe the project should not be blamed, and the better question would be why so many people chose to add this exact helper. Well, because Google itself also had the same helper [3]! So at this point protobuf should just standardize this simple framing format (with an explicitly different name though), instead of claiming that protobuf has no obligation to define one.

   [1] https://github.com/protocolbuffers/protobuf/blob/main/docs/t...

   [2] https://github.com/tafia/quick-protobuf/issues/130

   [3] https://protobuf.dev/reference/java/api-docs/com/google/prot...

   [4] https://github.com/protocolbuffers/protobuf/blob/main/src/go...

5. protobuf-conformance

   4 1 26 8.6 JavaScript

   A repository running the Protobuf conformance tests against various libraries

   

   It looks like Protobuf actually has a test suite to ensure compliance with the protocol:

   https://github.com/bufbuild/protobuf-conformance

   Seems like a good idea for protocols in general to have an official test suite, as a way to address this problem

6. riegeli

   5 1 441 9.1 C++

   Riegeli/records is a file format for storing a sequence of string records, typically serialized protocol buffers.

   

   > didn’t find any standard for separating protobuf messages

   The fact that protobufs are not self-delimiting is an endless source of frustration, but I know of 2 standards:

   - SerializeDelimited* is part of the protobuf library: https://github.com/protocolbuffers/protobuf/blob/main/src/go...

   - Riegeli is "a file format for storing a sequence of string records, typically serialized protocol buffers. It supports dense compression, fast decoding, seeking, detection and optional skipping of data corruption, filtering of proto message fields for even faster decoding, and parallel encoding": https://github.com/google/riegeli

7. msgpack

   6 8 7,217 0.0

   MessagePack is an extremely efficient object serialization library. It's like JSON, but very fast and small.

   

8. ntfy

   7 304 23,591 9.1 Go

   Send push notifications to your phone or desktop using PUT/POST

   

   If it was for fun and to learn how, that's fair. But are you aware of https://ntfy.sh?

9. SaaSHub

   www.saashub.com featured

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

Suggest a related project