MessagePack VS Cap'n Proto

so appreciate such a detailed reply, thanks. btw, why did you choose tinylib/msgp from 4 available go-impls?

If you find you're running the serial connection at maximum speed and it's still not fast enough, try switching to a more compact binary encoding that has both Serde and Arduino implementations, like MsgPack... though I don't remember enough about its format off the top of my head to tell you the easiest way to put an unambiguous header on each packet/message to make the protocol self-synchronizing.

The information can be stored in a database or as files, serialized in a standard format and with a schema agreed with your Data Engineering team. Depending on your information and requirements, it can be as simple as CSV, XML or JSON, or Big Data formats such as Parquet, Avro, ORC, Arrow, or message serialization formats like Protocol Buffers, FlatBuffers, MessagePack, Thrift, or Cap'n Proto.

MessagePack Similar to JSONs, just more compact, although not as much as the ones above. Still, it's usefull to retain some readability in your messages.

First of all, you're comparing this to GSON and Kryo, how does it compare to Msgpack, fast-serialization, but also Elsa and I'm sure, many others? Are there any limitations and/or trade-offs?

Upvote for msgpack, one of the great undervalued message protocols available.

I have two suggestions Capnproto, MessagePack (those are only the two examples that came to mind first, i bet there are even one or two especially developed for rust). Both of these are better than json in nearly every way.

Encode messages as JSON or MessagePack.

If you've ever inquired about MessagePack before, you probably know the phrase from its official website, msgpack.org: "It's like JSON, but fast and small." In fact, if you compare how much memory space the same data occupies in JSON and MessagePack, you'll see why the latter is a much more compact format. For example, the number 100 takes 3 bytes in JSON and only 1 in MessagePack. The difference becomes more significant as the number's order of magnitude grows. For the maximum value of int64 (9223372036854775807), the size of the stored data differs by as much as 10 bytes (19 against 9)! The same is true for boolean values---4 or 5 bytes in JSON against 1 byte in MessagePack. It is also true for arrays because many syntactic symbols---such as commas separating the elements, semicolons separating the key-value pairs, and brackets marking the array limits---don't exist in binary format. Obviously, the larger the array is, the more syntactic litter accumulates along with the payload. String values, however, are a little more complicated. If your strings don't consist entirely of quotation marks, line feeds, and other special symbols that require escaping, then you won't see a difference between their sizes in JSON and in MessagePack. For example, "foobar" has a length of 8 bytes in JSON and 7 in MessagePack. Note that the above only applies to UTF-8 strings. For binary strings, JSON's disadvantage against MessagePack is obvious.

Site oficial do MessagePack

capnproto

Are there any synergies with capnproto [1] or is the focus here purely on huge payloads?

I'm just an interested hobbyist when it comes to performant RPC frameworks but had some fun benchmarking capnproto for a small gamedev-related project and it was pretty awesome.

[1] https://capnproto.org/

While Go support isn't on par with the more popular gRPC ecosystem, I highly recommend looking at https://capnproto.org/ as a better alternative!

Similarly does anyone use https://capnproto.org/? It's a project I was really interested in a few years back, but I haven't heard much in the way of it lately.

I'd have a look at Cap'n'proto https://capnproto.org/

> ship working memory without costly XDR

Yes, you can have zero serialization latency: https://capnproto.org/

Capnproto by Kenton Varda <https://capnproto.org>.

There have certainly been other contributors <https://github.com/capnproto/capnproto/blob/master/CONTRIBUT...>, but Kenton does most of the development and maintenance on his own.

There are few things I lament more than having to spin up on a new serialization / file format and it isn't using Capnproto or Protobufs or some similar invention. Capnproto has truly spoiled me in terms of how consistent and feature rich it is for what seems like a "simple" task.

The most popular framework today in this space is gRPC by Google, but there are also Cap'n'Proto, tRPC, JSON-RPC (less of a framework, rather a convention) and others. And then there used to be SOAP for a long time. It's considered outdated and deprectated today. But in essence, it was doing the exact same thing - just in a slightly different way. Of course, gRPC has a few advantages over SOAP, especially more efficient message representation, thanks to binary encoding, and communication, thanks to HTTP/2 – on the other hand, though, SOAP has discoverability, thanks to to WSDL. The point I'm trying to make is that the fundamental concepts are almost exactly the same.