lizard VS MessagePack

Another fun trick is to compress less often used stuff in RAM memory, because decompressing something like LZ4 or Lizard is still potentially orders of magnitude faster than reading from disk.

Lizard - Formerly LZ5; an efficient compressor with fast decompression. Achieves compression ratios comparable with zip and zlib at decompression speeds of 1000MB/s and faster. BSD-2-Clause

so appreciate such a detailed reply, thanks. btw, why did you choose tinylib/msgp from 4 available go-impls?

If you find you're running the serial connection at maximum speed and it's still not fast enough, try switching to a more compact binary encoding that has both Serde and Arduino implementations, like MsgPack... though I don't remember enough about its format off the top of my head to tell you the easiest way to put an unambiguous header on each packet/message to make the protocol self-synchronizing.

The information can be stored in a database or as files, serialized in a standard format and with a schema agreed with your Data Engineering team. Depending on your information and requirements, it can be as simple as CSV, XML or JSON, or Big Data formats such as Parquet, Avro, ORC, Arrow, or message serialization formats like Protocol Buffers, FlatBuffers, MessagePack, Thrift, or Cap'n Proto.

MessagePack Similar to JSONs, just more compact, although not as much as the ones above. Still, it's usefull to retain some readability in your messages.

I'll try that msgpack.org website.

First of all, you're comparing this to GSON and Kryo, how does it compare to Msgpack, fast-serialization, but also Elsa and I'm sure, many others? Are there any limitations and/or trade-offs?

Upvote for msgpack, one of the great undervalued message protocols available.

I have two suggestions Capnproto, MessagePack (those are only the two examples that came to mind first, i bet there are even one or two especially developed for rust). Both of these are better than json in nearly every way.

Encode messages as JSON or MessagePack.