MessagePack VS ZLib

so appreciate such a detailed reply, thanks. btw, why did you choose tinylib/msgp from 4 available go-impls?

If you find you're running the serial connection at maximum speed and it's still not fast enough, try switching to a more compact binary encoding that has both Serde and Arduino implementations, like MsgPack... though I don't remember enough about its format off the top of my head to tell you the easiest way to put an unambiguous header on each packet/message to make the protocol self-synchronizing.

The information can be stored in a database or as files, serialized in a standard format and with a schema agreed with your Data Engineering team. Depending on your information and requirements, it can be as simple as CSV, XML or JSON, or Big Data formats such as Parquet, Avro, ORC, Arrow, or message serialization formats like Protocol Buffers, FlatBuffers, MessagePack, Thrift, or Cap'n Proto.

MessagePack Similar to JSONs, just more compact, although not as much as the ones above. Still, it's usefull to retain some readability in your messages.

First of all, you're comparing this to GSON and Kryo, how does it compare to Msgpack, fast-serialization, but also Elsa and I'm sure, many others? Are there any limitations and/or trade-offs?

Upvote for msgpack, one of the great undervalued message protocols available.

I have two suggestions Capnproto, MessagePack (those are only the two examples that came to mind first, i bet there are even one or two especially developed for rust). Both of these are better than json in nearly every way.

Encode messages as JSON or MessagePack.

If you've ever inquired about MessagePack before, you probably know the phrase from its official website, msgpack.org: "It's like JSON, but fast and small." In fact, if you compare how much memory space the same data occupies in JSON and MessagePack, you'll see why the latter is a much more compact format. For example, the number 100 takes 3 bytes in JSON and only 1 in MessagePack. The difference becomes more significant as the number's order of magnitude grows. For the maximum value of int64 (9223372036854775807), the size of the stored data differs by as much as 10 bytes (19 against 9)! The same is true for boolean values---4 or 5 bytes in JSON against 1 byte in MessagePack. It is also true for arrays because many syntactic symbols---such as commas separating the elements, semicolons separating the key-value pairs, and brackets marking the array limits---don't exist in binary format. Obviously, the larger the array is, the more syntactic litter accumulates along with the payload. String values, however, are a little more complicated. If your strings don't consist entirely of quotation marks, line feeds, and other special symbols that require escaping, then you won't see a difference between their sizes in JSON and in MessagePack. For example, "foobar" has a length of 8 bytes in JSON and 7 in MessagePack. Note that the above only applies to UTF-8 strings. For binary strings, JSON's disadvantage against MessagePack is obvious.

Site oficial do MessagePack

A minimal viable Deflate decompressor is not exactly complex[1], although slower than mainline zlib.

[1] https://github.com/madler/zlib/tree/master/contrib/puff

ZLibrary not zlib for anyone worried about archiving algorithms. https://zlib.net/ and all the Linux repositories are safe (AFAIK).

>The following is my educated guess at what to do next and I am a week-end warrior programmer at best. If someone with more experience comments, I would listen to them over me< If you have no programming experience, this may be a lost cause. It looks like the save file has a header area followed by a list of "zlib" compressed chunks. zlib.net may have a program that can open the save file and let you see the raw information inside it. Something after the header is garbled and it is preventing Save-Editors from decompressing that chunk OR from parsing what is inside of it. If you are lucky, the garbled bit will jump out at you and you can repair it.

These appears to be the relevant changes:

2022-07-30: https://github.com/madler/zlib/commit/eff308af425b67093bab25...

2022-08-08: https://github.com/madler/zlib/commit/1eb7682f845ac9e9bf9ae3...

The second commit definitely fixed a null pointer dereference, I am not sure if the CVE is referencing something else that was fixed by the first commit.

It is actually possible to estimate the compression level without actually compressing everything again, because different levels use different strategies which can be identified in some cases. zlib in particular has three strategies [1] and preflate [2] leverages this to store the deflate stream reconstruction data without much bits.

[1] https://github.com/madler/zlib/blob/21767c6/deflate.c#L134-L...

[2] https://github.com/deus-libri/preflate/blob/master/preflate_...

It's still a heap of old school C spaghetti https://github.com/madler/zlib/commit/eff308af425b67093bab25...

The non-forked zlib hasn't been accepting optimisations: https://github.com/madler/zlib/issues/346

Hopefully changes will be merged from my obscure private fork into four other obscure private forks (zlib-chromium (https://crbug.com/1354990), zlib-ng, zlib-cloudflare, and the one I personally care about, which doesn't take pull requests, zlib-apple), and possibly incorporated into libdeflate. That's the point of the blog post - to help maintainers understand the changes.

This seems to be a minor bug, in that it is only triggered using unusual (and rather unlikely) deflate parameters.

But, looking into this bug, I was sort-of interested to see how it was handled. The change log for 1.2.12 (2022-03-27) indicates the issue was resolved ("Fix a bug that can crash deflate on some input when using Z_FIXED"). Yet, in what seems to be the canonical Zlib repository (https://github.com/madler/zlib), I was unable to find a corresponding commit.

None of commits this year (7, so not too hard to review) seem to be particularity meaningful changes, and in particular yesterday's 'zlib 1.2.12' commit seems to only consist of version/copyright updates.

So, does anyone have any idea where to find the commits related to the change log entry? (Note that I'm not disputing the issue is actually fixed, I'm just trying to improve my Github reading skills...)