MessagePack VS Protobuf

so appreciate such a detailed reply, thanks. btw, why did you choose tinylib/msgp from 4 available go-impls?

If you find you're running the serial connection at maximum speed and it's still not fast enough, try switching to a more compact binary encoding that has both Serde and Arduino implementations, like MsgPack... though I don't remember enough about its format off the top of my head to tell you the easiest way to put an unambiguous header on each packet/message to make the protocol self-synchronizing.

The information can be stored in a database or as files, serialized in a standard format and with a schema agreed with your Data Engineering team. Depending on your information and requirements, it can be as simple as CSV, XML or JSON, or Big Data formats such as Parquet, Avro, ORC, Arrow, or message serialization formats like Protocol Buffers, FlatBuffers, MessagePack, Thrift, or Cap'n Proto.

MessagePack Similar to JSONs, just more compact, although not as much as the ones above. Still, it's usefull to retain some readability in your messages.

First of all, you're comparing this to GSON and Kryo, how does it compare to Msgpack, fast-serialization, but also Elsa and I'm sure, many others? Are there any limitations and/or trade-offs?

Upvote for msgpack, one of the great undervalued message protocols available.

I have two suggestions Capnproto, MessagePack (those are only the two examples that came to mind first, i bet there are even one or two especially developed for rust). Both of these are better than json in nearly every way.

Encode messages as JSON or MessagePack.

If you've ever inquired about MessagePack before, you probably know the phrase from its official website, msgpack.org: "It's like JSON, but fast and small." In fact, if you compare how much memory space the same data occupies in JSON and MessagePack, you'll see why the latter is a much more compact format. For example, the number 100 takes 3 bytes in JSON and only 1 in MessagePack. The difference becomes more significant as the number's order of magnitude grows. For the maximum value of int64 (9223372036854775807), the size of the stored data differs by as much as 10 bytes (19 against 9)! The same is true for boolean values---4 or 5 bytes in JSON against 1 byte in MessagePack. It is also true for arrays because many syntactic symbols---such as commas separating the elements, semicolons separating the key-value pairs, and brackets marking the array limits---don't exist in binary format. Obviously, the larger the array is, the more syntactic litter accumulates along with the payload. String values, however, are a little more complicated. If your strings don't consist entirely of quotation marks, line feeds, and other special symbols that require escaping, then you won't see a difference between their sizes in JSON and in MessagePack. For example, "foobar" has a length of 8 bytes in JSON and 7 in MessagePack. Note that the above only applies to UTF-8 strings. For binary strings, JSON's disadvantage against MessagePack is obvious.

Site oficial do MessagePack

Check out protocol buffers https://protobuf.dev/overview/#benefits https://protobuf.dev/ https://github.com/protocolbuffers/protobuf

#!/bin/sh echo "Installing Rust and Wasm toolchain..." curl https://sh.rustup.rs -sSf | sh -s -- -y rustup default stable rustup update rustup update nightly rustup target add wasm32-unknown-unknown --toolchain nightly echo "Installing protoc..." sudo apt update sudo apt install -y unzip PROTOC_VERSION=$(curl -s "https://api.github.com/repos/protocolbuffers/protobuf/releases/latest" | grep -Po '"tag_name": "v\K[0-9.]+') curl -Lo protoc.zip "https://github.com/protocolbuffers/protobuf/releases/latest/download/protoc-${PROTOC_VERSION}-linux-x86_64.zip" sudo unzip -q protoc.zip bin/protoc -d /usr/local sudo chmod a+x /usr/local/bin/protoc rm -rf protoc.zip echo "Building Substrate Node Template..." cargo build --release --locked echo "Installing cargo-contract CLI tool..." cargo install --force --locked cargo-contract --version 2.0.0-rc

gRPC uses protocol buffers (it is an open source message format) as the default method of communication between client and server. Also, gRPC uses HTTP/ 2 as the default protocol.

Switch back to your own user. You have to install protobuf, "Protocol Buffers - Google's data interchange format":

Install the Protobuf Compiler (protoc) on your system. On MacOS, you could use Homebrew, as described here.

Protocol Buffers (protobuf) is an open-source data format used to serialize structured data in binary form. It is used for transmitting data over networks or storing it in files. Protobuf allows you to send data in small packet sizes and at a faster rate than other messaging formats. USP specifies Protocol Buffers Version 3 as a mechanism to serialize data to be sent over a Message Transfer Protocol, such as MQTT.

This implementation is not meant to be bug-free and production-ready and is just a glimpse into what the proposed solution might look like. Additionally, we assume the producer application is already in place and produces events. The consumer application receives events as they are produced, process them, commits them on Kafka, and sends a new event containing the results to a separate topic to inform the producer of the result of the process. We also use Protocol Buffers to serialize our communication.

Download it from https://github.com/protocolbuffers/protobuf/releases/tag/v3.20.3 and add the directory where the protoc file is located under systempath.