JSMN VS FlatBuffers

Like how https://github.com/zserge/jsmn works. I thought it would be neat to have such as parser for https://github.com/vshymanskyy/muon

One more JSON implementation using this approach is https://github.com/zserge/jsmn.

Yep! I'm using JSMN (https://github.com/zserge/jsmn), which is a streaming parser that visits each token sequentially, so there's only one copy of each JSON response in memory. I also avoid allocating new intermediate memory whenever possible; for example, to unescape backslashes in the JSON strings, I use a destructive loop that moves the non-backslash characters forward in memory, and truncates the string by moving the null terminator earlier in the string. Not something I'd imagine doing in most environments today, but as you said, it saves a bit of space at the expense of CPU time :)

  void DestructivelyUnescapeStr(LPSTR lpInput) {

After reviewing several options, I’ve settled on plain old JSON. For parsing, I use https://github.com/zserge/jsmn. For serialization I use https://github.com/rdpoor/jems (disclaimer: I wrote the latter, but others use it as well).

As mentioned here, it appears that tiny-json is a parser, not a serializer. If you're looking for parsers, I've been very happy with jsmn.

Very small, 4-5 fields but I'm still going to write in binary because I'm trying to reduce dependencies and https://github.com/zserge/jsmn looks like good fit but jsmn only does parsing which I need for parsing some Oauth json data and config.json file. I will be able to dump the state struct in a state.bin file and read it later for comparing it with system time. Not having to write in text fits well for this particular use case. Benefits: Reduced dependencies and almost cost less decoding of the state struct(which the user will never see).

This is my very straight-forward implementation that came to be from the lack of JSON encoding in jsmn:

I don't work at Cloudflare but follow their work and occasionally work on performance sensitive projects.

If I had to guess, they looked at the landscape a bit like I do and regarded Cap'n Proto, flatbuffers, SBE, etc. as being in one category apart from other data formats like Avro, protobuf, and the like.

So once you're committed to record'ish shaped (rather than columnar like Parquet) data that has an upfront parse time of zero (nominally, there could be marshalling if you transmogrify the field values on read), the list gets pretty short.

https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... goes into some of the trade-offs here.

Cap'n Proto was originally made for https://sandstorm.io/. That work (which Kenton has presumably done at Cloudflare since he's been employed there) eventually turned into Cloudflare workers.

Another consideration: https://github.com/google/flatbuffers/issues/2#issuecomment-...

Much more in the discussion of this particular issue onGitHub: flatbuffers:Reverse engineering #4258

Related but not Rust-specific: FlatBuffers, Cap'n Proto.

If json files are a concern because of space, you can always look into something like protobuffers or flatbuffers. But whatever you use, you should try to find a solution where you don't have to think about the actual serialization/deserialization of your objects, and can just concentrate on the data.

Or, as an alternative to protobuffers, there's also flatbuffers, which is lighter weight and needs less memory: https://google.github.io/flatbuffers/

As the title suggests, I am considering using Flatbuffers as a way to parse sensor data that has been stored in local datafiles. The project language is python.