marshmallow VS GJSON

Also, JSON marshaling and unmarshaling may also be an issue. C/C++ are quite varied - I suppose they either use the fields of the JSON as is, or use some ugly-ass macros (u/xkcd-Hyphen-bot, do your thing. It actually fits here), but Go entries would probably used the standard way to do this in Go - the encoding/json package - which needs to read and parse struct field tags at runtime. Does it at least cache the parsed definitions? From the complaints I see about it, I doubt it. Also, there seems to be Marshmallow that can do caching (among other optimizations) and greatly outperforms the build in one. I guess that means the standard library implementation doesn't do it? That would explain why the TechEmpower benchmark entries are so slow - I don't think they use Marshmallow. Marshmallow has less than 300 starts on GitHub, which is far less than what you would expect it to have if it was commonly used.

I believe this was one of the reasons for marshmallow being written: https://github.com/PerimeterX/marshmallow The idea to partially unmarshal, inspect the type field, and then make a second pass.

Try out https://github.com/PerimeterX/marshmallow

I think this is what marshmallow was made for: https://github.com/PerimeterX/marshmallow

Marshmallow is used internally at PerimeterX for some time, and we've recently decided to open-source it and share a blog post about how it helped us trim 70% of our JSON parsing costs in production.

Someone made a benchmark of serialization libraries in go [1], and I was surprised to see gobs is one of the slowest ones, specially for decoding. I suspect part of the reason is that the API doesn't not allow reusing decoders [2]. From my explorations it seems like both JSON [3], message-pack [4] and CBOR [5] are better alternatives.

By the way, in Go there are a like a million JSON encoders because a lot of things in the std library are not really coded for maximum performance but more for easy of usage, it seems. Perhaps this is the right balance for certain things (ex: the http library, see [6]).

There are also a bunch of libraries that allow you to modify a JSON file "in place", without having to fully deserialize into structs (ex: GJSON/SJSON [7] [8]). This sounds very convenient and more efficient that fully de/serializing if we just need to change the data a little.

--

1: https://github.com/alecthomas/go_serialization_benchmarks

2: https://github.com/golang/go/issues/29766#issuecomment-45492...

--

3: https://github.com/goccy/go-json

4: https://github.com/vmihailenco/msgpack

5: https://github.com/fxamacker/cbor

--

6: https://github.com/valyala/fasthttp#faq

--

7: https://github.com/tidwall/gjson

8: https://github.com/tidwall/sjson

```

I don't think there is a way to sort an array, though. However, there is an option to have keys sorted. Personally, I don't think there is much annoyance in that. One could just pipe `jj` output to `sort | uniq -c`.

[0]: https://github.com/tidwall/gjson/blob/master/SYNTAX.md

I’m using GJSON, so far so good!

If the fields you need are just a small subset of the whole json, maybe https://github.com/tidwall/gjson might be of use to read only those (using jsonpath) without needing to create complete corresponding structs.

Thank you for the reminder, it's never too much of it :) Didn't say it, but the code was pprof-iled many times and i can really say it's well optimized. I use own libraries with on-the-fly equations (sums, avgs, emas, stds, ...) wherever possible and also made custom json parser as json messages are in fixed format, so the parser is about 10x faster than gjson. I optimized it to the point that I avoided using maps, and rather iterate via slice where ever possible.

You are right, for learning purposes this fit my needs, but I can imagine an approach similar to this repo: https://github.com/tidwall/gjson

https://github.com/tidwall/gjson is a relatively sensible library if this is something you need to deal with and the structure is actually unknowable.