network VS GJSON

So funny you say this. I think it's the insight of many developers including my own. I hacked together a framework that did this before the existence of GRPC. Now I'm trying to formalise it as a protocol. https://github.com/micro/network/blob/main/PROTOCOL.md

Alright, let me throw my hat into this ring with a totally unfinished idea. I started working on a design for something called the Micro Communication Protocol (MUCP) [1]. It's a header based protocol that's transport agnostic and focuses on service-to-service communication. An early prototype existed in Micro [2] but I'm primarily focused on redesigning the protocol before re-implementing it. Micro was geared towards API first services but I'm looking to expand the scope and try to build a UI layer on top. Most of the protocols focused very much on communication between people but I think if you focus on service-to-service communication more broadly it opens up the avenue to all sorts of multiplayer collaboration.

- [1] https://github.com/micro/network/blob/main/PROTOCOL.md

Yea like you're part of this club that has exclusive access to something, you contribute to it, deliver value, see it grow and then it's gone when you leave. It exists within a silo and for the better part of a decade that's really irked me but I haven't quite figured out how to solve for that problem beyond doing it in a shared open source repo and a shared platform. I think I the issue is it's bigger than any one person and you have to find a way to sell thousands of people on the idea. My starting point was code and now I wonder could I have approached this differently? Is there another path in which this would actually succeed? I'm still trying to figure it out and it's driving me crazy. Next I'll be writing a protocol no joke https://github.com/micro/network

Someone made a benchmark of serialization libraries in go [1], and I was surprised to see gobs is one of the slowest ones, specially for decoding. I suspect part of the reason is that the API doesn't not allow reusing decoders [2]. From my explorations it seems like both JSON [3], message-pack [4] and CBOR [5] are better alternatives.

By the way, in Go there are a like a million JSON encoders because a lot of things in the std library are not really coded for maximum performance but more for easy of usage, it seems. Perhaps this is the right balance for certain things (ex: the http library, see [6]).

There are also a bunch of libraries that allow you to modify a JSON file "in place", without having to fully deserialize into structs (ex: GJSON/SJSON [7] [8]). This sounds very convenient and more efficient that fully de/serializing if we just need to change the data a little.

--

1: https://github.com/alecthomas/go_serialization_benchmarks

2: https://github.com/golang/go/issues/29766#issuecomment-45492...

--

3: https://github.com/goccy/go-json

4: https://github.com/vmihailenco/msgpack

5: https://github.com/fxamacker/cbor

--

6: https://github.com/valyala/fasthttp#faq

--

7: https://github.com/tidwall/gjson

8: https://github.com/tidwall/sjson

```

I don't think there is a way to sort an array, though. However, there is an option to have keys sorted. Personally, I don't think there is much annoyance in that. One could just pipe `jj` output to `sort | uniq -c`.

[0]: https://github.com/tidwall/gjson/blob/master/SYNTAX.md

I’m using GJSON, so far so good!

If the fields you need are just a small subset of the whole json, maybe https://github.com/tidwall/gjson might be of use to read only those (using jsonpath) without needing to create complete corresponding structs.

Thank you for the reminder, it's never too much of it :) Didn't say it, but the code was pprof-iled many times and i can really say it's well optimized. I use own libraries with on-the-fly equations (sums, avgs, emas, stds, ...) wherever possible and also made custom json parser as json messages are in fixed format, so the parser is about 10x faster than gjson. I optimized it to the point that I avoided using maps, and rather iterate via slice where ever possible.

You are right, for learning purposes this fit my needs, but I can imagine an approach similar to this repo: https://github.com/tidwall/gjson

https://github.com/tidwall/gjson is a relatively sensible library if this is something you need to deal with and the structure is actually unknowable.