sonic VS simdjson

Small (400B, 11 keys, 3 layers)

Since most of the time would be spent decoding json, you could try to cut this time using https://github.com/bytedance/sonic or https://github.com/json-iterator/go, both are drop-in replacements for the stdlib, sonic is faster.

Also worth looking at https://github.com/bytedance/sonic

The article inside does not mention this.

Good morning.I hope this is not the wrong place to post… so let me introduce my first public Golang package. This is a JSON decoder called Sonnet ( https://github.com/sugawarayuuta/sonnet ) that has given faster results (at least in my environment) than Sonic - https://github.com/bytedance/sonic (which is said that it's the fastest) without the help of assembly!

JSON is a very well-known file format. It is used by everyone who does programming. However, it is also not uncommon to find problems with encoding/json and other third-party libraries. for more… (see https://github.com/sugawarayuuta/sonnet#problems-we-had )

I decided to create a new, standard library-compatible decoder that would be both easy to use and fast.

Thanks for reading, feel free to use, help, or ask questions, I look forward to hearing from you. All benchmarks and other information can be found in the link at the top.

I think parsing once into a faster format (sqlite3 or parquet) would be more beneficial.

https://simdjson.org/

AVX-512 is on Zen5. Ultra-popular libraries like simdjson can use it. https://github.com/simdjson/simdjson/issues/10

In general my hope is that runtimes pick up the good stuff & roll with it. Io_uring hasn't exactly been a stunning success on nidejs/libuv but the promise is so real that runtimes can take sweet io capabilities like io_uring or usersoace interrupts & boom, now everyone's ok is faster.

State machines are great for complex situations, but when it comes to performance, it's not at all clear to me that they're the most scalable approach with modern systems.

The data dependency between a loop iteration for each character might be pipelined really well when executed, and we can assume large enough L1/L2 cache for our lookup tables. But we're still using at least one lookup per character.

Projects like https://github.com/simdjson/simdjson?tab=readme-ov-file#abou... are truly fascinating, because they're based on SIMD instructions that can process 64 or more bytes with a single instruction. Very much worth checking out the papers at the link above.

Can you point to some of these benchmarks? https://news.ycombinator.com/item?id=26934854 suggests that in at least one synthetic benchmark (with a 7.5KB protobuf message which expands to a 17KB JSON payload), protobuf parsing at 2GB/s would be comparable to JSON parsing at 5GB/s.

Meanwhile, simdjson's numbers (https://github.com/simdjson/simdjson/blob/master/doc/gbps.pn...) show a peak parsing speed of about 3GB/s depending on the workload. Of course, it's not clear you can compare these directly, since they were probably not run on systems with comparable specs. But it's not clear to me that there's a 5x difference.

Perhaps my experience differs because I'm used to seeing very large messages being passed around, but I'd be happy to reconsider. (Or maybe I should go all-in on Cap'n Proto.)

I intentionally said "more towards embarrassingly parallel" rather than "only embarrassingly parallel". I don't think there's a hard cutoff, but there is a qualitative difference. One example that springs to mind is https://github.com/simdjson/simdjson - afaik there's no similarly mature GPU-based JSON parsing.

It's also supported by simdjson [0] (which has a lot of language bindings [1]):

> Multithreaded processing of gigantic Newline-Delimited JSON (ndjson) and related formats at 3.5 GB/s

[0] https://simdjson.org/

[0] https://github.com/simdjson/simdjson?tab=readme-ov-file#bind...