sonic
simdjson
sonic | simdjson | |
---|---|---|
25 | 72 | |
8,322 | 20,795 | |
2.5% | 1.4% | |
8.3 | 8.7 | |
19 days ago | 3 days ago | |
Go | C++ | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
sonic
- Sonic: A blazingly fast Golang JSON serializing and deserializing
-
ByteDance/Sonic: A Lightning-Fast JSON Library for Go
Small (400B, 11 keys, 3 layers)
- How to Visualize and Analyze Data in Open Source Communities
-
Handling high-traffic HTTP requests with JSON payloads
Since most of the time would be spent decoding json, you could try to cut this time using https://github.com/bytedance/sonic or https://github.com/json-iterator/go, both are drop-in replacements for the stdlib, sonic is faster.
- Building a Streaming Platform in Go for Postgres
-
Building a high performance JSON parser
Also worth looking at https://github.com/bytedance/sonic
- Sonic: A fast JSON serializing and deserializing library in Go
- sonic/INTRODUCTION.md at main · bytedance/sonic
-
High-performance JSON parsing in Go
The article inside does not mention this.
-
Toward the Fastest, Compatible JSON Decoder – Sonnet
Good morning.I hope this is not the wrong place to post… so let me introduce my first public Golang package. This is a JSON decoder called Sonnet ( https://github.com/sugawarayuuta/sonnet ) that has given faster results (at least in my environment) than Sonic - https://github.com/bytedance/sonic (which is said that it's the fastest) without the help of assembly!
JSON is a very well-known file format. It is used by everyone who does programming. However, it is also not uncommon to find problems with encoding/json and other third-party libraries. for more… (see https://github.com/sugawarayuuta/sonnet#problems-we-had )
I decided to create a new, standard library-compatible decoder that would be both easy to use and fast.
Thanks for reading, feel free to use, help, or ask questions, I look forward to hearing from you. All benchmarks and other information can be found in the link at the top.
simdjson
- The messy reality of SIMD (vector) functions
-
Make Ubuntu packages 90% faster by rebuilding them
I think parsing once into a faster format (sqlite3 or parquet) would be more beneficial.
https://simdjson.org/
-
User-Space Interrupts (2021)
AVX-512 is on Zen5. Ultra-popular libraries like simdjson can use it. https://github.com/simdjson/simdjson/issues/10
In general my hope is that runtimes pick up the good stuff & roll with it. Io_uring hasn't exactly been a stunning success on nidejs/libuv but the promise is so real that runtimes can take sweet io capabilities like io_uring or usersoace interrupts & boom, now everyone's ok is faster.
-
Wc2: Investigates optimizing 'wc', the Unix word count program
State machines are great for complex situations, but when it comes to performance, it's not at all clear to me that they're the most scalable approach with modern systems.
The data dependency between a loop iteration for each character might be pipelined really well when executed, and we can assume large enough L1/L2 cache for our lookup tables. But we're still using at least one lookup per character.
Projects like https://github.com/simdjson/simdjson?tab=readme-ov-file#abou... are truly fascinating, because they're based on SIMD instructions that can process 64 or more bytes with a single instruction. Very much worth checking out the papers at the link above.
-
Scan HTML faster with SIMD instructions – Chrome edition
Can you point to some of these benchmarks? https://news.ycombinator.com/item?id=26934854 suggests that in at least one synthetic benchmark (with a 7.5KB protobuf message which expands to a 17KB JSON payload), protobuf parsing at 2GB/s would be comparable to JSON parsing at 5GB/s.
Meanwhile, simdjson's numbers (https://github.com/simdjson/simdjson/blob/master/doc/gbps.pn...) show a peak parsing speed of about 3GB/s depending on the workload. Of course, it's not clear you can compare these directly, since they were probably not run on systems with comparable specs. But it's not clear to me that there's a 5x difference.
Perhaps my experience differs because I'm used to seeing very large messages being passed around, but I'd be happy to reconsider. (Or maybe I should go all-in on Cap'n Proto.)
-
SIMD < SIMT < SMT: Parallelism in Nvidia GPUs (2011)
I intentionally said "more towards embarrassingly parallel" rather than "only embarrassingly parallel". I don't think there's a hard cutoff, but there is a qualitative difference. One example that springs to mind is https://github.com/simdjson/simdjson - afaik there's no similarly mature GPU-based JSON parsing.
- The Simdjson Library
-
Tips on adding JSON output to your command line utility. (2021)
It's also supported by simdjson [0] (which has a lot of language bindings [1]):
> Multithreaded processing of gigantic Newline-Delimited JSON (ndjson) and related formats at 3.5 GB/s
[0] https://simdjson.org/
[0] https://github.com/simdjson/simdjson?tab=readme-ov-file#bind...
- 1BRC Merykitty's Magic SWAR: 8 Lines of Code Explained in 3k Words
- Training great LLMs from ground zero in the wilderness as a startup
What are some alternatives?
jsoniter - Using encoding/json to load parts of a large json document
RapidJSON - A fast JSON parser/generator for C++ with both SAX/DOM style API
encoding - Go package containing implementations of efficient encoding, decoding, and validation APIs.
json - JSON for Modern C++
fastjson - Fast JSON parser and validator for Go. No custom structs, no code generation, no reflection
cJSON - Ultralightweight JSON parser in ANSI C