rebar
encoding
rebar | encoding | |
---|---|---|
23 | 8 | |
197 | 964 | |
- | 0.7% | |
8.5 | 3.6 | |
about 2 months ago | 5 months ago | |
Python | Go | |
The Unlicense | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
rebar
-
Needle: A DFA Based Regex Library That Compiles to JVM ByteCode
The set of regex engines being compared here is pretty small, and even among backtracking regex engines, Java's is pretty slow. See: https://github.com/BurntSushi/rebar?tab=readme-ov-file#summa...
The backtracking engines ahead of are pcre2/jit, javascript/v8, d/ldc/std-regex (technically a hybrid I believe) and regress. Java's engine is about on par with Python's and Perl's (which are both written in C).
-
Knuth–Morris–Pratt Illustrated
https://github.com/BurntSushi/rebar
For regex, you can't really distill it down to one single fastest algorithm.
It's somewhat similar even for substring search. But certainly, the fastest algorithms are going to be the ones that make use of SIMD in some way.
-
Regex character "$" doesn't mean "end-of-string"
I'll add two notes to this:
* Finite automata based regex engines don't necessarily have to be slower than backtracking engines like PCRE. Go's regexp is in practice slower in a lot of cases, but this is more a property of its implementation than its concept. See: https://github.com/BurntSushi/rebar?tab=readme-ov-file#summa... --- Given "sufficient" implementation effort, backtrackers and finite automata engines can both perform very well, with one beating the other in some cases but not in others. It depends.
* Fun fact is that if you're iterating over all matches in a haystack (e.g., Go's `FindAll` routines), then you're susceptible to O(m * n^2) search time. This applies to all regex engines that implement some kind of leftmost match priority. See https://github.com/BurntSushi/rebar?tab=readme-ov-file#quadr... for a more detailed elaboration on this point.
-
Re2c
They are extremely fast too: https://github.com/BurntSushi/rebar?tab=readme-ov-file#summa...
-
C# Regex engine is now 3rd fastest in the world
I love the flourish of "in the world." I had never thought about it that way. Which makes me think if there are any regex engines that aren't in rebar that could conceivably by competitive with the top engines in rebar. I do maintained a WANTED list of engines[1], but none of them jump out to me except for maybe Nim's engine.
Of course, there's also the question of whether the benchmarks are representative enough to make such extrapolations. I don't have a good answer for that one. All models are wrong, but, some are useful.
[1]: https://github.com/BurntSushi/rebar/blob/96c6779b7e1cdd850b8...
-
Ugrep – a more powerful, ultra fast, user-friendly, compatible grep
I'm the author of ripgrep and its regex engine.
Your claim is true to a first approximation. But greps are line oriented, and that means there are optimizations that can be done that are hard to do in a general regex library.
If you read my commentary in the ripgrep discussion above, you'll note that it isn't just about the benchmarks themselves being accurate, but the model they represent. Nevertheless, I linked the hypergrep benchmarks not because of Hyperscan, but because they were done by someone who isn't the author of either ripgrep or ugrep.
As for regex benchmarks, you'll want to check out rebar: https://github.com/BurntSushi/rebar
You can see my full thoughts around benchmark design and philosophy if you read the rebar documentation. Be warned though, you'll need some time.
There is a fork of ripgrep with Hyperscan support: https://sr.ht/~pierrenn/ripgrep/
-
Translations of Russ Cox's Thompson NFA C Program to Rust
Before getting to your actual question, it might help to look at a regex benchmark that compares engines (perhaps JITs are not the fastest in all cases!): https://github.com/BurntSushi/rebar
In particular, the `regex-lite` engine is strictly just the PikeVM without any frills. No prefilters or literal optimizations. No other engines. Just the PikeVM.
As to your question, the PikeVM is, essentially, an NFA simulation. The PikeVM just refers to the layering of capture state on top of the NFA simulation. But you can peel back the capture state and you're still left with a slow NFA simulation. I mention this because you seem to compare the PikeVM with "big graph structures with NFAs/DFAs." But the PikeVM is using a big NFA graph structure.
At a very high level, the time complexity of a Thompson NFA simulation and a DFA hints strongly at the answer to your question: searching with a Thompson NFA has worst case O(m*n) time while a DFA has worst case O(n) time, where m is proportional to the size of the regex and n is proportional to the size of the haystack. That is, for each character of the haystack, the Thompson NFA is potentially doing up to `m` amount of work. And indeed, in practice, it really does need to do some work for each character.
A Thompson NFA simulation needs to keep track of every state it is simultaneously in at any given point. And in order to compute the transition function, you need to compute it for every state you're in. The epsilon transitions that are added as part of the Thompson NFA construction (and are, crucially, what make building a Thompson NFA so fast) exacerbate this. So what happens is that you wind up chasing epsilon transitions over and over for each character.
A DFA pre-computes these epsilon closures during powerset construction. Of course, that takes worst case O(2^m) time, which is why real DFAs aren't really used in general purpose engines. Instead, lazy DFAs are used.
As for things like V8, they are backtrackers. They don't need to keep track of every state they're simultaneously in because they don't mind taking a very long time to complete some searches. But in practice, this can make them much faster for some inputs.
Feel free to ask more questions. I'll stop here.
-
Compile time regular expression in C++
I'd love for someone to add this to rebar[1] so that we can get a good sense of how well it does against other general purpose regex engines. It will be a little tricky to add (since the build step will require emitting a C++ program and compiling it), but it should be possible.
[1]: https://github.com/BurntSushi/rebar
- Stringzilla: Fastest string sort, search, split, and shuffle using SIMD
-
Rust vs. Go in 2023
https://github.com/BurntSushi/rebar#summary-of-search-time-b...
Further, Go refusing to have macros means that many libraries use reflection instead, which often makes those parts of the Go program perform no better than Python and in some cases worse. Rust can just generate all of that at compile time with macros, and optimize them with LLVM like any other code. Some Go libraries go to enormous lengths to reduce reflection overhead, but that's hard to justify for most things, and hard to maintain even once done. The legendary https://github.com/segmentio/encoding seems to be abandoned now and progress on Go JSON in general seems to have died with https://github.com/go-json-experiment/json .
Many people claiming their projects are IO-bound are just assuming that's the case because most of the time is spent in their input reader. If they actually measured they'd see it's not even saturating a 100Mbps link, let alone 1-100Gbps, so by definition it is not IO-bound. Even if they didn't need more throughput than that, they still could have put those cycles to better use or at worst saved energy. Isn't that what people like to say about Go vs Python, that Go saves energy? Sure, but it still burns a lot more energy than it would if it had macros.
Rust can use state-of-the-art memory allocators like mimalloc, while Go is still stuck on an old fork of tcmalloc, and not just tcmalloc in its original C, but transpiled to Go so it optimizes much less than LLVM would optimize it. (Many people benchmarking them forget to even try substitute allocators in Rust, so they're actually underestimating just how much faster Rust is)
Finally, even Go Generics have failed to improve performance, and in many cases can make it unimaginably worse through -- I kid you not -- global lock contention hidden behind innocent type assertion syntax: https://planetscale.com/blog/generics-can-make-your-go-code-...
It's not even close. There are many reasons Go is a lot slower than Rust and many of them are likely to remain forever. Most of them have not seen meaningful progress in a decade or more. The GC has improved, which is great, but that's not even a factor on the Rust side.
encoding
- Handling high-traffic HTTP requests with JSON payloads
-
Rust vs. Go in 2023
https://github.com/BurntSushi/rebar#summary-of-search-time-b...
Further, Go refusing to have macros means that many libraries use reflection instead, which often makes those parts of the Go program perform no better than Python and in some cases worse. Rust can just generate all of that at compile time with macros, and optimize them with LLVM like any other code. Some Go libraries go to enormous lengths to reduce reflection overhead, but that's hard to justify for most things, and hard to maintain even once done. The legendary https://github.com/segmentio/encoding seems to be abandoned now and progress on Go JSON in general seems to have died with https://github.com/go-json-experiment/json .
Many people claiming their projects are IO-bound are just assuming that's the case because most of the time is spent in their input reader. If they actually measured they'd see it's not even saturating a 100Mbps link, let alone 1-100Gbps, so by definition it is not IO-bound. Even if they didn't need more throughput than that, they still could have put those cycles to better use or at worst saved energy. Isn't that what people like to say about Go vs Python, that Go saves energy? Sure, but it still burns a lot more energy than it would if it had macros.
Rust can use state-of-the-art memory allocators like mimalloc, while Go is still stuck on an old fork of tcmalloc, and not just tcmalloc in its original C, but transpiled to Go so it optimizes much less than LLVM would optimize it. (Many people benchmarking them forget to even try substitute allocators in Rust, so they're actually underestimating just how much faster Rust is)
Finally, even Go Generics have failed to improve performance, and in many cases can make it unimaginably worse through -- I kid you not -- global lock contention hidden behind innocent type assertion syntax: https://planetscale.com/blog/generics-can-make-your-go-code-...
It's not even close. There are many reasons Go is a lot slower than Rust and many of them are likely to remain forever. Most of them have not seen meaningful progress in a decade or more. The GC has improved, which is great, but that's not even a factor on the Rust side.
-
Quickly checking that a string belongs to a small set
We took a similar approach in our JSON decoder. We needed to support sets (JSON object keys) that aren't necessarily known until runtime, and strings that are up to 16 bytes in length.
We got better performance with a linear scan and SIMD matching than with a hash table or a perfect hashing scheme.
See https://github.com/segmentio/asm/pull/57 (AMD64) and https://github.com/segmentio/asm/pull/65 (ARM64). Here's how it's used in the JSON decoder: https://github.com/segmentio/encoding/pull/101
-
80x improvements in caching by moving from JSON to gob
Binary formats work well for some cases but JSON is often unavoidable since it is so widely used for APIs. However, you can make it faster in golang with this https://github.com/segmentio/encoding.
-
Speeding up Go's builtin JSON encoder up to 55% for large arrays of objects
Would love to see results from incorporating https://github.com/segmentio/encoding/tree/master/json!
-
Fastest JSON parser for large (~888kB) API response?
Try this one out https://github.com/segmentio/encoding it's always worked well for me
-
📖 Go Fiber by Examples: Delving into built-in functions
Converts any interface or string to JSON using the segmentio/encoding package. Also, the JSON method sets the content header to application/json.
-
In-memory caching solutions
If you're interested in super fast & easy JSON for that cache give this a try I've used it in prod & never had a problem.
What are some alternatives?
Rebar3 - Erlang build tool that makes it easy to compile and test Erlang applications and releases.
sonic - A blazingly fast JSON serializing & deserializing library
cl-ppcre - Common Lisp regular expression library
groupcache - Clone of golang/groupcache with TTL and Item Removal support
hypergrep - Recursively search directories for a regex pattern
parquet-go - Go library to read/write Parquet files
StringZilla - Up to 10x faster strings for C, C++, Python, Rust, and Swift, leveraging SWAR and SIMD on Arm Neon and x86 AVX2 & AVX-512-capable chips to accelerate search, sort, edit distances, alignment scores, etc 🦖
base64 - Faster base64 encoding for Go
moar - Moar is a pager. It's designed to just do the right thing without any configuration.
buntdb - BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support
bat - A cat(1) clone with wings.
hilbert - Go package for mapping values to and from space-filling curves, such as Hilbert and Peano curves.