hyperscan VS RE2

Maybe some other PCRE-compatible implementation offers streaming. For instance, https://www.intel.com/content/www/us/en/developer/articles/t... says it has this feature, but of course given who it's from it may be tied to a single brand of CPU.

github seems to be https://github.com/intel/hyperscan

I think of WAFs as an extra safety net. Defense in depth.

The author complained about the performance cost of WAFs in general, but not all WAFs have be structured like ModSecurity. They could for example be based on something like https://github.com/intel/hyperscan and perf is at a very different level.

Another example of old timey Unix code just breaking things in minor point releases. See https://abi-laboratory.pro/index.php?view=changelog&l=glibc&... and https://github.com/intel/hyperscan/issues/359.

It’s a bespoke scanning setup designed to deal with GitHub’s scale. Under the hood it’s using Intel’s hyperscan as the regex engine.

https://github.com/intel/hyperscan

Hyperscan is an Intel regular expression library.

We are using hyperscan [3] instead of grepping byte sequences with Python, which is orders of magnitudes faster. It can also handle 4Gb+ files because of this which binwalk cannot.

It's used for a year now in production and it's way more precise and faster than binwalk. We are getting less false-positives too, and even if unblob fails to extract everything, we still get meaningful information out of firmwares, where binwalk just failed with no output previously.

[1]: https://github.com/onekey-sec/unblob/blob/main/unblob/handle...

[2]: https://github.com/onekey-sec/unblob/blob/main/unblob/proces...

[3]: https://github.com/intel/hyperscan

Looking at the benchmark where C++ is worst compared to other languages, it's depending on the library used. I would guess if they used Google's re2 Regex library instead of Boost's, the result would be different.

https://github.com/google/re2

https://github.com/greensoftwarelab/Energy-Languages/blob/ma...

That page says it just includes "some of the most common special characters", and following the link to the Examples page in turn includes a link to the full list.

Python uses backtracking, so this probably isn't O(n), especially with the ability to choose the dictionary.

But with there are non-backtracking matchers which would make this O(n). Here's re2 from https://github.com/google/re2 :

  >>> import re2

RE2 is a Google regular expression library

google's regex library: https://github.com/google/re2

I am currently working with Regex, specifically Re2, and was wondering if there is a real difference between the above expressions for repeated sub-regex.

I ported Google RE2, a regular expression library, to SPVM as Resource::Re2, and created SPVM::Regex, a wrapper for it.

I searched for I found that there is a Perl compatible regular expression called Google RE2. It is written in C++, and with Google RE2, I can use Perl-compatible regular expressions as a library.

Yes, but there is an interesting clarification here. RE2 has used the "caching" approach documented in the Ruby bug ticket linked for quite some time (since its birth?): https://github.com/google/re2/blob/954656f47fe8fb505d4818da1...

It is mentioned only briefly in Cox's article on regex matching in the wild. Look for the word "bitstate": https://swtch.com/~rsc/regexp/regexp3.html

I didn't know Perl had implemented this trick too.

The paper[1] cited in the Ruby bug ticket was published very recently. When I first read the Ruby bug ticket, I immediately wondered how they sidestepped the memory use problem. The paper's abstract seems to suggest there is some technique for doing so, as it rebuffs the idea of doing "full" memoization. Alas, I do not have access the paper. (Which is fucking ridiculous.)

[1]: https://ieeexplore.ieee.org/abstract/document/9519427