Rusoto VS hyperscan

A small note on the Rust AWS SDK. It is currently in Developer Preview. However, the project's latest README indicates that it's production-ready, but not production-supported. More of a use-at-your-own-risk type of thing. At this point, I personally would be comfortable shipping with it, but I know that some might prefer something that is marked production-ready. If you want to explore another AWS SDK, rusoto might be for you. However, I imagine the SDK will go GA soon. That's a hunch and NOTHING official. I am not speaking for AWS here.

rusoto

For AWS, there is both the new official SDK preview (https://github.com/awslabs/aws-sdk-rust), and the older unofficial one (https://github.com/rusoto/rusoto). I've not used the new one, but rusoto was excellent when I used it a few years ago.

There's an draft PR to fix this, but it seems to have stalled.

And if the "preview" bit turns you off, Rusoto still works perfectly well and is mature (but in maintenance mode)

And i created an issue on GitHub https://github.com/rusoto/rusoto/issues/1955

Maybe start by having an AWS SDK for Rust that isn't community supported by one or two employees on their off time?

https://github.com/rusoto/rusoto

While not official, there is at least one or two Amazonians working on https://github.com/rusoto/rusoto which is a pretty great Rust AWS SDK.

Maybe some other PCRE-compatible implementation offers streaming. For instance, https://www.intel.com/content/www/us/en/developer/articles/t... says it has this feature, but of course given who it's from it may be tied to a single brand of CPU.

github seems to be https://github.com/intel/hyperscan

I think of WAFs as an extra safety net. Defense in depth.

The author complained about the performance cost of WAFs in general, but not all WAFs have be structured like ModSecurity. They could for example be based on something like https://github.com/intel/hyperscan and perf is at a very different level.

Another example of old timey Unix code just breaking things in minor point releases. See https://abi-laboratory.pro/index.php?view=changelog&l=glibc&... and https://github.com/intel/hyperscan/issues/359.

It’s a bespoke scanning setup designed to deal with GitHub’s scale. Under the hood it’s using Intel’s hyperscan as the regex engine.

https://github.com/intel/hyperscan

Hyperscan is an Intel regular expression library.

We are using hyperscan [3] instead of grepping byte sequences with Python, which is orders of magnitudes faster. It can also handle 4Gb+ files because of this which binwalk cannot.

It's used for a year now in production and it's way more precise and faster than binwalk. We are getting less false-positives too, and even if unblob fails to extract everything, we still get meaningful information out of firmwares, where binwalk just failed with no output previously.

[1]: https://github.com/onekey-sec/unblob/blob/main/unblob/handle...

[2]: https://github.com/onekey-sec/unblob/blob/main/unblob/proces...

[3]: https://github.com/intel/hyperscan