unblob
hyperscan
Our great sponsors
| unblob | hyperscan | |
|---|---|---|
| 16 | 25 | |
| 2,054 | 4,629 | |
| 2.5% | 1.0% | |
| 9.5 | 2.0 | |
| about 13 hours ago | 5 months ago | |
| Python | C++ | |
| GNU General Public License v3.0 or later | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
unblob
- Reverse-engineering an encrypted IoT protocol
-
How are zlib, gzip and zip related?
If you are interested in implementation details, how to unpack/decompress them, check out these Python implementations:
- https://github.com/onekey-sec/unblob/blob/main/unblob/handle...
- https://github.com/onekey-sec/unblob/blob/main/unblob/handle...
- https://github.com/onekey-sec/unblob/blob/main/unblob/handle...
- GNU poke: The extensible editor for structured binary data
- unblob: Extract files from any kind of container formats
-
FLiP Stack Weekly for 21 Jan 2023
Extract all the things https://unblob.org/
-
Show HN: Unblob – extraction suite for 30+ file formats
and the initial work they already merged: https://github.com/onekey-sec/unblob/pull/475
- Show HN: Unblob – accurate, fast, and easy-to-use extraction suite
hyperscan
-
Ask HN: Regex on a File or Stream
Maybe some other PCRE-compatible implementation offers streaming. For instance, https://www.intel.com/content/www/us/en/developer/articles/t... says it has this feature, but of course given who it's from it may be tied to a single brand of CPU.
github seems to be https://github.com/intel/hyperscan
- Aho-Corasick Algorithm
-
Stop deploying web application firewalls
I think of WAFs as an extra safety net. Defense in depth.
The author complained about the performance cost of WAFs in general, but not all WAFs have be structured like ModSecurity. They could for example be based on something like https://github.com/intel/hyperscan and perf is at a very different level.
-
Be careful of the examples you use. They stick
Another example of old timey Unix code just breaking things in minor point releases. See https://abi-laboratory.pro/index.php?view=changelog&l=glibc&... and https://github.com/intel/hyperscan/issues/359.
- hypergrep: A new "fastest grep" to search directories recursively for a regex pattern
- Accelerating Regular Expressions with AVX-512 at 1.5 GB/s/core
-
GitHub push protection is free for all public repositories
It’s a bespoke scanning setup designed to deal with GitHub’s scale. Under the hood it’s using Intel’s hyperscan as the regex engine.
https://github.com/intel/hyperscan
-
RE2 VS hyperscan - a user suggested alternative
2 projects | 17 Mar 2023
Hyperscan is an Intel regular expression library.
-
hyperscan VS RE2 - a user suggested alternative
2 projects | 17 Mar 2023
-
Show HN: Unblob – extraction suite for 30+ file formats
We are using hyperscan [3] instead of grepping byte sequences with Python, which is orders of magnitudes faster. It can also handle 4Gb+ files because of this which binwalk cannot.
It's used for a year now in production and it's way more precise and faster than binwalk. We are getting less false-positives too, and even if unblob fails to extract everything, we still get meaningful information out of firmwares, where binwalk just failed with no output previously.
[1]: https://github.com/onekey-sec/unblob/blob/main/unblob/handle...
[2]: https://github.com/onekey-sec/unblob/blob/main/unblob/proces...
[3]: https://github.com/intel/hyperscan
What are some alternatives?
binwalk - Firmware Analysis Tool
regex-benchmark - It's just a simple regex benchmark of different programming languages.
EMBA - EMBA - The firmware security analyzer
go - The Go programming language
dtrx - Do The Right Extraction
lunatic - Lunatic is an Erlang-inspired runtime for WebAssembly
append-zip - append a file into an existing zip file, overwriting the existing file of the same name if needed
Rusoto - AWS SDK for Rust
python-btrfs - Python Btrfs module
sqlx - 🧰 The Rust SQL Toolkit. An async, pure Rust SQL crate featuring compile-time checked queries without a DSL. Supports PostgreSQL, MySQL, and SQLite.
binspector - A binary format analysis tool
RE2 - RE2 is a fast, safe, thread-friendly alternative to backtracking regular expression engines like those used in PCRE, Perl, and Python. It is a C++ library.