hachoir
osv.dev
hachoir | osv.dev | |
---|---|---|
3 | 19 | |
586 | 1,407 | |
- | 2.1% | |
6.4 | 9.7 | |
3 months ago | 1 day ago | |
Python | Python | |
GNU General Public License v3.0 only | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
hachoir
-
Magika: AI powered fast and efficient file type identification
https://github.com/vstinner/hachoir/blob/main/hachoir/subfil...
File signature:
-
Kaitai Struct: A new way to develop parsers for binary structures
I contributed a number of file formats a few years ago (and attempted numerous others) but ran into a number of problems with certain file formats:
1. It's not possible to read from the file until a multiple byte termination sequence is detected. [1]
2. You can't read sections of a file where the termination condition is the presence of a sequence of bytes denoting the next unrelated section of the file (and you don't want to consume/read these bytes) [2]
3. The WebIDE at the time couldn't handle very large file format specifications such as Photoshop (PSD) [3]
4. Files containing compressed or encrypted sections require a compression/encryption algorithm to be hardcoded into Kaitai struct libraries for each programming language it can output to.
The WebIDE I particularly liked as it makes it easy to get started and share results. I also liked how Kaitai Struct allows easy definition of constraints (simple ones at least) into the file format specification so that you can say "this section of the file shall have a size not exceeding header.length * 2 bytes".
Some alternative binary file format specification attempts for those interested in seeing alternatives, each with their own set of problems/pros/cons:
1. 010 Editor [4]
2. Synalysis [5]
3. hachoir [6]
4. DFDL [7]
[1] https://github.com/kaitai-io/kaitai_struct/issues/158
[2] https://github.com/kaitai-io/kaitai_struct/issues/156
[3] https://raw.githubusercontent.com/davidhicks/kaitai_struct_f...
[4] https://www.sweetscape.com/010editor/repository/templates/
[5] https://github.com/synalysis/Grammars
[6] https://github.com/vstinner/hachoir/tree/main/hachoir/parser
[7] https://github.com/DFDLSchemas/
-
PyWhat: Identify Anything
Another one sort of related is hachoir, and specifically the hachoir-metadata script: https://github.com/vstinner/hachoir
osv.dev
-
Magika: AI powered fast and efficient file type identification
Is it safe to assume that hashing (1) every file on disk, or (2) any given file on disk at random, will yield random bits with uniform probability; and (3) why Argon2 instead of e.g. only two rounds of SHA256?
https://github.com/google/osv.dev/blob/master/README.md#usin... :
> We provide a Go based tool that will scan your dependencies, and check them against the OSV database for known vulnerabilities via the OSV API. ... With package metadata, not (a file hash, package) database that could be generated from OSV and the actual package files instead of their manifest of already-calculated checksums.
Might as well be heating a pool on the roof with all of this waste heat from hashing binaries build from code of unknown static and dynamic quality.
Add'l useful formats:
> Currently it is able to scan various lockfiles, debian docker containers, SPDX and CycloneDB SBOMs, and git repositories
-
Dependabot vs RenovateBot
Also it supports an alternative sources of CVEs with https://osv.dev
-
Pyscan: A command-line tool to detect security issues in your python dependencies.
https://osv.dev its open source and even has a free API with almost all the popular languages. One of the inspirations for my project.
-
Announcing Pyscan: A dependency vulnerability scanner for python projects.
pyscan uses OSV as its database for now. There are plans to add a few more.
-
We should start to add “ai.txt” as we do for “robots.txt”
JSON-LD or RDFa (RDF in HTML attributes) in at least the /index.html the HTML footer should be sufficient to indicate that there is structured linked data metadata for crawlers that then don't need an HTTP request to a .well-known URL /.well-known/ai_security_reproducibility_carbon.txt.jsonld.json
OSV is a new format for reporting security vulnerabilities like CVEs and an HTTP API for looking up CVEs from software component name and version. https://github.com/google/osv.dev#third-party-tools-and-inte... :
> We provide a Go based tool that will scan your dependencies, and check them against the OSV database for known vulnerabilities via the OSV API.
> Currently it is able to scan various lockfiles, -- (repo2docker REES config files) -- debian docker containers, SPDX and CycloneDB SBOMs, and git repositories.
-
Ask HN: Most interesting tech you built for just yourself?
Something I've recently worked on is building an SQLite database of all the dependencies my organisation uses, which makes it possible to write our own queries and reports. The tool is all Open Source (https://dmd.tanna.dev) and has a CLI as well as the SQLite data.
Ive used it to look for software that's out of date (via https://endoflife.date), to find vulnerablilities (via https://osv.dev) and get license information (via https://deps.dev)
It's been hugely useful for us understanding use of internal and external dependencies, and I wish I'd built it earlier in my career so I could've had it for other companies I've worked at!
-
Malicious library inits?
Something like osv-scanner (https://osv.dev/), Github's depandabot etc would help to a great extend in detecting malicious dependencies. Not much else be done I guess
- Distributed vulnerability database for Open Source
- SBOM and dependencies check tool and vulnerabilities database from Google
-
Free tool for generating SBOM and CVEs against source or binaries
Have a look at https://security.googleblog.com/2022/12/announcing-osv-scanner-vulnerability.html they have plans for c++ https://github.com/google/osv.dev/issues/783
What are some alternatives?
binrw - A Rust crate for helping parse and rebuild binary data using ✨macro magic✨.
osv-scanner - Vulnerability scanner written in Go which uses the data provided by https://osv.dev
usaddress - :us: a python library for parsing unstructured United States address strings into address components
certspotter - Certificate Transparency Log Monitor
fuckitjs - The Original Javascript Error Steamroller
betterscan-ce - Code Scanning/SAST/Static Analysis/Linting using many tools/Scanners + OpenAI GPT with One Report (Code, IaC) - Betterscan Community Edition (CE)
pyWhat - 🐸 Identify anything. pyWhat easily lets you identify emails, IP addresses, and more. Feed it a .pcap file or some text and it'll tell you what it is! 🧙♀️
Awesome-Fuzzing - A curated list of fuzzing resources ( Books, courses - free and paid, videos, tools, tutorials and vulnerable applications to practice on ) for learning Fuzzing and initial phases of Exploit Development like root cause analysis.
probablepeople - :family: a python library for parsing unstructured western names into name components.
certificate-transparency-go - Auditing for TLS certificates (Go code)
smm2-documentation - Documentation for the game Super Mario Maker 2.
tincan-tls - A cleanroom implementation of TLS 1.3