magika VS osv.dev

As someone that has worked in a space that has to deal with uploaded files for the last few years, and someone who maintains a WASM libmagic Node package ( https://github.com/moshen/wasmagic ) , I have to say I really love seeing new entries into the file type detection space.

Though I have to say when looking at the Node module, I don't understand why they released it.

Their docs say it's slow:

https://github.com/google/magika/blob/120205323e260dad4e5877...

It loads the model an runtime:

https://github.com/google/magika/blob/120205323e260dad4e5877...

They mark it as Experimental in the documentation, but it seems like it was just made for the web demo.

Also as others have mentioned. The model appears to only detect 116 file types:

https://github.com/google/magika/blob/120205323e260dad4e5877...

Where libmagic detects... a lot. Over 1600 last time I checked:

https://github.com/file/file/tree/4cbd5c8f0851201d203755b76c...

I guess I'm confused by this release. Sure it detected most of my list of sample files, but in a sample set of 4 zip files, it misidentified one.

We are very excited to announce the release of Magika our AI powered fast and efficient file type identification lib and tool - https://opensource.googleblog.com/2024/02/magika-ai-powered-fast-and-efficient-file-type-identification.html

Thanks to its optimized Keras model, large scale training dataset, and Onnx Magika massively outperform other file identification tools while be very fast even on CPU.

Magika python code and model is open sourced on Github: https://github.com/google/magika and we also provide an experimental TFJS based npm package https://www.npmjs.com/package/magika

With the team we hope you will find Magika useful for your own projects. Let us know what you think or if you have any question!

Is it safe to assume that hashing (1) every file on disk, or (2) any given file on disk at random, will yield random bits with uniform probability; and (3) why Argon2 instead of e.g. only two rounds of SHA256?

https://github.com/google/osv.dev/blob/master/README.md#usin... :

> We provide a Go based tool that will scan your dependencies, and check them against the OSV database for known vulnerabilities via the OSV API. ... With package metadata, not (a file hash, package) database that could be generated from OSV and the actual package files instead of their manifest of already-calculated checksums.

Might as well be heating a pool on the roof with all of this waste heat from hashing binaries build from code of unknown static and dynamic quality.

Add'l useful formats:

> Currently it is able to scan various lockfiles, debian docker containers, SPDX and CycloneDB SBOMs, and git repositories

Also it supports an alternative sources of CVEs with https://osv.dev

https://osv.dev its open source and even has a free API with almost all the popular languages. One of the inspirations for my project.

pyscan uses OSV as its database for now. There are plans to add a few more.

JSON-LD or RDFa (RDF in HTML attributes) in at least the /index.html the HTML footer should be sufficient to indicate that there is structured linked data metadata for crawlers that then don't need an HTTP request to a .well-known URL /.well-known/ai_security_reproducibility_carbon.txt.jsonld.json

OSV is a new format for reporting security vulnerabilities like CVEs and an HTTP API for looking up CVEs from software component name and version. https://github.com/google/osv.dev#third-party-tools-and-inte... :

> We provide a Go based tool that will scan your dependencies, and check them against the OSV database for known vulnerabilities via the OSV API.

> Currently it is able to scan various lockfiles, -- (repo2docker REES config files) -- debian docker containers, SPDX and CycloneDB SBOMs, and git repositories.

Something I've recently worked on is building an SQLite database of all the dependencies my organisation uses, which makes it possible to write our own queries and reports. The tool is all Open Source (https://dmd.tanna.dev) and has a CLI as well as the SQLite data.

Ive used it to look for software that's out of date (via https://endoflife.date), to find vulnerablilities (via https://osv.dev) and get license information (via https://deps.dev)

It's been hugely useful for us understanding use of internal and external dependencies, and I wish I'd built it earlier in my career so I could've had it for other companies I've worked at!

Something like osv-scanner (https://osv.dev/), Github's depandabot etc would help to a great extend in detecting malicious dependencies. Not much else be done I guess

Have a look at https://security.googleblog.com/2022/12/announcing-osv-scanner-vulnerability.html they have plans for c++ https://github.com/google/osv.dev/issues/783