hachoir VS magika

https://github.com/vstinner/hachoir/blob/main/hachoir/subfil...

File signature:

I contributed a number of file formats a few years ago (and attempted numerous others) but ran into a number of problems with certain file formats:

1. It's not possible to read from the file until a multiple byte termination sequence is detected. [1]

2. You can't read sections of a file where the termination condition is the presence of a sequence of bytes denoting the next unrelated section of the file (and you don't want to consume/read these bytes) [2]

3. The WebIDE at the time couldn't handle very large file format specifications such as Photoshop (PSD) [3]

4. Files containing compressed or encrypted sections require a compression/encryption algorithm to be hardcoded into Kaitai struct libraries for each programming language it can output to.

The WebIDE I particularly liked as it makes it easy to get started and share results. I also liked how Kaitai Struct allows easy definition of constraints (simple ones at least) into the file format specification so that you can say "this section of the file shall have a size not exceeding header.length * 2 bytes".

Some alternative binary file format specification attempts for those interested in seeing alternatives, each with their own set of problems/pros/cons:

1. 010 Editor [4]

2. Synalysis [5]

3. hachoir [6]

4. DFDL [7]

[1] https://github.com/kaitai-io/kaitai_struct/issues/158

[2] https://github.com/kaitai-io/kaitai_struct/issues/156

[3] https://raw.githubusercontent.com/davidhicks/kaitai_struct_f...

[4] https://www.sweetscape.com/010editor/repository/templates/

[5] https://github.com/synalysis/Grammars

[6] https://github.com/vstinner/hachoir/tree/main/hachoir/parser

[7] https://github.com/DFDLSchemas/

Another one sort of related is hachoir, and specifically the hachoir-metadata script: https://github.com/vstinner/hachoir

As someone that has worked in a space that has to deal with uploaded files for the last few years, and someone who maintains a WASM libmagic Node package ( https://github.com/moshen/wasmagic ) , I have to say I really love seeing new entries into the file type detection space.

Though I have to say when looking at the Node module, I don't understand why they released it.

Their docs say it's slow:

https://github.com/google/magika/blob/120205323e260dad4e5877...

It loads the model an runtime:

https://github.com/google/magika/blob/120205323e260dad4e5877...

They mark it as Experimental in the documentation, but it seems like it was just made for the web demo.

Also as others have mentioned. The model appears to only detect 116 file types:

https://github.com/google/magika/blob/120205323e260dad4e5877...

Where libmagic detects... a lot. Over 1600 last time I checked:

https://github.com/file/file/tree/4cbd5c8f0851201d203755b76c...

I guess I'm confused by this release. Sure it detected most of my list of sample files, but in a sample set of 4 zip files, it misidentified one.

We are very excited to announce the release of Magika our AI powered fast and efficient file type identification lib and tool - https://opensource.googleblog.com/2024/02/magika-ai-powered-fast-and-efficient-file-type-identification.html

Thanks to its optimized Keras model, large scale training dataset, and Onnx Magika massively outperform other file identification tools while be very fast even on CPU.

Magika python code and model is open sourced on Github: https://github.com/google/magika and we also provide an experimental TFJS based npm package https://www.npmjs.com/package/magika

With the team we hope you will find Magika useful for your own projects. Let us know what you think or if you have any question!