mimemagic VS hachoir

If you're curious, here's how I solved it for ruby back in the day. Still used magic bytes, but added an overlay on top of the freedesktop.org DB: https://github.com/mimemagicrb/mimemagic/pull/20

Get it directly from github commit.

[AO3-6152] - Due to a licensing incident with a Rails dependency known as mimemagic, we had to update Rails to 5.2.5 and mimemagic to 0.3.6.

I don't think mimemagic works on Windows after the drama. I opened a PR for that a month ago but no one seems to care: https://github.com/mimemagicrb/mimemagic/pull/141

"License, motherfucker"

I know the vulgarity of the statements is tongue in cheek, but this one has been reinforced lately by the "MIME Magic" debacle[1], mama mia.

[1] https://github.com/mimemagicrb/mimemagic/issues/98

it has a licensing issue

https://github.com/mimemagicrb/mimemagic/commit/749a7e59de480b7c0373acc4f8ceb4444352ba46#diff-2ea7e2364883967953ab518a8316b639e612b8a6f20eadb7b97939d91c8e2612

On the other hand mimemagic provides by_magic https://github.com/mimemagicrb/mimemagic#usage which does detection by heuristic. It's a radically different method for a radically different use case.

Anyway, I created a PR addressing new Mimemagic not working on Windows https://github.com/mimemagicrb/mimemagic/pull/141

Someone broke the internet for rails https://github.com/mimemagicrb/mimemagic/issues/98

https://github.com/vstinner/hachoir/blob/main/hachoir/subfil...

File signature:

I contributed a number of file formats a few years ago (and attempted numerous others) but ran into a number of problems with certain file formats:

1. It's not possible to read from the file until a multiple byte termination sequence is detected. [1]

2. You can't read sections of a file where the termination condition is the presence of a sequence of bytes denoting the next unrelated section of the file (and you don't want to consume/read these bytes) [2]

3. The WebIDE at the time couldn't handle very large file format specifications such as Photoshop (PSD) [3]

4. Files containing compressed or encrypted sections require a compression/encryption algorithm to be hardcoded into Kaitai struct libraries for each programming language it can output to.

The WebIDE I particularly liked as it makes it easy to get started and share results. I also liked how Kaitai Struct allows easy definition of constraints (simple ones at least) into the file format specification so that you can say "this section of the file shall have a size not exceeding header.length * 2 bytes".

Some alternative binary file format specification attempts for those interested in seeing alternatives, each with their own set of problems/pros/cons:

1. 010 Editor [4]

2. Synalysis [5]

3. hachoir [6]

4. DFDL [7]

[1] https://github.com/kaitai-io/kaitai_struct/issues/158

[2] https://github.com/kaitai-io/kaitai_struct/issues/156

[3] https://raw.githubusercontent.com/davidhicks/kaitai_struct_f...

[4] https://www.sweetscape.com/010editor/repository/templates/

[5] https://github.com/synalysis/Grammars

[6] https://github.com/vstinner/hachoir/tree/main/hachoir/parser

[7] https://github.com/DFDLSchemas/

Another one sort of related is hachoir, and specifically the hachoir-metadata script: https://github.com/vstinner/hachoir