hachoir
Kaitai Struct
hachoir | Kaitai Struct | |
---|---|---|
3 | 44 | |
586 | 3,839 | |
- | 1.1% | |
6.4 | 7.5 | |
3 months ago | 19 days ago | |
Python | Shell | |
GNU General Public License v3.0 only | GPL-3.0-or-later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
hachoir
-
Magika: AI powered fast and efficient file type identification
https://github.com/vstinner/hachoir/blob/main/hachoir/subfil...
File signature:
-
Kaitai Struct: A new way to develop parsers for binary structures
I contributed a number of file formats a few years ago (and attempted numerous others) but ran into a number of problems with certain file formats:
1. It's not possible to read from the file until a multiple byte termination sequence is detected. [1]
2. You can't read sections of a file where the termination condition is the presence of a sequence of bytes denoting the next unrelated section of the file (and you don't want to consume/read these bytes) [2]
3. The WebIDE at the time couldn't handle very large file format specifications such as Photoshop (PSD) [3]
4. Files containing compressed or encrypted sections require a compression/encryption algorithm to be hardcoded into Kaitai struct libraries for each programming language it can output to.
The WebIDE I particularly liked as it makes it easy to get started and share results. I also liked how Kaitai Struct allows easy definition of constraints (simple ones at least) into the file format specification so that you can say "this section of the file shall have a size not exceeding header.length * 2 bytes".
Some alternative binary file format specification attempts for those interested in seeing alternatives, each with their own set of problems/pros/cons:
1. 010 Editor [4]
2. Synalysis [5]
3. hachoir [6]
4. DFDL [7]
[1] https://github.com/kaitai-io/kaitai_struct/issues/158
[2] https://github.com/kaitai-io/kaitai_struct/issues/156
[3] https://raw.githubusercontent.com/davidhicks/kaitai_struct_f...
[4] https://www.sweetscape.com/010editor/repository/templates/
[5] https://github.com/synalysis/Grammars
[6] https://github.com/vstinner/hachoir/tree/main/hachoir/parser
[7] https://github.com/DFDLSchemas/
-
PyWhat: Identify Anything
Another one sort of related is hachoir, and specifically the hachoir-metadata script: https://github.com/vstinner/hachoir
Kaitai Struct
- Reverse-engineering an encrypted IoT protocol
-
Parsing an Undocumented File Format
- ImHex [2], which has a pattern language [3] which allows parsing, and it seems more powerful than what Kaitai offers. I stumbled upon some limitations with it but it was still useful.
[1]: https://kaitai.io/
- Kaitai Struct – a declarative language used to describe binary data structures
-
HTTPie Desktop: cross-platform API testing client for humans
Beautiful. Didn't know something like this exists. Reminds me of Katai[0]
[0]. https://kaitai.io/
-
Hacking the LG Monitor's EDID
An EDID override like this would be helpful for macOS as well, where the monitors swapping around after standby is a real annoyance [0] [1]
EDID rewrites are 99% of the time blocked by the monitor firmware: https://notes.alinpanaitiu.com/Decoding-monitor-EDID-on-macO...
By the way, one helpful tool that helped me navigate the EDID dump was Kaitai Struct [2]. It shows a side by side view with the hex view and the EDID structure, and it highlights the hex values in real time as you navigate the structure. Unfortunately [3] it doesn't support the extension blocks that the author needs.
[0] https://notes.alinpanaitiu.com/Weird-monitor-bugs
[1] https://forums.macrumors.com/threads/external-displays-swapp...
[2] https://kaitai.io/
[3] https://github.com/kaitai-io/edid.ksy
- Kaitai Struct: new way to develop parsers for binary structures
-
Fq: Jq for Binary Formats
Kaitai Struct might be a good choice for that: https://kaitai.io/
-
Ingesting, parsing and making sense of device log data
For binary log format, there's the excellent Kaitai Struct frameworks, that make it very easy to generate parsers from a declarative schema
-
What is this tool? More info in comments
kaitai
-
Visual Programming with Elixir: Learning to Write Binary Parsers (2019)
https://kaitai.io/
Worth a look if you are writing binary parsers.
What are some alternatives?
binrw - A Rust crate for helping parse and rebuild binary data using ✨macro magic✨.
Protobuf - Protocol Buffers - Google's data interchange format
usaddress - :us: a python library for parsing unstructured United States address strings into address components
csvkit - A suite of utilities for converting to and working with CSV, the king of tabular file formats.
fuckitjs - The Original Javascript Error Steamroller
Camelot - A Python library to extract tabular data from PDFs
pyWhat - 🐸 Identify anything. pyWhat easily lets you identify emails, IP addresses, and more. Feed it a .pcap file or some text and it'll tell you what it is! 🧙♀️
tablib - Python Module for Tabular Datasets in XLS, CSV, JSON, YAML, &c.
probablepeople - :family: a python library for parsing unstructured western names into name components.
PDFMiner - Python PDF Parser (Not actively maintained). Check out pdfminer.six.
smm2-documentation - Documentation for the game Super Mario Maker 2.
PyYAML