binrw
Grammars
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
binrw
-
How to write deserializer for custom binary protocol?
Serde can work but won't be easy, because It wants to deserialize everything. Take a look at the binrw crate(https://github.com/jam1garner/binrw) and define your binary layout with It, then you can actually create a wrapper which reads the file from any reader(be it a Cursor over the mmap'd slice or a file).
- Kaitai Struct: A new way to develop parsers for binary structures
-
Post-Monomorphization Dead Code Elimination and Other Unwritten Blog Posts
Sure! The commit is here (I haven’t pushed the version of this commit I made on binread itself but the actual changes are identical)
Grammars
-
Kaitai Struct: A new way to develop parsers for binary structures
I contributed a number of file formats a few years ago (and attempted numerous others) but ran into a number of problems with certain file formats:
1. It's not possible to read from the file until a multiple byte termination sequence is detected. [1]
2. You can't read sections of a file where the termination condition is the presence of a sequence of bytes denoting the next unrelated section of the file (and you don't want to consume/read these bytes) [2]
3. The WebIDE at the time couldn't handle very large file format specifications such as Photoshop (PSD) [3]
4. Files containing compressed or encrypted sections require a compression/encryption algorithm to be hardcoded into Kaitai struct libraries for each programming language it can output to.
The WebIDE I particularly liked as it makes it easy to get started and share results. I also liked how Kaitai Struct allows easy definition of constraints (simple ones at least) into the file format specification so that you can say "this section of the file shall have a size not exceeding header.length * 2 bytes".
Some alternative binary file format specification attempts for those interested in seeing alternatives, each with their own set of problems/pros/cons:
1. 010 Editor [4]
2. Synalysis [5]
3. hachoir [6]
4. DFDL [7]
[1] https://github.com/kaitai-io/kaitai_struct/issues/158
[2] https://github.com/kaitai-io/kaitai_struct/issues/156
[3] https://raw.githubusercontent.com/davidhicks/kaitai_struct_f...
[4] https://www.sweetscape.com/010editor/repository/templates/
[5] https://github.com/synalysis/Grammars
[6] https://github.com/vstinner/hachoir/tree/main/hachoir/parser
[7] https://github.com/DFDLSchemas/
What are some alternatives?
nom - Rust parser combinator framework
Kaitai Struct - Kaitai Struct: declarative language to generate binary data parsers in C++ / C# / Go / Java / JavaScript / Lua / Nim / Perl / PHP / Python / Ruby
wuffs - Wrangling Untrusted File Formats Safely
smm2-documentation - Documentation for the game Super Mario Maker 2.
hachoir - Hachoir is a Python library to view and edit a binary stream field by field
restruct - Rich binary (de)serialization library for Golang
inlinable-dyn-extension-traits - An exploration into the various ways optional trait methods can be implemented in Rust.
intertrait - A library providing direct casting among trait objects implemented by a type
deku - Declarative binary reading and writing: bit-level, symmetric, serialization/deserialization
FortniteReplayDecompressor - Read Fortnite replay files