Fq: Jq for Binary Formats

• fq

  43 9,384 9.4 Go

  jq for binary formats - tool, language and decoders for working with binary and text formats

Hi, sorry for the delay, on vacation. There is no process really more then convincing me :) and i think i will accept any decoder that is for a format used in public, standardized or proprietary.

I do want to add some kind of runtime format support and i'm working adding kaitai support but it's not ready yet, it's not an easy thing to do :) but i've made very good progress. ideally it will be something like: fq -d format.ksy file

Subscribe or keep an eye on this issue for updates https://github.com/wader/fq/issues/627

And feel free to ask any questions!

• rizin

  46 2,426 9.8 C

  UNIX-like reverse engineering framework and command-line toolset.

  

For this kind of task, using low-level debugger tools is probably better. Rizin[1][2]/Cutter[3][4] could help. We also have GSoC participant this year who works hard on improving debuginfo and debugging support[5]. I personally also like Binary Ninja, they recently made their debugger stable enough[6].

[1] https://rizin.re/

[2] https://github.com/rizinorg/rizin

[3] https://cutter.re/

[4] https://github.com/rizinorg/cutter

[5] https://rizin.re/posts/gsoc-2023-announcement/

[5] https://binary.ninja/2023/05/03/3.4-finally-freed.html#debug...

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• cutter

  39 14,926 8.2 C++

  Free and Open Source Reverse Engineering Platform powered by rizin

  

For this kind of task, using low-level debugger tools is probably better. Rizin[1][2]/Cutter[3][4] could help. We also have GSoC participant this year who works hard on improving debuginfo and debugging support[5]. I personally also like Binary Ninja, they recently made their debugger stable enough[6].

[1] https://rizin.re/

[2] https://github.com/rizinorg/rizin

[3] https://cutter.re/

[4] https://github.com/rizinorg/cutter

[5] https://rizin.re/posts/gsoc-2023-announcement/

[5] https://binary.ninja/2023/05/03/3.4-finally-freed.html#debug...

• Kaitai Struct

  44 3,830 7.5 Shell

  Kaitai Struct: declarative language to generate binary data parsers in C++ / C# / Go / Java / JavaScript / Lua / Nim / Perl / PHP / Python / Ruby

  

Kaitai Struct might be a good choice for that: https://kaitai.io/

• jqjq

  10 485 8.6 jq

  jq implementation of jq

Being a long time personal friend with the author I can assure you the more obscure the better :-) His interest in esoteric things and solutions are "well documented" if you browse around his github repos.

Some examples:

https://github.com/wader/jqjq

• catgolf

  2 38 7.3

  cat(1) golf

• 010-Editor-Templates

  2 36 6.0 C++

There is 010, at heart a cross-platform scriptable hex editor with a template language [1], which has a central template repository [2] as well as templates around the internet (e.g. 3, 4).

But it being a paid tool means there are fewer contributions from 13 year olds, which if we are all honest make up the majority of unpaid open source contributions - they simply have more spare time.

1: https://www.sweetscape.com/010editor/

2: https://www.sweetscape.com/010editor/repository/templates/

3: https://github.com/tge-was-taken/010-Editor-Templates/tree/m...

4: https://wiki.redmodding.org/cyberpunk-2077-modding/modding-k...

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• kaitai_struct_formats

  3 682 6.3 Kaitai Struct

  Kaitai Struct: library of binary file formats (.ksy)

  

Kaitai has a repository of binary formats[1] that can be used in visualizers or to auto-generate parsers.

[1] https://formats.kaitai.io/

• spyql

  23 902 0.0 Jupyter Notebook

  Query data on the command line with SQL-like SELECTs powered by Python expressions

I prefer a SQL-like format. It’s not as complete but it cover most of the day-to-day use cases. Take a look at https://github.com/dcmoura/spyql (I am the author). Congrats on fq!

• ImHex

  46 32,832 9.9 C++

  🔍 A Hex Editor for Reverse Engineers, Programmers and people who value their retinas when working at 3 AM.

I'd also like to throw https://github.com/WerWolv/ImHex in the mix here.

• feng

  1 11 8.3 Go

  Binary file format template system

I'm working on something, that is a open template format for binary file formats. It is usable today as a universal file extractor, with some bugs and limitations.

Check it out at https://github.com/martinlindhe/feng

• flac.tcl

  2 11 10.0 Tcl

  Probably the slowest FLAC decoder in the world

fun fact: fq is kind of the spiritual predecessor of https://github.com/wader/flac.tcl, you can see traces of it in flac_frame.go in fq, was used to prototype some things :)

Suggest a related project