Why isn't there a Swagger/OpenAPI for binary formats?

• awesome-jsonschema

  70 101 3.4 Handlebars

  A curated list of awesome JSON Schema resources, tutorials, tools, and more.

  

Is OP conflating OpenAPI with JSON Schema? OpenAPI [1] specifies HTTP endpoints while JSON Schema [2] specifies data schema of JSON documents. And though OpenAPI specs frequently use the application/json media type with a schema specified using JSON Schema, can’t it also be used with other media types?

[1]: https://swagger.io/specification/

[2]: https://json-schema.org/

• Kaitai Struct

  44 3,844 7.5 Shell

  Kaitai Struct: declarative language to generate binary data parsers in C++ / C# / Go / Java / JavaScript / Lua / Nim / Perl / PHP / Python / Ruby

  

you are looking for katai struct: https://kaitai.io/

katai allows you to declaratively define a binary format bit for bit - they even have an IDE

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• RecordFlux

  2 100 9.6 Ada

  Formal specification and generation of verifiable binary parsers, message generators and protocol state machines

  

• gRPC

  201 40,820 9.9 C++

  The C based gRPC (C++, Python, Ruby, Objective-C, PHP, C#)

  

• grpc-swagger

  1 412 0.0 Java

  Debugging gRPC application with swagger-ui.

  

• grpcui

  11 4,921 7.5 Go

  An interactive web UI for gRPC, along the lines of postman

  

You can see that reflection in action through tools like grpcui:

https://github.com/fullstorydev/grpcui

• kaitai-to-wireshark

  1 37 0.0 Lua

  Converts a Kaitai Struct file description to a Wireshark LUA plugin

Would something like this help? https://github.com/joushx/kaitai-to-wireshark

Happened to just see this article when reading Kaitai docs: https://www.incibe-cert.es/en/blog/understanding-industrial-...

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• binspector

  4 225 1.8 C++

  A binary format analysis tool

  

• restruct

  3 345 3.2 Go

  Rich binary (de)serialization library for Golang

  

My project Restruct[1] does Kaitai-like things but also supports serialization. Unfortunately, it only supports Go and only deals with Go struct tags rather than YAML manifests. Still, it totally can be used for serialization. I use it to sketch out quick projects against arbitrary binary formats. Two examples: one, parsing PNG headers to implement a quick binwalk-like program for just PNG that looks for the IEND chunk to extract accurately[2], two, a program that splits FL Studio FLP projects by playlist track[3].

I feel like I’ve self-promoted Restruct like four times on Hacker News, and I feel kind of bad because it could use improvements and even some bug fixes and I never seem to get around to it. Oh well. It’s still useful for me, I hope it’s useful for others, too.

That said, Kaitai has a fairly clear path towards adding serialization from a design PoV; many things that would be calculated for parsing structures in deserialization could just become checks/assertions in serialization. As an example, checking that an expression calculates out to the expected value would be a reasonable approach. Reversible expressions could be implemented for some cases, too, if you want it to do more of the heavy lifting. I think the biggest obstacle is actually implementing it, and frankly my Scala is too weak to help with such a relatively big undertaking.

I’ve also played with the rust nom library, which implements functional programming style parser combinators. It is quite cool how it can express fairly complex grammars and binary formats pretty much equally well, albeit optimizing it effectively requires serious magic that I do not think nom has. (I assume in Haskell, the same thing can be done with mind-boggling optimization power.)

[1]: https://github.com/go-restruct/restruct

[2]: https://github.com/jchv/pngextract

[3]: https://github.com/jchv/flsplit

• pngextract

  1 0 1.8 Go

  Tool to extract PNG files out of a binary blob.

My project Restruct[1] does Kaitai-like things but also supports serialization. Unfortunately, it only supports Go and only deals with Go struct tags rather than YAML manifests. Still, it totally can be used for serialization. I use it to sketch out quick projects against arbitrary binary formats. Two examples: one, parsing PNG headers to implement a quick binwalk-like program for just PNG that looks for the IEND chunk to extract accurately[2], two, a program that splits FL Studio FLP projects by playlist track[3].

I feel like I’ve self-promoted Restruct like four times on Hacker News, and I feel kind of bad because it could use improvements and even some bug fixes and I never seem to get around to it. Oh well. It’s still useful for me, I hope it’s useful for others, too.

That said, Kaitai has a fairly clear path towards adding serialization from a design PoV; many things that would be calculated for parsing structures in deserialization could just become checks/assertions in serialization. As an example, checking that an expression calculates out to the expected value would be a reasonable approach. Reversible expressions could be implemented for some cases, too, if you want it to do more of the heavy lifting. I think the biggest obstacle is actually implementing it, and frankly my Scala is too weak to help with such a relatively big undertaking.

I’ve also played with the rust nom library, which implements functional programming style parser combinators. It is quite cool how it can express fairly complex grammars and binary formats pretty much equally well, albeit optimizing it effectively requires serious magic that I do not think nom has. (I assume in Haskell, the same thing can be done with mind-boggling optimization power.)

[1]: https://github.com/go-restruct/restruct

[2]: https://github.com/jchv/pngextract

[3]: https://github.com/jchv/flsplit

• flsplit

  1 2 10.0 Go

  Simple tool to split FL Studio project files by playlist track.

My project Restruct[1] does Kaitai-like things but also supports serialization. Unfortunately, it only supports Go and only deals with Go struct tags rather than YAML manifests. Still, it totally can be used for serialization. I use it to sketch out quick projects against arbitrary binary formats. Two examples: one, parsing PNG headers to implement a quick binwalk-like program for just PNG that looks for the IEND chunk to extract accurately[2], two, a program that splits FL Studio FLP projects by playlist track[3].

I feel like I’ve self-promoted Restruct like four times on Hacker News, and I feel kind of bad because it could use improvements and even some bug fixes and I never seem to get around to it. Oh well. It’s still useful for me, I hope it’s useful for others, too.

That said, Kaitai has a fairly clear path towards adding serialization from a design PoV; many things that would be calculated for parsing structures in deserialization could just become checks/assertions in serialization. As an example, checking that an expression calculates out to the expected value would be a reasonable approach. Reversible expressions could be implemented for some cases, too, if you want it to do more of the heavy lifting. I think the biggest obstacle is actually implementing it, and frankly my Scala is too weak to help with such a relatively big undertaking.

I’ve also played with the rust nom library, which implements functional programming style parser combinators. It is quite cool how it can express fairly complex grammars and binary formats pretty much equally well, albeit optimizing it effectively requires serious magic that I do not think nom has. (I assume in Haskell, the same thing can be done with mind-boggling optimization power.)

[1]: https://github.com/go-restruct/restruct

[2]: https://github.com/jchv/pngextract

[3]: https://github.com/jchv/flsplit

Suggest a related project