restruct VS awesome-jsonschema

My project Restruct[1] does Kaitai-like things but also supports serialization. Unfortunately, it only supports Go and only deals with Go struct tags rather than YAML manifests. Still, it totally can be used for serialization. I use it to sketch out quick projects against arbitrary binary formats. Two examples: one, parsing PNG headers to implement a quick binwalk-like program for just PNG that looks for the IEND chunk to extract accurately[2], two, a program that splits FL Studio FLP projects by playlist track[3].

I feel like I’ve self-promoted Restruct like four times on Hacker News, and I feel kind of bad because it could use improvements and even some bug fixes and I never seem to get around to it. Oh well. It’s still useful for me, I hope it’s useful for others, too.

That said, Kaitai has a fairly clear path towards adding serialization from a design PoV; many things that would be calculated for parsing structures in deserialization could just become checks/assertions in serialization. As an example, checking that an expression calculates out to the expected value would be a reasonable approach. Reversible expressions could be implemented for some cases, too, if you want it to do more of the heavy lifting. I think the biggest obstacle is actually implementing it, and frankly my Scala is too weak to help with such a relatively big undertaking.

I’ve also played with the rust nom library, which implements functional programming style parser combinators. It is quite cool how it can express fairly complex grammars and binary formats pretty much equally well, albeit optimizing it effectively requires serious magic that I do not think nom has. (I assume in Haskell, the same thing can be done with mind-boggling optimization power.)

[1]: https://github.com/go-restruct/restruct

[2]: https://github.com/jchv/pngextract

[3]: https://github.com/jchv/flsplit

I’m a big fan of Kaitai Struct, to the point where I’ve even contributed a small bit of improvements to its Go support, and I use it in a handful of small projects. It’s indispensable for spelunking blobs of binary data.

I’ve also taken some inspiration with a Go library I wrote, restruct:

https://github.com/go-restruct/restruct

… which is a bit like Go’s JSON encoding/decoding library, but with kaitai-like annotations for binary encoding. (Check the PNG example to see some of what can be done with it.)

Honestly, I dislike plaintext formats a lot. It is more accessible because it’s human readable. But, this only extends to humans who happen to speak the language the protocol uses for keywords. While it’s not a huge ask, I still suggest this is mostly not that interesting of a benefit.

Parsing and emitting plaintext formats, meanwhile, is a rabbit hole. It’s human readable which makes you tempted to make it human writable. Should you accept extraneous whitespace? Tabs vs spaces? Terminating new line? Unix or DOS line endings? Etc.

Binary data may seem less accessible, but I blame the libraries. There’s tons of easy ways to parse text. You can use string.split, atoi and scanf in your language of choice. What is there for binary?

In Go, the encoding/binary package actually implements something really cool. A simple reflection-based mechanism that can read and write binary data into a structure in a defined and simple way.

lunixbochs extended this to struc[1], which adds additional tags for advanced reading and writing of binary structures, including variable length structures. I went further and maybe a bit off into the deep end with Restruct[2], a similar concept but with a lot more features, designed specifically so I could handle advanced structures quickly.

The end result is that I can define some Go structs with integers, strings, byte arrays and corresponding tags, and be able to serialize and deserialize from those structures to their corresponding binary representation. For an overdone demo of what you could do with Restruct for example, see this (incomplete) PNG demo: https://github.com/go-restruct/restruct/blob/master/formats/... (It is mainly incomplete because I had moved focus to develop a codegen for restruct, to improve runtime performance, although such work has since stalled.)

[1]: https://pkg.go.dev/github.com/lunixbochs/struc

[2]: https://pkg.go.dev/github.com/go-restruct/restruct

Truth. Zod is comparable to JSON Schema plus AJV, and it doesn't compare well at all. Your Zod code is all locked inside TypeScript so not only can it not be shared to any other language in your stack but it also cannot be serialized, which introduces many limitations. You also miss out on all the JSON Schema ecosystem tooling. (1, 2) For example the intellisense you get in VS Code for config files is powered by JSON Schema and schemastore.

The very first line of text below the header on the json-schema.org homepage is:

The framework's official website mentions a number of pros of FastAPI. In my opinion, the most useful features from a microservice perspective are: the simplicity of code (easy to use and avoid boilerplate), high operational capacity thanks to Starlette and Pydantic and compatibility with industry standards - OpenAPI and JSON Schema.

I've used Felte to reduce form boilerplate. Felte supports several different validation libraries like Zod. I actually used a custom validation function with ajv (which uses JSON schema).

(There is already a JSON Schema definition at https://json-schema.org/)

Like you said - standard XML isn't terrible. Adding on an XSD isn't terrible, because now you can enforce structure and datatypes on files provided by outside parties. Creating an XSLT is much more of a mental challenge, and probably should be left to tools to define.

Anything beyond those technologies is someone polishing up their resume.

While the aws-sdk is being installed to simulate DynamoDB locally, let me explain a few things about this command. First Comment will be the name of the model the scaffold creates. This model will be codified under app/models/schemas/comment.mjs as a JSON Schema object. Each of the parameters after Comment will be split into a property name and type (e.g. property name “subject”, property type “string”). This JSON Schema document will be used to validate the form data both on the client and server sides.

What you are looking is called Json-schema. Have a look at the implementations page, which will give you an idea of what you can do with json-schema, which also includes UI rendering.