bincode VS lalrpop

Ermm... actually I meant something like this: playground, but then I realized it's basically (de)serialization, and I just found that we already have a crate for that: bincode.

One, figure out the bincode format (documented here: https://github.com/bincode-org/bincode/blob/trunk/docs/spec.md) and write your own parser. Maybe a one-off that specifically only handles this one data structure would be fairly straightforward.

Step 1: The Deserialize type requests data from the Deserializer with one of the deserialize_type methods. This gives it an opportunity to provide certain metadata about the type: structs provide a list of fields, enums provide a list of variants, tuples provide a length, etc. Some data formats (notably bincode) require this metadata to drive deserializing, as the wire format is not self-describing. Crucially, the Deserialize type also provides a visitor that is capable of receiving the requested data from the Deserializer.

Alternatively, give Bincode a try.

Like separate instructions? I was thinking if a instruction have unknown length I make sure I have some kind of header field that tells the data length of the instruction so receiver knows when next instruction starts. And I was planning on using Bincode with serde to serialize and dezerialize like structs and stuff.

Isn't this essentially bincode?

This is similar in practice to using abi_stable, and end-users will still receive compiled files, but your plugins will be sandboxed and a single build will work on all platforms. The downside is that it's a bit more work because WebAssembly's support for passing complex data types between the host and the WebAssembly code is in the preliminary stages, so you need to do something like using Serde to encode your data into something like Bincode or MessagePack (or JSON and friends) to hand it off between the host and the plugin.

What kind of access do you need to the data ? You should be able to make a safe api to the Vec class by iterating on in in chunks, and using a closure to translate data between u8 and other representations. ( f32, u32 has the fomr_ne_bytes() / to_ne_bytes() methods ) You could make a helper function that takes a format description ( i.e. "fffuucc" , and calculates the size of the chunk, and generates a closure for reading accessing the data, of the layout is completely dynamic. This closure could use an enum to wrap the different primitive types. ) Or if the layouts are known at compile time , you could use procedural macros to generate code for serializaion / deserialization inot the the [u8] , though https://crates.io/crates/bincode may already do that for you )

Apparently there's a lot of discussion going on about that (3 of the 4 open tickets on the bincode implementation are about it), for example this one.

And some related parser tools: - https://github.com/kevinmehall/rust-peg - https://github.com/pest-parser/pest - https://github.com/lalrpop/lalrpop

lalrpop

Rust is a very nice langage for implementing compilers, and has a nice ecosystem for it (logos, rust-peg, lalrpop, astmaker -- this one is mine --, etc...).

rust-langdev has a lot of libraries for building compilers in Rust. Perhaps you could use these to make your implementation easier, and revisit it later if you want to build things from scratch. I'd suggest logos for lexing, LALRPOP / chumsky for parsing, and rust-gc for garbage collection.

Hi! For one of my projects I am currently using lalrpop (https://github.com/lalrpop/lalrpop/tree/master/doc/calculator/src), which is far from complete, but has the basic syntax I was looking for. I took some examples and worked around some lexer stuff but I’m currently happy with it. If you use it and have Intellij stuff installed, you can also use a plug-in for highlighting and SOMETIMES error checking. Otherwise, even VSCode had a great plug-in for highlighting!

How would you compare and contrast this with, say, lalrpop?

lalrpop is great. It's a completely different approach from nom, but for parsing a programming language, I would at least consider it. RustPython uses it.

I presume LALRPOP handles left recursion just fine.

I wanted to share an experimental side project I have been working on for some time. I constantly use commands like `jq` and `yq` for processing structured data in my day job and I was curious if a similar idea could be applied to images.

Another goal of mine was to get some exposure to with rust. I discovered the LALRPOP parser generator which really helped moved the project along (https://github.com/lalrpop/lalrpop)

The key point here is that we are going to depend on the lalrpop library to generate the parser based on the formal grammar we define. Note that we have it as part of the [build-dependencies] section and we only depend on a tiny utility crate called lalrpop-util at runtime. The reason for that is the main lalrpop "magic" would happen during the crate compilation (in the build.rs file) when lalrpop would generate the deterministic pushdown automaton based on our grammar. The code generation logic is not required to be part of our interpreter, we only need a few utility methods from the lalrpop-util for the automaton to operate. You might have noticed that we also enable the lexer feature of lalrpop, because we are going to use lexer provided by lalrpop as well (please refer to the Part I if you do not know what the lexer is).