cranelift-jit-demo VS lineiform

I'm working on a small stack-based programming language. I'm currently at a stage where I'm trying to compile it using Cranelift. Altrough the Cranelift documentation is extensive, I'm lacking a broader picture on how to approach some things like heap-allocations and stack-management. The only example project I found are cranelift-jit-demo and this wonderful post.

First, it's much easier than you think, I swear. I strongly suggest that you start with the cranelift JIT toy language demo, it has everything that you need to get started.

Fyi I've checked a few (from here; https://github.com/appcypher/awesome-wasm-langs): - assembly script complier is written is typescript/javascript and in theory could be compiled to wasm, and hence could be embedded, but it is only theory as noone has managed to complete this flow - rust-driver requires the linker and calls it as an external tool to link the rustcore to the user code. without the core lib i could not manage to create anything usable. - zig (somewhat similar to rust): on discord some experr said it cannot be embedded and he see no option/plan for it. - lua: they have lua runtime running in wasm, but no transpiller to wasm I've also checked a few other without any success and closest I coild get was the example language for cranelift (https://github.com/bytecodealliance/cranelift-jit-demo)

JITs are somehow more tricky and differ in the a few points including: a) Codegen is much more time critical. b) JITs must know what's allready generated and what isn't. c) JITs often rely on informations only generated at runtime and must respond to that. See here for a JIT example witten with cranelift: https://github.com/bytecodealliance/cranelift-jit-demo.

You could also try Cranelift. The resulting code isn't as optimized as with LLVM, but it's faster and pleasant to use (and is written in Rust).

Backend IRs for code generation: - Cranelift (see https://github.com/bytecodealliance/cranelift-jit-demo as well as the messages on the Zulip chat if you get stuck)

As for LLVM, I'm not sure if there are any tutorials but I would really advise writing a bytecode interpreter first, unless you already have some grasp of assembly. However, this repository: https://github.com/bytecodealliance/cranelift-jit-demo is really great for learning cranelift which is essentially an LLVM alternative.

It was mainly built for wasm compilation. So no it is not married to rust. https://github.com/bytecodealliance/cranelift-jit-demo

Sounds similar to lineiform. Which isn't all that stable or actively developed, but it is a cute approach to writing a meta-jit in rust. It's a weird approach, but IMO it's worth more experimentation.

Working on Lineiform, my meta-JIT library, some more.

In response to Cranelift, switching to my own Tangle IR won't be using Cranelift at all (it uses raw dynasm-rs for emitting instructions). I go into a bit in https://github.com/chc4/lineiform/issues/19, but Cranelift specifically has some rules about iflags, the type they use to conceptualize processor flags effects (e.g. add's carryout or overflow). You can only have one iflags value live at a time, and it can't overlap with any other math operation. This is a problem because the x86 we're lifting doesn't always follow that rule, so if we just emit Cranelift as we go it will panic and say we built an invalid function.

I go into a bit in https://github.com/chc4/lineiform/issues/19, but it's less a problem with its optimizer and more a problem with its IR constraints. Cranelift specifically has some rules about `iflags`, the type they use to conceptualize processor flags effects (e.g. add's carryout or overflow). You can only have one `iflags` value live at a time, and it can't overlap with any other math operation. This is a problem because the x86 we're lifting doesn't always follow that rule, so if we just emit Cranelift as we go it will panic and say we built an invalid function.

The iflags design in general is kinda awkward too, and was being rethought a few months ago when I was first getting this working; I think they're planning on redesigning the add carryout interface and things to be slightly more streamlined. I suspect that any redesigned interface will have similar problems with mismatch between what I want from Cranelift and what 90% of other uses of Cranelfit want, though, and so I decided to just make my own IR instead.

Working on the codegen backend for Lineiform again. I sketched out a plan on how to implement register allocation in a way that hopefully doesn't have horrible behavior in the majority of cases, and implemented ~half of it last week, and hopefully I'll implement the other half and instruction scheduling this week.

Yup, https://github.com/chc4/lineiform. It's not usable at all yet - I was building it on top of Cranelift, which turned out to be a fairly bad idea, so I'm going to have to essentially rewrite all of it with my own codegen backend I think. I've been hacking on it on and off but it's been much slower progress due to work (and writing a codegen backend is hard...)

I got struck by either a very dumb or very good idea a few days ago, and finally have a working (minimal) proof-of-concept for it: Lineiform is a meta-JIT library to nearly automatically get an optimizing method JIT from a Rust interpreter. It does dynamic recompilation on closures by lifting from x86 to Cranelift IR for runtime function inlining and constant propagation.