luajit-remake VS Pyjion

It's really cool that Haoran Xu copy-and-patch technique is catching on, I remember discovering it through his blog posts on applying these techniques to his own LuaJIT remake project[0][1] (and I probably found those through a post here). I highly recommend them if you're into that sort of thing, BTW. They're incredible deep dives, but he uses the details-element to keep the metaphorical descents into Mariana Trench optional so it doesn't get too overwhelming.

I even had the privilege of congratulating him the 1000th star of the GH repo[2], where he reassured me and others that he's still working on it despite the long pause after the last blog post, and that this mainly has to do with behind-the-scenes rewrites that make no sense to publish in part.

[0] https://sillycross.github.io/2022/11/22/2022-11-22/

[1] https://sillycross.github.io/2023/05/12/2023-05-12/

[2] https://github.com/luajit-remake/luajit-remake/issues/11

This seems like an awesome way of writing faster interpreters – i.e. not in assembly, but in C++ snippets you stitch together with a tool.

I did peek at the deegen tool a bit, and it seems quite large? https://github.com/luajit-remake/luajit-remake/tree/master/d...

I would be interested in an overview of all the analysis it has to do, which as I understand is basically “automated Mike Pall”

FWIW I think this is the hand-written equivalent with LuaJIT’s dynasm tool: https://github.com/LuaJIT/LuaJIT/blob/v2.1/src/vm_x64.dasc (just under 5000 lines)

Also there are several of these files with no apparent sharing, as you would get with deegen:

https://github.com/LuaJIT/LuaJIT/blob/v2.1/src/vm_x86.dasc

https://github.com/LuaJIT/LuaJIT/blob/v2.1/src/vm_ppc.dasc

It exists, was created by microsoft employees, and is referenced in the article: https://www.trypyjion.com/

I've actually come across and started using Pyjion recently (https://github.com/tonybaloney/pyjion); how does Pypy compare, both in terms of performance and purpose? There seems to be a lot of overlap...

Python is capable of doing things at runtime that are really hard to statically compile around, such as monkeypatching methods onto existing objects. You can compile it, but it's complicated. One strategy is to use a JIT that can observe application state at runtime and then invalidate code as it becomes obsoleted by changes, but it's complicated. See pyjion for an example.

When I say "Python" I am referring to the standard CPython interpreter which most people use. But there is also PyPy, which includes a Just In Time compile that compiles selected code into machine language on the fly, as needed. pyjion is another JIT compiler that generates machine language on the fly, and you can install it with pip. Or you could work for Facebook and use Cinder. Cython, Nuitka and Pyston are other alternatives.

and if you need more speed you can try https://www.pypy.org/ or https://github.com/tonybaloney/Pyjion or https://www.pyston.org/

Finally, there is also Pyjion which based on its website is “A drop-in JIT Compiler for Python 3.10” (https://www.trypyjion.com/). We will be covering it on a separate writeup. See you next time ;-).

Though some exciting stuff happening over the next few years, Python is getting faster, has been for awhile, and stuff like Pyjion https://www.trypyjion.com/, a drop in C# powered JIT compiler is starting to approach usable. Rust and Python seem to be best buds right now, so more extension libraries in rust, a newer more approachable language than say C/C++ but with a similar speed. Sign me up!

Pyjion Easy to use, small compiler. Increase performance of our 🐌 CPython.