LjTools VS Som

I dug a little more and came across this tool which does seem to have the capability to view all LuaJIT Bytecode. https://github.com/rochus-keller/LjTools

> a large lua game code base, over 4000 files, 1.5 million lines of code

Interesting; how do you manage to keep consistency? Do you have special tools to e.g. detect inadvertent global variables? I once wrote a Smalltalk VM in Lua (https://github.com/rochus-keller/Smalltalk/blob/master/Inter...) which is a much smaller code base but even with this size I quickly would have lost track of e.g. scopes and names without tools I had to write myself (https://github.com/rochus-keller/LJTools).

Most of my languages have VM backends; see e.g. https://github.com/rochus-keller/Oberon; I implemented different backends generating LuaJIT bytecode; a year ago I switched to Mono which is based on ECMA-335; here is a discussion why I switched: https://github.com/rochus-keller/Oberon/releases/tag/IDEv0.9.0; I implemented utility libraries for both LuaJIT and CIL bytecode; see https://github.com/rochus-keller/LjTools/, https://github.com/rochus-keller/Pelib/ and https://github.com/rochus-keller/MonoTools/. I evaluated many VMs and think the mentioned ones are best suited. There were a lot of challenges with both technologies, what is to be expected, and too much to describe here.

LuaJIT is well suited as a backend/runtime environment for custom languages; I did it several times (see e.g. https://github.com/rochus-keller/Smalltalk, https://github.com/rochus-keller/Som/, https://github.com/rochus-keller/Oberon/). I also implemented a bit of infrastructure to ease the reuse: https://github.com/rochus-keller/LjTools. LuaJIT has some limitations though; if you require closures you have to know that the corresponding LuaJIT FNEW bytecode is not yet supported by the JIT, i.e. switches to the interpreter; as a work-around I implemented my own closures; LuaJIT also doesn't support multi-threading, but co-routines; and there is no debugger, and the infrastructure to implement one has limitations (i.e. performance is low when running to breakpoints). For most of my projects this was no issue. Recently I switched to CIL/Mono for my Oberon+ implementation which was a good move. But still I consider LuaJIT a good choice if you can cope with the mentioned limitations. The major advantage of LuaJIT is the small footprint and impressive performance for dynamic languages.

Implementing a VM is certainly interesting, but if you just need a fast backend you could generate LuaJIT bytecode (see e.g. https://github.com/rochus-keller/ljtools/ LuaJitComposer.h/cpp).

If you want to re-use LuaJIT as a backend, have e.g. a look at https://github.com/rochus-keller/ljtools

Recently I mostly use https://github.com/rochus-keller/LjTools#lua-parser-and-ide-features

> Smalltalkish

Have a look at the SOM dialect which is successfully used in education: http://som-st.github.io/

Here is an implementation in C++ which runs on LuaJIT: https://github.com/rochus-keller/Som/

> unfortunately out of print book Smalltalk 80: the language and its implementation is commonly recommended

I assume you know this link: http://stephane.ducasse.free.fr/FreeBooks/BlueBook/Bluebook....

Here is an implementation in C++ and Lua: https://github.com/rochus-keller/Smalltalk

- https://github.com/rochus-keller/Smalltalk/ Parser, code model, interpreter and navigable browser for the original Xerox Smalltalk-80 v2 sources and virtual image file

- https://github.com/rochus-keller/Som/ Parser, code model, navigable browser and VM for the SOM Smalltalk dialect

- https://github.com/rochus-keller/Simula A Simula 67 parser written in C++ and Qt

> do you regret those endeavours?

No, not in any way; the projects were very entertaining and gave me interesting insights.

If your DSL is statically typed then I recommend that you have a look at the Mono CLR; it's compatible with the ECMA-335 standard and the IR (CIL) is well documented, even with secondary literature.

If your DSL is dynamically typed I recommend LuaJIT; the bytecode is lean and documented (not as good as CIL though). LuaJIT also works well with statically typed languages, but Mono is faster in the latter case. Even if it was originally built for Lua any compiler can generate LuaJIT bytecode.

Both approaches are lean (Mono about 8 MB, LuaJIT about 1 MB), general purpose, available on many platforms and work well (see e.g. https://github.com/rochus-keller/Oberon/ and https://github.com/rochus-keller/Som/).

At the latest when you run a benchmark suite like Are-we-fast-yet; here are some measurment results: http://software.rochus-keller.info/are-we-fast-yet_crystal_lua_node_som_pharo_i386_results_2020-12-29.pdf. See also https://github.com/rochus-keller/Som/ and https://github.com/rochus-keller/Smalltalk.

LuaJIT is well suited as a backend/runtime environment for custom languages; I did it several times (see e.g. https://github.com/rochus-keller/Smalltalk, https://github.com/rochus-keller/Som/, https://github.com/rochus-keller/Oberon/). I also implemented a bit of infrastructure to ease the reuse: https://github.com/rochus-keller/LjTools. LuaJIT has some limitations though; if you require closures you have to know that the corresponding LuaJIT FNEW bytecode is not yet supported by the JIT, i.e. switches to the interpreter; as a work-around I implemented my own closures; LuaJIT also doesn't support multi-threading, but co-routines; and there is no debugger, and the infrastructure to implement one has limitations (i.e. performance is low when running to breakpoints). For most of my projects this was no issue. Recently I switched to CIL/Mono for my Oberon+ implementation which was a good move. But still I consider LuaJIT a good choice if you can cope with the mentioned limitations. The major advantage of LuaJIT is the small footprint and impressive performance for dynamic languages.

One option is to leverage someone else's JIT: you could, for example, rewrite the interpreter to transpile to Lua source, which is then run in LuaJIT. There's a Smalltalk dialect which does this successfully; the Lua version runs in 1/12th the time of the C interpreted version. https://github.com/rochus-keller/Som You can use LuaJIT's FFI to call back into the Stunt server, or else just rewrite it completely in Lua --- large parts of the Stunt server will just go away in a native Lua implementation (e.g. the object database is just a table). Javascript would be another candidate for this.

This is a completely different kind of measurement; unfortunately this is not clear enough from my Readme. I wanted to find out, how well my naive Bluebook interpreter performs on LuaJIT (using my virtual meta tracing approach) compared to Cog, which is a dedicatd Smalltalk VM optimized with whatever genious approaches over two decades (or even longer considering the long experience record by Elliot). This experiment continues in https://github.com/rochus-keller/Som, because I didn't want to modify the original Smalltalk image. I found that my naive LuaJIT based approach is about factor seven behind the highly optimized Cog/Spur, and further improvements would require similar optimization tricks as in the latter.