xv6-public VS Fennel

+1 for mini operating system.

Us, application developers, rely on many OS features: memory management, filesystem, etc. I'm sure eventually we'll ask "how such things are done behind the scene?"

That's why I tinker with xv6 (https://github.com/mit-pdos/xv6-public) during sparetime. Learning various process scheduling algorithms from textbook is a thing. Implementing it is another thing. I learn a lot. And it's definitely fun, even though there's almost zero chance the knowledge gained is relevant for my job (I'm a mobile app dev).

Recently I compiled xv6 using gcc 7.5.0 on Ubuntu 18 , everything is ok. But when I try to compile it using gcc 13.2.1 on latest Arch, it's failed: result

https://github.com/mit-pdos/xv6-public has under 10,000 lines of C and assembly including some user space programs.

xv6 was originally written for 32-bit x86; the RISC-V port is a relatively recent development. See e.g. https://github.com/mit-pdos/xv6-public for some of the earlier history.

rxv64 was written for a specific purpose: we had to ramp up professional engineers on both 64-bit x86_64 and kernel development in Rust; we were pointing them to the MIT materials, which at the time still focused on x86, but they were getting tripped up 32-bit-isms and the original PC peripherals (e.g., accessing the IDE disk via programmed IO). Interestingly, the non sequitur about C++ aside, porting to Rust exposed several bugs or omissions in the C original; fixes were contributed back to MIT and applied to the original (and survived into the RISC-V port).

Oh, by the way, the use of the term "SMP" predates Intel's usage by decades.

I'd define an arena as the pattern where the arena itself owns N objects. So you free the arena to free all objects.

My first job was at EA working on console games (PS2, GameCube, XBox, no OS or virtual memory on any of them), and while at the time I was too junior to touch the memory allocators themselves, we were definitely not malloc-ing and freeing all the time.

It was more like you load data for the level in one stage, which creates a ton of data structures, and then you enter a loop to draw every frame quickly. There were many global variables.

---

Wikipedia calls it a region, zone, arena, area, or memory context, and that seems about right:

https://en.wikipedia.org/wiki/Region-based_memory_management

It describes history from 1967 (before C was invented!) and has some good examples from Apache ("pools") and Postgres ("memory contexts").

I also just looked at these codebases:

https://github.com/mit-pdos/xv6-public (based on code from the 70's)

https://github.com/id-Software/DOOM (1997)

I looked at allocproc() in xv6, and gives you an object from a fixed global array. A lot of C code in the 80's and 90's was essentially "kernel code" in that it didn't have an OS underneath it. Embedded systems didn't run on full-fledges OSes.

DOOM tends to use a lot of what I would call "pools" -- arrays of objects of a fixed size, and that's basically what I remember from EA.

Though in g_game.c, there is definitely an arena of size 0x20000 called "demobuffer". It's used with a bump allocator.

---

So I'd say

- malloc / free of individual objects was NEVER what C code looked like (aside from toy code in college)

- arena allocators were used, but global vars and pools are also very common.

- arenas are more or less wash for memory safety. they help you in some ways, but hurt you in others.

The reason C programmers don't malloc/free all the time is for speed, not memory safety. Arenas are still unsafe.

When you free an arena, you have no guarantee there's nothing that points to it anymore.

Also, something that shouldn't be underestimated is that arena allocators break tools like ASAN, which use the malloc() free() interface. This was underscored to me by writing a garbage collector -- the custom allocator "broke" ASAN, and that was actually a problem:

https://www.oilshell.org/blog/2023/01/garbage-collector.html

If you want memory safety in your C code, you should be using ASAN (dynamically instrumented allocators) and good test coverage. Arenas don't help -- they can actually hurt. An arena is a trivial idea -- the problem is more if that usage pattern actually matches your application, and apps evolve over time.

Magma is proudly licensed under the MIT license, and uses code from Xv6 and Yagura.

My OS classes used xv6, a reimplementation of Unix Version 6 for a RISC-V architecture. Accompanying that was the OSTEP textbook.

Check out xv6 if you are only getting started with operating systems and want something simpler.

Here is kernel.ld.

It's interesting: minimal software is out there, but folks don't tend to choose it. I spend a fair amount of time thinking about how to be conservative in my dependencies, and this encourages a lightweight stack that tends to perform pretty well. These days, I'm favoring tools like Lua, SQLite, Fennel[0], Althttpd[1], Fossil[2], and the Mako Server[3] and find that great, lightweight, stable, efficient software is to be had, for free, but you have to go a bit off the beaten path. This isn't stuff you hear about on Stack Overflow.

In terms of frontend, which the post focuses on (Google Docs and a 30MB doc), I guess I'm conflicted. While I tend to favor native apps + web pages, I'm also a daily Tiddlywiki user, and I really think web apps have their place (heck, one idea I'm working on is a lightweight local server that lets you run web apps like Tiddlywiki). But without a doubt, Tiddlywiki is more resource intensive than Emacs (my go-to for notetaking when I'm not on TW). My tab for a 6MB Tiddlywiki file uses 155MB of RAM, and my (heavily customized, dozens of open buffers) Emacs session uses 88MB. So I do think the author has a good point.

[0]: https://fennel-lang.org/

Eh it's not just luajit and luajit didn't create that problem either. It's a symptom of lua actually succeeding at its design goal of being easily embedded as an extension language. A significant number of incompatible runtimes are more popular than the most recent puc lua, including I believe the older official lua 5.2 released in 2011.

I've done a fair bit of professional lua development and I don't think I've ever written standalone up-to-date puc lua except maybe for some tooling & scripts. It's such a small language and used in such a way that the runtime, distribution method, and available APIs have much more impact on your use (and compatibility) than the version.

Virtually everyone shipping a lua environment is also shipping changes to it that make it a unique target, if only extensions to the standard library. This is why I think syntax layer-only approach like fennel's is the correct choice for improving on lua. It mirrors lua's runtime semantics exactly, and allows you to access the implementation peculiars on their own terms and so can just be run on time of any lua system.

https://fennel-lang.org

Just learned about https://fennel-lang.org/ , could have probably used that as well to avoid Lua.

> I’m positive that there is a Lispy language out there (actually in existence, or the aether) that is appropriate for embedded work, but the constraints of the target make it difficult to envision.

Perhaps Fennel* fits the bill?

* https://fennel-lang.org/

I've also seen neovim plugins written in fennel [0], so if you want something lispy, that's possible now.

[0]: a Lisp that compiles to Lua, https://github.com/bakpakin/Fennel

Something similar: Fennel (https://fennel-lang.org/) is a lisp that compiles into Lua, which nvim can use as plugins, so you can write nvim plugins in a lisp. Aniseed (https://github.com/Olical/aniseed) makes this really easy.

If you want a more FP language on the Lua runtime, you might be interested in Fennel. I wrote a post about adding Fennel compiler to a hslua interpreter a while back, which might be useful for you.