libuv VS llfio

I wound up on a different team with pre-existing Python code so temporarily shelved my use of Go for a bit, and we used Sanic (an async Python framework built on top of the excellent uvloop & libuv that also powers Node.js) to build some APIs for live channel management & operations. We hand-wrote our OpenAPI and used it to generate documentation and a CLI, which was an improvement over what was there (or not) before. Other teams used the OpenAPI document to generate SDKs to interact with our service.

In the source code of the Node.js opensource project, lib folder contains JavaScript code, mostly wrappers over C++ and function definitions. On the contrary, src folder contains C++ implementations of the functions, which pulls dependencies from the V8 project, the libuv project, the zlib project, the llhttp project, and many more - which are all placed at the deps folder.

Libuv, the C library that gives Node.js its asynchronous, non-blocking I/O capability is responsible for managing the thread pool. Node.js gives you the capability of using additional threads for computationally expensive and long-lasting operations to avoid blocking the event loop.

Node.js is written in C, C++, and JavaScript. The core components of Node.js - the V8 engine and the libuv library - are written in C++ and C, respectively, since these languages provide low-level access to system resources, making them well-suited for building high-performance and efficient applications. JavaScript is mainly used to write the application logic.

x8 apparently https://github.com/libuv/libuv/pull/3952

Notably upgrades to libuv 1.45 which has io_uring support. Faster file system access! Awhh yeah, it's on.

Remarkable what a mild & unintrusive PR adding io_uring was. https://github.com/libuv/libuv/pull/3952

Well, the single-threaded nature ultimately leads to its biggest downfall. Node.js utilizes a synchronous event loop engineered using Libuv that takes in code from the call stack and executes it.

The reference library can be found at https://ned14.github.io/llfio/

Thanks for the suggestions, which I have transposed into https://github.com/ned14/llfio/issues/106

There was an effort to get an afio library accepted into boost in the past. I believe the most current work on that library is happening here nowadays : https://github.com/ned14/llfio I'm not sure if it is considered production-ready or not. But I couldn't see any mention of it in the replies so I figured I would fix that!

It has an implementation: LLFIO

I'm a fan of Interprocess, used it for over a decade. But for mmapping I've switched to LLFIO and recommend it highly. (Plugging so Niall doesn't have to.)

Since that post, I have the reference implementation library very nearly passing its test suite https://github.com/ned14/llfio/pull/89. Once it's done I'll start very slowly writing its proposal paper for WG21 SG4. Should land before this summer.

FYI it doesn't solve quite what you're solving, but I've been careful to ensure https://github.com/ned14/llfio works well on Freestanding and < 64 Kb microcontrollers and I know Victor has been careful to ensure a good subset of std::format could work well on embedded. In other words, the i/o story for embedded C++ may improve greatly in the next few years.

If you want stronger guarantees than iostreams can give you, either use the OS-specific calls or a wrapper of said calls (e.g. https://github.com/ned14/llfio, disclaimer I'm the owner of that). Note that even in LLFIO, there is no concept of "seek to the end" because that's racy so we don't implement that. All you get is atomic append, otherwise you're on your own to coordinate what "end of file" means.