Our great sponsors
-
rapiddisk
An Advanced Linux RAM Drive and Caching kernel modules. Dynamically allocate RAM as block devices. Use them as stand alone drives or even map them as caching nodes to slower local disk drives. Access those volumes locally or export them across an NVMe Target network. Manage it all from a web API.
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
As for ramdisk cache, what I was looking into is Primo Ramdisk which was mentioned a LOT around here, especially with high density plotting. Unfortunately that is only for Windows, but I found something similar for Linux: https://github.com/pkoutoupis/rapiddisk/ At this point I havent tried it, maybe indeed kernel module compiling will be enough, maybe not.
on my list above is just hard to do perfectly unless you specifically engineer the system that way. Thsi project probably manages to hit erasure blocks just right by virtue of the fact that it uses fairly large files, but it could probably be tuned to work better. You could maybe get partway there by changing the internal buffer size of the buffered disk in the plotter and recompiling/pip install-ing the plotter. In that case the buffer size would need to be the same size as the erasure block size of the SSD (or a multiple of it), but that is generally ~64-128MB these days. This approach would also cause somewhat higher memory consumption because the internal buffer size is for each bucket in each sort manager that is being actively written to, so more buckets with a larger buffer -> more RAM use (this is outside the memory limit you set on the command line).