Ask HN: What are some good resources for learning about low level disk/file IO?

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured
  • go-caskdb

    (educational) build your own disk based KV store in Go

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
  • py-caskdb

    (educational) build your own disk based KV store

  • dokany

    User mode file system library for windows with FUSE Wrapper

    I lead a project that included shipping a filesystem driver and a virtual disk on Windows.

    What I did to learn the lower-level APIs, and perform initial testing on the driver, was write a "mirror" drive. The user-mode code pointed to a folder on disk, the driver made a virtual disk drive, and all reads and writes in the virtual disk drive went to the mirror folder.

    On Windows, you can implement something like that using Dokany, Dokan, or Winfsp. On linux, there's the Fuse API. On Mac, there's MacFUSE.

    Even if you don't do a "mirror" drive, understanding the callbacks that libraries like Dokany, Dokan, Winfsp, and Fuse do helps you understand how IO happens in the driver. Many IO methods provided in popular languages provide abstractions above what the OS does. (For example, the Windows kernel has no concept of the "Stream" that's in your C# program. The "Stream"'s Position property is purely a construct within the .Net framework.)

    https://dokan-dev.github.io/

    https://github.com/dokan-dev/dokany

    https://osxfuse.github.io/

    Another place to start is the OS's documentation itself. For example, you can start with Window's CreateFileA function. This typically is what gets called "under the hood" in most programming languages when you open or create a file: https://learn.microsoft.com/en-us/windows/win32/api/fileapi/...

  • osxfuse

    FUSE extends macOS by adding support for user space file systems

    I lead a project that included shipping a filesystem driver and a virtual disk on Windows.

    What I did to learn the lower-level APIs, and perform initial testing on the driver, was write a "mirror" drive. The user-mode code pointed to a folder on disk, the driver made a virtual disk drive, and all reads and writes in the virtual disk drive went to the mirror folder.

    On Windows, you can implement something like that using Dokany, Dokan, or Winfsp. On linux, there's the Fuse API. On Mac, there's MacFUSE.

    Even if you don't do a "mirror" drive, understanding the callbacks that libraries like Dokany, Dokan, Winfsp, and Fuse do helps you understand how IO happens in the driver. Many IO methods provided in popular languages provide abstractions above what the OS does. (For example, the Windows kernel has no concept of the "Stream" that's in your C# program. The "Stream"'s Position property is purely a construct within the .Net framework.)

    https://dokan-dev.github.io/

    https://github.com/dokan-dev/dokany

    https://osxfuse.github.io/

    Another place to start is the OS's documentation itself. For example, you can start with Window's CreateFileA function. This typically is what gets called "under the hood" in most programming languages when you open or create a file: https://learn.microsoft.com/en-us/windows/win32/api/fileapi/...

  • fio

    Flexible I/O Tester

    Not specifically addressing your question, but when you get to the point of wanting to start doing some experiments you may find that 'fio' [1] is very handy.

    [1] https://github.com/axboe/fio

  • build-your-own-x

    Master programming by recreating your favorite technologies from scratch.

    In the spirit of building complex systems from scratch, you’ll enjoy this repo https://github.com/codecrafters-io/build-your-own-x that contains several ideas for projects around building things from scratch (eg Build your own Git, Docker, etc)

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • File systems can change in Sequoia with FSKit

    1 project | news.ycombinator.com | 1 Jul 2024
  • Projected File System

    2 projects | news.ycombinator.com | 22 Feb 2024
  • why is my mac able to read the left sd card but not the right?

    1 project | /r/mac | 11 Dec 2023
  • Please someone save me from file sharing hell to windows

    2 projects | /r/MacOS | 9 Dec 2023
  • Spacedrive – an open source cross-platform file explorer

    7 projects | news.ycombinator.com | 10 Oct 2023

Did you konow that C is
the 7th most popular programming language
based on number of metions?