Disk write buffering and its interactions with write flushes

• nbdkit

  17 - -

Direct IO is very inflexible. A better way is this (recommended by Linus):

https://lkml.iu.edu/hypermail/linux/kernel/1005.2/01845.html

I implemented something along the same lines but a bit less spicy here:

https://gitlab.com/nbdkit/nbdkit/-/commit/aa5a2183a6d16afd91...

• recordmydesktop

  2 13 4.6 C

  Fork of recordMyDesktop X Window System desktop recording abandonware, much bug fixing and general cleanups/rewriting has been done. (by recordmydesktop)

  

One of the things I improved in my recordMyDesktop fork [0] was an awful tendency for the frame cache writer to accumulate heaps of dirty pages until background writeback would flush them out.

I had 16GiB of RAM which meant quite large swaths of dirty pages would become buffered while the SSD sat idle until writeback began. This would cause high-FPS full-screen recordings in particular to just become backlogged and start dropping frames / audio dropouts. Just generally broken behavior for a desktop recorder, especially for a defferred-encode mode that's supposed to be optimized for minimizing system-wide effects/overheads during the recording.

The simple solution I found was to proactively initiate writeback regularly via fdatasync() on the cache fd. [1] I haven't decided yet if more should be done to constrain its buffer cache effects though. The cache files will be read back during encoding in post, so if there's enough RAM it can be desirable to enable reading them back entirely from memory instead of having to hit the disks again... but it would also be nice to let the rest of the system's processes keep their stuff in the page cache. memcg can probably be used to find a balanced solution, but I haven't done any experiments yet. Have any of you handled similar scenarios? What did you do?

[0] https://github.com/recordmydesktop/recordmydesktop

[1] https://github.com/recordmydesktop/recordmydesktop/commit/42...

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

Suggest a related project