dedupfs VS mfsbsd

Is it possible to use tarsnap's deduplication code on my own server? We're setting up an ML dataset distribution box, and I was hoping to avoid storing e.g. imagenet as a tarball + untar'd (so that nginx can serve each photo individually) + imagenet in TFDS format.

https://github.com/xolox/dedupfs was the closest I found, but it has a lot of downsides.

Has anyone made an interface to tarsnap's tarball dedup code? A python wrapper around the block dedup code would be ideal, but I doubt it exists.

(Sorry for the random question -- I was just hoping for a standalone library along the lines of tarsnap's "filesystem block database" APIs. I thought about emailing this to you instead, but I'm crossing my fingers that some random HN'er might know. I'm sort of surprised that filesystems don't make it effortless. In fact, I delayed posting this for an hour to go research whether ZFS is the actual solution -- apparently "no, not unless you have specific brands of SSDs: https://www.truenas.com/community/resources/my-experiments-i..." which rules out my non-SSD 64TB Hetzner server. But like, dropbox solved this problem a decade ago -- isn't there something similar by now?)

I'm very new to BSD in general, but I find it very fun and interesting!

However, I need pointers to get started.

> You won't be spoon-fed, and are expected to have read the manuals and other documentations...

I read a lot of FreeBSD and NetBSD documentation to get to the point of compiling my own kernels, but I don't think I ever read about the equivalent concept of Linux cpio/initramfs for BSD. My minimal images use a UFS filesystem.

Here, after checking https://mfsbsd.vx.sk/ and https://github.com/mmatuska/mfsbsd/blob/master/scripts/mdini... I think mfsbsd is just a using tmpfs so it may not exactly the same thing as initramfs, that allows booting linux from a bzImage + initrd

I'll keep searching, it's not super high priority at the moment, but it's something I'd like to do with (Free|Net)BSD.

I suspect it depends on how much support and/or hand-holding you need from your hosting provider. I'd hesitate to run an unofficial build/image but I believe the alternative on DO is to use mfsbsd (a memory-file-system installer for FreeBSD) which is also an unofficial build/image.

The common answer here is to use mfsbsd which puts all the installer's requirements onto a RAM disk so you should (in theory) be able to pull the install media and plug in other devices as needed

I looked into this for a project a couple of years ago and ended up using mfsbsd instead.

https://github.com/mmatuska/mfsbsd