dedupfs VS tinyssh

Is it possible to use tarsnap's deduplication code on my own server? We're setting up an ML dataset distribution box, and I was hoping to avoid storing e.g. imagenet as a tarball + untar'd (so that nginx can serve each photo individually) + imagenet in TFDS format.

https://github.com/xolox/dedupfs was the closest I found, but it has a lot of downsides.

Has anyone made an interface to tarsnap's tarball dedup code? A python wrapper around the block dedup code would be ideal, but I doubt it exists.

(Sorry for the random question -- I was just hoping for a standalone library along the lines of tarsnap's "filesystem block database" APIs. I thought about emailing this to you instead, but I'm crossing my fingers that some random HN'er might know. I'm sort of surprised that filesystems don't make it effortless. In fact, I delayed posting this for an hour to go research whether ZFS is the actual solution -- apparently "no, not unless you have specific brands of SSDs: https://www.truenas.com/community/resources/my-experiments-i..." which rules out my non-SSD 64TB Hetzner server. But like, dropbox solved this problem a decade ago -- isn't there something similar by now?)

While on topic of sshd having minimal dependencies, shout-out to Jan Mojžíš and his minimalist implementation:

https://github.com/janmojzis/tinyssh/

> [after] hardening steps [...] most of the bots can't even negotiate a connection

Yep, same here, except I'm using [tinyssh], which organically does not support anything other than ed25519/curve25519, sha256, and chacha-poly.

[tinyssh] https://tinyssh.org/

djb suggested that for openssh instead of the tinydns kex, so tinydns switched also:

https://github.com/janmojzis/tinyssh/issues/50