rdfind VS duperemove

I use rdfind for this.

rdfind is available for macOS; I've been using it on linux: https://github.com/pauldreik/rdfind

You can use rdfind to find all duplicates in your experiments dir and replace files with hardlinks. This way files will occupy disk space only once and all inode references will be to the same disk location.

I use rdfind to deal with this: https://github.com/pauldreik/rdfind

Yeah, I periodically download the full drives and just deduplicate with rdfind hardlinking identical files.

I use rdfind on my Linux NAS. https://github.com/pauldreik/rdfind

Very useful for identifying files that may need to get deduplicate or that can be removed entirely. Unfortunately, I don't think this will also find identical directories.

If deleting files isn't what you want, I'd suggest looking into deduplicating tools.

ZFS has its own de duplicator built in, which is nice. It should just deduplicate files and individual extents of files by itself once you enable it. Probably not a good idea on very write-heavy disks, but it's an option.

Other file systems with extent level deduplication can use https://github.com/markfasheh/duperemove to not only deduplicaye files, but also deduplicate individual extents. This can be very useful for file systems that store a lot of duplicate content, like different WINE prefixes. For filesystems without extent deduplication, duperemove should try hard linking files to make them take up practically no disks space.

ZFS now has reflink support, which doesn't require lots of RAM, but isn't done automatically while writing. You need to run something like https://github.com/markfasheh/duperemove

Then duperemove

Filesystems like Btrfs and XFS have support for deduplication, you can use a program like duperemove to save space.

Maybe you could use a file deduplication instead of a block based? https://github.com/markfasheh/duperemove

If not bees, do you run other programs for deduping? I see jdupes has support for BTRFS, https://github.com/jbruchon/jdupes, and also duperemove, https://github.com/markfasheh/duperemove.

It's a good filesystem, I use it with a special setup that needs a filesystem with snapshots. It's been stable for me, I run a duperemovehttps://github.com/markfasheh/duperemove) occasionally and that's about all the maintenance it needs.

With Btrfs or XFS you can easily deduplicate the data with tools like duperemove, potentially saving a lot of space if you've installed many small games.