rdfind
duperemove
rdfind | duperemove | |
---|---|---|
16 | 16 | |
883 | 662 | |
- | - | |
4.1 | 9.2 | |
about 1 month ago | about 1 month ago | |
C++ | C | |
GNU General Public License v3.0 or later | GNU General Public License v3.0 only |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
rdfind
- Rdfind: A utilty to find duplicate files, delete them or replace with hardlinks
-
Self hosted, web gui, file duplication scanner
I use rdfind for this.
-
Is there a Mac app that will allow me to recursively go through thousands of folders, calculate the total folder size, then compare against all other folder sizes, and if the size is identical, delete the newer one?
rdfind is available for macOS; I've been using it on linux: https://github.com/pauldreik/rdfind
-
Deduplication on EXT4
You can use rdfind to find all duplicates in your experiments dir and replace files with hardlinks. This way files will occupy disk space only once and all inode references will be to the same disk location.
- How do I show non-duplicate files across 2 drives?
-
Pip and cargo are not the same
I use rdfind to deal with this: https://github.com/pauldreik/rdfind
- Backing Up Data: Tips/Advice for Tons of Unorganized Data and Duplicate Files from Multiple Sources
-
This has probably happened to all of us at least once
Yeah, I periodically download the full drives and just deduplicate with rdfind hardlinking identical files.
- AMD/Xilinx Vivado rant
-
recommends for de-duplication?
I use rdfind on my Linux NAS. https://github.com/pauldreik/rdfind
duperemove
-
fdupes: Identify or Delete Duplicate Files
Very useful for identifying files that may need to get deduplicate or that can be removed entirely. Unfortunately, I don't think this will also find identical directories.
If deleting files isn't what you want, I'd suggest looking into deduplicating tools.
ZFS has its own de duplicator built in, which is nice. It should just deduplicate files and individual extents of files by itself once you enable it. Probably not a good idea on very write-heavy disks, but it's an option.
Other file systems with extent level deduplication can use https://github.com/markfasheh/duperemove to not only deduplicaye files, but also deduplicate individual extents. This can be very useful for file systems that store a lot of duplicate content, like different WINE prefixes. For filesystems without extent deduplication, duperemove should try hard linking files to make them take up practically no disks space.
-
Bcachefs Merged into the Linux 6.7 Kernel
ZFS now has reflink support, which doesn't require lots of RAM, but isn't done automatically while writing. You need to run something like https://github.com/markfasheh/duperemove
- ZFS 2.2.0 (RC): Block Cloning merged
- Craziest thing I ever used SQLite for: partial file deduplication
-
Deduplication on EXT4
Then duperemove
-
What is a wineprefix and should I make a new one every time I add a new game to lutris?
Filesystems like Btrfs and XFS have support for deduplication, you can use a program like duperemove to save space.
-
File deduplication report?
Maybe you could use a file deduplication instead of a block based? https://github.com/markfasheh/duperemove
-
Anyone running Bees? Or deduping data some other way?
If not bees, do you run other programs for deduping? I see jdupes has support for BTRFS, https://github.com/jbruchon/jdupes, and also duperemove, https://github.com/markfasheh/duperemove.
-
Is switching to BTRFS useful for my use-cases?
It's a good filesystem, I use it with a special setup that needs a filesystem with snapshots. It's been stable for me, I run a duperemovehttps://github.com/markfasheh/duperemove) occasionally and that's about all the maintenance it needs.
-
With Proton being as good as it is now, do we still need separate prefixes for every game?
With Btrfs or XFS you can easily deduplicate the data with tools like duperemove, potentially saving a lot of space if you've installed many small games.
What are some alternatives?
fdupes - FDUPES is a program for identifying or deleting duplicate files residing within specified directories.
bees - Best-Effort Extent-Same, a btrfs dedupe agent
jdupes - A powerful duplicate file finder and an enhanced fork of 'fdupes'.
rmlint - Extremely fast tool to remove duplicates and other lint from your filesystem
dduper - Fast block-level out-of-band BTRFS deduplication tool.
fclones - Efficient Duplicate File Finder
btrfs - WinBtrfs - an open-source btrfs driver for Windows
dupeguru - Find duplicate files
kindfs - Index filesystem into a database, then easily make queries e.g. to find duplicates files/dirs, or mount the index with FUSE.
compsize - btrfs: find compression type/ratio on a file or set of files