duperemove
rmlint
duperemove | rmlint | |
---|---|---|
16 | 16 | |
662 | 1,778 | |
- | - | |
9.2 | 5.8 | |
about 1 month ago | 5 months ago | |
C | C | |
GNU General Public License v3.0 only | GNU General Public License v3.0 only |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
duperemove
-
fdupes: Identify or Delete Duplicate Files
Very useful for identifying files that may need to get deduplicate or that can be removed entirely. Unfortunately, I don't think this will also find identical directories.
If deleting files isn't what you want, I'd suggest looking into deduplicating tools.
ZFS has its own de duplicator built in, which is nice. It should just deduplicate files and individual extents of files by itself once you enable it. Probably not a good idea on very write-heavy disks, but it's an option.
Other file systems with extent level deduplication can use https://github.com/markfasheh/duperemove to not only deduplicaye files, but also deduplicate individual extents. This can be very useful for file systems that store a lot of duplicate content, like different WINE prefixes. For filesystems without extent deduplication, duperemove should try hard linking files to make them take up practically no disks space.
-
Bcachefs Merged into the Linux 6.7 Kernel
ZFS now has reflink support, which doesn't require lots of RAM, but isn't done automatically while writing. You need to run something like https://github.com/markfasheh/duperemove
- ZFS 2.2.0 (RC): Block Cloning merged
- Craziest thing I ever used SQLite for: partial file deduplication
-
Deduplication on EXT4
Then duperemove
-
What is a wineprefix and should I make a new one every time I add a new game to lutris?
Filesystems like Btrfs and XFS have support for deduplication, you can use a program like duperemove to save space.
-
File deduplication report?
Maybe you could use a file deduplication instead of a block based? https://github.com/markfasheh/duperemove
-
Anyone running Bees? Or deduping data some other way?
If not bees, do you run other programs for deduping? I see jdupes has support for BTRFS, https://github.com/jbruchon/jdupes, and also duperemove, https://github.com/markfasheh/duperemove.
-
Is switching to BTRFS useful for my use-cases?
It's a good filesystem, I use it with a special setup that needs a filesystem with snapshots. It's been stable for me, I run a duperemovehttps://github.com/markfasheh/duperemove) occasionally and that's about all the maintenance it needs.
-
With Proton being as good as it is now, do we still need separate prefixes for every game?
With Btrfs or XFS you can easily deduplicate the data with tools like duperemove, potentially saving a lot of space if you've installed many small games.
rmlint
-
fdupes: Identify or Delete Duplicate Files
My preferred solution is rmlint [https://github.com/sahib/rmlint] mostly because it also looks at duplicate directories. It produces a bash script instead of deleting anything itself, so you can examine it before running the script it made.
-
ZFS 2.2.0 (RC): Block Cloning merged
After I removed duplicates (with help of https://github.com/sahib/rmlint ), I migrated my photos to an ordinary zpool instead.
-
I decluttered 14,000 digital items within a few hours. Here's how I did it.
For the technically savvy among you there is an excellent open source program called ‘rmlint’ (aka. Remove Lint). It is excellent at finding duplicates and saved me terabytes of space.
-
Looking for Powerful Deduplication software
You don’t say if you are on Windows or Unix. I have used rmlint successfully in the past.
-
the very best anti-duplicate app ?
dupeguru or rmlint
-
deleting duplicates programs?
rmlint, my friend, is the last tool you will ever need for this
- script to remove redundant parent directories
- Is there software that scans for duplicates?
- data hoarding software
-
Go Find Duplicates: blazingly-fast simple-to-use tool to find duplicate files
I use and test assorted duplicate finders regularly.
fdupes is the classic (going way way back) but it's really very slow, not worth using anymore.
The four I know are worth trying these days (depending on data set, hardware, file arrangement and other factors, any one of these might be fastest for a specific use case) are https://github.com/jbruchon/jdupes , https://github.com/pauldreik/rdfind , https://github.com/jvirkki/dupd , https://github.com/sahib/rmlint
Had not encountered fclones before, will give it a try.
What are some alternatives?
bees - Best-Effort Extent-Same, a btrfs dedupe agent
jdupes - A powerful duplicate file finder and an enhanced fork of 'fdupes'.
rdfind - find duplicate files utility
dduper - Fast block-level out-of-band BTRFS deduplication tool.
fdupes - FDUPES is a program for identifying or deleting duplicate files residing within specified directories.
btrfs - WinBtrfs - an open-source btrfs driver for Windows
fclones - Efficient Duplicate File Finder
compsize - btrfs: find compression type/ratio on a file or set of files
dupeguru - Find duplicate files
bees-docker - docker container for zygo/bees
czkawka - Multi functional app to find duplicates, empty folders, similar images etc.