jdupes
duperemove
Our great sponsors
jdupes | duperemove | |
---|---|---|
44 | 16 | |
1,681 | 659 | |
- | - | |
0.0 | 9.2 | |
7 months ago | 28 days ago | |
C | C | |
MIT License | GNU General Public License v3.0 only |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
jdupes
-
File Servers... how are you handling duplicates
I recommend the use of jdupes, a fork of the well-known fdupes, to find duplicate files.
-
fdupes: Identify or Delete Duplicate Files
200 lines of Nim [1] seems to run about 9X faster than the 8000 lines of C in fdupes on a little test dir I have. If you need C, I think jdupes [2] is faster as @TacticalCoder points out a couple of times here. In my testing, `dups` is usually faster than `jdupes`, though.
[1] https://github.com/c-blake/bu/blob/main/dups.nim
[2] https://github.com/jbruchon/jdupes
-
I'm amazed how I find anything & why I have so many dupes!
There's always the well-respected tool, Czkawka. Or, of the CLI is your thing, jdupes is a good option.
- Anyone know of any good file deduplication tools?
-
Johnny Decimal
My research into this many years ago turned out that jdupes was the right / best solution I could find for my usecase.
https://github.com/jbruchon/jdupes
Though that works fine from a script perspective I'd like some more interactive way of sorting directories etc. Identifying is just the first step, jdupes helps with linking the files (both soft and hard links comes with caveats though!) but that is mostly to save space, not to help in reorganisation.
- Jdupes: A powerful duplicate file finder
-
Does jdupes do a 'dry run' if you just specify directory(s) and no other options
I can work it out by looking at https://github.com/jbruchon/jdupes.
-
replace duplicates with hard links - I think jdupes is the answer, or maybe fclones (I have questions)
I have looked at a few alternatives and think jdupes is the one for me. Then I found out it was not multi-threaded so will give it a go but the developer of jdupes recomended fclones (https://github.com/jbruchon/jdupes/issues/186) if you were dealing with large file systems and wanted multi-threading. But as I am using a HD it may not be necessary.
-
De-Duping a file server
jdupes is a fork of the old standby fdupes, but it has a Win32 release as well as supporting POSIX.
-
Any good duplicate file finder for windows?
jdupes is a tuned fork of the well-known fdupes, and has Win32 releases.
duperemove
-
fdupes: Identify or Delete Duplicate Files
Very useful for identifying files that may need to get deduplicate or that can be removed entirely. Unfortunately, I don't think this will also find identical directories.
If deleting files isn't what you want, I'd suggest looking into deduplicating tools.
ZFS has its own de duplicator built in, which is nice. It should just deduplicate files and individual extents of files by itself once you enable it. Probably not a good idea on very write-heavy disks, but it's an option.
Other file systems with extent level deduplication can use https://github.com/markfasheh/duperemove to not only deduplicaye files, but also deduplicate individual extents. This can be very useful for file systems that store a lot of duplicate content, like different WINE prefixes. For filesystems without extent deduplication, duperemove should try hard linking files to make them take up practically no disks space.
-
Bcachefs Merged into the Linux 6.7 Kernel
ZFS now has reflink support, which doesn't require lots of RAM, but isn't done automatically while writing. You need to run something like https://github.com/markfasheh/duperemove
- ZFS 2.2.0 (RC): Block Cloning merged
- Craziest thing I ever used SQLite for: partial file deduplication
-
Deduplication on EXT4
Then duperemove
-
What is a wineprefix and should I make a new one every time I add a new game to lutris?
Filesystems like Btrfs and XFS have support for deduplication, you can use a program like duperemove to save space.
-
File deduplication report?
Maybe you could use a file deduplication instead of a block based? https://github.com/markfasheh/duperemove
-
Anyone running Bees? Or deduping data some other way?
If not bees, do you run other programs for deduping? I see jdupes has support for BTRFS, https://github.com/jbruchon/jdupes, and also duperemove, https://github.com/markfasheh/duperemove.
-
Is switching to BTRFS useful for my use-cases?
It's a good filesystem, I use it with a special setup that needs a filesystem with snapshots. It's been stable for me, I run a duperemovehttps://github.com/markfasheh/duperemove) occasionally and that's about all the maintenance it needs.
-
With Proton being as good as it is now, do we still need separate prefixes for every game?
With Btrfs or XFS you can easily deduplicate the data with tools like duperemove, potentially saving a lot of space if you've installed many small games.
What are some alternatives?
fdupes - FDUPES is a program for identifying or deleting duplicate files residing within specified directories.
bees - Best-Effort Extent-Same, a btrfs dedupe agent
dupeguru - Find duplicate files
dduper - Fast block-level out-of-band BTRFS deduplication tool.
rmlint - Extremely fast tool to remove duplicates and other lint from your filesystem
btrfs - WinBtrfs - an open-source btrfs driver for Windows
rdfind - find duplicate files utility
czkawka - Multi functional app to find duplicates, empty folders, similar images etc.
compsize - btrfs: find compression type/ratio on a file or set of files
fclones - Efficient Duplicate File Finder
bees-docker - docker container for zygo/bees