jdupes
kopia
Our great sponsors
jdupes | kopia | |
---|---|---|
44 | 224 | |
1,681 | 6,241 | |
- | 5.4% | |
0.0 | 9.6 | |
7 months ago | 5 days ago | |
C | Go | |
MIT License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
jdupes
-
File Servers... how are you handling duplicates
I recommend the use of jdupes, a fork of the well-known fdupes, to find duplicate files.
-
fdupes: Identify or Delete Duplicate Files
200 lines of Nim [1] seems to run about 9X faster than the 8000 lines of C in fdupes on a little test dir I have. If you need C, I think jdupes [2] is faster as @TacticalCoder points out a couple of times here. In my testing, `dups` is usually faster than `jdupes`, though.
-
I'm amazed how I find anything & why I have so many dupes!
There's always the well-respected tool, Czkawka. Or, of the CLI is your thing, jdupes is a good option.
- Anyone know of any good file deduplication tools?
-
Johnny Decimal
My research into this many years ago turned out that jdupes was the right / best solution I could find for my usecase.
https://github.com/jbruchon/jdupes
Though that works fine from a script perspective I'd like some more interactive way of sorting directories etc. Identifying is just the first step, jdupes helps with linking the files (both soft and hard links comes with caveats though!) but that is mostly to save space, not to help in reorganisation.
- Jdupes: A powerful duplicate file finder
-
Does jdupes do a 'dry run' if you just specify directory(s) and no other options
I can work it out by looking at https://github.com/jbruchon/jdupes.
-
replace duplicates with hard links - I think jdupes is the answer, or maybe fclones (I have questions)
I have looked at a few alternatives and think jdupes is the one for me. Then I found out it was not multi-threaded so will give it a go but the developer of jdupes recomended fclones (https://github.com/jbruchon/jdupes/issues/186) if you were dealing with large file systems and wanted multi-threading. But as I am using a HD it may not be necessary.
-
De-Duping a file server
jdupes is a fork of the old standby fdupes, but it has a Win32 release as well as supporting POSIX.
-
Any good duplicate file finder for windows?
jdupes is a tuned fork of the well-known fdupes, and has Win32 releases.
kopia
-
DwarFS – The Deduplicating Warp-Speed Advanced Read-Only File System
I think Kopia would be great for your use case
It has a great system to snapshot files but only store data if it's changed. I use it in an environment where I can't use something like zfs to snapshot data because I don't have the ability to make decisions about what filesystem we're using. It's been amazing, love it so much!
-
Ask HN: Open-source Windows 11 backup solutions
Thanks for the tip on Kopia. Setting it up now, looks perfect.
-
Building a Managed Service Provider Business With Open Source
Kopia - GitHub
-
I Backup
I've been happy with: https://kopia.io/
Fairly easy to configure, does snapshots to S3 and has a icon in my tray I can watch :)
-
Very strange behavior/bug - devices stuck together
Btw, kopia is one fine backup tool. Apparently borgbackup is good too.
-
Kopia: Open-Source, Fast and Secure Open-Source Backup Software
Kopia is great, though it's worth noting for folks on Linux: non-UTF-8 paths aren't stored correctly [1] and xattrs aren't stored [2]. While most folks probably won't care about the former, the latter can could cause issues (eg. losing SELinux labels makes it difficult to restore a backup of the root filesystem on distros that use SELinux).
- Kopia: Open-Source Backup Software
-
How I backup my servers (2023)
I think Kopia [1] is on its way to be that. I am sticking to Restic for now but it seems like the strongest contender.
-
Borgbase backups have been unavailable for 3 days – recovery is at 26%
I used their trial for a bit to test it out with Vorta [1] in a container. Vorta (and Borg) seemed to work fine, until I wanted to restore an archive and I noticed that my recent snapshots were completely empty. Probably because of a misconfiguration on my end though. But it made me look elsewhere. For me backups should be a fire, test and forget solution.
Recently I made the switch to Kopia [2] which seems to have feature parity with Borg (and Restic [3]). It also has a web UI which is way easier to work with than Vorta. And I can easily view, extract and restore individual files or folders from there. This gave me way more confidence about this solution. The only thing I really miss is that I cannot chose different targets for different paths. For instance, with Borg I was able to backup a partial of my Docker appdata to an external source. And I haven't found a way to do this with Kopia. Besides that I'm pretty happy with this solution and I would recommend it.
-
Show HN: Gdańsk AI – full stack AI voice chatbot (STT, LLM, TTS, auth, payments)
There's a few. Off the top of my head
What are some alternatives?
fdupes - FDUPES is a program for identifying or deleting duplicate files residing within specified directories.
restic - Fast, secure, efficient backup program
dupeguru - Find duplicate files
BorgBackup - Deduplicating archiver with compression and authenticated encryption.
rmlint - Extremely fast tool to remove duplicates and other lint from your filesystem
Duplicati - Store securely encrypted backups in the cloud!
rdfind - find duplicate files utility
UrBackup - UrBackup - Client/Server Open Source Network Backup for Windows, MacOS and Linux
czkawka - Multi functional app to find duplicates, empty folders, similar images etc.
rclone - "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files
duperemove - Tools for deduping file systems
restic-wrapper - Simple bash wrapper to source .env configuration files for Restic. Facilitates both manual CLI execution and scheduled (cron) execution.