File Deduplication

• fdupes

  17 2,354 2.5 C

  FDUPES is a program for identifying or deleting duplicate files residing within specified directories.

I recently used [fdupes](https://github.com/adrianlopezroche/fdupes) to figure out duplicate files from my amazon cloud drive / photos migration. Took about 2 days to scour through about 1.5TB worth of day.

• scripts

  1 20 10.0 Awk

  Miscellaneous scripts that serve a stand-alone purpose that might be useful for others. (by taltman)

Prior to settling on this approach, I found [this](https://unix.stackexchange.com/questions/277697/whats-the-quickest-way-to-find-duplicated-files) post to be very helpful. One of the respondent wrote [this](https://github.com/taltman/scripts/blob/master/unix_utils/find-dupes.awk) awk script that is supposedly very fast. However, it leverages the FreeBSD flavor of things. I [tried](https://github.com/taltman/scripts/issues/4) getting it to work on Linux, but couldn't get it to work given my awk-fu skills aren't so good.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• czkawka

  361 17,501 7.7 Rust

  Multi functional app to find duplicates, empty folders, similar images etc.

You can use czkawka to find and remove duplicates. It's free, easy-to-use and pretty reliable. I also sometime use Starwinds dedupe analyzer to check if there still any data that can be deduplicated.

Suggest a related project