Ask HN: Identify duplicate files in my data hoard?

This page summarizes the projects mentioned and recommended in the original post on

Our great sponsors
  • Sonar - Write Clean Python Code. Always.
  • InfluxDB - Access the most powerful time series database as a service
  • ONLYOFFICE ONLYOFFICE Docs — document collaboration in your environment
  • czkawka

    Multi functional app to find duplicates, empty folders, similar images etc.

  • restic

    Fast, secure, efficient backup program

    Duplicates aren’t always bad… some files naturally exist in many places, and removing them from some of the places make that directory/app incomplete.

    If you do want to save space by storing one copy of the bits/blocks, and still retain the index of all the original locations… you can store all your backups on a ZFS file system with Dedup turned on… (this uses memory and has performance implications)..

    Or back everything up with restic:

    restic stores files encrypted in a tree by the hash, so it naturally stores 1 file but as many references to it as needed. It has lookup and list functions that can tell you what’s duplicated.

    To simply find/report dups to be dealt with manually, you could quite easily md5/Sha1 your entire file tree, storing the output in a text file , which you can then pipe through sort , awk, and uniq to see which hashes occupy multiple lines … this is labor intensive… I just let my backup tools “compress” by saving one copy of each hash, and then it doesn’t matter as much (in my opinion).

    If its pictures or some other specific file type that you want to focus on the most… I’d pick an app that’s intended for cataloging those. Example: Adobe Lightroom shows me my duplicate pics and I can deal with those easily there.

  • Sonar

    Write Clean Python Code. Always.. Sonar helps you commit clean code every time. With over 225 unique rules to find Python bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.

  • organize

    The file management automation tool.

  • dupeguru

    Find duplicate files

    It can even search for similar files (binary, music and pictures).

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts