De-duplication tool

This page summarizes the projects mentioned and recommended in the original post on /r/DataHoarder
Python Deduplication

InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured

  • rusty-dedupe

    • I made a file duplicate locator in rust which runs quite fast - could help you solve your problem. I compiled for both linux & windows - you can grab the EXE file on the release page here: https://github.com/devmaster24/rusty-dedupe/releases/tag/v0.1.0

  • dupeguru

    Find duplicate files

    • Czkawka should be fine. It can search for duplicates by size and hash, not just file name. Not sure how well Dupeguru will do but it also scans by filename or content: https://dupeguru.voltaicideas.net/. Also, for overall estimation of dedup savings, Starwind Dedup Analyzer is good: https://www.starwindsoftware.com/starwind-deduplication-analyzer

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • Duplicity

    14 projects | news.ycombinator.com | 24 Jan 2024

  • How to use onedrive for culling photos

    1 project | /r/onedrive | 11 Dec 2023

  • Does anyone know any freeware duplicate file checkers without an upsell similar to awesome duplicate photo finder?

    1 project | /r/software | 6 Dec 2023

  • DupeGuru: Open-source, cross-platform GUI tool to find duplicate files

    1 project | news.ycombinator.com | 30 Aug 2023

  • App recommendation for finding duplicate images

    1 project | /r/mac | 10 Jul 2023