dedup
Find duplicate text files. (by zyocum)
image-ndd-lsh
Near-duplicate image detection using Locality Sensitive Hashing (by mendesk)
dedup | image-ndd-lsh | |
---|---|---|
1 | 2 | |
11 | 64 | |
- | - | |
0.0 | 0.0 | |
about 1 month ago | almost 3 years ago | |
Python | Python | |
MIT License | - |
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
dedup
Posts with mentions or reviews of dedup.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2021-04-04.
-
[D] Hashing techniques to compare large datasets?
There is actually a whole family of hashing functions called locality sensitive hashing functions (LSH) that have the property that the likelihood of a hash collision is proportional to the similarity of the hashed data values. I’ve used Simhash myself for textual similarity, but LSHs can be used for finding similar images, audio, or other data types.
image-ndd-lsh
Posts with mentions or reviews of image-ndd-lsh.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2023-03-07.
-
Discussion Thread
You might look into locally sensitive hashing. Here's some software I found with a quick search
-
File consolidation software/strategy for merging and deduplicating photos from multiple TB drives into one place with possible corruptions.
What would be nice is something that used locality-sensitive hashing to bin very similar files together for further examination.
What are some alternatives?
When comparing dedup and image-ndd-lsh you can also consider the following projects:
datasketch - MinHash, LSH, LSH Forest, Weighted MinHash, HyperLogLog, HyperLogLog++, LSH Ensemble and HNSW
LSH - Locality Sensitive Hashing using MinHash in Python/Cython to detect near duplicate text documents
dupeguru - Find duplicate files
videoduplicatefinder - Video Duplicate Finder - Crossplatform
ssim - 🖼🔬 JavaScript Image Comparison