dedup
Find duplicate text files. (by zyocum)
LSH
Locality Sensitive Hashing using MinHash in Python/Cython to detect near duplicate text documents (by mattilyra)
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
dedup
Posts with mentions or reviews of dedup.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2021-04-04.
-
[D] Hashing techniques to compare large datasets?
There is actually a whole family of hashing functions called locality sensitive hashing functions (LSH) that have the property that the likelihood of a hash collision is proportional to the similarity of the hashed data values. I’ve used Simhash myself for textual similarity, but LSHs can be used for finding similar images, audio, or other data types.
LSH
Posts with mentions or reviews of LSH.
We have used some of these posts to build our list of alternatives
and similar projects.
-
D Efficient Way To Cluster Millions Of Face
I'm looking into this, after reading the wikipedia entry of this it sound promissing! I already found a python lib https://github.com/mattilyra/LSH for this, I will get back to you once I tested this!
What are some alternatives?
When comparing dedup and LSH you can also consider the following projects:
datasketch - MinHash, LSH, LSH Forest, Weighted MinHash, HyperLogLog, HyperLogLog++, LSH Ensemble and HNSW
dupeguru - Find duplicate files
image-ndd-lsh - Near-duplicate image detection using Locality Sensitive Hashing
Neural-Scam-Artist - Web Scraping, Document Deduplication & GPT-2 Fine-tuning with a newly created scam dataset.
Cython - The most widely used Python to C compiler
intertext - Detect and visualize text reuse