datasketch
MinHash, LSH, LSH Forest, Weighted MinHash, HyperLogLog, HyperLogLog++, LSH Ensemble and HNSW (by ekzhu)
dedup
Find duplicate text files. (by zyocum)
| datasketch | dedup | |
|---|---|---|
| 1 | 1 | |
| 2,352 | 11 | |
| - | - | |
| 6.4 | 0.0 | |
| about 1 month ago | 5 days ago | |
| Python | Python | |
| MIT License | MIT License |
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
datasketch
Posts with mentions or reviews of datasketch.
We have used some of these posts to build our list of alternatives
and similar projects.
-
D Efficient Way To Cluster Millions Of Face
A great library that implements all those "big data" algos is https://github.com/ekzhu/datasketch
dedup
Posts with mentions or reviews of dedup.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2021-04-04.
-
[D] Hashing techniques to compare large datasets?
There is actually a whole family of hashing functions called locality sensitive hashing functions (LSH) that have the property that the likelihood of a hash collision is proportional to the similarity of the hashed data values. I’ve used Simhash myself for textual similarity, but LSHs can be used for finding similar images, audio, or other data types.
What are some alternatives?
When comparing datasketch and dedup you can also consider the following projects:
image-ndd-lsh - Near-duplicate image detection using Locality Sensitive Hashing
LSH - Locality Sensitive Hashing using MinHash in Python/Cython to detect near duplicate text documents
dupeguru - Find duplicate files
searx - Privacy-respecting metasearch engine [Moved to: https://github.com/searx/searx]
Neural-Scam-Artist - Web Scraping, Document Deduplication & GPT-2 Fine-tuning with a newly created scam dataset.
jina - ☁️ Build multimodal AI applications with cloud-native stack
whoogle-search - A self-hosted, ad-free, privacy-respecting metasearch engine
intertext - Detect and visualize text reuse