Top 7 locality-sensitive-hashing Open-Source Projects

annoy

40 12,692 5.3 C++

Approximate Nearest Neighbors in C++/Python optimized for memory usage and loading/saving to disk

Project mention: Do we think about vector dbs wrong? | news.ycombinator.com | 2023-09-05

The focus on the top 10 in vector search is a product of wanting to prove value over keyword search. Keyword search is going to miss some conceptual matches. You can try to work around that with tokenization and complex queries with all variations but it's not easy.
Vector search isn't all that new a concept. For example, the annoy library (https://github.com/spotify/annoy) has been around since 2014. It was one of the first open source approximate nearest neighbor libraries. Recommendations have always been a good use case for vector similarity.
Recommendations are a natural extension of search and transformers models made building the vectors for natural language possible. To prove the worth of vector search over keyword search, the focus was always on showing how the top N matches include results not possible with keyword search.
In 2023, there has been a shift towards acknowledging keyword search also has value and that a combination of vector + keyword search (aka hybrid search) operates in the sweet spot. Once again this is validated through the same benchmarks which focus on the top 10.
On top of all this, there is also the reality that the vector database space is very crowded and some want to use their performance benchmarks for marketing.
Disclaimer: I am the author of txtai (https://github.com/neuml/txtai), an open source embeddings database

datasketch

1 2,348 6.4 Python

MinHash, LSH, LSH Forest, Weighted MinHash, HyperLogLog, HyperLogLog++, LSH Ensemble and HNSW
InfluxDB

www.influxdata.com sponsored

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
soundfingerprinting

6 909 8.1 C#

Open source audio fingerprinting in .NET. An efficient algorithm for acoustic fingerprinting written purely in C#.

Project mention: Ask HN: How many of you are self employed? | news.ycombinator.com | 2024-02-05

Started 10 years ago as an open-source project, building an algorithm for audio fingerprinting. Added a commercial offering, selling storage built specifically for audio fingerprints, targeting enterprise customers. Since the offering was too technical (it's hard to sell solutions to problems that are too narrow and domain-specific), pivoted to more "business-oriented problems". This last year's pivot is a chance to finally grow. Running a business in single-player mode is, at times, too stressful. Aside from the technical part, which I very much enjoy, I need to wear marketing, sales, and customer support hats.
[1] - https://emysound.com

elastiknn

1 352 8.8 Scala

Elasticsearch plugin for nearest neighbor search. Store vectors and run similarity search using exact and approximate algorithms.
image-ndd-lsh

2 60 0.0 Python

Near-duplicate image detection using Locality Sensitive Hashing
gaoya

1 48 4.8 Rust

Locality Sensitive Hashing
dedup

1 11 0.0 Python

Find duplicate text files.
WorkOS

workos.com sponsored

The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).