magnitude
finalfusion-rust
Our great sponsors
| magnitude | finalfusion-rust | |
|---|---|---|
| 5 | 1 | |
| 1,610 | 85 | |
| -0.1% | - | |
| 0.0 | 6.3 | |
| 9 months ago | 6 months ago | |
| Python | Rust | |
| MIT License | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
magnitude
-
Text Classification Library for a Quick Baseline
(3) FastText now supports multiple languages [2].
[1] https://github.com/plasticityai/magnitude#pre-converted-magn...
-
Pgvector – vector similarity search for Postgres
Check out Magnitude, we built it to solve that problem: https://github.com/plasticityai/magnitude
It's still loaded from a file, but heavily uses memory-mapping and caching to be speedy and not overload your RAM immediately. And in production scenarios, multiple worker processes can share that memory due to the memory mapping.
Disclaimer: I'm the author.
Our startup made a package powered by SQLite for this very purpose: https://github.com/plasticityai/magnitude
Might be worth checking out :)
-
Build an Embeddings index from a data source
General language models from pymagnitude
-
Tutorial series on txtai
Backed by the pymagnitude library. Pre-trained word vectors can be installed from the referenced link.
finalfusion-rust
-
Compressing high-dimensional vectors by 97%
Nice article that explains product quantization very well!
PQ is really a nice compression technique. I implemented PQ and Optimized PQ [1] a while back in our word embedding package for Rust:
https://github.com/finalfusion/finalfusion-rust/
https://github.com/finalfusion/reductive/
Particularly Optimized PQ was effective in reducing vector sizes ~10 times with virtually no reconstruction loss. This made it much easier to ship models (no more 3GB embedding matrix with a neural net that is just a few megabytes large).
What are some alternatives?
flashtext - Extract Keywords from sentence or Replace keywords in sentences.
faiss - A library for efficient similarity search and clustering of dense vectors.
excalidraw-animate - A tool to animate Excalidraw drawings
pgvector - Open-source vector similarity search for Postgres
Milvus - A cloud-native vector database, storage for next generation AI applications
txtai - 💡 All-in-one open-source embeddings database for semantic search, LLM orchestration and language model workflows
Resume-Matcher - Resume Matcher is an open source, free tool to improve your resume. It works by using language models to compare and rank resumes with job descriptions.
pretty-print-confusion-matrix - Confusion Matrix in Python: plot a pretty confusion matrix (like Matlab) in python using seaborn and matplotlib
Romanian-Word-Embeddings - Romanian Word Embeddings. Here you can find pre-trained corpora of word embeddings. Current methods: CBOW, Skip-Gram, Fast-Text (from Gensim library). The .vec and .model files are available for download (all in one archive).
sentence-transformers - Multilingual Sentence & Image Embeddings with BERT
excalidraw - Virtual whiteboard for sketching hand-drawn like diagrams
cleora - Cleora AI is a general-purpose model for efficient, scalable learning of stable and inductive entity embeddings for heterogeneous relational data.