LASER VS fast_vector_similarity

I'm curious how people are handling multi-lingual embeddings.

I've found LASER[1] which originally had the idea to embed all languages in the same vector space, though it's a bit harder to use than models available through SentenceTransformers. LASER2 stuck with this approach, but LASER3 switched to language-specific models. However, I haven't found benchmarks for these models, and they were released about 2 years ago.

Another alternative would be to translate everything before embedding, which would introduce some amount of error, though maybe it wouldn't be significant.

1. https://github.com/facebookresearch/LASER

You can check out some of our materials and open sourced artifacts here:  - Our latest blog post: https://ai.facebook.com/blog/nllb-200-high-quality-machine-translation - Project Overview: https://ai.facebook.com/research/no-language-left-behind/  - Product demo: https://nllb.metademolab.com/  - Research paper: https://research.facebook.com/publications/no-language-left-behind - NLLB-200: https://github.com/facebookresearch/fairseq/tree/nllb - FLORES-200: https://github.com/facebookresearch/flores - LASER3: https://github.com/facebookresearch/LASER   Joining us today for the AMA are: - Angela Fan (AF), Research Scientist  - Jean Maillard (JM), Research Scientist - Maha Elbayad (ME), Research Scientist - Philipp Koehn (PK), Research Scientist - Shruti Bhosale (SB), Software Engineer   We’ll be here from 07/21/2022 @09:00AM PT - 10:00AM PT    Thanks and we’re looking forward to answering your questions!

This may be helpful: https://github.com/facebookresearch/LASER

Check FB's LASER: https://github.com/facebookresearch/LASER/tree/master/tasks/CCMatrix Also , Sentence-Transformers has a pretty neat model for crosslingual sentence similarity: https://huggingface.co/sentence-transformers/stsb-xlm-r-multilingual

You want LASER its a superbig model trained on tons of languages you can use it with sentence_transformers in python to compute embedings. Then you can use faiss or datasketch to find matches at K

Yes, check out my library for vector similarity that has various other measures which are more discriminative:

https://github.com/Dicklesworthstone/fast_vector_similarity

pip install fast_vector_similarity

Got it. I'd encourage you to expose more of that functionality at the level of your application if possible. I think there is a lot of potential in using more than just cosine similarity, especially when there are lots of candidates and you really want to sharpen up the top few recommendations to the best ones. You might find this open-source library I made recently useful for that:

https://github.com/Dicklesworthstone/fast_vector_similarity

I've had good results from starting with cosine similarity (using FAISS) and then "enriching" the top results from that with more sophisticated measures of similarity from my library to get the final ranking.

You can see how I did something similar in my library here:

https://github.com/Dicklesworthstone/fast_vector_similarity/...

Basically you use ndarray instead of numpy, try to vectorize anything you can, and for the for loops that can’t be vectorized, you can use rayon to do them in parallel.

Yeah, like the other commenter said, everything is in this file here:

https://github.com/Dicklesworthstone/fast_vector_similarity/...

If you also make your project using Rust and Maturin, you can literally just copy and paste that into your project because it's totally generic, and if the repo is public, GitHub will just run it all for you for free.

The only thing is you need to create an account on PyPi (pip) and add 2-Factor Auth so you can generate an API key. Then you go into the repo settings and go to secrets, and create a Github Actions secret with the name PYPI_API_TOKEN and make the value your PyPi token. That's it! It will not only compile all the wheels for you but even upload the project to PyPi for you using the settings found in your pyproject.toml file, like this:

https://github.com/Dicklesworthstone/fast_vector_similarity/...