stopes
LASER
Our great sponsors
| stopes | LASER | |
|---|---|---|
| 1 | 5 | |
| 237 | 3,520 | |
| 3.4% | 0.5% | |
| 5.8 | 7.0 | |
| 5 months ago | 5 months ago | |
| Python | Jupyter Notebook | |
| MIT License | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
stopes
-
[D] Hey Reddit! We're a bunch of research scientists and software engineers and we just open sourced a new state-of-the-art AI model that can translate between 200 different languages. We're excited to hear your thoughts so we're hosting an AMA on 07/21/2022 @ 9:00AM PT. Ask Us Anything!
We have a bunch! The model and data are available here: https://github.com/facebookresearch/fairseq/tree/nllb/examples/nllb/modeling , LASER3 here: https://github.com/facebookresearch/fairseq/tree/nllb/examples/nllb/laser\_distillation , training data here: https://github.com/facebookresearch/fairseq/tree/nllb/examples/nllb/data , FLORES and our other human translated datasets here: https://github.com/facebookresearch/flores , and an entire modular pipeline for data cleaning here: https://github.com/facebookresearch/stopes. It's also available on HuggingFace! [angela]
LASER
-
SentenceTransformers: Python framework for sentence, text and image embeddings
I'm curious how people are handling multi-lingual embeddings.
I've found LASER[1] which originally had the idea to embed all languages in the same vector space, though it's a bit harder to use than models available through SentenceTransformers. LASER2 stuck with this approach, but LASER3 switched to language-specific models. However, I haven't found benchmarks for these models, and they were released about 2 years ago.
Another alternative would be to translate everything before embedding, which would introduce some amount of error, though maybe it wouldn't be significant.
1. https://github.com/facebookresearch/LASER
-
[D] Hey Reddit! We're a bunch of research scientists and software engineers and we just open sourced a new state-of-the-art AI model that can translate between 200 different languages. We're excited to hear your thoughts so we're hosting an AMA on 07/21/2022 @ 9:00AM PT. Ask Us Anything!
You can check out some of our materials and open sourced artifacts here: - Our latest blog post: https://ai.facebook.com/blog/nllb-200-high-quality-machine-translation - Project Overview: https://ai.facebook.com/research/no-language-left-behind/ - Product demo: https://nllb.metademolab.com/ - Research paper: https://research.facebook.com/publications/no-language-left-behind - NLLB-200: https://github.com/facebookresearch/fairseq/tree/nllb - FLORES-200: https://github.com/facebookresearch/flores - LASER3: https://github.com/facebookresearch/LASER Joining us today for the AMA are: - Angela Fan (AF), Research Scientist - Jean Maillard (JM), Research Scientist - Maha Elbayad (ME), Research Scientist - Philipp Koehn (PK), Research Scientist - Shruti Bhosale (SB), Software Engineer We’ll be here from 07/21/2022 @09:00AM PT - 10:00AM PT Thanks and we’re looking forward to answering your questions!
-
School project : sentiments analysis with my country Arabic Dialect
This may be helpful: https://github.com/facebookresearch/LASER
-
[P] Bilingual text alignment tools for NMT - help needed
Check FB's LASER: https://github.com/facebookresearch/LASER/tree/master/tasks/CCMatrix Also , Sentence-Transformers has a pretty neat model for crosslingual sentence similarity: https://huggingface.co/sentence-transformers/stsb-xlm-r-multilingual
-
Help with aligned word embeddings
You want LASER its a superbig model trained on tons of languages you can use it with sentence_transformers in python to compute embedings. Then you can use faiss or datasketch to find matches at K
What are some alternatives?
NeMo - A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)
MUSE - A library for Multilingual Unsupervised or Supervised word Embeddings
tm2tb - Bilingual term extractor
electra - ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
flores - Facebook Low Resource (FLoRes) MT Benchmark
Arraymancer - A fast, ergonomic and portable tensor library in Nim with a deep learning focus for CPU, GPU and embedded devices via OpenMP, Cuda and OpenCL backends
nematus - Open-Source Neural Machine Translation in Tensorflow
fairseq - Facebook AI Research Sequence-to-Sequence Toolkit written in Python.
Thirukkural-English-Translation-Dataset - Thirukural in English