clean-discord
Cleaning discord data for NLP (by JEF1056)
stopes
A library for preparing data for machine translation research (monolingual preprocessing, bitext mining, etc.) built by the FAIR NLLB team. (by facebookresearch)
clean-discord | stopes | |
---|---|---|
2 | 1 | |
22 | 237 | |
- | 1.3% | |
0.0 | 5.8 | |
over 2 years ago | 5 months ago | |
Python | Python | |
- | MIT License |
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
clean-discord
Posts with mentions or reviews of clean-discord.
We have used some of these posts to build our list of alternatives
and similar projects.
-
Massive multi-turn conversational dataset based on cleaned discord data
Splitting up the data currently isn't in the plans yet; if someone (or you) could create a classifier (NOTE: please, please optimize it, the amount of data to process here is not trivial) to split the data into the relevant groups, go ahead and create a branch and pull request to https://github.com/JEF1056/clean-discord and hopefully I can do that for the next release, which is slated toward the end of the year. I can provide small snippets of some of the raw JSON data so you can understand how it's formatted.
-
[R] Massive multi-turn conversational dataset based on cleaned discord data
Included in the raw data are a lot of unwanted, non-language behaviors; these include massive blocks of code, bot commands, bot messages, ASCII art, messages with only images attached, messages containing only unicode (non-standard) spaces, etc. Other messages are considered "toxic", e.g. falling under the categories: toxic, obscene, threatening, insulting, or identity hate. The regex and cleaning code can be found here.
stopes
Posts with mentions or reviews of stopes.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2022-07-21.
-
[D] Hey Reddit! We're a bunch of research scientists and software engineers and we just open sourced a new state-of-the-art AI model that can translate between 200 different languages. We're excited to hear your thoughts so we're hosting an AMA on 07/21/2022 @ 9:00AM PT. Ask Us Anything!
We have a bunch! The model and data are available here: https://github.com/facebookresearch/fairseq/tree/nllb/examples/nllb/modeling , LASER3 here: https://github.com/facebookresearch/fairseq/tree/nllb/examples/nllb/laser\_distillation , training data here: https://github.com/facebookresearch/fairseq/tree/nllb/examples/nllb/data , FLORES and our other human translated datasets here: https://github.com/facebookresearch/flores , and an entire modular pipeline for data cleaning here: https://github.com/facebookresearch/stopes. It's also available on HuggingFace! [angela]
What are some alternatives?
When comparing clean-discord and stopes you can also consider the following projects:
NeMo - A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)
tm2tb - Bilingual term extractor
flores - Facebook Low Resource (FLoRes) MT Benchmark
nematus - Open-Source Neural Machine Translation in Tensorflow
Thirukkural-English-Translation-Dataset - Thirukural in English