tatoeba2
common-voice
tatoeba2 | common-voice | |
---|---|---|
47 | 66 | |
669 | 3,250 | |
1.5% | 0.5% | |
0.0 | 10.0 | |
1 day ago | 1 day ago | |
PHP | TypeScript | |
GNU Affero General Public License v3.0 | Mozilla Public License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
tatoeba2
-
The AI Revolution Is Crushing Thousands of Languages
Alternate take, it can also help people learn niche languages if native speakers contribute to data sets. For example, I've been using Clozemaster for the past few months as a way to work on vocabulary on some languages, and they pull their dataset from Tatoeba [1]. I was very surprised to see that my father's native language, Kabylie, which is admittedly a somewhat niche language, is one of the top languages by sentence contribution in the dataset (over 700k entries, more than French or Spanish or German). I showed him the sentences once and he confirmed that yes, they all seem like what a native speaker would say. Not all of them have translations into other languages of course, and a lot of them are slight variations on each other, but some native speakers are there contributing. It's not currently an option to use in Clozemaster -- I'm guessing the TTS isn't really there -- but I totally could see these as gaps that are easily filled.
Same with my wife's native language (Bengali). There are surprisingly few language learning resources for Bangla, even though it's the 7th most spoken language in the world. But there it is in the data set with TTS and the ability for Clozemaster to have ChatGPT "explain" what's going on in the sentence (a very useful feature for new speakers).
Anyway, I don't view AI as good or bad, just another tool that we should be intentional about when we cultivate the data sets underlying the tool.
[1] https://tatoeba.org
-
Where can I find reliable example sentences?
Maybe on tatoeba.org with filters
-
Best vocab (not writing) app
I use both. I make a lot of my own cards so I get to focus on the vocab I want. Generally find a word I want to learn, use https://forvo.com/ to find native audio for it, then use https://tatoeba.org/ to find sentences use that word. Once you get a bit of practise it's pretty quick to make a word note, then make 2 or 3 sentence notes for it*. However I do use some pre-made decks like this set of sentence decks for each HSK level with native audio: https://ankiweb.net/shared/byauthor/933449107
-
Anyone else spend heaps of time searching for sentences for Anki?
You can try tatoeba https://tatoeba.org but I don't know if it's good with arabic ...
-
GPT-4's toki pona capabilities
tatoeba if anything because that has sentences so at least a modicum of context
-
Is there an app or website where I can paste a word/phrase and get examples of how it’s used in a sentence?
I use Tatoeba https://tatoeba.org : it's a collection of phrases with sometimes translations and audio recordings. You could use Forvo but it's only audio recordings.
- maneiras de falar "no pasa nada / it's okay/all right" em BR-PT?
- How do I get audio data from from native speakers for Anki?
-
anyone know a site like Reverso but for simpler sentences?
As someone else suggested, Tatoeba is also a good option. Nowadays, I use it less and less because I prefer the more didactic sentences found on online dictionaries. Nonetheless, it's still very good, especially due to the sheer quantity of sentences you can find there.
-
Nihongo Lessons has launched on the App Store
Appearances in the Tatoeba example sentence database.
common-voice
-
OpenAI's Whisper is another case study in Colonisation
Mozillas Common Voice Project (https://commonvoice.mozilla.org/) is creating an open dataset for many minority languages to make it easier to support them in STT systems. If you speak one of these languages please consider donating a few minutes of your voice.
- Mozilla Launching a Public Voice Dataset
-
Common Voice
> it was not at all obvious to me there was some way of speeding up getting a language in the first place.
Yeah, that's the biggest failing of Common Voice in my opinion. Getting a new language up to speed could be much improved by simply adding a few links to documentation, but even the existing links are broken, which I reported in March 2022... https://github.com/common-voice/common-voice/issues/3637
> I have no interest in wasting time contributing to a UI translation I actively don't want to be subjected to
Translating the UI may still help you get other people to record, even if you don't want to use it yourself.
> I'll see if I can submit some sentences at least
If you want to go faster, there's also a project to extract sentences from Wikipedia etc. in small doses Mozilla's lawyers and Wikimedia's lawyers have agreed are fair use. I think you'd only need to define how Norwegian Bokmål separates sentences. (E.g. after a period but not if it's a common abbreviation like "etc." in the preceding sentence.)
- Practice speaking and listening of your target language on Common Voice
-
Web Speech API is (still) broken on Linux circa 2023
There is a lot of TTS and SST development going on (https://github.com/mozilla/TTS; https://github.com/mozilla/DeepSpeech; https://github.com/common-voice/common-voice). That is the only way they work: Contributions from the wild.
- How do I get audio data from from native speakers for Anki?
-
Web Speech API is not available in the Quest browser
Since you're interested in STT and TTS, let me just plug in Mozilla's Common Voice, a way for everyone to contribute to an open source data set for STT. You can record yourself or verify other people's recordings!
-
Mozilla Common Voice - Korean Language is live - Help Build a Korean Corpus for Training AI/Navi/etc
[커먼보이스 전자우편](mailto:[email protected]) || Common Voice || Korean Language Homepage || FAQs || Speaking Aloud and Reviewing Recordings || Sentence Collector || NVidia/NeMo
-
Ask HN: Open-source video transcribing software?
How can it be used for transcription?
In their website I only see an interface for either uploading audio or submitting transcriptions:
https://commonvoice.mozilla.org/es
The Github repo they mention (https://github.com/common-voice/common-voice) seems to be just that sample collection software. I do not see where I can download the software to transcribe audio.
-
[D] Will continuous model development inevitably lead to data leakage and overfitting?
A practical example: in the speech community, getting good results on datasets such as TIMITwould not be revolutionary, since it's super small and very old. On the contrary, things like Common Voice (which is constantly crowd-sourced) would be much more impactful. Just my two cents :)
What are some alternatives?
gutensearch - Search engine for Project Gutenberg books
TTS - 🐸💬 - a deep learning toolkit for Text-to-Speech, battle-tested in research and production
river-runner - Uses USGS/MERIT Basin data to visualize the path of a rain droplet to its endpoint.
vosk-server - WebSocket, gRPC and WebRTC speech recognition server based on Vosk and Kaldi libraries
FrequencyWords - Repository for Frequency Word List Generator and processed files
DeepSpeech - DeepSpeech is an open source embedded (offline, on-device) speech-to-text engine which can run in real time on devices ranging from a Raspberry Pi 4 to high power GPU servers.
rum - Simple, decomplected, isomorphic HTML UI library for Clojure and ClojureScript
NeMo - A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)
PaddleSpeech - Easy-to-use Speech Toolkit including Self-Supervised Learning model, SOTA/Streaming ASR with punctuation, Streaming TTS with text frontend, Speaker Verification System, End-to-End Speech Translation and Keyword Spotting. Won NAACL2022 Best Demo Award.
forced-alignment-tools - A collection of links and notes on forced alignment tools
DeepSpeech-Italian-Model - Tooling for producing Italian model (public release available) for DeepSpeech and text corpus
TTS - :robot: :speech_balloon: Deep learning for Text to Speech (Discussion forum: https://discourse.mozilla.org/c/tts)