common-voice VS AIDungeon

Mozillas Common Voice Project (https://commonvoice.mozilla.org/) is creating an open dataset for many minority languages to make it easier to support them in STT systems. If you speak one of these languages please consider donating a few minutes of your voice.

> it was not at all obvious to me there was some way of speeding up getting a language in the first place.

Yeah, that's the biggest failing of Common Voice in my opinion. Getting a new language up to speed could be much improved by simply adding a few links to documentation, but even the existing links are broken, which I reported in March 2022... https://github.com/common-voice/common-voice/issues/3637

> I have no interest in wasting time contributing to a UI translation I actively don't want to be subjected to

Translating the UI may still help you get other people to record, even if you don't want to use it yourself.

> I'll see if I can submit some sentences at least

If you want to go faster, there's also a project to extract sentences from Wikipedia etc. in small doses Mozilla's lawyers and Wikimedia's lawyers have agreed are fair use. I think you'd only need to define how Norwegian Bokmål separates sentences. (E.g. after a period but not if it's a common abbreviation like "etc." in the preceding sentence.)

There is a lot of TTS and SST development going on (https://github.com/mozilla/TTS; https://github.com/mozilla/DeepSpeech; https://github.com/common-voice/common-voice). That is the only way they work: Contributions from the wild.

Since you're interested in STT and TTS, let me just plug in Mozilla's Common Voice, a way for everyone to contribute to an open source data set for STT. You can record yourself or verify other people's recordings!

[커먼보이스 전자우편](mailto:[email protected]) || Common Voice || Korean Language Homepage || FAQs || Speaking Aloud and Reviewing Recordings || Sentence Collector || NVidia/NeMo

How can it be used for transcription?

In their website I only see an interface for either uploading audio or submitting transcriptions:

https://commonvoice.mozilla.org/es

The Github repo they mention (https://github.com/common-voice/common-voice) seems to be just that sample collection software. I do not see where I can download the software to transcribe audio.

A practical example: in the speech community, getting good results on datasets such as TIMITwould not be revolutionary, since it's super small and very old. On the contrary, things like Common Voice (which is constantly crowd-sourced) would be much more impactful. Just my two cents :)

I have a GPT-Neo AI-Dungeon game in my host. Don't know how to take that to do the same project as yours. But would be interesting. https://github.com/Latitude-Archives/AIDungeon

Its fine-tuning training data is here and it is in the form of choose your own adventures. Programs involving text parsing couldn't be used as training data because it requires the player to type something in.

For what it's worth, the code for the original version of AI Dungeon can be found on the AI Dungeon Github page. If the language stats at the bottom of the page are accurate, then it would seem that, at the very least, the original version of AI Dungeon was primarily written in Python. This is the only info I could find relating to what programming language was used for AID.

Is there a new version of this (or does anyone have a ZIP I could download)? The most recent version here (https://github.com/Latitude-Archives/AIDungeon) is completely broken for python 3.9, tensorflow 2.6, and with my computer's OS (ubuntu 21). It also only includes a .torrent but I have no idea how to make that into the json model.

Its open source, so you can go and look for yourself to see exactly how it was done: https://github.com/Latitude-Archives/AIDungeon

Well, the base GPT-3 model, made by OpenAI, was just scraped text data from all around the internet. GPT-3 was trained on more text than what any person could realistically read in their lifetime. I'm pretty sure that even OpenAI, themselves, don't really know what exactly the AI was trained with. As for Latitude, they finetuned their models on a 30MB dataset, which you can download and look at here. Their finetuning dataset consists of a total of about 90 CYOA games from the website, ChooseYourStory.

Warning NSFW/NSFL AID training material

AI Dungeon's "Censored Words"