clean-discord VS datasets

Splitting up the data currently isn't in the plans yet; if someone (or you) could create a classifier (NOTE: please, please optimize it, the amount of data to process here is not trivial) to split the data into the relevant groups, go ahead and create a branch and pull request to https://github.com/JEF1056/clean-discord and hopefully I can do that for the next release, which is slated toward the end of the year. I can provide small snippets of some of the raw JSON data so you can understand how it's formatted.

Included in the raw data are a lot of unwanted, non-language behaviors; these include massive blocks of code, bot commands, bot messages, ASCII art, messages with only images attached, messages containing only unicode (non-standard) spaces, etc. Other messages are considered "toxic", e.g. falling under the categories: toxic, obscene, threatening, insulting, or identity hate. The regex and cleaning code can be found here.

I tried Librispeech, a very common dataset for speech recognition, in both HF and TFDS.

TFDS performed extremely bad.

First it failed because the official hosting server only allows 5 simultaneous connections, and TFDS totally ignored that and makes up to 50 simultaneous downloads and that breaks. I wonder if anyone actually tested this?

Then you need to have some computer with 30GB to do the preparation, which might fail on your computer. This is where I stopped. https://github.com/tensorflow/datasets/issues/3887. It might be fixed now but it took them 8 months to respond to my issue.

On HF, it just worked. There was a smaller issue in how the dataset was split up but that is fixed now, and their response was very fast and great.

but it looks like none of your plants are supported by the plantvillage model, or do I understand something wrong? https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/image_classification/plant_village.py#L57

To do our example, we're going to use some audio files released by Google.