clean-discord VS doccano

Splitting up the data currently isn't in the plans yet; if someone (or you) could create a classifier (NOTE: please, please optimize it, the amount of data to process here is not trivial) to split the data into the relevant groups, go ahead and create a branch and pull request to https://github.com/JEF1056/clean-discord and hopefully I can do that for the next release, which is slated toward the end of the year. I can provide small snippets of some of the raw JSON data so you can understand how it's formatted.

Included in the raw data are a lot of unwanted, non-language behaviors; these include massive blocks of code, bot commands, bot messages, ASCII art, messages with only images attached, messages containing only unicode (non-standard) spaces, etc. Other messages are considered "toxic", e.g. falling under the categories: toxic, obscene, threatening, insulting, or identity hate. The regex and cleaning code can be found here.

Huh?

I wrote my own system for classifying a stream of texts in Python, I might Open Source it one of these days but I have to get it to the point where it is modular enough that I can customize it to do the particular things I want without subjecting people to my whims... I use it every day and I'm not afraid to demo it because it is rock solid.

My understanding is that my system would not be hard to adapt to work on images for certain kinds of tasks.

Pytorch is open source, Huggingface is open source. CUDA isn't. This is

https://labelstud.io/

and for annotating text spans there are so many open source tools

https://github.com/doccano/doccano

I worked for a company a few years back that built annotation tools for projects we sold to customers but never quite got to a polished general purpose annotator. Today there are an overwhelming number of companies in this space and products I never heard of, many of which are cloud based or paid. Looks like a gold rush to me.

I followed the guide to have doccano https://github.com/doccano/doccano setup in a notebook server in Kubeflow. It is running and the django connection is established, but it fails to connect with my localmachine so when I try to open the link, it does not respond. Is there a way to connect apps running on remove notebook servers to local in Kubeflow?

Why not just use the docker instructions on the GitHub page? https://github.com/doccano/doccano

doccano is built on a Django backend though I believe it uses extensive front end code for the annotation UI.

I worked at a place where we developed a system for doing this kind of tagging but it was for making training sets and there was no expectation that you could export the document from the system for normal use.

Quite a few NLP annotation systems are out there

https://prodi.gy/

https://brat.nlplab.org/

https://doccano.herokuapp.com/