toxicity VS hate-speech-and-offensive-language

I'm a PM at a human data company (https://www.surgehq.ai) that helps the large language model companies ensure their models are safe (we're the “clever prompt engineers” who helped Redwood assess their model performance).

We actually just published a blog today that includes our perspective on building “AI red teams” and best practices for AI alignment/safety: https://www.surgehq.ai/blog/ai-red-teams-for-adversarial-tra...

I'd love to chat. Want to reach out to the email in my profile? I'm the founder of a much higher-quality data startup (https://www.surgehq.ai), and previously built the human computation platforms at a couple FAANGs.

We work with a lot of the top AI/NLP companies and research labs, and do both the "typical" data labeling work (sentiment analysis, text categorization, etc), but also a lot more advanced stuff (e.g., training coding assistants, evaluating the new wave of large language models, adversarial labeling, etc -- so not just distinguishing cats and dogs, but rather making full use of the power of the human mind!).

> Rather than operating under a strict definition of toxicity, we asked our team to identify comments that they personally found toxic.

[0]: https://github.com/surge-ai/toxicity

Love language? So do we, and our mission is to infuse AI with that same love. At Surge, we're building the human infrastructure to power NLP — from detecting hate speech, to parsing complex documents, to injecting human values into the next wave of language models. Our first product is a platform that helps ML teams create amazing, human-powered datasets to train AI in the richness of language. We're a team of former Google, Facebook, and Airbnb engineering leads, and we work with top companies at the forefront of machine learning. Our tech stack is Ruby on Rails, React, and Python. We’re rapidly growing, and we're looking for full-stack engineers to join the team and develop our product. To apply, please email [email protected] with a resume and 2-3 sentences describing your interest in Surge. We love personal projects and writings too!

More information: https://www.surgehq.ai/about#careers

A blog post explaining the problems we are working to solve: https://www.surgehq.ai/blog/the-ai-bottleneck-high-quality-h...

I've used NRCLex to perform sentiment analysis on some Twitter data. I have hate speech classifier code (https://github.com/t-davidson/hate-speech-and-offensive-language/blob/master/classifier/final_classifier.ipynb) I want to pass the dataset through, but before I can I need to have a "class" column for the model. For those not familiar, NRCLex returns scores for 10 emotions: anticipation, joy, anger, fear, surprise, disgust, positive, negative, sadness and trust. The table looks like this (letters denoting emotions):

Some of this exists, and both Quora and Facebook (among others) use it extensively. Both hate speech and porn are good targets for machine learning. It needs supervision, but it can take a lot of load off human moderators.

Open source implementations exist, e.g.:

https://github.com/t-davidson/hate-speech-and-offensive-lang...

I suspect more message board will want to start applying these sooner rather than later. Most have already figured out that they need anti-spam tools, rather than it coming as a surprise when they roll things out and it fills up with bots. The technology is similar.

You mention being able to share that information across boards, and I don't know of any widespread implementation of that. You can, at least, let somebody else handle your authentication, which slightly slows their ability to create new accounts when you blacklist one. I'd like to see those sites distinguish "aged" accounts, so that it at least takes some effort or cost to use a new account.