rmarkdown VS List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words

I'm surprised to see no one has pointed out [RMarkdown + RStudio](https://rmarkdown.rstudio.com) as one way to immediately interface with Pandoc.

I used to write papers and slides in LaTeX (using vim, because who needs render previews), then eventually switched to Pandoc (also vim). I eventually discovered RMarkdown+RStudio. I was looking for a nice way to format a simple table and discovered that rmarkdown had nice extensions of basic markdown (this was many years ago so maybe that is incorporated into vanilla markdown/pandoc).

The RMarkdown page claims:

> R Markdown supports dozens of static and dynamic output formats including HTML, PDF, MS Word, Beamer, HTML5 slides, Tufte-style handouts, books, dashboards, shiny applications, scientific articles, websites, and more.

...which I think is largely due to using pandoc as the core generator.

RStudio shows you the pandoc command it runs to generate your document, which I've used to figure out the pandoc command I want to run when I've switched to using pandoc directly.

This is a bit of a "lazy" way to interact with pandoc. Maybe the "laziest" aspect: when I get a new computer, I can install the entire stack by installing Rstudio, then opening a new rmarkdown document. Rstudio asks whether I'd like to install all the necessary libraries -- click "yes" and that's it. Maybe that sounds silly but it used to be a lot of work to manage your LaTeX install. These days I greatly favor things that save me time, which seems to get more precious every year.

Then, I worked on a Shiny project where I had to learn R Markdown. I was very excited about it because being paid to learn a new technology is something I have always preferred. I also worked with Highcharts graphs, which I didn’t do for years. It was also the first time I was being paid to design something. I didn’t enjoy that part as much as development, but I cannot say it was a bother either.

files/figure-latex/unnamed-chunk-2-1.pdf) Try to find the following text in midterm-question.Rmd: . See https://github.com/rstudio/rmarkdown/issues/385 for more info.

We used R Markdown for cleaning and analysis, creating updateable web pages we could share with everyone involved. Similarweb’s categories were useful, but too niche for us. So we spent a lot of time recategorizing and redefining the groupings. We used the token count for each website — how many words or phrases — to measure it’s importance in the overall training data.

In [R Markdown](https://rmarkdown.rstudio.com/) or [Quarto](https://quarto.org/), I can include inline code in a math equation, e.g.,

you probably want an rmd file with html output

GitHub issue code folding

R Markdown / Quarto https://quarto.org/ https://rmarkdown.rstudio.com/ ; can dynamically generate a document and compile it to HTML, PDF, others

Hi! At this moment I'm not opening the source code, but I can explain you the tech used. This site is based on another site I created before named https://r-charts.com/ and it was created with blogdown (HUGO + R Markdown). Hence, each tutorials is an R markdown file. For PYTHON CHARTS, in order to run Python within an R markdown file I had to use an R package named reticulate. In addition, the template depends on shuffle.js for filtering and fuse.js for searching

Good point. I am already checking against the naughty-words list from here:

https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and...

https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words is one

We know that C4 was used to train Google’s influential T5 model, Facebook’s LLaMA, as well as the open source model Red Pajama. C4 is a very cleaned-up version of a scrape of the internet from the non-profit CommonCrawl taken in 2019. OpenAI’s model GPT-3 used a training dataset that began with 41 scrapes of the web from CommonCrawl from 2016 to 2019 so I think it’s safe to say that something akin to C4 was part of GPT-3. (The researchers who originally looked into C4 argue that these issues are common to all web-scraped datasets.) When we reached out to OpenAI and Google for comment, both companies emphasized that they undergo extensive efforts to weed out potentially problematic data from their training sets. But within the industry, C4 is known as being a heavily filtered dataset and has been criticized, in fact, for eliminating content related to LGBTQ+ identities because of its reliance on a heavy-handed blocklist. (https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words ) We are working on some reporting to try to address your last and very crucial question, but it’s an open area of research and one that even AI developers are struggling to answer.

"The Colossal Clean Crawled Corpus, used to train a trillion parameter LM in , is cleaned, inter alia, by discarding any page containing one of a list of about 400 “Dirty, Naughty, Obscene or Otherwise Bad Words”. This list is overwhelmingly words related to sex, with a handful of racial slurs and words related to white supremacy (e.g. swastika, white power) included. While possibly effective at removing documents containing pornography (and the associated problematic stereotypes encoded in the language of such sites) and certain kinds of hate speech, this approach will also undoubtedly attenuate, by suppressing such words as twink, the influence of online spaces built by and for LGBTQ people. If we filter out the discourse of marginalized populations, we fail to provide training data that reclaims slurs and otherwise describes marginalized identities in a positive light"

from "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? " https://dl.acm.org/doi/10.1145/3442188.3445922

That list of words is https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and...

Yeah, This is shutterstocks one which they shared

I always turn off the chatfilter, so defo let them choose if they want to have it censored or not. For the actual words themselves, there are plenty of lists out there that you can use (like this one). Although these are just regular words, none of the circumvention methods are included

That it's very crudely filtered for naughty words. According to the paper, "We removed any page that contained any word on the “List of Dirty, Naughty, Obscene or Otherwise Bad Words”." That list is here. While it contains a lot of unquestionably ugly words, it also contains words like "tit".

No problem! I wrote a short json file and Swift script to remove the nsfw words from the prompt during the image generation process, therefore it's not based on the negative prompt. The json file is a txt full with nsfw words so the app can check and remove unwanted prompts, e.g.: https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words

This is the closest I’ve come to finding one. It’s not that great.