v-diffusion-pytorch VS dalle-2-preview

This is dumb.

We employed Eleuther team members as Stability AI employees/contractors and incubated them until the 501(c)3 was set up and we managed to bring in other funders too: https://techcrunch.com/2023/03/02/stability-ai-hugging-face-...

I am on the board and delighted to continue to support their work as an independent organisation for LM evaluation, alignment and interpretability which is much needed.

Indeed though our approach was handing out significant compute for no control, no equity, no IP.

Anyone who has received Stability AI grants will be able to attest to this with multiple breakthroughs as a result, for example funding https://github.com/BlinkDL/RWKV-LM, the work of https://github.com/lucidrains and others.

Similarly we funded the beta of MidJourney with a cash grant for compute without ever even floating asking for equity etc as it is a market-creating innovation.

At the time MidJourney was using cc12m_1, a model developed by one of our lead (employed) generative AI developers Katherine Crownson / RiversHaveWings (https://github.com/crowsonkb/v-diffusion-pytorch)

Our model is simply to take open innovation and create commercial variants of that (our stable series models) from scratch and on our own, plus variants of that for private data - https://twitter.com/EMostaque/status/1649152422634221593?s=2...

This means we can be hands off versus other funders and trust researchers and help them succeed, something others do not.

From what I've gathered by being involved early in the beta / in other discords, Midjourney was originally based on a fine-tuned version of classifier-free guided v-diffusion. The fine-tuning dataset was a manually curated set largely from LAION-2B similar to the laion-art / laion-hd. To make it so fast they were using Progressive Distillation (possibly distilling on PLMS steps rather than p/ddim?) and settings optimized to let them skip a few of the first steps like Quick CLIP-Guided Diffusion. There's a good chance they were doing some prompt augmentation as well, although I think this would be susceptible to prompt discovery attacks which I haven't seen any examples of for Midjourney.

Source. This tweet is from a person whose organization has been publicly credited with providing compute for others in the past (example: "Thank you to stability.ai for compute to train these models!").

com/RiversHaveWings/status/1462859669454536711, 2021.

[8] Katherine Crowson. v-diffusion. https://github.com/crowsonkb/v-diffusion-pytorch, 2021.

More parameters are available as seen in this code. The fix was adapted from this.

I'm not sure if this Colab notebook was mentioned in this sub previously, but it's been available since January 2022. The cc12m_1_cfg model used by this Colab notebook is different than the cc12m_1 model from this December 2021 post (reference).

A few more here https://twitter.com/metasemantic/status/1486334535436488705. Samples careful constructed with a heavily modified diffusion model by @rivershavewings https://github.com/crowsonkb/v-diffusion-pytorch

Most of the code was from Katherine Crowson's (@RiversHaveWings) v-diffusion-pytorch library (https://github.com/crowsonkb/v-diffusion-pytorch), which is an implementation of denoising diffusion probabilistic models (https://arxiv.org/abs/2006.11239). I used the CC12M_1 CFG checkpoint.

We believe that many decisions about our defaults and hard bounds should be made collectively, and while practical implementation is a challenge, we aim to include as many perspectives as possible. As a starting point, we’ve sought external input on our technology in the form of red teaming. We also recently began soliciting public input on AI in education (one particularly important context in which our technology is being deployed).

From this OpenAI document:

It's also of note that you can't sell a game using this method, as Dalle-2's terms of service prevent use in commercial projects. It's hard to justify rate of return considering you can only ever give it away for free, and even in that case there are some uncertain legal elements regarding copyright and the images that are used to train the dataset.

On their GitHub https://github.com/openai/dalle-2-preview/blob/main/system-card.md I can only see references to v1.

"b) Exploration of the bolded part of OpenAI's comment "Each generated image includes a signature in the lower right corner, with the goal of indicating when DALL·E 2 helped generate a certain image." (source)." (source link: https://github.com/openai/dalle-2-preview/blob/main/system-c...)

I feel the DALL-E 2 watermark signature could be a seed or something.

DALL-E's docs for example mention it can output whole copyrighted logos and characters[1] and understands it's possible to generate human faces that are bear the likeness of those in the training data. We've also seen people recently critique Stable Diffusion's output for attempting to recreate artists' signatures that came from the commercial trained data.

That said by a certain point the kinks will be ironed out and likely skirt around such issues by only incorporating/manipulating just enough to be considered fair use and creative transformation.

[1] "The model can generate known entities including trademarked logos and copyrighted characters." https://github.com/openai/dalle-2-preview/blob/main/system-c...

Some types of content were purposely excluded from the training dataset(s) (source).