stable-diffusion-videos VS MultiDiffusion

Main Code: https://github.com/nateraw/stable-diffusion-videos/

So I've continued to experiment with how many papers I can fit into a single pipe and have them play nicely together. The images below were created by combining the panorama code from omerbt/MultiDiffusion with the ideas from albarji/mixture-of-diffusers. Also turns out nateraw/stable-diffusion-videos can be seen as a special case of a panorama (in latent space rather than prompt space).

https://huggingface.co/spaces/tomg-group-umd/pez-dispenser https://huggingface.co/spaces/AIML-TUDA/safe-stable-diffusion https://huggingface.co/spaces/AIML-TUDA/semantic-diffusion https://github.com/nateraw/stable-diffusion-videos

The goal is to make a (short) video out of a given first and last frame. It is similar to what this guy does (https://github.com/nateraw/stable-diffusion-videos (7sec example video half way down page)). But instead of starting and ending with a prompt, I want to start and end with 2 different frames.

Hey Y'all! We've been working on a tool that extends Nate Raw's Stable Diffusion Videos repo and makes it as easy as possible to use for artists and are having a competition this week to stress test the beta and see who can use it to make the most compelling short video (40 seconds max)

Of course! There might be a better process but mainly used: 1.) Nate Raw's repo for morphing between prompts https://github.com/nateraw/stable-diffusion-videos 2.) Google FILM interpolation to smooth out transitions https://github.com/google-research/frame-interpolation

There are directions in the readme and an example script.

This does exactly that - https://github.com/nateraw/stable-diffusion-videos

For composing this approach works pretty well

https://multidiffusion.github.io/

So I've continued to experiment with how many papers I can fit into a single pipe and have them play nicely together. The images below were created by combining the panorama code from omerbt/MultiDiffusion with the ideas from albarji/mixture-of-diffusers. Also turns out nateraw/stable-diffusion-videos can be seen as a special case of a panorama (in latent space rather than prompt space).

Essentially, I fine-tuned Stable Diffusion 2.1 base (the 512x512) model on the ESA Hubble Deep Space Images & Captions dataset I collected from public Hubble images & captions. After around 33,000 training steps, I saved the model and was really impressed by the results. But I really wanted to be able to generate wallpaper-level quality space images, so I stumbled upon MultiDiffusion: a new project for generating massive panorama images using stable diffusion models. I then used hubble-diffusion-2 along with MultiDiffusion to generate each one of these amazing 2560x1536 images. Each image took a little over an hour to generate on a Google Colab T4 GPU. I used the following prompts for each of these images:

If you need generated and not upscaled 4k for some reason, try something like https://github.com/omerbt/MultiDiffusion

Project: https://multidiffusion.github.io/ Paper: https://arxiv.org/abs/2302.08113 GitHub: https://github.com/omerbt/MultiDiffusion

Quick Read: https://www.marktechpost.com/2023/02/24/meet-multidiffusion-a-unified-ai-framework-that-enables-versatile-and-controllable-image-generation-using-a-pre-trained-text-to-image-diffusion-model/ Paper: https://arxiv.org/abs/2302.08113 Github: https://github.com/omerbt/MultiDiffusion Project: https://multidiffusion.github.io/

So I haven't made many images with Stable Diffusion despite using it heavily. The reason is I've been messing with the internals of the diffusion pipe, to interfere with the diffusion process in different ways. Todays fun result is based on omerbt/MultiDiffusion for making panoramas.

If we combine Mixture of Diffusers + MultiDiffusion+ Composer+ cross-domain-compositing and probably some more I'm not thinking of.