nix-stable-diffusion VS BLIP

I have been playing around with a memory efficient version of [stable-diffusion](https://github.com/basujindal/stable-diffusion), an AI model that outputs images based on input text. I think it is rather neat, and [this post](https://xeiaso.net/blog/stable-diffusion-nixos) by Xeiaso (good blog for nix resources and rants) inspired me to try and make the installation process more nix-friendly. So I have forked the original [stable-diffusion repo](https://github.com/CompVis/stable-diffusion) and added a [`default.nix`](https://github.com/Danielhp95/nix-stable-diffusion/blob/main/default.nix) which dumps you in a `nix-shell` with all dependencies: cuda (nvidia) stuff, python dependencies and C++ libraries for pytorch (deep learning framework). Although **note** that in order to run it you must first download the model weights from [this repo](https://huggingface.co/CompVis/stable-diffusion-v1-4) and place them in `models/ldm/stable-diffusion-v1/model.ckpt` directory.

I suggest trying BLIP for this. I've had really good results from that.

https://github.com/salesforce/BLIP

I built a tiny Python CLI wrapper for it to make it easier to try: https://github.com/simonw/blip-caption

You can download the source and run it yourself from here: https://github.com/salesforce/BLIP

Then there's also BLIP (Bootstrapping Language-Image Pre-training).

Or BLIP

GitHub - salesforce/BLIP: PyTorch code for BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

From what I understand, what's most important to you is having a model that's already trained on something, rather than the architecture. Yolo is probably fine, as would be some of the older ones. You should be able to find a model that's been pretrained on COCO - you can look at see what classes are included. I don't know if there are other broadly trained models available that will serve your purpose. What I'd do is just run your picture through a COCO trained object detection model and see if the annotations do what you want.

Though backing up a bit, there are also image captioning models that may better do what you want to do for organizing your photos. I'm not really familiar with any - though I did come across BLIP the other day but I haven't used it: https://github.com/salesforce/BLIP

This may be a better way to get at what you want

git clone https://github.com/salesforce/BLIP.git repositories/BLIP

# Copy models as described in README cp ~/Downloads/model.ckpt . cp ~/Downloads/GFPGANv1.3.pth . # Clone other repos as mentioned in README mkdir repositories git clone https://github.com/CompVis/stable-diffusion.git repositories/stable-diffusion git clone https://github.com/CompVis/taming-transformers.git repositories/taming-transformers git clone https://github.com/sczhou/CodeFormer.git repositories/CodeFormer git clone https://github.com/salesforce/BLIP.git repositories/BLIP export NIXPKGS_ALLOW_UNFREE=1 nix-shell default.nix pip install torch --extra-index-url https://download.pytorch.org/whl/cu113 # Also from linux instructions. Can probably be added to default.nix python webui.py

Also check img2text (basically to prompt): https://github.com/salesforce/BLIP