clip-interrogator VS alpaca.cpp

It's image -> words, the inverse of stable diffusion.

see: https://github.com/pharmapsychotic/clip-interrogator

I have been using this on a CPU https://github.com/pharmapsychotic/clip-interrogator, I tried a lot of pre-trained models combinations, all are slow.

I'm addition to controlnet, try the clip interrogator to see how clip would describe the image and then use that language in your prompt. You can try the whole image or cropped portions. There is a colab available if you don't want to run it locally.

In my current process, I use CLIP Interrogator to produce a high level caption and wd14 tagger for more granular booru tags. Typically in that order, because you can append the results from the latter to the former. Both tools perform with greater accuracy than the standard interrogators in img2img and give you more flexibility and features as well. You still have to do some manual adjustments, but I generally prefer this process over starting from scratch.

If you are interested in this, definitely see if you can get some of the OSS models running and get a feel for how to interrogate them. Maybe see if you can get some mileage out of the CLIP-Interrogator

Next, I used the lists of artists, flavors, mediums, movements, and negatives that are used for the clip-interrogator and pasted these in the chat and told the bot to categorize them accordingly. As you can only paste up to certain characters in single message (4-5K in 3.5 and 6-8K in 4).

Here’s the specific one I’m using (runs in browser)

I really enjoy using the CLIP Interrogator on huggingspaces, but it is often super slow and sometimes straight up breaks. Now it is possible to locally install it, https://github.com/pharmapsychotic/clip-interrogator but I don't know if its viable to run on a laptop with 6gb videocard anyway.

Where's the 30B-in-6GB claim? ^FGB in your GH link finds [0] which is neither by jart nor by ggerganov but by another user who promptly gets told to look at [1] where Justine denies that claim.

  [0] https://github.com/antimatter15/alpaca.cpp/issues/182

You can just download the language model, dude!!! Everyone doesn’t need to make their own and the open source models literally get better every day.

Alpaca

Square could have also trained a more specific data source for their NLP, very similar to Alpaca. Alpaca was trained from interactions from a larger dataset. So while it isn't as smart, it's still able to understand instructions and act upon them.

The model I found was in 8 parts. The alpaca.cpp chat client (chat.cpp) needs to be modified to run the 8 part model, documented here: https://github.com/antimatter15/alpaca.cpp/issues/149

try the instructions on this github repo https://github.com/antimatter15/alpaca.cpp, its not the best one but I was able to run this model on my linux machine with 16GB memory, I think its a good starting point.

Doesn't run offline. If you're running something without a beefy-ish GPU, there's https://github.com/antimatter15/alpaca.cpp .

Ye, atm. toying with alpaca 7B/13B in a local install.