open_clip
stable-diffusion
open_clip | stable-diffusion | |
---|---|---|
28 | 383 | |
8,571 | 65,739 | |
4.7% | 1.3% | |
8.2 | 0.0 | |
6 days ago | about 5 hours ago | |
Jupyter Notebook | Jupyter Notebook | |
GNU General Public License v3.0 or later | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
open_clip
- FLaNK AI Weekly for 29 April 2024
-
A History of CLIP Model Training Data Advances
While OpenAI’s CLIP model has garnered a lot of attention, it is far from the only game in town—and far from the best! On the OpenCLIP leaderboard, for instance, the largest and most capable CLIP model from OpenAI ranks just 41st(!) in its average zero-shot accuracy across 38 datasets.
-
How to Build a Semantic Search Engine for Emojis
Whenever I’m working on semantic search applications that connect images and text, I start with a family of models known as contrastive language image pre-training (CLIP). These models are trained on image-text pairs to generate similar vector representations or embeddings for images and their captions, and dissimilar vectors when images are paired with other text strings. There are multiple CLIP-style models, including OpenCLIP and MetaCLIP, but for simplicity we’ll focus on the original CLIP model from OpenAI. No model is perfect, and at a fundamental level there is no right way to compare images and text, but CLIP certainly provides a good starting point.
-
Database of 16,000 Artists Used to Train Midjourney AI Goes Viral
It is a misconception that Adobe's models have not been trained on copyrighted work. Nobody should be repeating their marketing claims.
Adobe has not shown how they train the text encoders in Firefly, or what images were used for the text-based conditioning (i.e. "text to image") part of their image generation model. They are almost certainly using CLIP or T5, which are trained on LAION2b, an image dataset with the very problems they are trying to address, C4 (a text dataset similarly encumbered) and similar.
I welcome anyone who works at Adobe to simply answer this question of how they trained the text encoders for text conditioning and put it to rest. There is absolutely nothing sensitive about the issue, unless it exposes them in a lie.
So no chance. I think it's a big fat lie. They'd have to have made some other scientific breakthrough, which they didn't.
Using information from https://openai.com/research/clip and https://github.com/mlfoundations/open_clip, it's possible to investigate the likelihood that using just their stock image dataset, can they make a working text encoder?
It's certainly not impossible, but it's impracticable. On 248m images (roughly the size of Adobe Stock), CLIP gets 37% on ImageNet, and on the 2000m from LAION, it performs 71-80%. And even with 2000m images, CLIP is substantially worse performing than the approach that Imagen uses for "text comprehension," which relies on essentially many billions more images and text tokens.
-
MetaCLIP – Meta AI Research
https://github.com/mlfoundations/open_clip/blob/main/docs/op...
-
COMFYUI SDXL WORKFLOW INBOUND! Q&A NOW OPEN! (WIP EARLY ACCESS WORKFLOW INCLUDED!)
in the modal card it says: pretrained text encoders (OpenCLIP-ViT/G and CLIP-ViT/L).
-
Is Nicholas Renotte a good guide for a person who knows nothing about ML?
also, if you describe your task a bit more, we might be able to direct you to a fairly out-of-the-box solution, e.g. you might be able to use one of the pretrained models supported by https://github.com/mlfoundations/open_clip without any additional training
-
Generate Image from Vector Embedding
It says on the Stable Diffusion Github repo that it uses the “OpenCLIP-ViT/H” https://github.com/mlfoundations/open_clip model as a text encoder, and from my prior experience with CLIP, I have found that it is very easy to generate image and text embeddings (because CLIP is a multimodal model).
-
What's up in the Python community? – April 2023
https://replicate.com/pharmapsychotic/clip-interrogator
using:
cfg.apply_low_vram_defaults()
interrogate_fast()
I tried lighter models like vit32/laion400 and others etc all are very very slow to load or use (model list: https://github.com/mlfoundations/open_clip)
I'm desperately looking for something more modest and light.
-
Low accuracy on my CNN model.
A library that is very useful for this kind of application is timm. You may also find the feature representation provided by a CLIP model particularly powerful.
stable-diffusion
-
Top 7 Text-to-Image Generative AI Models
Stable Diffusion: It is based on a kind of diffusion model called a latent diffusion model, which is trained to remove noise from images in an iterative process. It is one of the first text-to-image models that can run on consumer hardware and has its code and model weights publicly available.
-
Go is bigger than crab!
Which is a 1-click install of Stable Diffusion with an alternative web interface. You can choose a different approach but this one is pretty simple and I am new to this stuff.
-
Why & How to check Invisible Watermark
an invisible watermarking of the outputs, to help viewers identify the images as machine-generated.
-
How to create an Image generating AI?
It sounds like you just want to set up Stable Diffusion to run locally. I don't think your computer's specs will be able to do it. You need a graphics card with a decent amount of VRAM. Stable diffusion is in Python as is almost every AI open source project I've seen. If you can get your hands on a system with an Nvidia RTX card with as much VRAM as possible, you're in business. I have an RTX 3060 with 12 gigs of VRAM and I can run stable diffusion and a whole variety of open source LLMs as well as other projects like face swap, Roop, tortoise TTS, sadtalker, etc...
-
Two video cards...one dedicated to Stable Diffusion...the other for everything else on my PC?
Use specific GPU on multi GPU systems · Issue #87 · CompVis/stable-diffusion · GitHub
- Automatic1111 - Multiple GPUs
- Ist Google inzwischen einfach unbrauchbar?
-
Why are people so against compensation for artists?
I dealt with this in one of my posts. At least SD 1.1 till 1.5 are all trained on a batch size of 2048. The version pretty much everyone uses (1.5) is first pretrained at a resolution of 256x256 for 237K steps on laion2B-en, at the end of those training steps it will have seen roughly 500M images in laion2B-en. After that it is pre-trained for 194K steps on laion-high-resolution at a resolution of 512x512, which is a subset of 170M images from laion5B. Finally it is trained for 1.110K steps on LAION aesthetic v2 5+. This is easily verified by taking a glance at the model card of SD 1.5. Though that one doesn't specify for part of the training exactly which aesthetic set was used for part of the training, for that you have to look at the CompVis github repo. Thus at the end of it all both the most recent images and the majority of images will have come from LAION aesthetic v2 5+ (seeing every image approx 4 times). Realistically a lot of the weights obtained from pretraining on 2B will have been lost, and only provided a good starting point for the weights.
-
Is SDXL really open-source?
stable diffusion · CompVis/stable-diffusion@2ff270f · GitHub
- I want to ask the AI to draw me as a Pokemon anime character then draw six of Pokemon of my choice next to me. What are my best free, 15$ or under and 30$ or under choices?
What are some alternatives?
CLIP - CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image
GFPGAN - GFPGAN aims at developing Practical Algorithms for Real-world Face Restoration.
DALLE-pytorch - Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch
Real-ESRGAN - Real-ESRGAN aims at developing Practical Algorithms for General Image/Video Restoration.
taming-transformers - Taming Transformers for High-Resolution Image Synthesis
diffusers-uncensored - Uncensored fork of diffusers
Dreambooth-Stable-Diffusion - Implementation of Dreambooth (https://arxiv.org/abs/2208.12242) with Stable Diffusion
diffusers - 🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch and FLAX.
bitsandbytes - Accessible large language models via k-bit quantization for PyTorch.
VQGAN-CLIP - Just playing with getting VQGAN+CLIP running locally, rather than having to use colab.
clip-retrieval - Easily compute clip embeddings and build a clip retrieval system with them
onnx - Open standard for machine learning interoperability