imgbeddings VS screenshot-to-code

Seconding the recommendation of CLIP embeddings, especially compared to image histograms + requiring OpenCV.

I wrote a naive, minimal dependency Python package to calculate image embeddings (https://github.com/minimaxir/imgbeddings) with some lookup demo notebooks and it works well in a pinch, although it's due for an upgrade.

Normally, this is Hacker News reductiveness, but yes, image classification via CLIP is that easy, especially with Hugging Face's API for it: https://huggingface.co/docs/transformers/model_doc/clip

I created a Python package to generate image embeddings from CLIP's vision model (without requiring a ML framework), and a simple linear classifier on those embeddings does the trick: https://github.com/minimaxir/imgbeddings

Thanks!

There's a dropdown where you can choose a stack for screenshots (Tailwind, React, Vue, etc.). I haven't updated the prompts for the video feature just yet. You can tweak the prompt yourself here: https://github.com/abi/screenshot-to-code/blob/6069c2a118592...

The quality of the output code is solid, I think. You can see the code for the examples: https://codepen.io/abi/pen/ExJPdop and https://codepen.io/abi/pen/jORWeYB

I think the biggest thing LLM code is typically missing is better abstractions/componentization. You could probably prompt around some of that.

You can use GPT-V to create functional components from images directly:

There was an open source project (https://github.com/abi/screenshot-to-code) that I borrowed the prompt from and made a custom GPT for myself where I just drag and drop the image. It’s not perfect but it’s pretty great overall!

Here are the prompts:

PS: I got the idea from tldraw/make-real and abi/screenshot-to-code projects. So all credit to them 🙌

Screenshot to Code GitHub Repo: https://github.com/abi/screenshot-to-code