LAVIS VS clipseg

For multimodal, there's China (*many), then Salesforce.

Been trying for all day to get a working inference for this example: https://github.com/salesforce/LAVIS/tree/main/projects/instructblip

There's also been big breakthroughs in computer vision. Not that long ago it was hard to recognize if a photo contained a bird; that's solved now by models like CLIP, Yolo, or Segment Anything. Now research has moved on to generating 3D scenes from images or interactively answering questions about images.

The new hotness is multimodal vision-language models like InstructBLIP that can interactively answer questions about images. Check out the examples in the github repo, I would not have thought this was possible a few years ago.

Salesforce’s BLIP family has a new member– InstructBLIP, a vision-language instruction-tuning framework using BLIP-2 models. It has achieved state-of-the-art zero-shot generalization performance on a wide range of vision-language tasks, substantially outperforming BLIP-2 and Flamingo. (Source)

Github

Most of my workflows are self-made. For captioning I used Blip-2 in a custom script I made that automates the process by going into directories and their sub-directories and creates a .txt file beside each image. This way I can keep my images organized in their proper directories, without having to put dump them all in a single place.

Following the comments to this old post, I tried to use in-painting with manual mask selection. I didn't get beautiful results but I'm sure with some tweaking, I could make it better. The main problem I had was having to manually select the area where I wanted to place the logo and trying to resize my logo mask to the fit the segment. I tried some automatic segmentation tools (Clipseg and Segment Anything). I couldn't tell the segmentation models to find a good area to for logo placement (i.e. some small flat surface). Given the complexity of what I was dealing with, I think there could be a better way (XY problem).

The addon utilizes clipseg for region masking, which was trained on "an extended version of the PhraseCut dataset" (many thousands of images.)

You'll also need to make sure to update your clipseg repo. The script won't do this for you. Namely you just need to update this models/clipseg.py file to ensure your clipseg has support for the new model.

For localisation at search time I ended up using OWL-ViT. This worked really well. I did not try Detic or CLIPseg but would be interested to hear if anyone else has tried these?

clipseg is an image segmentation method used to find a mask for an image from a prompt. I implemented it as an executor for dalle-flow and added it to my bot yasd-discord-bot.

Automated Replacement (txt2mask) by clipseg