LAVIS VS examples

For multimodal, there's China (*many), then Salesforce.

Been trying for all day to get a working inference for this example: https://github.com/salesforce/LAVIS/tree/main/projects/instructblip

There's also been big breakthroughs in computer vision. Not that long ago it was hard to recognize if a photo contained a bird; that's solved now by models like CLIP, Yolo, or Segment Anything. Now research has moved on to generating 3D scenes from images or interactively answering questions about images.

The new hotness is multimodal vision-language models like InstructBLIP that can interactively answer questions about images. Check out the examples in the github repo, I would not have thought this was possible a few years ago.

Salesforce’s BLIP family has a new member– InstructBLIP, a vision-language instruction-tuning framework using BLIP-2 models. It has achieved state-of-the-art zero-shot generalization performance on a wide range of vision-language tasks, substantially outperforming BLIP-2 and Flamingo. (Source)

Github

Most of my workflows are self-made. For captioning I used Blip-2 in a custom script I made that automates the process by going into directories and their sub-directories and creates a .txt file beside each image. This way I can keep my images organized in their proper directories, without having to put dump them all in a single place.

For “cloud-native” apps, JuiceFS is not needed.

S3 is not designed for intensive metadata operations, like listing, renaming etc. For these operations, you will need a somewhat POSIX-complaint system. For example, if you want to train on ImageNet dataset, the “canonical” way [1] is to extract the images and organize them into folders, class by class. The whole dataset is discovered by directory listing. This where JuiceFS shines.

Of course, if the dataset is really massive, you will mostly end-up with in-house solutions.

[1]: https://github.com/pytorch/examples/blob/main/imagenet/extra...

Pytorch includes a simple neural network example for the MNIST data: https://github.com/pytorch/examples/blob/main/mnist/main.py

It only takes a few minutes to train with default parameters and will have >99% accuracy on the MNIST test set.

pytorch-examples

Pretty much just grab one of these, swap in your own database, go home early: https://pytorch.org/examples/

[5] https://github.com/pytorch/examples/tree/main/word_language_...

The cherry on top of the cake I've added my father's name at the top of the code in the comments. I hope that for the next upcoming 200-300 years, someone will read modify and improve or perform experiments with my code.(Vivek V patel), My code can be found at official PyTorch's Website https://pytorch.org/examples/(Image Classification Using Forward-Forward Algorithm)

I am straying out of my domain knowledge to attempt a basic reinforcement learning task in a toy environment and have become fairly familiar with the REINFORCE algorithm for policy gradient agents, especially PyTorch’s implementation (found here). It is clear to me now that there are superior methods to train RL agents (PPO for instance), but as I read, these feel beyond my current intellectual or time resources. As such, I’d like to eek out as much power through modifications of REINFORCE as possible before determining how I might move on.

import torch import torch.nn as nn import torch.nn.functional as F # Simplified version of https://github.com/pytorch/examples/blob/main/mnist/main.py#L21 class Net(nn.Module): def __init__(self): super(Net, self).__init__() self.conv1 = nn.Conv2d(1, 32, 3, 1) def forward(self, x): x = self.conv1(x) output = F.relu(x) return output