Recommended open LLMs with image input modality?

• Awesome-Multimodal-Large-Language-Models

  2 8,991 9.7

  :sparkles::sparkles:Latest Papers and Datasets on Multimodal Large Language Models, and Their Evaluation.

https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Evaluation this is pretty comprehensive. tldr; blip is probably the best, though i've heard it does need a lot of vram. In my experience its the most responsive to prompt engineering.

• unilm

  40 18,358 9.0 Python

  Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

  

It is missing kosmos-2. I remember its image captioning was(demo currently down) really good and it's almost as fast as llava and lavin.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• instructblip-pipeline

  1 27 4.1 Python

  A multimodal inference pipeline that integrates InstructBLIP with textgen-webui for Vicuna and related models.

I've been using it in oobabooga. There's a repo for the extension here: https://github.com/kjerk/instructblip-pipeline/tree/main

Suggest a related project