Top 10 Python vision-language Projects

GroundingDINO

5 4,978 6.6 Python

Official implementation of the paper "Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection"

Project mention: Autodistill: A new way to create CV models | /r/developersIndia | 2023-09-30

Some of the foundation/base models include: * GroundedSAM (Segment Anything Model) * DETIC * GroundingDINO

marqo

114 4,111 9.3 Python

Unified embedding generation and search engine. Also available on cloud - cloud.marqo.ai

Project mention: Are we at peak vector database? | news.ycombinator.com | 2024-01-25

We (Marqo) are doing a lot on 1 and 2. There is a huge amount to be done on the ML side of vector search and we are investing heavily in it. I think it has not quite sunk in that vector search systems are ML systems and everything that comes with that. I would love to chat about 1 and 2 so feel free to email me (email is in my profile). What we have done so far is here -> https://github.com/marqo-ai/marqo

WorkOS

workos.com sponsored

The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
Chinese-CLIP

1 3,590 7.6 Python

Chinese version of CLIP which achieves Chinese cross-modal retrieval and representation generation.
OFA

3 2,323 2.8 Python

Official repository of OFA (ICML 2022). Paper: OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework
ONE-PEACE

2 838 8.6 Python

A general representation model across vision, audio, language modalities. Paper: ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities

Project mention: A general representation modal across vision, audio, language modalities | news.ycombinator.com | 2023-05-25

SEED

1 464 8.5 Python

Official implementation of SEED-LLaMA (ICLR 2024). (by AILab-CVC)

Project mention: List of Stable Diffusion research softwares that I don't think gotten widespread adoption. | /r/StableDiffusion | 2023-12-10

https://github.com/AILab-CVC/SEED -

LViT

2 249 5.9 Python

[IEEE Transactions on Medical Imaging/TMI] This repo is the official implementation of "LViT: Language meets Vision Transformer in Medical Image Segmentation"
InfluxDB

www.influxdata.com sponsored

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
image-captioning

1 29 3.1 Python

Image captioning using python and BLIP
rtic-gcn-pytorch

2 20 0.0 Python

Official PyTorch Implementation of RITC
vision-language-examples

1 7 6.4 Python

Vision-lanugage model example code.

Project mention: [D] how to accelerate ViT models more faster | /r/MachineLearning | 2023-07-13

I have conducted experiments and examples on accelerating ViT (Vision Transformer) using methods such as TensorRT, FasterTransformer, and xFormers. The experiments were conducted using a single A100 as a baseline. - https://github.com/bnabis93/vision-language-examples/tree/main/acceleration

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Python vision-language related posts

MetaCLIP – Meta AI Research
6 projects | news.ycombinator.com | 26 Oct 2023
A general representation modal across vision, audio, language modalities
1 project | news.ycombinator.com | 25 May 2023
[D] Object Detection Machine Learning
1 project | /r/MachineLearning | 25 Apr 2023
Code for RTIC Is Released
2 projects | news.ycombinator.com | 25 Oct 2021

Index

What are some of the best open-source vision-language projects in Python? This list will help you:

	Project	Stars
1	GroundingDINO	4,978
2	marqo	4,111
3	Chinese-CLIP	3,590
4	OFA	2,323
5	ONE-PEACE	838
6	SEED	464
7	LViT	249
8	image-captioning	29
9	rtic-gcn-pytorch	20
10	vision-language-examples	7