LLaVA
CogVLM
Our great sponsors
LLaVA | CogVLM | |
---|---|---|
20 | 16 | |
16,101 | 4,968 | |
- | 13.5% | |
9.4 | 9.0 | |
6 days ago | 12 days ago | |
Python | Python | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
LLaVA
-
Show HN: I Remade the Fake Google Gemini Demo, Except Using GPT-4 and It's Real
Update: For anyone else facing the commercial use question on LLaVA - it is licensed under Apache 2.0. Can be used commercially with attribution: https://github.com/haotian-liu/LLaVA/blob/main/LICENSE
-
Image-to-Caption Generator
https://github.com/haotian-liu/LLaVA (fairly established and well supported)
-
Llamafile lets you distribute and run LLMs with a single file
That's not a llamafile thing, that's a llava-v1.5-7b-q4 thing - you're running the LLaVA 1.5 model at a 7 billion parameter size further quantized to 4 bits (the q4).
GPT4-Vision is running a MUCH larger model than the tiny 7B 4GB LLaVA file in this example.
LLaVA have a 13B model available which might do better, though there's no chance it will be anywhere near as good as GPT-4 Vision. https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZO...
- FLaNK Stack Weekly for 27 November 2023
-
Using GPT-4 Vision with Vimium to browse the web
There are open source models such as https://github.com/THUDM/CogVLM and https://github.com/haotian-liu/LLaVA.
-
Is supervised learning dead for computer vision?
Hey Everyone,
I’ve been diving deep into the world of computer vision recently, and I’ve gotta say, things are getting pretty exciting! I stumbled upon this vision-language model called LLaVA (https://github.com/haotian-liu/LLaVA), and it’s been nothing short of impressive.
In the past, if you wanted to teach a model to recognize the color of your car in an image, you’d have to go through the tedious process of training it from scratch. But now, with models like LLaVA, all you need to do is prompt it with a question like “What’s the color of the car?” and bam – you get your answer, zero-shot style.
It’s kind of like what we’ve seen in the NLP world. People aren’t training language models from the ground up anymore; they’re taking pre-trained models and fine-tuning them for their specific needs. And it looks like we’re headed in the same direction with computer vision.
Imagine being able to extract insights from images with just a simple text prompt. Need to step it up a notch? A bit of fine-tuning can do wonders, and from my experiments, it can even outperform models trained from scratch. It’s like getting the best of both worlds!
But here’s the real kicker: these foundational models, thanks to their extensive training on massive datasets, have an incredible grasp of image representations. This means you can fine-tune them with just a handful of examples, saving you the trouble of collecting thousands of images. Indeed, they can even learn with a single example (https://www.fast.ai/posts/2023-09-04-learning-jumps)
-
Adept Open Sources 8B Multimodal Modal
Fuyu is not open source. At best, it is source-available. It's also not the only one.
A few other multimodal models that you can run locally include IDEFICS[0][1], LLaVA[2], and CogVLM[3]. I believe all of these have better licenses than Fuyu.
[0]: https://huggingface.co/blog/idefics
[1]: https://huggingface.co/HuggingFaceM4/idefics-80b-instruct
[2]: https://github.com/haotian-liu/LLaVA
[3]: https://github.com/THUDM/CogVLM
-
AI — weekly megathread!
Researchers released LLaVA-1.5. LLaVA (Large Language and Vision Assistant) is an open-source large multimodal model that combines a vision encoder and Vicuna for general-purpose visual and language understanding. LLaVA-1.5 achieved SoTA on 11 benchmarks, with just simple modifications to the original LLaVA and completed training in ~1 day on a single 8-A100 node [Demo | Paper | GitHub].
- LLaVA: Visual Instruction Tuning: Large Language-and-Vision Assistant
-
LLaVA gguf/ggml version
Hi all, I’m wondering if there is a version of LLaVA https://github.com/haotian-liu/LLaVA that works with gguf and ggml models?? I know there is one for miniGPT4 but it just doesn’t seem as reliable as LLaVA but you need at least 24gb of vRAM for LLaVA to run it locally by the looks of it. The 4bit version still requires 12gb vram.
CogVLM
-
Mixtral: Mixture of Experts
CogVLM is very good in my (brief) testing: https://github.com/THUDM/CogVLM
The model weights seem to be under a non-commercial license, not true open source, but it is "open access" as you requested.
-
IT Employment Grew by Just 700 Jobs in 2023, Down From 267,000 in 2022
increasing growth most places in world
https://twitter.com/elonmusk/status/1743028102446408026
heres a total feature map of what was released in 2023:
https://twitter.com/enriquebrgn/status/1740950767325024387
I think thats definitely a signal that the B and C teams werent needed, considering they cut 90% of staff LOL.
As for the bots, AI is making it easier than ever to bypass those systems. CogVLM is just sitting there menacingly on github https://github.com/THUDM/CogVLM
- Show HN: I built an open source AI video search engine to learn more about AI
-
CogAgent-18B – visual-based GUI Agent capabilities
Jump to heading for benchmarks and examples: https://github.com/THUDM/CogVLM/tree/main?tab=readme-ov-file...
-
What do you think. When should we expect the next SDXL version?
Honestly at this point there is no need for human for captioning except maybe for NSFW content. Img2text is just good enough for nearly all images. GPTVision or open source equivalent (like CogVLM https://github.com/THUDM/CogVLM ) are just good enough.
-
shinning the spotlight on CogVLM
A core Llama.cpp contributor, named cmp-nct, discovered stumbled upon what might be the next leap forward for vision/language models. CogVLM (which uses a Vicuna 7B language model combined with a 9B vision tower) excels particularly in OCR (Optical Character Recognition), detail detection, and minimal hallucinations. It effectively understands both handwritten and typed text, context, fine details, and background graphics. It even provides pixel coordinates for small visual targets. CovVLM surpasses other models like llava-1.5 and Qwen-VL in performance.
-
Image-to-Caption Generator
https://github.com/THUDM/CogVLM (really impressive)
-
Gemini: Google's most capable AI model yet
I'm researching using LLMs for alt-text suggestion for forum users, can you share your finding so far?
Outside of GPT-4V I had good first results with https://github.com/THUDM/CogVLM
-
Open-source LLMs with Image Interpretation
I've got some decent results with CogVLM. Resolution kinda sucks at 490x490, though.
- FLaNK Stack Weekly for 27 November 2023
What are some alternatives?
MiniGPT-4 - Open-sourced codes for MiniGPT-4 and MiniGPT-v2 (https://minigpt-4.github.io, https://minigpt-v2.github.io/)
ollama - Get up and running with Llama 3, Mistral, Gemma, and other large language models.
FastChat - An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
ComfyUI - The most powerful and modular stable diffusion GUI, api and backend with a graph/nodes interface.
mPLUG-Owl - mPLUG-Owl & mPLUG-Owl2: Modularized Multimodal Large Language Model
Qwen-VL - The official repo of Qwen-VL (通义千问-VL) chat & pretrained large vision language model proposed by Alibaba Cloud.
llama.cpp - LLM inference in C/C++
vimGPT - Browse the web with GPT-4V and Vimium
image2dsl - This repository contains the implementation of an Image to DSL (Domain Specific Language) model. The model uses a pre-trained Vision Transformer (ViT) as an encoder to extract image features and a custom Transformer Decoder to generate DSL code from the extracted features.
uform - Pocket-Sized Multimodal AI for content understanding and generation across multilingual texts, images, and 🔜 video, up to 5x faster than OpenAI CLIP and LLaVA 🖼️ & 🖋️
llamafile - Distribute and run LLMs with a single file.
LinkBERT - [ACL 2022] LinkBERT: A Knowledgeable Language Model 😎 Pretrained with Document Links