LLaVA VS llamafile

Update: For anyone else facing the commercial use question on LLaVA - it is licensed under Apache 2.0. Can be used commercially with attribution: https://github.com/haotian-liu/LLaVA/blob/main/LICENSE

https://github.com/haotian-liu/LLaVA (fairly established and well supported)

That's not a llamafile thing, that's a llava-v1.5-7b-q4 thing - you're running the LLaVA 1.5 model at a 7 billion parameter size further quantized to 4 bits (the q4).

GPT4-Vision is running a MUCH larger model than the tiny 7B 4GB LLaVA file in this example.

LLaVA have a 13B model available which might do better, though there's no chance it will be anywhere near as good as GPT-4 Vision. https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZO...

There are open source models such as https://github.com/THUDM/CogVLM and https://github.com/haotian-liu/LLaVA.

Hey Everyone,

I’ve been diving deep into the world of computer vision recently, and I’ve gotta say, things are getting pretty exciting! I stumbled upon this vision-language model called LLaVA (https://github.com/haotian-liu/LLaVA), and it’s been nothing short of impressive.

In the past, if you wanted to teach a model to recognize the color of your car in an image, you’d have to go through the tedious process of training it from scratch. But now, with models like LLaVA, all you need to do is prompt it with a question like “What’s the color of the car?” and bam – you get your answer, zero-shot style.

It’s kind of like what we’ve seen in the NLP world. People aren’t training language models from the ground up anymore; they’re taking pre-trained models and fine-tuning them for their specific needs. And it looks like we’re headed in the same direction with computer vision.

Imagine being able to extract insights from images with just a simple text prompt. Need to step it up a notch? A bit of fine-tuning can do wonders, and from my experiments, it can even outperform models trained from scratch. It’s like getting the best of both worlds!

But here’s the real kicker: these foundational models, thanks to their extensive training on massive datasets, have an incredible grasp of image representations. This means you can fine-tune them with just a handful of examples, saving you the trouble of collecting thousands of images. Indeed, they can even learn with a single example (https://www.fast.ai/posts/2023-09-04-learning-jumps)

Fuyu is not open source. At best, it is source-available. It's also not the only one.

A few other multimodal models that you can run locally include IDEFICS[0][1], LLaVA[2], and CogVLM[3]. I believe all of these have better licenses than Fuyu.

[0]: https://huggingface.co/blog/idefics

[1]: https://huggingface.co/HuggingFaceM4/idefics-80b-instruct

[2]: https://github.com/haotian-liu/LLaVA

[3]: https://github.com/THUDM/CogVLM

Researchers released LLaVA-1.5. LLaVA (Large Language and Vision Assistant) is an open-source large multimodal model that combines a vision encoder and Vicuna for general-purpose visual and language understanding. LLaVA-1.5 achieved SoTA on 11 benchmarks, with just simple modifications to the original LLaVA and completed training in ~1 day on a single 8-A100 node [Demo | Paper | GitHub].

Hi all, I’m wondering if there is a version of LLaVA https://github.com/haotian-liu/LLaVA that works with gguf and ggml models?? I know there is one for miniGPT4 but it just doesn’t seem as reliable as LLaVA but you need at least 24gb of vRAM for LLaVA to run it locally by the looks of it. The 4bit version still requires 12gb vram.

I think the llamafile[0] system works the best. Binary works on the command line or launches a mini webserver. Llamafile offers builds of Mixtral-8x7B-Instruct, so presumably they may package this one up as well (potentially a quantized format).

You would have to confirm with someone deeper in the ecosystem, but I think you should be able to run this new model as is against a llamafile?

[0] https://github.com/Mozilla-Ocho/llamafile

Thermostats: https://www.sinopetech.com/en/products/thermostat/

I haven't tried running a local text-to-speech engine backed by an LLM to control Home Assistant. Maybe someone is working on this already?

TTS: https://github.com/SYSTRAN/faster-whisper

LLM: https://github.com/Mozilla-Ocho/llamafile/releases

LLM: https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-D...

It would take some tweaking to get the voice commands working correctly.

While I did not succeed in making the matmul code from https://github.com/Mozilla-Ocho/llamafile/blob/main/llamafil... work in isolation, I compared eigen, openblas, and mkl: https://gist.github.com/Dobiasd/e664c681c4a7933ef5d2df7caa87...

In this (very primitive!) benchmark, MKL was a bit better than eigen (~10%) on my machine (i5-6600).

Since the article https://justine.lol/matmul/ compared the new kernels with MLK, we can (by transitivity) compare the new kernels with Eigen this way, at least very roughly for this one use-case.

Yes, they're just ZIP files that also happen to be actually portable executables.

https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file...

have you seen llamafile[0]?

[0] https://github.com/Mozilla-Ocho/llamafile

llama.cpp has integrated gemma support. So you can use llamafile for this. It is a standalone executable that is portable across most popular OSes.

https://github.com/Mozilla-Ocho/llamafile/releases

So, download the executable from the releases page under assets. You want either just main or just server. Don't get the huge ones with the model inlined in the file. The executable is about 30MB in size,

https://github.com/Mozilla-Ocho/llamafile/releases/download/...

The improvements in ease of use for locally hosting LLMs over the last few months have been amazing. I was ranting about how easy https://github.com/Mozilla-Ocho/llamafile is just a few hours ago [1]. Now I'm torn as to which one to use :)

1: Quite literally hours ago: https://euri.ca/blog/2024-llm-self-hosting-is-easy-now/

Slightly off topic, here is the best local llama.cpp wrapper I've run into:

https://github.com/Mozilla-Ocho/llamafile

You can download any .gguf model (not just the ones in their examples) and run it locally (as long as you have the ram for it). I was running 7B models with ease on an old FX8350 and now 13B models on a 5600X (32GB RAM on both machines).

This wrapper spins up a local web server that runs a simple web frontend to use immediately with no code, but also exposes an OpenAI compatible API for dev work and alt frontends (like SillyTavern).