ollama VS exllamav2

Download: Go to https://ollama.com/ and click on the download link for your operating system.

Most AI resume tools call out to OpenAI or Anthropic and charge you for every request. Persona supports Ollama — which means you can run the AI model locally on your own hardware, with zero API costs and zero data leaving your machine.

To solve these, we built a stack that prioritizes integrity over ease. The centerpiece is Ollama, running the mxbai-embed-large model locally. This is the engine that translates human thought into high-dimensional coordinates.

Ollama for model serving

Fully offline: Point it at Ollama or LM Studio. Zero cost, nothing leaves your network.

Recommended hardware: The RTX 3060 with 12 GB VRAM is the budget king here — all these models fit with room to spare for KV cache overhead, even Gemma 4:12B (which needs ~8.5–9 GB with overhead). An RTX 4060 Ti 16 GB gives you more headroom. On the Apple side, any M2 or M3 MacBook with 16 GB unified memory handles these models comfortably via Ollama's Metal backend.

This guide shows how to swap out every cloud API with a local Ollama server running qwen3-coder:30b. Same tools, same workflows, no data leaving your network.

Related: 35B MoE on 2× 1080 Ti · Ollama

The sample is a tiny console app running entirely against a local Ollama model — no cloud keys, and every HTTP call is traced so I can see exactly what goes over the wire (complete sample code). There's a single skill on disk:

irm https://ollama.com/install.ps1 | iex

Today, I’m going to focus on my most popular project, TabbyAPI. TabbyAPI is a python based FastAPI server that allows users to interact with Large Language Models (or LLMs) using the ExllamaV2 library and adheres to the OpenAI API specification.

There are already some implementations out there which attempt to accomplish this!

Here's an example: https://github.com/silphendio/sliced_llama

A gist pertaining to said example: https://gist.github.com/silphendio/535cd9c1821aa1290aa10d587...

Here's a discussion about integrating this capability with ExLlama: https://github.com/turboderp/exllamav2/pull/275

And same as above but for llama.cpp: https://github.com/ggerganov/llama.cpp/issues/4718#issuecomm...

Sorry, I'm somewhat familiar with this term (I've seen it as a model loader in Oobabooga), but still not following the correlation here. Are you saying I should instead be using this project in lieu of llama.cpp? Or are you saying that there is, perhaps, an exllamav2 "extension" or similar within llama.cpp that I can use?

EDIT: I reported the bug to the exllamav2 Github. It's actually already fixed, just not on any current built release.

install exllamav2 from git with pip install git+https://github.com/turboderp/exllamav2.git. Make sure you have flash attention 2 as well.

Recent releases for exllamav2 brings working fp8 cache support, which I've been very excited to test. This feature doubles the maximum context length you can run with your model, without any visible downsides.

Without batching, I was actually thinking that's kind of modest.

ExllamaV2 will get 48 tokens/s on a 4090, which is much slower/cheaper than an H100:

https://github.com/turboderp/exllamav2#performance

I didn't test codellama, but the 3090 TI figures are in the ballpark of my generation speed on a 3090.

First, you need the convert.py script from turboderp's Exllama2 repo. You can read all about the convert.py arguments here.