ollama VS vllm

Download: Go to https://ollama.com/ and click on the download link for your operating system.

Most AI resume tools call out to OpenAI or Anthropic and charge you for every request. Persona supports Ollama — which means you can run the AI model locally on your own hardware, with zero API costs and zero data leaving your machine.

To solve these, we built a stack that prioritizes integrity over ease. The centerpiece is Ollama, running the mxbai-embed-large model locally. This is the engine that translates human thought into high-dimensional coordinates.

Ollama for model serving

Fully offline: Point it at Ollama or LM Studio. Zero cost, nothing leaves your network.

Recommended hardware: The RTX 3060 with 12 GB VRAM is the budget king here — all these models fit with room to spare for KV cache overhead, even Gemma 4:12B (which needs ~8.5–9 GB with overhead). An RTX 4060 Ti 16 GB gives you more headroom. On the Apple side, any M2 or M3 MacBook with 16 GB unified memory handles these models comfortably via Ollama's Metal backend.

This guide shows how to swap out every cloud API with a local Ollama server running qwen3-coder:30b. Same tools, same workflows, no data leaving your network.

Related: 35B MoE on 2× 1080 Ti · Ollama

The sample is a tiny console app running entirely against a local Ollama model — no cloud keys, and every HTTP call is traced so I can see exactly what goes over the wire (complete sample code). There's a single skill on disk:

irm https://ollama.com/install.ps1 | iex

Here's a real, runnable config that uses EAGLE for offline batched generation. It's straight from the vLLM repo's eagle.md example:

Say I'm tinkering with deathclaw-alpha, a little self-hosted AI assistant I run at home on OpenClaw and a local vLLM instance:

Compilation is cleaner on dense. llama.cpp, MLX, and vLLM all support both, but the dense path has had more attention. Fewer corner cases in expert routing, GQA, and KV layout interactions. If you've ever had a custom kernel mis-handle expert dispatch, you know.

vLLM: Focuses on efficient multi-model serving and inference optimization

The compute is real, and the compute is not the part that decides. Self-hosting means picking and operating a serving framework — vLLM and Text Generation Inference are the dominant choices in 2026 — and picking and operating the orchestration around it: Kubernetes or a comparable runtime, container registries, model artifact storage, an evaluation harness that runs on every model update, an on-call rotation that knows what to do when an inference replica wedges, a release-gating process for new model versions, and a documented procedure for the cases when an OSS model is deprecated by its publisher and needs to be rotated to a successor.

Most of the public technical writing about vLLM focuses on its GPU-side innovations — PagedAttention, continuous batching, the V1 engine. Less has been written about the CPU backend, which is where I spent the last couple of weeks: building vLLM from source, working through some rough edges, and shipping a small PR that clarifies three confusing error messages.

Seems like a pull request for vLLM was just approved a few minutes ago:

https://github.com/vllm-project/vllm/pull/41745

("Add Gemma4 MTP speculative decoding support")

Llama 4-70B-Instruct requires ~140GB of GPU memory in full precision (FP16), which means you need 2x A100 80GB instances per replica, costing ~$16/hour on AWS. Our benchmarks show 4-bit quantization using bitsandbytes (https://github.com/TimDettmers/bitsandbytes) reduces memory usage to ~35GB, allowing a single A100 40GB instance to run the model at 1/4 the cost. We tested 4-bit, 8-bit, and FP16 quantization across 10,000 queries and found 4-bit quantization has only a 1.2% drop in response accuracy for RAG use cases, which is negligible for most production applications. Avoid 8-bit quantization unless you have strict accuracy requirements: it only reduces memory usage by 40% vs 4-bit’s 75%, making it a poor cost-accuracy tradeoff. For high-throughput pipelines, use vLLM (https://github.com/vllm-project/vllm) instead of the default HuggingFace generate method: vLLM’s PagedAttention reduces latency by 3x for batch sizes over 8, and supports continuous batching for dynamic query loads. We saw throughput jump from 14 queries/sec to 89 queries/sec after switching to vLLM for Llama 4 inference. Always test quantization with your specific dataset: medical and legal RAG pipelines may see higher accuracy drops, so we recommend a 2-week A/B test before rolling out quantized models to production.

This post is a small step in that direction: serving an LLM using vLLM, deployed on Amazon EKS, provisioned the infra using AWS CDK, and wrapped into a simple chatbot using Streamlit.

You are deploying on Kubernetes: vLLM's official Helm chart (https://github.com/vllm-project/vllm/tree/main/helm) has native support for horizontal pod autoscaling and Prometheus metrics.