ollama VS litellm

Download: Go to https://ollama.com/ and click on the download link for your operating system.

Most AI resume tools call out to OpenAI or Anthropic and charge you for every request. Persona supports Ollama — which means you can run the AI model locally on your own hardware, with zero API costs and zero data leaving your machine.

To solve these, we built a stack that prioritizes integrity over ease. The centerpiece is Ollama, running the mxbai-embed-large model locally. This is the engine that translates human thought into high-dimensional coordinates.

Ollama for model serving

Fully offline: Point it at Ollama or LM Studio. Zero cost, nothing leaves your network.

Recommended hardware: The RTX 3060 with 12 GB VRAM is the budget king here — all these models fit with room to spare for KV cache overhead, even Gemma 4:12B (which needs ~8.5–9 GB with overhead). An RTX 4060 Ti 16 GB gives you more headroom. On the Apple side, any M2 or M3 MacBook with 16 GB unified memory handles these models comfortably via Ollama's Metal backend.

This guide shows how to swap out every cloud API with a local Ollama server running qwen3-coder:30b. Same tools, same workflows, no data leaving your network.

Related: 35B MoE on 2× 1080 Ti · Ollama

The sample is a tiny console app running entirely against a local Ollama model — no cloud keys, and every HTTP call is traced so I can see exactly what goes over the wire (complete sample code). There's a single skill on disk:

irm https://ollama.com/install.ps1 | iex

For calibration I took the seven-dimension methodology to LiteLLM, a mature and widely-deployed proxy, pinned at a commit. It came back 4 green / 3 yellow / 0 red — production-ready, with the usual edges a structured pass surfaces in a good codebase: one authorization resolver that fails open where its siblings fail closed, unpinned third-party MCP servers, and per-tool least-privilege left opt-in. Then I built a deliberately-broken reference gateway that fails the same checks by construction. The scanner here automates the static slice of that methodology — the part you can run in seconds, before any deeper review.

Yes — set llm_provider: "openai_compatible_llm" and point base_url at any OpenAI-compatible endpoint (Anthropic via LiteLLM, OpenRouter, Groq, Together, Fireworks, or self-hosted vLLM). You lose offline-mode and gain quality. Most users do a hybrid: local Whisper + Cloud LLM + local TTS for cost vs latency tuning.

— Getting Started

No multi-provider fan-out. It is Bedrock-shaped. If you need OpenAI direct, Vertex, Gemini, Ollama, and so on in one proxy, LiteLLM is the right tool — it speaks 100+ providers and has a much richer commercial budgeting tier.

"Local-first" still routes summaries to cloud LLMs by default. Until local Gemma 4 ships, the transcript stays local but the summary prompt (which includes the transcript) goes to whichever API key you configured. If that's a dealbreaker, run a local OpenAI-compatible endpoint like LiteLLM or Ollama's OpenAI-compatible mode and point OpenBrief at http://localhost:11434.

LiteLLM GitHub

LiteLLM: https://github.com/BerriAI/litellm

Source: https://github.com/BerriAI/litellm/issues/27639 (open, updated 2026-05-19)

LiteLLM GitHub. Source for the most widely deployed open-source gateway.

We didn't want to lock BowerBot to one LLM provider. Studios have different policies (some can't send data to OpenAI, others have Anthropic enterprise contracts, some run local models). We wrapped LLM calls through litellm, which gives us a common interface across OpenAI, Anthropic, Google, Cohere, local Ollama, and more.