If you want to run an open model on your own hardware, the tooling looks like a crowded field — but read the source and it collapses into three camps, split by which layer you want to operate at. llama.cpp is the raw engine: a self-contained binary that loads a model file and runs it on whatever hardware you have. Ollama is the friendly manager that wraps that engine in a nicer package. vLLM is the throughput server that throws the engine out and builds its own. I read llama.cpp at 2026-07-01 master (a6647b1), Ollama at v0.31.1, and vLLM at v0.24.0rc2, and the comparison is clean once you see the axis. Every claim points at a line.
The one question
What layer do you want to operate at? That single choice sets your hardware range, your concurrency, and how much the tool does for you.
- Raw engine — you drive the inference library directly. Maximum portability and control, minimum hand-holding. (llama.cpp)
- Manager — something wraps the engine and adds a model registry, an API, and lifecycle management. Convenience for one machine. (Ollama)
- Throughput server — a purpose-built engine optimized to serve many concurrent requests. Throughput at the cost of needing real GPUs. (vLLM)
Camp 1 — The raw engine: llama.cpp
llama.cpp is a self-contained C/C++ inference engine. It builds small single-purpose binaries — llama-cli and the llama-server HTTP server (CMakeLists.txt:57) — that load a GGUF file, ggml’s own quantized model format (llama-model-loader.cpp:550), and run it. Its superpower is portability: the compute backends (CPU, CUDA, Metal, Vulkan, ROCm, and a dozen more) sit behind a C vtable and can be dynamically loaded at runtime by a single build (ggml-backend-dl.cpp:35). It runs on a Raspberry Pi or an H100.
You control the GPU split with -ngl (arg.cpp:2482), which defaults to auto — and auto genuinely measures your free VRAM and fits as many layers as it can, trimming the count to what will fit (fit.cpp:375). That memory-fitting is a real piece of engineering living in llama.cpp itself, which matters for the next camp. This is the camp for maximum control and reach — and the least hand-holding: no model registry, no daemon, you manage the files and flags yourself.
Camp 2 — The friendly manager: Ollama
Ollama doesn’t reimplement inference — it wraps llama.cpp’s llama-server as a subprocess (server.go:107) and adds everything around it: a model registry with ollama pull, a REST API, and a keep-alive scheduler that unloads idle models after five minutes. As I found in its teardown, it even delegates the GPU layer split down to llama.cpp — by default it passes no -ngl at all (llama_server.go:396) and lets llama.cpp’s auto fitting decide, then reads the resulting layer count back out of the server’s logs. So the two camps nest: Ollama is the convenience layer, and the engine underneath is exactly Camp 1. What you’re buying is the packaging — pull-and-run, an API, and lifecycle management for a single-user machine.
Camp 3 — The throughput server: vLLM
vLLM is the opposite bet: it is the engine, purpose-built for serving many requests at once. Where llama.cpp optimizes for running one model anywhere, vLLM optimizes for keeping a GPU saturated. As I found in its teardown, it pages the KV cache into fixed 16-token blocks so memory is tightly packed and shareable (cache.py:46), and its scheduler has no prefill/decode phases — it rebuilds the batch every step from a token budget (scheduler.py:391), mixing new prompts and ongoing generations so nothing idles. That’s continuous batching, and it’s why vLLM’s throughput pulls away from the other two under concurrent load — at the cost of wanting a real GPU with the whole model resident.
The comparison
| llama.cpp — raw engine | Ollama — manager | vLLM — throughput server | |
|---|---|---|---|
| You operate the | inference library directly | wrapper around llama.cpp | dedicated serving engine |
| Model format | GGUF (quantized) | GGUF (via llama.cpp) | HF weights (various quant) |
| Hardware range | anything: CPU → any GPU | same as llama.cpp | real GPU(s), model resident |
| Concurrency | basic | simple per-model scheduling | continuous batching (high) |
| VRAM fitting | -ngl auto fits to your VRAM |
Ollama’s estimate + llama.cpp | PagedAttention blocks |
| Convenience | low — you drive it | high — pull & REST API | server ops, not a laptop tool |
| Relationship | the engine | wraps the engine | replaces the engine |
Why the split decides everything
Notice the nesting, because it’s the real story. Ollama and llama.cpp aren’t rivals — Ollama is llama.cpp with a bow on it. If you’re choosing between them, you’re choosing convenience-vs-control over the same engine, GGUF models and all. vLLM is the genuine fork in the road: a different engine, a different model format, a different optimization target. So the decision isn’t three-way symmetric. It’s first “do I need serving throughput?” (vLLM vs. the llama.cpp family), and only then “do I want the manager or the raw engine?” (Ollama vs. llama.cpp).
Where do you fit?
- One model, one machine, you just want it to work → Ollama. It’s llama.cpp underneath with a registry, an API, and idle-unloading on top; you give up nothing in the engine and gain all the packaging.
- Weird hardware, tight control, or embedding inference into your own app → llama.cpp directly. It runs where nothing else will (CPU-only, Metal, edge devices), and you hold every knob — including which layers land on the GPU.
- Serving many concurrent users and throughput is the metric → vLLM. PagedAttention and continuous batching are built for exactly that load, and they leave the single-stream engines behind once the requests pile up. Just bring the GPU.
Scope & honesty
I read llama.cpp at master a6647b1 (2026-07-01), Ollama v0.31.1 (710292f), and vLLM v0.24.0rc2 (6d37570). The Ollama and vLLM mechanisms were each verified against their source in the linked teardowns; llama.cpp’s engine shape (self-contained binaries, GGUF loading, dynamic ggml backends) was run through an adversarial claim-verifier at this commit and confirmed.
Two honesty notes. First, a wrinkle I had to correct mid-read: llama.cpp’s -ngl auto does measure free VRAM and trim layers to fit — that logic lives in common/fit.cpp, invoked before the model loads, and it fires only for auto (not all or an explicit count). An initial pass that only searched the core model loader missed it and concluded “no fitting”; the claim-verifier caught the error before it reached this page. That fit is exactly what makes Ollama’s “just delegate to llama.cpp” strategy work. Second, llama.cpp is a fast-moving target — pinning a date matters here more than anywhere, and Ollama bundles its own build of it, which can lag or lead upstream master.
