If Ollama is the single-user convenience layer over llama.cpp, vLLM is the other end of the local-inference spectrum: a throughput engine built to keep a GPU saturated with many concurrent requests. Its reputation rests on two words — PagedAttention and continuous batching — and I read them at v0.24.0rc2, commit 6d37570 (the rewritten “v1” engine). They turn out to be one idea about memory and one idea about scheduling, and they only work because of each other. Every claim below points at a line.
The problem both ideas solve
A transformer’s KV cache grows one token at a time, and you don’t know how long a request will run. The naive implementations waste the GPU twice: they reserve a big contiguous KV buffer per request (so short requests strand memory and long ones fragment it), and they batch statically (gather N requests, run them all to completion, then start the next N — so the whole batch waits on its slowest member and the GPU idles as requests finish at different times). vLLM attacks both.
PagedAttention: the KV cache is paged, like virtual memory
Instead of one contiguous buffer per request, vLLM slices the GPU’s KV memory into fixed-size blocks — 16 tokens each by default (cache.py:46). The pool is sized once at startup from whatever GPU memory is left after weights: num_blocks = available_memory // page_size // num_layers (kv_cache_utils.py:987), where “available” is gpu_memory_utilization — 0.92 by default (cache.py:67) — of the card minus the model.
Each request then holds a list of block indices, not a contiguous range. When it needs room for new tokens, allocate_slots grabs ceil(new_tokens / 16) blocks off the free queue (kv_cache_manager.py:244); if the pool can’t cover it, it returns None (kv_cache_manager.py:416) — a signal we’ll come back to. The attention kernel never assumes contiguity: it’s handed a per-request block table (flash_attn.py:582) mapping logical token positions to physical block ids, and gathers the KV from wherever those blocks live. That’s the whole trick — the “paged” in PagedAttention is literally OS-style paging for the KV cache, and it kills fragmentation.
The payoff on top is prefix sharing. Every full block is hashed by chaining the previous block’s hash with this block’s token ids — hash((parent_block_hash, curr_block_token_ids, extra_keys)) (kv_cache_utils.py:603) — so the hash covers the entire prefix, not one block. Two requests that begin with the same system prompt produce the same hashes, so the second one just touches the existing physical blocks (ref-count +1, no copy) instead of allocating its own (block_pool.py:597). A block only returns to the free pool when its ref-count hits zero, and cached-but-free blocks sit at the tail of the free queue so they’re evicted last (LRU).
PagedAttention gives each request a block table. Logical token positions can point to non-contiguous physical KV blocks, and shared prefixes can reuse the same blocks.
Continuous batching: rebuild the batch every step
The scheduler’s design comment says it outright: “There’s no ‘decoding phase’ nor ‘prefill phase’ in the scheduler” (scheduler.py:391). Every request is just a (num_computed_tokens, num_tokens_with_spec) pair, and each step the scheduler hands out a token budget — max_num_batched_tokens (scheduler.py:408) — trying to let every request’s computed count catch up to its total. A brand-new 2,000-token prompt (a “prefill”) and a request generating its 500th token (a “decode”) are the same kind of thing: both just want tokens from the budget.
So one forward pass mixes prompts and generations freely. The scheduler packs already-running requests (scheduler.py:432) and then admits waiting ones (scheduler.py:629) until the budget runs out, and a long prompt can be split across steps via chunked prefill (scheduler.py:468). Because the batch is rebuilt from scratch every step, the moment a request finishes its slot is reused by whatever’s waiting — the GPU never sits idle waiting for a batch to drain. That’s “continuous” (in-flight) batching, and it’s the throughput multiplier.
What happens when the blocks run out
The two ideas meet at the failure case. When allocate_slots returns None mid-schedule, there’s no free memory to grow a running request — so vLLM preempts one. Under the default FCFS policy it evicts the newest running request, self.running.pop() (scheduler.py:562); under priority scheduling it evicts the lowest-priority, latest-arrival one (scheduler.py:538). The victim’s blocks are freed, its num_computed_tokens is reset to 0 (scheduler.py:1120), and it goes back to the head of the waiting queue to be recomputed later.
Two things worth knowing here. First, v1 preemption is recompute-only — the old CPU-swap path is gone; a preempted request throws away its KV and regenerates it. Second, prefix caching softens that: because freeing evicts a request’s tail blocks first, its prompt prefix often survives in the pool, so on rescheduling get_computed_blocks (kv_cache_manager.py:202) can hand back the still-cached prefix and skip recomputing it. The same paging that packs memory tightly is what makes the recompute cheap.
Where does this fit?
The local-inference axis I’d draw is convenience vs. throughput, and it tracks a deeper split: manager over an engine vs. purpose-built engine.
Ollama is a manager. It shells out to llama.cpp and, as I found in its teardown, even delegates the GPU layer split to the upstream server. It’s optimized for one developer running one model with a friendly UX. vLLM is the engine — its own PagedAttention CUDA kernels, its own paged KV allocator, its own continuous-batching scheduler — and every design choice above exists to serve many concurrent requests at high throughput. Neither is “better”; they’re built for different rooms. Ollama makes one model easy on your laptop. vLLM keeps an A100 busy under a thousand users.
When would you reach for it?
Reach for vLLM when you’re serving — many concurrent requests, throughput and GPU utilization matter, and shared prefixes (a big common system prompt) are money on the table that prefix caching collects for free. Go in knowing the shape of its trade-offs: it wants a beefy GPU and pre-reserves ~92% of it for the cache; under memory pressure it preempts and recomputes rather than swaps, so pathological over-subscription shows up as wasted recompute; and it’s an operational component, not a brew install. If you just want to chat with a local model on your own machine, that’s Ollama’s room, not this one.
Methodology & scope
I read vLLM v0.24.0rc2 at commit 6d37570, focused on the v1 engine: vllm/v1/core/sched/scheduler.py, vllm/v1/core/kv_cache_manager.py, vllm/v1/core/block_pool.py, vllm/v1/core/kv_cache_utils.py, and vllm/config/cache.py. Three headline claims — the fixed-block paged KV cache with prefix sharing, the per-step token-budget continuous batching with no prefill/decode phases, and preempt-and-recompute on block exhaustion — were each run through an adversarial claim-verifier at this commit and confirmed. Line numbers are exact there.
Scope caveats, stated plainly:
- This is the v1 engine. The older v0 architecture had a CPU-swap preemption path; v1 is recompute-only, so version matters for this claim.
- The actual KV gather over the block table runs in a compiled FlashAttention CUDA kernel, a binary dependency I did not read — I verified that vLLM hands the kernel a per-request block table, not the kernel’s internals.
- I read the scheduler and KV-cache subsystem, not the distributed paths (tensor/pipeline parallelism, the KV-connector offload feature that lives separately from preemption).
