How Ollama Loads a Model — and Why It Doesn't Actually Decide Your GPU Layers

Ask most people what Ollama does when you ollama run a model and you’ll hear some version of “it figures out how much VRAM you have and decides how many layers to offload to the GPU.” I read the source at v0.31.1, commit 710292f, and that’s not the mechanism. Ollama is a scheduler and model manager that spawns the upstream llama.cpp llama-server as a subprocess — and by default it lets llama.cpp pick the GPU layer split, then reads the result back out of the server’s logs. Ollama isn’t its own inference engine, and it isn’t the thing deciding your -ngl. Every claim below points at a line.

What is it, really?

For a standard GGUF model, the actual token-generating engine is the upstream llama.cpp llama-server, run as a child process. The file that drives it says so plainly: “wraps the llama-server binary as a subprocess” (llama_server.go:1), and the constructor’s comment is even blunter — “All GGML models are served via the upstream llama-server subprocess” (server.go:107). Ollama finds a llama-server-{cuda,rocm,darwin} binary (llama_binary.go:167), starts it (llama_server.go:428), and talks to it over HTTP (llama_server.go:110).

So Ollama’s real job is everything around inference: pull and register models, pick which one is loaded, fit it in memory, run several at once, and unload them when idle. That’s the scheduler, and it’s where the interesting decisions live.

Who decides how many layers go on the GPU?

Here’s the part that surprised me. When it builds the llama-server command line, Ollama only passes the -ngl (number-of-GPU-layers) flag if you set num_gpu yourself (llama_server.go:396):

if launch.opts.NumGPU > 0 {
    params = append(params, "-ngl", strconv.Itoa(launch.opts.NumGPU))
} else if launch.opts.NumGPU == 0 {
    params = append(params, "-ngl", "0") // CPU only
}
// NumGPU == -1 (default): don't pass -ngl, let llama-server auto-detect

And num_gpu’s default is -1 (types.go:1112). So on a normal run, Ollama passes no -ngl at all and lets llama.cpp’s own fitting logic choose the split. How does Ollama then know how many layers ended up on the GPU? It parses the number out of llama-server’s log output. There’s a regex — offloaded (\d+)/(\d+) layers to GPU (llama_server.go:2626) — whose captures are assigned straight into the runner’s state (llama_server.go:2676). The field even documents itself: gpuLayers // model layers loaded on GPU, parsed from llama-server logs (llama_server.go:123).

Read that back: in the default path, the layer split is llama.cpp’s decision, and Ollama scrapes the answer. If you want to force it, num_gpu is your lever — that’s the only thing that makes Ollama emit -ngl.¹

Then what is Ollama’s memory math for?

Ollama does compute a VRAM estimate — but for scheduling, not for the layer split, and it’s deliberately crude. PredictServerVRAM uses the model file size as the weights estimate plus a rough f16 KV-cache term (llama_server.go:2562):

weights = os.Stat(modelPath).Size()
kvCache = 2 (K+V) * layers * kv_heads * head_dim * numCtx * 2 bytes

Its own comment calls it out: “intentionally conservative — it overestimates to avoid VRAM contention” (llama_server.go:2564). Every caller of it lives in the scheduler (sched.go:542), where it answers one question: will this model fit, or do I need to evict something first? There’s a much more detailed per-layer estimator in the tree (GraphSize, ggml.go:609) — but at this commit nothing calls it for the fit decision. Once the server is up, Ollama replaces the estimate with reality by parsing the actual buffer sizes from llama-server’s post-load logs (llama_server.go:2520).

The mental-model correction is the whole story here: Ollama estimates coarsely to schedule; llama.cpp fits precisely to run; the truth is reconciled from logs afterward.

Why does my model get unloaded?

Because of a timer. Loaded runners are reaped on a keep_alive duration whose default is 5 minutes (config.go:130; a negative value means “stay forever,” zero means “unload immediately”). When a request finishes and the model goes idle, the scheduler arms a timer for keep_alive; when it fires, the model is unloaded (sched.go:401). Note the default is resolved deep in the scheduler’s load loop, not at the HTTP handler (sched.go:518) — so an unset keep_alive inherits the 5-minute env default there.

The other reason a model disappears is pressure. Ollama caps how many models stay resident at 3 × your GPU count by default (defaultModelsPerGPU = 3, sched.go:87, multiplied by the GPU count at sched.go:292). When a new model won’t fit — or that cap is hit — it evicts one, chosen by findRunnerToUnload (sched.go:1682): sort by shortest keep_alive, prefer an idle runner, and only take a busy one if nothing idle is available. All of this runs in a single scheduler goroutine (sched.go:231), so load and evict decisions are serialized — there’s no race between two models trying to claim the same VRAM.

Where does this fit?

Local inference has a clean dividing line: are you the engine, or are you the manager on top of one?

llama.cpp (and its llama-server) is the engine — it does the GGUF loading, the GPU fitting, the token generation. Ollama is the manager: a model registry, a REST API, and the scheduler you just saw. It deliberately delegates the hard numerical part (the layer split) downward and keeps the orchestration part (what’s loaded, what fits, what to evict, when to unload) for itself. That’s a coherent design — it’s why Ollama feels effortless — but it also means the answer to “why did my model only put 20 layers on the GPU?” lives in llama.cpp, not Ollama, unless you took the wheel with num_gpu.

There’s experimental scaffolding for a native Go engine in the tree (the model/ package), and separate runners for Apple’s MLX and for image generation (runner.go:10) — but for the GGUF LLMs almost everyone runs, the path is the llama-server subprocess, full stop.

When would you reach for it?

Reach for Ollama when you want the manager layer: a one-line pull-and-run, a local REST API, automatic GPU fitting (delegated to llama.cpp, which is a fine default), and hands-off multi-model loading with idle unloading. Go in knowing the trade: you don’t directly control the offload split unless you set num_gpu, the fit decision is a conservative overestimate (so Ollama may keep a model off the GPU that llama.cpp could have squeezed on), and you’re a release behind whatever llama.cpp just shipped. If you need the bleeding edge or exact placement control, driving llama-server yourself is the other camp — you trade the manager’s conveniences for direct control of the engine.

Methodology & scope

I read Ollama v0.31.1 at commit 710292f, focused on server/sched.go, llm/llama_server.go, llm/server.go, llm/llama_binary.go, envconfig/config.go, and api/types.go. Three headline claims — that the default path delegates the GPU layer split to llama-server and reads it back from logs, that the scheduler’s keep_alive default is 5 minutes with a 3×GPU load cap and idle-first eviction, and that Ollama’s scheduling memory math is a conservative file-size-plus-KV proxy rather than an exact per-layer estimate — were each run through an adversarial claim-verifier at this commit and confirmed. Line numbers are exact there.

Scope caveats, stated plainly:

• This is v0.31.1-specific. Earlier Ollama versions computed the layer split themselves (the classic estimateGPULayers); the delegation-to-llama.cpp behavior described here is what the code does at this commit, and it has clearly evolved.
• I did not read inside llama.cpp’s own auto-fit logic — that’s the upstream engine, out of this repo. My claims are about what Ollama passes it and reads back, not how llama-server decides.
• The native model/ engine and the MLX/imagegen runners exist but are off the standard GGUF path; I scoped the teardown to the mainstream flow.

Footnotes

1. There’s one automatic exception to “Ollama doesn’t touch the split”: a multimodal projector OOM. If the server fails to start with an out-of-memory error and the model has a vision projector, Ollama restarts it once with the projector forced onto the CPU (llama_server.go:998) — a targeted retry, not a general layer calculation. ↩