How LiteLLM's Router Picks a Deployment — Weighted Random by Default, Smart Only If You Ask

LiteLLM’s Router is pitched as smart load balancing across your model deployments. I read it at v1.92.0-dev.1, commit 88e03e5, and the default is a coin flip. Out of the box it uses simple-shuffle: pick a random healthy deployment, weighted by rpm/tpm if you bothered to set weights, uniform otherwise. It is not latency-aware or load-aware unless you opt into a strategy that is. What it does give you for free is a retry → cooldown → fallback failure ladder. And where it earns “smart” — the usage/latency strategies — it earns it by metering every key, which is the exact thing a subscription proxy can’t do. Every claim below points at a line.

What is it, and where does it sit?

The Router is the thing that sits in front of a model group — several deployments (API keys, regions, or providers) that all serve, say, gpt-4 — and picks one per request, then handles failure. It’s the API-key routing half of the LLM-gateway comparison. The other half is the subscription-account proxy I tore down separately (CLIProxyAPI); the contrast at the end is the whole point.

The default is dumber than the marketing

The default routing strategy is literally the string "simple-shuffle" (router.py:302), and it’s dispatched inline, bypassing the selector machinery the smart strategies use (router.py:10299). Here’s the entire selection logic:

# weighted pick if a weight/rpm/tpm is set...
selected_index = random.choices(range(len(weights)), weights=weights)[0]
# ...otherwise a plain coin flip
item = random.choice(healthy_deployments)

That’s simple_shuffle.py:59 and :68. No latency, no queue depth, no live load feeds into it. “Healthy” just means the deployment survived a set of filters — it isn’t cooling down, isn’t admin-blocked, and (if you enabled pre-call checks) fits the context window and is under its RPM limit. So the honest one-liner: by default, LiteLLM spreads load by weighted random choice, not by measuring anything. If you want it to measure, you opt in.

The smart strategies are opt-in — and they meter

Set routing_strategy to one of least-busy, latency-based-routing, usage-based-routing[-v2], or cost-based-routing and a selector is built and registered as a logging callback (router.py:865). The trick that makes proactive routing possible is that each strategy writes a per-deployment metric on every call and reads it at selection time, into a DualCache (in-memory plus optional Redis) keyed by deployment id:

• least-busy keeps an in-flight request counter, +1 before the call and -1 after (least_busy.py:44).
• lowest-latency appends each response’s latency to a rolling per-deployment list and picks the fastest band.
• usage-based v2 increments per-minute tpm/rpm counters (lowest_tpm_rpm_v2.py:258) and, before dispatching, reserves an RPM slot with a pre-call increment that raises RateLimitError if you’re over budget (lowest_tpm_rpm_v2.py:116, driven from router.py:7120).

That reservation is the sharpest bit: with Redis, the meter is authoritative across proxy instances, so LiteLLM can refuse a request before it ever hits the provider.¹ This is only possible because LiteLLM owns the key and sees every token spent through it.

The failure ladder

When a call fails, three mechanisms fire in order. The defaults live in one file (constants.py):

Step	What happens	Default
Retry	Re-pick a healthy deployment and try again	num_retries = 2 (:20)
Cooldown	Bench the failed deployment for a TTL	cooldown_time = 5s (:32)
Fallback	Cascade to a backup model group	ROUTER_MAX_FALLBACKS = 5 (:9)

The ordering matters: it exhausts the 1 + num_retries attempts within the model group first, and only if those all fail does it move to a fallback group (router.py:6193). Two details worth knowing:

Retries are usually instant. The backoff function returns 0 — retry immediately — as long as the group isn’t down to a single deployment and has a healthy one left (router.py:6635). Because each retry re-picks and the just-failed deployment is being cooled down, “retry” effectively means “try a different one right now.” Only when you’re down to a lone deployment does it fall back to exponential backoff (0.5s doubling, capped at 8s, honoring a Retry-After header).

Cooldowns are short and rate-based — and spare your only model. A deployment is benched when it returns 429/401/408/404 or any 5xx (cooldown_handlers.py:40), or when its failure rate crosses 50% over at least 5 requests in the window (:210). But if the model group has only one deployment, the 429 and rate-based cooldowns are deliberately skipped (:204) — cooling down your only model would just make every request fail. The cooldown itself is a cache key with a 5-second TTL that expires on its own. Fallbacks then split by failure type: a context-window overflow routes to context_window_fallbacks, a content-policy block to content_policy_fallbacks, everything else to the generic fallbacks list.

Where does this fit?

Same axis I used for CLIProxyAPI: what are you routing, and can you see its cost?

LiteLLM routes API keys, and it meters them — every request updates that deployment’s tpm/rpm/latency/in-flight count. That visibility is what lets it (optionally) route by least-busy or usage or latency, and reserve capacity before spending it. CLIProxyAPI routes subscription logins bound by opaque quota windows it can’t see, so its only lever is reaction — wait for the 429, then cool the account down. Read side by side, the two are the same failover-cooldown skeleton with a crucial difference: LiteLLM can be proactive because it holds the meter; the subscription proxy can only be reactive because the meter is hidden from it. (And a small tell of that philosophy: LiteLLM refuses to cool down your only deployment; CLIProxyAPI benches individual accounts regardless.)

When would you reach for it?

Reach for LiteLLM’s router when you have multiple keys/deployments for the same model and want one endpoint with retries, cooldowns, and fallbacks that mostly Just Work. Go in knowing two things. First, the default doesn’t balance intelligently — if you care about load or latency, set routing_strategy explicitly and, ideally, wire up Redis so the meter is shared; otherwise set per-deployment weight/rpm so the random pick at least respects your capacity. Second, the failure defaults are aggressive-but-short: a 5-second cooldown and instant re-picks are great for transient blips, but they mean a deployment having a bad minute gets hammered again quickly — tune cooldown_time and allowed_fails if your providers are flaky.

Methodology & scope

I read LiteLLM v1.92.0-dev.1 at commit 88e03e5, focused on litellm/router.py, litellm/router_strategy/*, litellm/router_utils/cooldown_*.py and fallback_event_handlers.py, and litellm/constants.py. Three headline claims — the weighted-random default with opt-in smart strategies, the retry/cooldown/fallback ladder and its exact defaults, and the per-deployment metering that makes proactive routing possible — were each run through an adversarial claim-verifier at this commit and confirmed. Line numbers are exact there.

Scope caveats, stated plainly:

• I read the router’s routing and failure paths, not every one of the 15+ strategy files (there are newer adaptive/complexity/quality routers I did not open); the “smart is opt-in” claim is about the classic five.
• The RPM reservation is best-effort under a Redis outage — the code swallows connection errors and lets the call through rather than failing closed (lowest_tpm_rpm_v2.py), so treat it as a strong reservation, not a hard guarantee.
• A doc/code drift I’ll flag rather than paper over: the cooldown_time docstring at router.py:343 says “Defaults to 1,” but the constant it actually resolves to is 5.

Footnotes

1. The atomicity is increment-then-check, so under heavy concurrency the reservation can transiently overshoot the limit between the increment and the check. It’s a real pre-call reservation, not a perfectly serialized lock — good enough to shed load, not a hard rate-limit guarantee. ↩