LLM Load Balancing & Fallbacks: A Practical Guide

TL;DR: LLM load balancing spreads requests across multiple models or providers, and fallbacks reroute a request when one fails. Together they turn provider outages, rate limits, and latency spikes into non-events. The key distinction most teams miss: a retry re-attempts the same target, while a fallback moves to a different one. This guide covers the strategies, the config that implements them, and how to do it at the gateway.

Ask anyone running LLMs in production about their worst incident and you’ll usually hear a version of the same story: a single provider had a bad hour, and because every request went straight to that provider, the whole feature went down with it. LLM providers post real outages, enforce hard rate limits, and swing wildly in latency depending on time, region, and model. If your architecture treats one provider as infallible, you’ve adopted its worst day as your own.

Provider incidents are routine, not rare — which is why single-provider dependency is a liability.

Load balancing and fallbacks are how you stop depending on any single point of failure. This is a practical guide to how they work, the different strategies available, and the configuration that actually implements them — with concrete examples you can adapt.

Why route across multiple targets

Before the how, the why. There are four recurring drivers, and most production setups are motivated by more than one.

Outages and overload. Providers go down or start returning 429/5xx under load. Routing to a healthy alternative keeps you serving while one provider recovers.

Rate limits. Every provider caps tokens and requests per minute. When you brush against a limit, spreading traffic across models or accounts is the difference between smooth operation and a wall of failed requests.

Latency variance. The “fastest” model isn’t fixed — it drifts by hour and region. The graph below, from TrueFoundry’s docs, shows how much model latency fluctuates over a single month.

Model latency varies constantly — dynamic routing can chase the fastest healthy option.

Canary and cost. Routing lets you send a small slice of traffic to a new model before committing, or send cheap tasks to cheaper models. It’s a control surface, not just a safety net.

Load balancing strategies

Not all load balancing is the same. The right strategy depends on what you’re optimizing for:

• Priority-based routing. Targets get a priority (0, 1, 2…); traffic goes to the primary and only shifts down on failure. Best for a clear primary with backups.
• Weight-based routing. Traffic splits by percentage — 90/10, 70/30 — ideal for canary rollouts or spreading load across accounts.
• Latency-based routing. The gateway races targets and sends traffic to the lowest-latency healthy one. Best for real-time, user-facing workloads.
• Sticky routing. Multi-turn conversations stay pinned to the same target for consistency, keyed by a session identifier.

Retries vs fallbacks: the distinction that matters

This trips up almost everyone. A retry re-attempts the same target — useful for transient blips. A fallback gives up on that target and tries a different one — useful when a provider is genuinely down. They’re configured separately and triggered by different status codes, and getting the split right is what makes routing robust instead of just noisy.

In TrueFoundry, a retry is governed by retry_config (with attempts, delay, and on_status_codes), which by default retries on 429, 500, 502, and 503. A fallback is governed by fallback_status_codes, which by default rolls over to another target on 401, 403, 404, 408, 429, 500, 502, and 503. Network errors always roll over.

Configuring it: the virtual model

The cleanest way to implement all of this is a virtual model — one logical model name that fronts several real targets with routing and fallback baked in. Your application calls the virtual model by name using the standard OpenAI API, and never has to know what’s behind it:

from openai import OpenAI

client = OpenAI(base_url="https://<your-gateway>.truefoundry.com/api/llm",
                api_key="<tfy-api-key>")

# “gpt-4o-router” is a virtual model — the gateway handles
# load balancing and fallback behind this single name.
client.chat.completions.create(
    model="gpt-4o-router",
    messages=[{"role": "user", "content": "Draft the release notes."}],
)

Behind that name, a weight-based canary with retries and an SLA cutoff looks like this (fields shown are illustrative of TrueFoundry’s virtual-model config):

name: gpt-4o-router
routing_config:
  type: weight-based-routing
load_balance_targets:
  - model: openai-main/gpt-4o
    weight: 90
    retry_config:
      attempts: 2
      delay: 200            # ms between retries on the same target
      on_status_codes: [429, 500, 502, 503]
    fallback_status_codes: [401, 403, 408, 429, 500, 502, 503]
    sla_cutoff:
      time_to_first_token_ms: 3000   # mark unhealthy if avg TTFT exceeds
      time_per_output_token_ms: 50   # mark unhealthy if avg TPOT exceeds
  - model: azure-openai/gpt-4o
    weight: 10
    fallback_candidate: true

For a strict primary-with-backups setup — say, keeping an on-prem model primary and bursting to the cloud only on failure — switch to priority routing:

routing_config:
  type: priority-based-routing
load_balance_targets:
  - model: onprem-vllm/llama-3         # primary
    priority: 0
  - model: anthropic/claude-sonnet     # first fallback
    priority: 1
  - model: bedrock/claude-sonnet       # second fallback
    priority: 2

That sla_cutoff block is the underrated part: it lets the gateway pull a target out of rotation when its time-to-first-token or per-output-token latency degrades, before users feel it.

How the gateway executes it

When a request hits the gateway, routing happens in a defined order: authenticate and authorize, check rate and budget limits, then select a target using the load-balancing rules, translate the request to the provider’s format, and — on a failing response — retry or fall back per your config.

The gateway selects the target and handles fallback inline, with logging done asynchronously.

Crucially, all of these checks run in-memory with no external calls in the request path, so routing adds minimal overhead — TrueFoundry benchmarks the AI Gateway at roughly +7 ms at 350 RPS on a single vCPU, scaling to tens of thousands of RPS with more replicas. You get resilience without turning the gateway into the new bottleneck. (TrueFoundry manages this through virtual models; note the older tenant-level routing config is deprecated in favor of them.)

Best practices

• Always define fallbacks across different providers, not just different models from one vendor — a vendor outage takes all its models down together.
• Use latency-based routing for user-facing paths and priority routing for cost-sensitive or on-prem-first setups.
• Set SLA cutoffs so a degrading target is removed before users notice.
• Separate retries from fallbacks — retry transient blips on the same target, fall back on genuine failures.
• Canary with weights before shifting a workload fully to a new model.
• Keep routing in the gateway, not scattered in application code, so it stays consistent and auditable.

FAQ

Q: What is LLM load balancing? A: It’s distributing LLM requests across multiple models or providers — by priority, weight, or latency — so no single provider is a bottleneck or a single point of failure. It’s typically implemented at an AI gateway.

Q: What’s the difference between a retry and a fallback? A: A retry re-attempts the same target after a transient error; a fallback abandons that target and routes to a different one. They’re configured separately and usually triggered by different status codes.

Q: How does an AI gateway handle provider outages? A: It detects failing responses (or network errors) and automatically routes to the next healthy target in your fallback configuration, so the request still succeeds and the outage doesn’t reach users.

Related reading

• LLM Load Balancing
• What is an LLM Gateway?
• Routing Across Multiple LLM Providers

Conclusion

Load balancing and fallbacks are what separate a demo from a production LLM system. The mechanics aren’t complicated once you internalize the retry-versus-fallback distinction and pick a routing strategy that fits your workload — and doing it at the gateway keeps the logic in one auditable place. If you’re hardening an LLM stack for real traffic, it’s worth seeing how TrueFoundry implements routing and fallbacks through virtual models across 1,000+ models.