Model routing in production: OpenAI, Anthropic and open-weight models by cost and latency

Routing is a product decision, not a model preference

A production application does not make one kind of language model call. It classifies a support ticket, extracts fields from a document, rewrites a search query, drafts a reply, and occasionally reasons through something genuinely hard. Those tasks have wildly different requirements, and sending all of them to the same frontier model means you overpay for the easy 90 percent without improving the results for the rest.

Costs and capabilities change constantly, so this article deliberately uses relative terms — small, mid and large tiers rather than named models with quoted prices. The routing structure survives model releases; a hard-coded model name does not.

Name your task classes first

Before writing a router, list every place your product calls a model and put each one in a class. In most products there are only five or six.

• Classification and routing — short input, tiny output, a fixed label set. Small models are usually indistinguishable from large ones here, and this is often your highest-volume call.
• Extraction — structured output from a document or a message. Constrained by a schema, so failures are visible and cheap to retry.
• Rewriting and summarising — medium input, medium output, quality is noticeable but the bar is not "frontier".
• Drafting user-facing text — quality is the product. This is where a better model actually pays for itself.
• Multi-step reasoning and agents — tool calls in a loop, where a weaker model fails in an expensive way: it does not produce a worse answer, it produces five extra tool calls and then a worse answer.
• Long-context work — the choice is made for you by the context window, whatever else you would have preferred.

Declare the class at each call site. Everything after this depends on the caller stating what kind of work it is rather than on the router guessing from the prompt text.

Signals that actually decide the route

Four signals do almost all the work, in this order.

• Context length — a hard constraint. If the input does not fit, nothing else matters, and truncation strategies belong upstream of the router, not inside it.
• Tool use — multi-step tool calling punishes weak instruction following far more than single-shot generation does. Route tool loops up a tier, and the cost usually goes down because the loop terminates sooner.
• Task complexity — approximate it from the class, not from clever prompt inspection. Heuristics that read the prompt to guess difficulty create unpredictable behaviour and bugs that are hard to diagnose.
• Streaming — if a human is watching tokens appear, time to first token dominates perceived quality. A model that is 30 percent slower to first token feels worse than one that is 30 percent worse on your eval set.

Two signals look tempting but are usually mistakes: using a classifier model to route each user prompt (you have added a network round trip and a new failure mode to every request), and routing on live price feeds (the system's behaviour now changes for reasons nobody can reproduce).

A small router

Keep the router a pure function. It takes a description of the task and returns a tier plus the limits that go with it. It makes no network calls, it has no state, and it can be unit tested exhaustively — which matters, because this is the function that decides your monthly bill.

type Tier = 'small' | 'mid' | 'large';

type Task = {
  kind: 'classify' | 'extract' | 'summarise' | 'draft' | 'reason' | 'agent';
  inputTokens: number;
  needsTools: boolean;
  streaming: boolean;
  tenantTier: 'free' | 'paid';
};

type Route = { tier: Tier; maxOutputTokens: number; timeoutMs: number };

export function route(task: Task): Route {
  // 1. Hard constraints first — they are not negotiable.
  if (task.inputTokens > 100_000) {
    return { tier: 'large', maxOutputTokens: 2_000, timeoutMs: 60_000 };
  }
  if (task.kind === 'agent' || task.needsTools) {
    // multi-step tool calling punishes weak instruction following
    return { tier: 'mid', maxOutputTokens: 1_500, timeoutMs: 45_000 };
  }

  // 2. Cheap, high-volume, well-specified work.
  if (task.kind === 'classify' || task.kind === 'extract') {
    return { tier: 'small', maxOutputTokens: 256, timeoutMs: 8_000 };
  }

  // 3. Everything user-facing and open-ended.
  if (task.kind === 'reason') {
    return { tier: 'large', maxOutputTokens: 2_000, timeoutMs: 60_000 };
  }

  // 4. Default, with a business rule on top.
  const tier: Tier = task.tenantTier === 'paid' ? 'mid' : 'small';
  return { tier, maxOutputTokens: 800, timeoutMs: task.streaming ? 30_000 : 20_000 };
}

The tier names then resolve to concrete provider models in one configuration file per environment. When a provider ships something better, you change one mapping and re-run your evals instead of grepping for model identifiers across the codebase.

Fallbacks, retries and timeouts

Providers have incidents. If a single upstream failure becomes a 500 error for your users, the routing layer is not finished. Use a chain of attempts: the preferred tier, then an alternative, with a strict timeout on each attempt.

const CHAIN: Record<Tier, Tier[]> = {
  small: ['small', 'mid'],
  mid:   ['mid', 'small', 'large'],
  large: ['large', 'mid'],
};

export async function complete(task: Task, messages: Message[]) {
  const { tier, timeoutMs, maxOutputTokens } = route(task);
  let lastError: unknown;

  for (const candidate of CHAIN[tier]) {
    const started = Date.now();
    try {
      const res = await withTimeout(
        callProvider(candidate, messages, { maxOutputTokens }),
        timeoutMs,
      );
      record({ task: task.kind, planned: tier, used: candidate,
               ms: Date.now() - started, usage: res.usage, ok: true });
      return res;
    } catch (err) {
      lastError = err;
      record({ task: task.kind, planned: tier, used: candidate,
               ms: Date.now() - started, ok: false, err });
      if (!isRetryable(err)) throw err;   // 4xx: do not shop around
    }
  }
  throw lastError;
}

Three rules keep this from becoming a cost incident of its own. Retry only on retryable failures — timeouts, rate limits and 5xx errors — and never on a 4xx error caused by your own malformed request. Cap total attempts, because a retry storm against a degraded provider is how a small outage becomes a large invoice. Keep a circuit breaker per provider so you stop waiting for the timeout on every request once a provider is clearly down.

Fall back across providers, not just across tiers, for anything you consider critical. That is the entire practical argument for keeping at least two providers wired up.

Caching, in three layers

• Prefix caching — most providers can reuse a repeated prompt prefix at a reduced rate. It costs almost nothing to use: put everything stable at the front (system prompt, tool definitions, long shared context) and everything variable at the end. Teams that interleave the two lose the benefit without ever noticing they had it.
• Response caching — hash the normalised prompt plus the tier and the parameters, and cache deterministic calls. Classification and extraction on repeated inputs hit the cache far more often than intuition suggests. Set a TTL and include a prompt version in the key so a prompt change invalidates the cache automatically.
• The layer above — the cheapest call is the one you do not make. Deduplicate identical in-flight requests, skip the model when a rule or a lookup answers the question, and do not summarise a document that has not changed since you last summarised it.

Token budgets

Set a maximum output length per task class and enforce it in the request, not in a code review comment. Unbounded output is the most common source of unexpected costs and p99 latency: one request that decides to write an essay can cost more than a thousand well-behaved ones.

On the input side, budget explicitly. Decide how many tokens each part of the prompt may occupy — system, retrieved context, conversation history, user input — and trim deterministically when the budget is exceeded, oldest history first. Increment a counter every time you trim, because a budget that trips constantly is a design problem rather than a runtime event.

Measure cost per request and p95 latency

Emit one structured record per model call: task class, planned tier, tier actually used, input and output tokens, cached token counts, latency, time to first token when streaming, retry count, and the outcome. You can derive every subsequent metric by aggregating these records.

• Cost per request by task class — the number that tells you where routing is worth the effort. It is almost never spread evenly.
• p95 and p99 latency by class, and time to first token separately for streaming calls. Averages hide the requests that make users leave.
• Fallback rate — how often the preferred tier failed. A rising fallback rate is an early warning that arrives before your users complain.
• Cache hit rate by class, split between prefix and response caching, so you can tell which one is actually paying off.

Convert token counts to money at reporting time using a small table of rates, rather than storing a computed cost. Rates change; your history should not be rewritten when they do.

Prove the cheaper model is good enough

Never downgrade a route based on instinct. For each task class, keep 50 to 200 real inputs with an accepted output or a grading rule. Then the decision becomes a measurement rather than an argument.

• Classification and extraction — exact match or field-level F1 against labels. This is objective, so automate it and gate deploys on it.
• Generation — a model grader scoring against a short rubric, calibrated once by hand on a sample so you know where the grader is generous.
• Always report latency and cost alongside quality. A model that is one point worse and three times cheaper is often the right call; that is a decision for the product owner, and your job is to present it clearly.

Then ship the change as a shadow run first: send a percentage of live traffic to both tiers, compare the results offline, and switch only when the gap is within your tolerance. Keep the eval in CI so nobody quietly reverses the decision three sprints later.

Guardrails

• A per-tenant spend limit and a global one, enforced in code, with defined behaviour when either is reached — degrade to a smaller tier or queue the work, but decide in advance instead of during an incident.
• Rate limiting per tenant, so one automation loop cannot consume the budget of every other customer.
• Validate structured output against a schema and retry once with the validation error appended. This is cheaper and far more reliable than a larger model.
• Never let untrusted text choose the route or the tools. Prompt injection that can escalate a request to your most expensive tier is a denial-of-wallet bug.
• Log prompts and completions with PII redaction and a retention window you can actually defend to a customer.

Checklist

• Every call site declares a task class; the router is a pure, unit-tested function.
• Tier names map to concrete models in one config file per environment.
• Hard constraints first: context length, then tool use, then complexity, then streaming.
• Timeout per attempt, retry only what is retryable, circuit breaker per provider.
• A second provider wired up for anything critical.
• Stable prompt prefix at the front; response cache keyed with a prompt version.
• Max output tokens set per class; input budget trimmed deterministically and counted.
• One structured record per call; cost computed at reporting time.
• An eval set per class in CI, with quality, latency and cost reported together.
• Spend limits, per-tenant rate limits and schema validation with a single repair retry.

If you only do one thing from this list, make it the per-call record. Routing decisions without measurement are guesses, and guesses about model cost tend to be wrong in the expensive direction.