What is Model Routing

Labs ship a ladder of models: small and cheap, large and expensive. Model routing is the policy above that ladder: use a cheaper, faster model for simple steps, and a stronger model only when the task needs it.

It is not a dropdown labelled “best.” Someone typing “use the best model” for a classify-this-ticket step is how a flagship invoice gets burned on work a compact model could have finished in a second.

Done well, extract runs on compact models and hard reasoning runs on frontier models. Done poorly, every step hits the most expensive model, spend becomes a surprise, and “we use the best model” becomes an unexamined religion.

OpenAI and Anthropic publish those ladders in public. The prices change. The shape does not: input, output, and sometimes tools meter differently, and the top rung is many times the compact rung. Finance cannot treat “always flagship” as a quality culture. It is an unbudgeted preference.

Words you’ll hear

• Frontier / flagship model. The strongest (and usually most expensive) model a lab currently sells. At work, this is for judgment: does this clause violate the playbook?
• Compact / small model. Faster and cheaper. Often enough for extract, classify, and summarise. At work, this is “pull the fields from the export.”
• Cascade. Try cheap first; spend the expensive call only when the cheap one is not enough.
• Fallback. If a provider is down or over budget, send the step somewhere else.
• Orchestration. What steps exist. Different from routing, which is which brain each step uses. You can orchestrate a brilliant multi-agent graph and still send every node to the flagship.
• Quality bar. The reject-rate or rework threshold that decides whether a compact model is good enough on this job.
• Data residency / data class. A cheap endpoint may be forbidden for a class of records. Routing is then a compliance table, not only a cost table.
• NTU. The normalised unit routing is trying to protect. See What is AI token economics.

Routing is also not fine-tuning (changing a model’s weights). Fine-tuning is a research and ops programme. Routing is an operating policy over models you already buy.

Constraints that belong in the route table: data residency, evaluation (you cannot route on vibes), and security (a model with web tools is a different actor than a model with none). Choosing a model is choosing a brain. Choosing tools is choosing hands. Decide them separately. See What is Model Context Protocol.

Why you should care

It affects you if you pay the bill, or if quality on a step is load-bearing.

Talk about it as a budget and quality conversation, not as an ML research project:

• Tag the steps. Extracting fields from an export is not the same as arguing whether a clause violates policy. If your platform cannot name steps, it cannot route them.
• Set a quality bar per step. “Compact model until human reject rate exceeds X on this job.” Without a bar, routing becomes “always escalate because someone was once unhappy.”
• Keep the gate regardless of model. A cheap model with a write tool is still a write tool. See What is write-back governance.

A spend ceiling without routing still lets every step hit the flagship until the ceiling kills the run. Routing is how you stay under the ceiling and finish the job. See What is an agentic workflow for why loops without stops dominate the bill.

What changes by role

Finance. Routing is the practical lever on unit cost. Quotes should assume the policy, not the flagship. Finance should ask for approved-updates per NTU, and for evidence that extract steps are not on the top rung. Locking one vendor forever is a pricing and outage choice; routing across providers is a second tape measure.

Legal. Data class and residency can forbid the cheap endpoint. Legal should sit on the route table for those classes, not discover them on an invoice. Customer-facing language may need a stronger model and a named signer; routing does not replace the gate.

Operations. Steps must be named or you cannot route them. Ops should own fallbacks when a provider is down, and should refuse a single “best” toggle that bypasses the table.

Go-to-market. Quality anxiety is strongest here. Measure reject rates on the job. A compact model that extracts next steps may be fine; a compact model that invents a concession is not. Routing on one unhappy anecdote will pin every step to flagship.

Security. A model with browsing or unconstrained tools is a different actor. Routing should not silently add hands. Prompt injection plus a flagship model plus write tools is a worse combination than a compact extract-only step behind a fail-closed gate.

What people get wrong

Always the smartest model. Use the weakest model that meets the quality bar for that step. Flagship is for judgment, not for labelling.

Routing as multi-agent. Several agents is a cast. Routing is which brain each step pays for. See What is multi-agent AI.

Routing as fine-tuning. Different programme.

Dropdown labelled “best.” That is not a policy. It is a preference that cannot be audited.

Dropping the write gate for a “trusted” model. Trust the gate. Models change weekly.

Routing on vibes. One anecdote becomes a permanent escalate. Measure rework.

Good looks like: named steps, a route table with cost, quality bar, and data class, cascade where it helps, fallback across providers, gates independent of model, falling unit cost as the wiki reduces re-derivation. Failure looks like flagship-everywhere and a board slide about the bill.

Adjacent ideas worth keeping separate: token economics is quote, cap, and attribute. Routing is which rung of the ladder a named step is allowed to use. Multi-agent AI is how many specialist roles run. You can route a single agent, and you can send a whole agent team to the flagship by mistake. MCP is hands, not brains: do not let a compact extract step inherit a write tool because “the helper was available.”

Evaluation has to live on the job, not in a model-arena screenshot. A compact model that extracts fields with a low reject rate is a success even if it would lose a public chatbot bake-off. A flagship model that drafts a concession the wiki forbids is a failure even if it is eloquent. Tie routing reviews to workstream outcomes — approved writes, rejects, rework — the same way you would review any other operating policy.

How this shows up in Nimbus

Nimbus treats routing as an operating decision tied to workstream steps: task type, sensitivity, and cost — not “best everywhere.” Release gates apply regardless of which model drafted the payload.

NTU quotes and ceilings sit around that policy so operators see a number before they commit. Everyday extract should not consume flagship credits.

See Models. For the unit of account routing sits inside, What is AI token economics. Product: Workstreams.

Questions people actually ask

Should we always use the smartest model?

No. Use the weakest model that meets the quality bar for that step. Flagship is for judgment, not for labelling.

Will routing make answers worse?

It can, if you under-route hard steps. Measure rejects and rework on the job. Do not route on a single anecdote.

Is this the same as having several agents?

No. Several agents is a cast. Routing is which brain each step pays for.

Can we lock one vendor forever?

You can. You will pay for it in price, outages, and lock-in. Routing across providers is how finance keeps a second tape measure.

What is a cascade?

Try the cheap model first. Escalate only when a confidence or quality check says the cheap pass is not enough. It is a tactic inside a policy, not a substitute for naming steps.

Does a better model remove the need for a wiki?

No. Stronger models are better at sounding like policy. Asserted playbooks still win over Drive folklore. See What is a company wiki for AI agents.

How do we set a quality bar?

Start with human reject rate and rework on that step. “Compact until rejects exceed X on this workstream” is a bar. “People like the flagship” is not.

Should customer-facing copy always use the flagship?

Not always. It should always use a human gate if it asserts a term or a price. Model size does not absorb Air Canada-style risk.

What if the cheap endpoint is in the wrong region?

Then it is not cheap; it is forbidden. Put residency in the route table beside price.

Does routing replace spend caps?

No. Caps stop unbounded loops. Routing makes legitimate work affordable under the cap. You want both. See What is AI token economics.

Can the model choose its own successor?

Letting the model always escalate is how every step becomes flagship. Escalation should be a policy check, not a preference the model expresses.

How does this show up in an RFP?

Ask whether steps are named, whether gates apply regardless of model, and whether finance sees a normalised unit. “We use the best models” is not an answer.

Related reading

What is AI token economics and What is multi-agent AI.

Sources

• OpenAI API pricing
• Anthropic pricing