A smaller-first LLM cascade saves inference spend when its first-pass and checking costs, plus the cost of escalated cases, stay below sending every case directly to the larger model. That comparison is useful only when both paths meet the same acceptance standard. A low escalation rate can mean efficient routing or errors that the checker failed to catch.
For a product owner, the decision is whether the routing system preserves acceptable work at a lower full cost. Start with the break-even calculation, then test the cases it sends down the wrong path.
Upfront routing and a smaller-first cascade have different bills
An upfront router classifies incoming work before choosing a model. A smaller-first cascade sends the work to the cheaper model, checks the result, then escalates selected cases. Escalated cases pay for both attempts.
Anthropic’s workflow guidance describes routing distinct input categories to specialised paths, including easier questions to smaller models and harder questions to more capable models. The important condition is that classification works accurately. The guidance establishes a useful pattern; it does not establish that a particular company’s requests can be separated reliably.
Consider a support-drafting workflow. A routine delivery-status question has a different evidence requirement from a disputed contractual promise. Message length alone does not distinguish them. A short request can require judgement about conflicting records, while a long message can ask for a straightforward summary.
An upfront rule can send known dispute categories directly to the stronger path. The calculation below addresses a different design: try the smaller model first, then decide whether its work is acceptable. Do not apply its formula unchanged to every routing architecture.
Calculate the maximum affordable escalation rate
Use costs per incoming case, with consistent currency and measurement boundaries:
| Symbol | What it includes |
|---|---|
| S | Smaller-model first pass, including its actual input and output usage |
| V | Checking cost paid on every case, including a model-based validator if used |
| L | Larger-model cost when used directly for the same task |
| E | Incremental larger-model cost on an escalated case, including any extra context passed forward |
| r | Fraction of incoming cases escalated |
The cascade’s expected inference and checking cost is:
C = S + V + r × E
It is cheaper than the direct path when:
r < (L − S − V) ÷ E
This boundary assumes E is positive and the costs describe the workload being evaluated. If the right-hand side is zero or negative, the first pass and checking already consume the direct-path budget. No reduction in escalation will make this cascade cheaper on those assumptions.
When costs vary by case, calculate the actual total across the replay. Difficult cases can have longer contexts, more output or additional tool calls. Multiplying an overall escalation rate by the average price of all larger-model requests can understate the cost of the subset that gets escalated.
The formula also leaves out fixed implementation work, maintenance and differences in human effort. Those belong in the investment decision. The accepted-outcome ROI guide explains how to connect operating spend to realised value rather than treating reduced token spend as a complete business case.
An illustrative cascade stops saving at 80% escalation
These are assumed costs for arithmetic, not current vendor prices or measured model performance. Suppose a replay contains 1,000 cases. Each smaller-model pass costs $0.003, checking costs $0.001 per case, and the larger-model path costs $0.020 either directly or after escalation.
Sending everything directly costs $20. The cascade pays $4 before any escalation. Its break-even escalation fraction is:
($0.020 − $0.003 − $0.001) ÷ $0.020 = 0.80
| Cases escalated | Escalation spend | Total cascade spend | Difference from direct path |
|---|---|---|---|
| 200 of 1,000 | $4 | $8 | $12 lower |
| 700 of 1,000 | $14 | $18 | $2 lower |
| 800 of 1,000 | $16 | $20 | Equal |
At 80% escalation, the smaller-first route has spent its entire advantage. A more expensive escalated context would bring that boundary down.
At 20% escalation, the apparent $12 saving is still an inference-and-checking result. If additional review costs $15 across the replay, the cascade costs $23 and loses its advantage over the $20 direct inference baseline, assuming the direct path has no corresponding extra review cost. Shared costs cancel only when they are genuinely equal.
Low escalation can hide the most expensive mistakes
Escalation rate measures where the system sends work. It does not measure whether that decision was correct.
A checker that accepts a plausible but unsupported contractual promise reduces the recorded escalation rate. The customer still receives a bad draft. Sending only rejected outputs for human inspection hides this failure because the accepted path never reaches the reviewer.
For each replay case, preserve the smaller-model result, the checker’s decision and the final result. Have an independent reviewer assess a sample of accepted first-pass outputs, including the consequential categories. Record escaped errors: unacceptable first-pass results that the cascade accepted without escalation.
Also inspect unnecessary escalations. They consume the larger-model budget without improving the outcome. These two mistakes have different consequences, so an aggregate routing-accuracy score is insufficient. Set a separate stop rule for consequential escaped errors rather than allowing cheap routine cases to average them away.
A model’s self-reported confidence does not replace this evidence. If confidence is used as a routing signal, test whether its thresholds separate acceptable and unacceptable results on held-out work. Repeated assurances from the same model are not independent checking.
Compare the routes on identical cases before changing traffic
OpenAI’s model-selection guidance recommends comparing models on the same inputs and retaining the lightest setting that meets the quality bar. For a cascade, compare the complete route against the direct path, not just the smaller model in isolation.
Use a replay with representative intake and a separately labelled set of difficult cases. Keep the latter visible rather than treating its deliberately enriched mix as a production escalation forecast. The pilot success-criteria guide covers that distinction and how to retain failed cases.
A practical decision record should capture:
- Record independent acceptance decisions for both paths on each case. Review the larger model’s work too; it can also be wrong.
- Separate consequential escaped errors from unnecessary escalations, and identify the categories producing each.
- Add both attempts and checking to the ledger, along with tool usage and additional reviewer time. Where acceptance differs, compare cost per accepted outcome.
- Measure completion time for first-pass acceptance and escalated cases separately. Report how many miss the workflow’s deadline.
The last item can veto an otherwise attractive saving. A sequential fallback adds an earlier attempt and a check before the larger model starts. OpenAI’s latency guidance identifies model speed and request design as optimisation levers. Measure the complete path: a faster first model does not guarantee a faster escalated result.
This is a proposed evaluation protocol. It has not been run against a named vendor pair, and the illustrative costs above do not predict a deployment’s savings.
Approve a bounded routing rule, with evidence for its boundary
A credible first change identifies the cases eligible for the smaller-first route, the escalation trigger and the conditions that send work directly to the existing path. Approval depends on acceptable outputs and deadlines as well as cost. Keep a held-out set for the decision; tuning the trigger and reporting success on the same examples overstates the evidence.
Model Operator is a Dubai-based AI product studio founded by Alexander Chrisostomou. Its AI Initiative Consulting work covers workflow analysis and implementation priorities, including deciding where additional AI complexity earns its cost.
Bring one workflow and a sample of its accepted and rejected outputs to a Model Operator build conversation. Use that evidence to decide whether model routing deserves an implementation trial before committing to another inference layer.