Routing for Cost/Quality Balance — Cost Engineering

Tiered Inference Strategies

Route easy requests to low-cost models and reserve premium models for complex or high-risk tasks. The price spread between tiers is often 10-50x per token - routing is the single biggest cost lever most teams never pull.

The Escalation Pattern in Practice

request → Nova Lite (cheap tier)
  ├─ output validates + confidence ≥ 0.8  → serve (≈85% of traffic)
  └─ validation fails / low confidence    → escalate to Nova Pro / Claude tier
        └─ still failing → human queue

# Result: premium prices paid only for the ~15% that needs premium capability

Batch: The Forgotten Half-Price Tier

For large, non-interactive workloads, use Batch inference: submit many prompts as JSONL to Amazon S3 and get responses back asynchronously at roughly 50% of on-demand pricing. Constraints to design around:

• No tool calling or structured-output enforcement - each record is processed independently with no back-and-forth.
• Asynchronous completion - hours, not seconds; wrong for anything a user is waiting on.
• Perfect for: nightly classification/enrichment, eval-suite runs, embedding backfills, content moderation sweeps.

Measure the Metric That Matters

A premium model at 3x the token price that succeeds first-time can beat a budget model that retries twice and escalates - only outcome-level accounting reveals this.