← Back to Dashboard
1. Token Economics and Caching2. Routing for Cost/Quality Balance

Token Economics and Caching

📚 Cost Engineering⏱ 10 min⭐ 95 XP

Cost Comes from Repetition and Over-Context

Large context payloads and repeated prompts dominate LLM cost curves. Because input tokens are billed on every request, a bloated system prompt is not a one-time cost - it is a tax on every single call, forever.

Where the Money Actually Goes

# A "small" inefficiency at scale
system prompt: 2,800 tokens (could be 900)
history sent:  6,000 tokens (relevant: ~1,500)
per-request waste: ~6,400 input tokens
� - 500k requests/month = 3.2B wasted tokens/month
→ at typical mid-tier input pricing, thousands of dollars for zero quality gain

The Optimization Ladder (Cheapest First)

LeverActionTypical impact
Trim contextCompact system prompts; window/summarise history; send top-3 chunks not top-1020-60% input reduction
Prompt cachingMark stable prefixes (system prompt, tool defs, few-shot examples) as cache checkpoints - cached tokens are billed at a steep discount on supported modelsUp to ~90% off cached prefix + latency win
Cap outputsSet maxTokens to the real need; instruct terse formats (JSON, bullets)Output tokens cost several � - input tokens
Right-size the modelRoute easy tasks to Nova Micro/Lite-class tiers (next lesson)Order-of-magnitude per-request difference
Batch inferenceMove non-interactive workloads to discounted async batch~50% off vs on-demand

Prompt Caching Mechanics on Bedrock

Structure prompts so the stable part comes first (policy, schema, examples) and the volatile part last (user message, retrieved context). Cache checkpoints only pay off when the prefix repeats within the cache window - design for prefix stability, and don't interleave dynamic content into the cached region.

  • Use compact system prompts - every sentence should earn its tokens.
  • Trim irrelevant history - summarise old turns instead of resending them.
  • Reuse cached context for repetitive workflows where supported.
Instrument first: log inputTokens/outputTokens per route before optimizing. The biggest saving is usually one chatty endpoint nobody suspected - measurement finds it in an afternoon.
🧪 Knowledge Check
Press 1-4 to select1 of 2
Most avoidable inference cost usually comes from:
TLS handshakes
Oversized repeated context payloads
CloudWatch metrics
S3 object tags