← Back to Dashboard
1. Latency Budgets and SLOs2. Concurrency and Backpressure

Latency Budgets and SLOs

📚 Performance and Throughput⏱ 10 min⭐ 100 XP⌨ Hands-on lab

Define Performance Targets Before Incidents

AI systems need explicit SLOs like any critical service. Without latency and reliability targets, scaling decisions become guesswork and every slowdown becomes a debate about whether it matters.

Decompose the Latency Budget

full latency is a sum of parts - budget each one so regressions are attributable:

User-perceived p95 target: 2,500ms
├─ API gateway + auth          ≤ 60ms
├─ input validation/guardrail  ≤ 120ms
├─ retrieval + rerank          ≤ 350ms
├─ model TTFT (time to first token) ≤ 800ms
├─ generation (streamed)       budget by output length
└─ post-processing + egress    ≤ 100ms

For streaming UX, time-to-first-token is often the SLO that matters - users forgive long completions that start instantly.

AI-Specific SLO Dimensions

DimensionExample SLONotes
Latencyp95 ≤ 2200ms, TTFT ≤ 900msPer endpoint, not global
Availability99.9% successful completionInclude fallback-served responses separately
Quality proxyTask completion ≥ baseline − 2%From online eval signals
Policy complianceGuardrail pass rate ≥ 99.5%Safety is an SLO, not a hope
  • Set p95 and p99 targets by endpoint - a chat UI and a batch enrichment job deserve different budgets.
  • Track queue wait vs inference time separately - they need different fixes (capacity vs model choice).
  • Alert on trend, not only hard breach - "p95 rising 10%/day" is actionable before users notice.
Error budget logic applies: if the quality SLO is burning too fast after a prompt release, that is a rollback trigger - the same discipline SRE applies to availability, extended to AI behavior.
⌨ HANDS-ON LABDefine AI SLOs
⭐ +150 XP

Set production service level objectives for latency, safety, and completion quality.

1Create an SLO policy file.
2Add alert thresholds for violations.
lab-sandbox — simulated environment
INFINITY LAB SANDBOX v2.6 — simulated shell
Type the command for the current objective. Helpers: "hint", "solution", "clear".
$
OBJECTIVE 1 / 2 — type "hint" if stuck
🧪 Knowledge Check
Press 1-4 to select1 of 2
Why are p95/p99 targets critical?
They are optional vanity metrics
They capture tail latency user pain
They replace all logs
They only matter offline