← Back to Dashboard
1. Regional Failover Patterns2. Chaos Testing for AI Services
Chaos Testing for AI Services
📚 Multi-Region Resilience⏱ 10 min⭐ 115 XP
Prove Resilience Before Real Incidents
Run game days where model endpoints, queues, or downstream tools fail. Validate fallback behavior, user impact, and operator response time. Chaos drills turn assumptions into measured resilience - and AI stacks have failure modes that classic chaos suites never exercise.
AI-Specific Fault Injections
| Injection | Simulates | What you verify |
|---|---|---|
| Force ThrottlingException on N% of calls | Quota exhaustion at peak | Backoff+jitter works; no retry storm; queues stay bounded |
| Add 5-15s latency to model calls | Degraded model endpoint | Timeout budgets fire; streaming UX degrades gracefully |
| Return malformed/truncated model output | stopReason max_tokens, schema drift | Validation catches it; bounded repair loop; no garbage to users |
| Kill a tool endpoint mid-plan | Downstream dependency outage | Circuit breaker opens; escalation with partial-state summary |
| Empty retrieval results | KB sync failure / index corruption | Model says "I don't know" instead of hallucinating confidently |
| Fail over to secondary region live | Regional impairment | Equivalence map holds; quotas absorb; residency respected |
Running a Credible Game Day
- Hypothesis first - "if the primary model throttles at 50%, users see cached answers within 2s and zero errors." A drill without a hypothesis is theatre.
- Measure operator experience - time-to-detect, time-to-diagnose (did traces make it obvious?), time-to-mitigate.
- Score the blast radius - which tenants/features degraded, and did priority queues protect the critical ones?
- File the gaps - every surprise becomes a backlog item with an owner and date, and the drill reruns after the fix.
Start in staging, graduate to production: mature teams run scoped production chaos (1% traffic, business hours, kill switch armed). An untested fallback path is a second bug waiting behind the first - and you'd rather meet it on a Tuesday morning drill than a Friday night incident.
🧪 Knowledge Check
Press 1-4 to select1 of 2
Chaos testing primarily validates:
Brand colors
Real behavior under failure
IAM creation speed
Markdown formatting