Operational Learning Loop

A strong postmortem is blameless, evidence-based, and action-oriented. It must result in concrete preventive controls and updated runbooks - for AI incidents, that specifically means new eval cases, adjusted gates, and tightened policies, so the same failure cannot ship twice.

AI Postmortem Structure

# INC-4402: Triage quality regression (P2)
Timeline:    14:02 deploy r7 → 14:40 escalation rate +35% → 15:10 rollback
Blast radius: 2,300 tickets misprioritised; 12 P1s delayed ~40min
Root cause:   prompt v3.3 reworded priority rules; eval golden set
              had NO cases for the reworded pattern → gate passed green
Contributing: canary held only 20min (peak traffic not observed)
Corrective actions:
  1. Add 45 misprioritised tickets to regression set   [owner: ai-plat, due: 07-24]
  2. Extend canary bake to include peak hour           [owner: sre,     due: 07-31]
  3. Alert on escalation-rate delta >20%/30min         [owner: obs,     due: 08-07]

The AI-Specific Root-Cause Questions

• Which layer failed? Prompt, model version, retrieval, guardrail, tool, or orchestration - the trace chain should answer this in minutes.
• Why didn't the eval catch it? Nearly every quality incident is also an eval-coverage gap. The failing cases become regression cases - that is the flywheel.
• Why didn't detection fire sooner? 38 minutes of user pain before a human noticed means a missing leading-indicator alarm.
• Did containment work as designed? If rollback took 30 minutes because the old prompt wasn't deployable, that's action item #1.

Actions That Actually Prevent Recurrence

Every action needs an owner and a due date, and the postmortem review verifies completion - an action list nobody tracks is a wish list.