Controlled Model Upgrades

A model swap is a production change with blast radius - treat it like a database migration, not a config tweak. Introduce model changes through staged rollout gates: canary traffic, scorecard comparison, and rollback criteria.

The Rollout Pipeline

1. Offline eval - run the candidate model against your golden + adversarial datasets. It must meet or beat the incumbent on quality, safety, latency, and cost before any traffic.
2. Shadow mode (optional) - duplicate real requests to the candidate without serving its answers; compare outputs offline at production traffic shape.
3. Canary - route 1-5% of traffic; segment metrics by model; watch p95 latency, guardrail intervention rate, task completion.
4. Progressive shift - 25% → 50% → 100%, holding each stage long enough to see representative traffic (including peak).
5. Post-rollout watch - drift can appear days later as users adapt their inputs.

Scorecard Dimensions

Rollback Discipline

Always define rollback triggers before rollout starts - e.g. "guardrail interventions +30% over 1h" or "task completion −5% sustained 30min". Decided in advance, rollback is a runbook step; decided during an incident, it is a debate. Keep the previous model configuration deployable at all times (config-driven model IDs make this a one-line revert).