← Back to Dashboard
1. Containerized Orchestration Services2. Release Strategies and Rollbacks
Release Strategies and Rollbacks
📚 Bedrock with Containers⏱ 10 min⭐ 110 XP
Safe Delivery for AI Backends
Use blue/green or canary releases with feature flags for prompt/model changes. Roll back quickly when quality or safety degrades. The twist versus classic services: an AI release can be code-identical and still behaviorally different - prompt text, model ID, guardrail config, and KB sync state are all release artifacts.
What Actually Constitutes a "Release"
release manifest v2026.07.16-r2
code: orchestration-svc:1.14.2
prompt: ticket-triage v3.2
model: amazon.nova-pro-v1:0 (pinned)
guardrail: gr-support vX7
kb: support-kb sync 2026-07-15
eval: golden 0.89 | safety 0.99 | p95 1780ms
If any element changes, the manifest version changes - and the eval gate reruns. This is what makes "what exactly did we ship?" answerable during an incident.
Rollout Mechanics
| Strategy | Mechanism | Best for |
|---|---|---|
| Blue/green | Two task sets behind the LB; shift 100% atomically after bake | Model/prompt swaps where mixed versions confuse users |
| Canary | Weighted routing (1% → 25% → 100%) with metric gates per stage | Most changes; catches issues at minimal blast radius |
| Feature flags | Per-tenant/percentage toggles inside the service | Prompt variants, new tools, A/B experiments |
Rollback Reality
- Rollback must revert the whole manifest - reverting code while keeping the new prompt is a half-rollback that debugs like a haunted house.
- Keep the previous manifest warm and deployable; config-driven model/prompt IDs make revert a one-line change.
- Keep release logs tied to eval scores and incident timelines - when an incident starts, the first question is "what changed and what were its scores?".
Bake time matters: LLM regressions often surface on traffic diversity, not volume. Hold canary stages long enough to see peak-hour and edge-case traffic before promoting.
🧪 Knowledge Check
Press 1-4 to select1 of 2
A safe release for AI backends includes:
No rollback path
Canary/blue-green with score-gated promotion
Immediate global rollout always
No traceability