← Back to Dashboard
1. Containerized Orchestration Services2. Release Strategies and Rollbacks

Release Strategies and Rollbacks

📚 Bedrock with Containers10 min110 XP

Safe Delivery for AI Backends

Use blue/green or canary releases with feature flags for prompt/model changes. Roll back quickly when quality or safety degrades. The twist versus classic services: an AI release can be code-identical and still behaviorally different - prompt text, model ID, guardrail config, and KB sync state are all release artifacts.

What Actually Constitutes a "Release"

release manifest v2026.07.16-r2
  code:        orchestration-svc:1.14.2
  prompt:      ticket-triage v3.2
  model:       amazon.nova-pro-v1:0 (pinned)
  guardrail:   gr-support vX7
  kb:          support-kb sync 2026-07-15
  eval:        golden 0.89 | safety 0.99 | p95 1780ms

If any element changes, the manifest version changes - and the eval gate reruns. This is what makes "what exactly did we ship?" answerable during an incident.

Rollout Mechanics

StrategyMechanismBest for
Blue/greenTwo task sets behind the LB; shift 100% atomically after bakeModel/prompt swaps where mixed versions confuse users
CanaryWeighted routing (1% → 25% → 100%) with metric gates per stageMost changes; catches issues at minimal blast radius
Feature flagsPer-tenant/percentage toggles inside the servicePrompt variants, new tools, A/B experiments

Rollback Reality

  • Rollback must revert the whole manifest - reverting code while keeping the new prompt is a half-rollback that debugs like a haunted house.
  • Keep the previous manifest warm and deployable; config-driven model/prompt IDs make revert a one-line change.
  • Keep release logs tied to eval scores and incident timelines - when an incident starts, the first question is "what changed and what were its scores?".
Bake time matters: LLM regressions often surface on traffic diversity, not volume. Hold canary stages long enough to see peak-hour and edge-case traffic before promoting.
🧪 Knowledge Check
Press 1-4 to select1 of 2
A safe release for AI backends includes:
No rollback path
Canary/blue-green with score-gated promotion
Immediate global rollout always
No traceability