← Back to Dashboard
1. Reference Architecture Blueprint2. Launch Checklist and Day-2 Operations
Launch Checklist and Day-2 Operations
📚 Capstone: Production Bedrock Stack⏱ 12 min⭐ 130 XP⌨ Hands-on lab
Day-2 Matters More Than Day-1
After launch, success depends on ongoing evaluation, incident learning, and controlled iteration. Day-1 proves the system works once; day-2 proves it keeps working while prompts, models, traffic, and attackers all change underneath it.
The Launch Gate (Pass/Fail, Signed)
| Category | Gate | Evidence |
|---|---|---|
| Quality | Golden ≥ threshold; regression suite green | CI eval report for the launch manifest |
| Safety | Adversarial block rate ≥ target; zero known false negatives; human gates wired | Safety suite scores + gate demo |
| Cost | Cost/outcome modelled at 3� - projected traffic; budgets + 80% alerts armed | Load-test token telemetry |
| Resilience | Failover drilled; backpressure verified; kill switch tested | Game-day report with timings |
| Governance | Policy-control-evidence map current; runbooks published; oncall trained | Signed readiness review |
Formal owner sign-off is not bureaucracy - it is the moment someone accountable confirms every category was actually checked, with their name on it.
The Operating Rhythm After Launch
daily: cost + safety anomaly review (5 min dashboard scan)
weekly: quality drift review - online signals vs launch baseline
· escalation-rate trend · new adversarial samples → suites
monthly: routing matrix sanity vs new model releases
· quota headroom check · stale-KB audit
quarterly: chaos drill · policy-map review · postmortem action auditDrift Is the Default
- Input drift - users learn the system and change how they ask; last quarter's golden set slowly stops representing reality. Refresh it from production samples.
- Ecosystem drift - new model versions, deprecations (e.g. Agents Classic → AgentCore), pricing changes: each one re-triggers your eval + routing review.
- Team drift - the builders move on. Runbooks, traces, and the release manifest are what make the system operable by their successors.
Bedrock Enterprise SOE (Standard Operating Environment)
| Layer | Baseline / Standard | Hardening & Pinning Rule |
|---|---|---|
| SDK Baseline | AWS SDK v3 (@aws-sdk/client-bedrock-runtime) / boto3 >= 1.35.x | Pin major/minor versions; lock dependencies via npm package-lock or poetry.lock |
| Identity & IAM | IAM Roles with Session Tags | Zero hardcoded AWS access keys; assume IAM roles dynamically with least-privilege action boundaries |
| Network Isolation | Bedrock VPC Endpoints (AWS PrivateLink) | Enforce VPC endpoint policies and Service Control Policies (SCPs) blocking direct internet egress |
| Encryption | AWS KMS Customer Managed Keys (CMK) | Mandatory CMK encryption for all model invocations, guardrails, and Knowledge Base vector stores |
Bedrock Enterprise AOE (AI Operations Engineering) Blueprint
- Cross-Region Inference (CRI) Resilience: Configure Bedrock Cross-Region Inference profiles (US/EU) with automatic dynamic traffic re-routing during regional throttle spikes.
- CloudWatch OpenTelemetry Metrics: Publish custom metrics for
InvocationLatency,GuardrailInterventions, andPromptCacheHitRate; alert when P95 latency exceeds 2500ms. - Automated CI/CD Quality Gates: Integrate AgentCore Evaluations in AWS CodePipeline; automatically block canary deployments if golden set accuracy < 85% or adversarial safety < 97%.
- Runbook-Driven Incident Remediation: Provide executable automated runbooks for quota increases, guardrail sensitivity tuning, and rollback to prior prompt-model manifests.
The graduation test: six months post-launch, a new on-call engineer handles a quality incident full using only your runbooks, dashboards, and traces - without paging anyone who built it. That is what "production-ready" actually means.
⌨ HANDS-ON LABRun a Pre-Launch Readiness Review
⭐ +180 XPUse a final readiness checklist across quality, safety, cost, resilience, and governance before launch.
1Create a launch checklist document.
2Record launch owner sign-off.
OBJECTIVE 1 / 2 — type "hint" if stuck
🧪 Knowledge Check
Press 1-4 to select1 of 2
Why is day-2 operations emphasized in AI systems?
Models never change
Behavior and usage drift require continuous control
No incidents occur
No monitoring needed