Protect the System Under Load

Use queueing, admission controls, and backpressure to prevent cascading failures during traffic spikes. LLM calls are seconds-long and quota-bound - a spike that a normal microservice absorbs will pile up in front of Bedrock quotas and take your whole request path down with it.

The Failure Cascade You're Preventing

traffic spike → Bedrock TPM/RPM quota hit → ThrottlingException
→ naive clients retry immediately → retry storm amplifies load
→ request queues grow unbounded → memory pressure + timeout waves
→ upstream services time out → user-facing outage

Control Points

Capacity Strategy

A common production shape: Provisioned Throughput for the predictable baseline, on-demand (with cross-region inference profiles) absorbing the burst above it, and batch inference handling everything that doesn't need to be synchronous at all.

• Cap concurrent expensive requests per tenant.
• Use graceful degradation for non-critical features - shorter outputs, cheaper tier, cached answers.
• Prioritize critical workflows in queue policy.