← Back to Dashboard
1. Latency Budgets and SLOs2. Concurrency and Backpressure
Concurrency and Backpressure
📚 Performance and Throughput⏱ 10 min⭐ 100 XP⌨ Hands-on lab
Protect the System Under Load
Use queueing, admission controls, and backpressure to prevent cascading failures during traffic spikes. LLM calls are seconds-long and quota-bound - a spike that a normal microservice absorbs will pile up in front of Bedrock quotas and take your whole request path down with it.
The Failure Cascade You're Preventing
traffic spike → Bedrock TPM/RPM quota hit → ThrottlingException
→ naive clients retry immediately → retry storm amplifies load
→ request queues grow unbounded → memory pressure + timeout waves
→ upstream services time out → user-facing outageControl Points
| Control | Mechanism | Effect |
|---|---|---|
| Admission control | Reject/queue at the front door when depth exceeds bound | Fail fast beats fail slow |
| Concurrency caps | Semaphore per tenant + per model tier | One noisy tenant cannot starve the rest |
| Bounded queues + priority | Critical workflows first; batch/enrichment waits | Graceful degradation, not random drops |
| Backoff + jitter on 429s | Exponential retry spacing across clients | Prevents synchronized retry storms |
| Provisioned Throughput | Reserved Model Units (no-commit, 1-mo, 6-mo terms) | Guaranteed capacity for baseline load |
Capacity Strategy
A common production shape: Provisioned Throughput for the predictable baseline, on-demand (with cross-region inference profiles) absorbing the burst above it, and batch inference handling everything that doesn't need to be synchronous at all.
- Cap concurrent expensive requests per tenant.
- Use graceful degradation for non-critical features - shorter outputs, cheaper tier, cached answers.
- Prioritize critical workflows in queue policy.
Load-test reality: soak-test at 2� - expected peak with production-shaped prompts (long contexts, real tool loops). Synthetic 50-token pings tell you nothing about how quotas behave under genuine load.
⌨ HANDS-ON LABConfigure Token Bucket Rate Limiter with Backoff
⭐ +150 XPSimulate concurrent request bursts against Bedrock quotas with exponential backoff and jitter.
1Run load test simulating 100 concurrent workers.
OBJECTIVE 1 / 1 — type "hint" if stuck
🧪 Knowledge Check
Press 1-4 to select1 of 3
Backpressure mechanisms primarily prevent:
Prompt engineering errors
System-wide meltdown during overload
IAM misconfiguration
Schema drift