← Back to Dashboard
1. Telemetry Baselines2. Trace Correlation Across Agent Steps
Telemetry Baselines
📚 Observability and Tracing⏱ 10 min⭐ 95 XP
What to Measure on Day One
At minimum, capture request counts, latency percentiles, error rates, guardrail outcomes, token usage, and fallback frequency. Everything here is available from Bedrock's own telemetry surfaces - you just have to turn them on and route them somewhere queryable.
The Baseline Metric Set
| Metric | Why it matters | Alarm hint |
|---|---|---|
| Invocation count | Traffic shape, quota headroom | Sudden spike = runaway loop or abuse |
| p50 / p95 / p99 latency | Tail latency is what users feel | p95 trending toward SLO ceiling |
| Error + throttle rate | Capacity and dependency health | ThrottlingException rising = quota pressure |
| inputTokens / outputTokens | The raw material of your bill | Tokens-per-request drift = context bloat |
| Guardrail interventions | Safety posture and attack detection | Intervention spike = probing or policy drift |
| Fallback / escalation rate | How often degraded paths engage | Rising = upstream quality problem |
AWS Building Blocks
- CloudWatch metrics - Bedrock publishes invocation counts, latency, and token metrics per model out of the box.
- Model invocation logging - opt-in capture of full request/response bodies to S3 or CloudWatch Logs (mind data classification before enabling in regulated environments).
- CloudTrail - who invoked what, from which principal - your audit backbone.
Segmentation Is the Difference Between Data and Answers
- Break down by model, endpoint, and tenant - an aggregate p95 hides one tenant's pathological prompts.
- Track p50/p95/p99 separately - averages are marketing, tails are engineering.
- Correlate quality signals with cost over time - cost per successful outcome is the metric leadership actually needs.
Day-one rule: emit a structured log line per invocation - {traceId, model, tokens, latency, guardrailAction, outcome}. Every hard production question later ("why did costs double Tuesday?") is a GROUP BY over this table.
🧪 Knowledge Check
Press 1-4 to select1 of 2
Which latency metric is most useful for user pain?
Average only
p95 and p99 percentiles
No latency metric
Disk throughput