Retries, Idempotency, and Timeouts
Operational Discipline for Agent Tools
Most production failures are orchestration failures: duplicate actions, runaway retries, or hung calls. An agent that creates three identical P1 tickets because a timeout retried is an agent nobody trusts again.
Idempotency: Same Intent, One Effect
# Deterministic key: same logical action always produces the same key
idempotency_key = hash(service + incident_signature + minute_bucket)
POST /tickets
Idempotency-Key: chk-7f3a2b-202607161430
→ first call: 201 Created (ticket #4402)
→ retry call: 200 OK (returns existing #4402, no duplicate)The receiving service stores processed keys and returns the original result on replay. Without this, every retry of a side-effect operation is a potential duplicate write.
Retry Policy That Doesn't Melt Downstream Systems
| Failure type | Retry? | Pattern |
|---|---|---|
| Transient (timeout, 429, 503) | Yes | Exponential backoff + jitter, budget of 2-3 attempts |
| Permanent (400 validation, 403 authz) | No | Return typed error to the agent - retrying cannot help |
| Ambiguous (connection dropped mid-call) | Only with idempotency key | Otherwise you risk duplicate side effects |
Timeout Budgets
Set a per-tool timeout (e.g. 5s) and a whole-plan budget (e.g. 60s). A 10-step agent where each step can hang for 30s is a 5-minute user wait hiding behind a spinner. When budget is exhausted: degrade gracefully - partial results plus an explicit "steps not completed" summary beats a silent hang.
- Use idempotency keys for every side-effect operation.
- Set timeout budgets and bounded retries with backoff + jitter.
- Emit structured logs for every tool decision - attempted, validated, executed, outcome, latency.
Define retry and idempotency rules for ticket creation and notification actions.