← Back to Dashboard
1. Offline Evaluation Design2. Online Eval and User Feedback Loops
Online Eval and User Feedback Loops
📚 Evaluation Frameworks⏱ 10 min⭐ 90 XP
Production Learning Loops
Offline evals prove readiness; online evals prove reality. Once real users arrive, their behavior becomes your richest - and cheapest - evaluation signal.
Signal Sources, Strongest First
| Signal | What it tells you | Caveat |
|---|---|---|
| Task completion | Did the user accomplish the goal (ticket resolved, doc accepted)? | Define "completion" per workflow explicitly |
| Behavioral signals | Copy/paste of answer, retry with rephrasing, session abandonment | Implicit - correlate, don't over-interpret single events |
| Escalations | Human takeover rate and reasons | Rising escalations = quality or trust regression |
| Thumbs up/down | Direct sentiment on specific responses | Low response rate, negativity bias - use as trend, not truth |
| Incident outcomes | Postmortems attributing impact to AI behavior | Lagging but highest-severity signal |
Closing the Loop
- Sample and label - pull flagged interactions (bad feedback, escalations, abandonments) into a review queue.
- Diagnose the layer - was it retrieval, prompt, routing, or guardrail? (Your trace IDs make this a query, not an investigation.)
- Feed the datasets - every confirmed failure becomes a regression-set case; every novel attack joins the adversarial set.
- Adjust the lever - prompt revision, routing threshold, guardrail tuning - then verify offline before redeploying.
This is a flywheel: production failures continuously harden the offline suite, which prevents their recurrence.
Anti-pattern: dashboards nobody acts on. Every online signal needs a routing rule - who reviews it, within what SLA, feeding which dataset. Telemetry without a consumer is decoration.
🧪 Knowledge Check
Press 1-4 to select1 of 2
What distinguishes online evals from offline evals?
They use no metrics
They include real user behavior and outcomes
They cannot run in production
They ignore safety