[ ABORT TO HUD ]
SEQ. 1
SEQ. 2
SEQ. 3
SEQ. 4

Agent Evaluation (Evals)

🏆 Evaluation & Production⏱ 10 min⭐ 100 BASE XP⌨ HANDS-ON LAB

Evaluating the Process, Not Just the Output

Standard LLM evals ask: "Is the final answer correct?"

Agent evals must use Trajectory Scoring. They ask:

  • Did the agent call the right tool?
  • Did it recover when the tool returned an error?
  • Did it loop infinitely?
  • Did it use the external data without hallucinating?

You must build a Golden Dataset of scenarios and use an LLM-as-a-Judge (e.g., prompting Claude 3.7 Sonnet or GPT-4o to grade a smaller agent's execution logs) to automatically score the agent on every pull request.

⌨ HANDS-ON LABRun Your First Eval Matrix
⭐ +150 XP

You changed one line of the system prompt - did quality go up or down? Stand up promptfoo, run a test matrix across prompts, and inspect results in the local viewer.

1Scaffold an eval config in the current directory with promptfoo init.
2Execute the evaluation matrix - every prompt � - provider � - test case.
3Open the web viewer to diff outputs side-by-side.
lab-sandbox — simulated environment
INFINITY LAB SANDBOX v2.6 — simulated shell
Type the command for the current objective. Helpers: "hint", "solution", "clear".
$
OBJECTIVE 1 / 3 — type "hint" if stuck
SYNAPSE VERIFICATION
QUERY 1 // 1
Why is evaluating an agent harder than evaluating a standard LLM chatbot?
Agents cost more
Because you must evaluate the entire trajectory (sequence of decisions and tool calls), not just the final text output
Agents output binary data
Agents cannot be evaluated