Agent Evaluation (Evals)

Evaluating the Process, Not Just the Output

Standard LLM evals ask: "Is the final answer correct?"

Agent evals must use Trajectory Scoring. They ask:

• Did the agent call the right tool?
• Did it recover when the tool returned an error?
• Did it loop infinitely?
• Did it use the external data without hallucinating?

You must build a Golden Dataset of scenarios and use an LLM-as-a-Judge (e.g., prompting Claude Opus to grade a smaller agent's execution logs) to automatically score the agent on every pull request.