FIELD NOTE 009 / AGENT EVALUATION
How to Evaluate AI Agent Trajectories and External Effects
A practical guide to testing an AI agent's full trajectory, approvals, tool calls, external effects, and final system state before release.
By Harrison Ndeke · Published August 28, 2026 · Updated August 28, 2026 · 14 min read
AGENT EVALUATION
TASK → TRAJECTORY → EFFECT → STATE
In this article
DIRECT ANSWEREvaluate an AI agent by testing the whole run: the task, retrieved evidence, decisions, tool calls, approvals, external side effects, and final state. Record each trial against explicit success, safety, and operational criteria. Prefer deterministic checks for state and policy, use human review for consequential or ambiguous cases, and inspect trajectories to distinguish agent failures from broken tests.
Key takeaways
- Grade the outcome and the path: a fluent final answer cannot prove that the correct record was changed—or that no unauthorized action occurred.
- Make external effects first-class: record what the agent proposed, what a human approved, what the tool executed, and what state actually changed.
- Use different checks for different questions: deterministic assertions for contracts and state, human review for ambiguity, and model-based grading only when calibrated for the task.
- Test both action and restraint: include cases where the agent should act, ask, refuse, escalate, or stop.
- Do not confuse a template with results: the record and matrix below are an implementable starting point, not evidence that Harrison has run formal agent evaluations or measured outcomes.
What exactly should an agent evaluation inspect?
Start with a distinction that polished demos often blur. Anthropic defines a trajectory—also called a transcript or trace—as the complete record of a trial, including outputs, tool calls, intermediate results, and other interactions. It defines the outcome separately as the final state in the environment. An agent can say “done” while the intended state never changed, or reach the intended state through an unsafe route. See Anthropic’s agent-evaluation definitions.
For a tool-using business agent, inspect six layers: the input and task contract; evidence retrieved; decisions and policy checks; tool names and typed arguments; approvals and execution receipts; and the resulting state. The point is not to force one perfect chain of steps. Anthropic cautions that exact tool-order checks can reject valid approaches, so grade required invariants and outcomes unless sequence itself is a safety requirement.
How should external effects be evaluated?
Treat every write, send, purchase, permission change, booking, deletion, or system-of-record update as a separately verifiable event. The evaluation should answer: Was the action allowed? Were its arguments valid? Was required approval attached to that exact proposal? Did it execute once? Did the resulting state match the request?
Harrison’s published engineering boundaries make those questions concrete. Tool arguments should be validated before authorization; consequential actions should pause after a concrete proposal and before execution; retries should not duplicate a side effect; and logs, metrics, and traces should make failure visible. These are public design boundaries, not a claim that Harrison has completed a formal evaluation programme.
Use an immutable correlation ID across the proposal, approval, tool call, execution receipt, and state check. Record a redacted hash or stable reference for sensitive payloads rather than copying secrets or unnecessary personal data into the evaluation store. A pass requires evidence of the intended effect and evidence that forbidden effects did not occur.
What should an implementable evaluation record contain?
Keep the record compact enough to review and complete enough to reproduce. GitHub’s account of production LLM evaluation records the prompt, model, dataset version, and system configuration for each run; it also recommends changing one major variable at a time so a regression is attributable. See GitHub’s evaluation lifecycle.
type AgentEvaluationRecord = {
runId: string;
taskId: string;
startedAt: string;
versions: {
model: string;
prompt: string;
tools: string;
policy: string;
dataset: string;
};
input: {
fixtureId: string;
expectedIntent: string;
sensitivity: "low" | "restricted";
};
trajectory: Array<{
step: number;
kind: "retrieve" | "decide" | "tool" | "approve" | "respond";
name: string;
inputRef?: string;
outputRef?: string;
startedAt: string;
endedAt: string;
status: "ok" | "blocked" | "error";
}>;
externalEffects: Array<{
effectId: string;
proposalRef: string;
approvalId?: string;
idempotencyKey?: string;
executionReceipt?: string;
stateBeforeRef: string;
stateAfterRef: string;
}>;
checks: Array<{
id: string;
level: "step" | "trajectory" | "outcome";
grader: "code" | "human" | "model";
result: "pass" | "fail" | "unknown";
evidenceRef: string;
}>;
finalDecision: "pass" | "fail" | "review";
reviewer?: string;
notes?: string;
};Store references to trace payloads rather than exposing hidden reasoning or sensitive content. AWS shows how OpenTelemetry spans can reconstruct a session from top-level agent, inference, and tool-execution spans, including tool names, parameters, and results. That is an observability pattern, not a requirement to use AWS. See AWS’s trace-based evaluation walkthrough.
Which tests belong in a practical agent-evaluation matrix?
| Test task | Expected trajectory invariant | Outcome check | Primary grader |
|---|---|---|---|
| Valid read-only request | Uses allowed retrieval; no write tool or approval request. | Answer is supported by the permitted source set. | Code checks plus sampled human review. |
| Malformed tool proposal | Schema validation blocks unknown, missing, or wrongly typed fields before execution. | No external state change. | Deterministic. |
| Consequential valid action | Produces an exact proposal, pauses, binds approval to unchanged arguments, then executes. | One authorized state change with a receipt. | Deterministic plus human approval audit. |
| Missing or rejected approval | Stops before the effect; does not reinterpret silence as consent. | No external state change. | Deterministic. |
| Duplicate delivery or retry | Reuses the business idempotency key and returns the prior result or a controlled duplicate. | Side effect occurs at most once. | State and uniqueness checks. |
| Tool timeout after uncertain completion | Checks state or receipt before retrying; escalates if completion cannot be established. | No blind second effect; uncertainty is visible. | Deterministic plus operator review. |
| Unsupported or conflicting evidence | Does not invent a fact; asks, limits the answer, or routes for review. | No unsupported claim or consequential effect. | Human-calibrated rubric. |
| Observability break | Missing correlation, tool result, or state evidence makes the run ungradeable. | Result is review or fail, never an assumed pass. | Deterministic completeness check. |
Balance positive and negative cases. Anthropic recommends testing both when a behaviour should occur and when it should not; otherwise, a suite can reward over-triggering. Begin with manual release checks and real failure patterns, then add a reference solution that proves each task and grader can work.
How would this apply to the Standout4Growth RAG Chatbot?
Harrison’s public Standout4Growth RAG Chatbot demonstrates a dual-context assistant: company retrieval from a Supabase vector store, a brand-coach persona, and conversation memory. Public evidence shows the retrieval architecture and a live answer to a company question while the persona is maintained. It does not show a formal evaluation suite, measured accuracy, latency results, or security certification.
A defensible test set would therefore be prospective. Ask an answerable company question and require supporting retrieved records; ask an unanswerable question and require a limitation rather than invention; provide distractor content and verify that persona does not override company facts; test a memory turn and verify it does not become unauthorized company evidence; and confirm that a retrieval failure is observable. Record retrieval references, response, citation support, tool errors, and whether the agent stopped safely.
How should a team run this before release?
- Define the product decision. State the user outcome, safety constraints, and operational guardrails before choosing a metric. GitHub’s primary-source case study separates these categories so one attractive number cannot hide a safety regression.
- Version the moving parts. Record model, prompt, tools, policy, data, and harness. Change one major variable at a time.
- Reset the environment. Isolate trials so cached data, prior files, or shared state do not create false passes or correlated failures.
- Run repeated trials where variation matters. A single success is not proof of reliable behaviour. Keep individual records and report the trial count rather than smoothing away failures.
- Read failed and passed trajectories. Determine whether the agent failed, the environment failed, or the grader rejected a valid solution. Anthropic explicitly recommends transcript review before taking scores at face value.
- Gate consequential changes. A release should not advance when a safety invariant fails, even if answer quality improves.
What are the limits of trajectory evaluation?
A trace records what the instrumentation captured, not everything that occurred. Missing spans, asynchronous jobs, hidden downstream retries, stale fixtures, and incomplete state checks can create a reassuring but false story. Model-based graders are non-deterministic and require human calibration; human review is slower and can vary between reviewers. Production monitoring reveals real use but may lack clean ground truth. Anthropic presents automated evaluation, monitoring, user feedback, transcript review, A/B testing, and systematic human studies as complementary methods—not substitutes for one another.
This Field Note proposes an evaluation design from primary-source guidance and Harrison’s published engineering boundaries. It reports no pass rate, benchmark score, client result, cost saving, accuracy measurement, formal audit, or completed evaluation for Harrison’s systems. Any future result should name the dataset, trial count, versions, graders, thresholds, and known coverage gaps.
Executive summary
Evaluate an agent as a state-changing system, not a text generator. Define the decision first; test realistic tasks in isolated environments; record versions, trajectory events, approvals, effects, and final state; and match graders to the evidence. Validation, approval, idempotency, and observability become testable invariants. Read trajectories before trusting aggregate results, and refuse to call an unobservable or ambiguously graded run a pass.
Primary sources
- Anthropic Engineering: “Demystifying evals for AI agents”—definitions, grader types, task design, transcript review, and complementary evaluation methods.
- AWS Machine Learning Blog: “Evaluate any agent framework with Amazon Bedrock AgentCore Evaluations”—trace structure, span roles, tool-call evidence, and session-level evaluation.
- GitHub Blog: “How to evaluate LLMs before production”—product decisions, repeatable run records, production-like data, error analysis, and human review.