This guide was prepared with AI assistance using the linked documentation. Code examples are illustrative; verify them against your environment and current API documentation before use.
Decide what a passing run means
An agent can give a good final answer after choosing the wrong tool, exposing a private document, or retrying a write twice. Those failures disappear if your test only compares the last paragraph.
Start with a single workflow and define acceptable behavior: which tools may be called, which evidence should be used, when the agent should stop, and which actions require approval. Write these as observable checks.
Build cases from real task shapes
Use synthetic or appropriately approved data. Include straightforward cases and the failures that matter to your application: missing information, conflicting sources, denied access, tool errors, interruptions, and requests for an unauthorized action.
Each case needs a stable identifier, input, permitted context, expected outcome, and a reason for the expectation. Keep a separate holdout set so repeated prompt tuning does not simply memorize the development cases.
- A valid request with the source present.
- An answer that is absent from the permitted documents.
- A tool that times out or returns an error.
- A request to act outside the user's permissions.
Capture the path through the run
A trace can record model calls, tool calls, handoffs, and guardrail decisions. OpenAI's agent evaluation guide describes trace grading as a way to inspect workflow-level behavior. Keep sensitive inputs out of a shared test report and control who can read traces.
The example below is a local assertion sketch. It does not call an evaluation service and assumes your application has already produced a sanitized trace.
type Trace = { calledTools: string[]; unauthorizedAction: boolean };
function checkTrace(trace: Trace, allowed: string[]): boolean {
return !trace.unauthorizedAction && trace.calledTools.every(name => allowed.includes(name));
}Choose a grader for each claim
Use deterministic checks for facts your code can verify, such as an allowed tool name or a valid identifier. A human reviewer can assess ambiguous cases and source support. Model-based grading may help scale subjective review, but compare it with human judgments before relying on it.
Provider tooling changes. OpenAI's Evals platform is in a deprecation transition, so check current documentation before choosing a managed surface. Your dataset and expected outcomes should remain understandable outside any one evaluation product.
Compare changes against a baseline
Run the same cases before and after changing a prompt, model, tool definition, or retrieval setup. Record the version of each component and inspect individual regressions, especially unauthorized actions and unsupported answers.
A single success percentage can hide an important failure. Break results down by case type and report the number of cases as well as the result. With a small dataset, treat the result as a debugging signal rather than a precise production forecast.
Keep the set alive
When a real failure is found, reduce it to a safe test case and add it to the regression set. Assign an owner to review results and decide when a change is ready to ship.
Measure latency and cost too, but never pass an unsafe run just because it is fast.
Make one case reproducible
A useful case gives a reviewer enough information to explain a failure without reading a whole transcript. For example, a policy assistant receives a question about expenses; the only permitted document says the deadline is 30 days; the expected outcome is an answer supported by that document. The case also names a tool the agent must not call.
Add a related case in which the deadline is absent. A passing result should acknowledge that the source does not answer the question. The two cases look similar to the model, but they test different decisions: finding evidence and recognizing a gap.
Record the input, synthetic source version, allowed tools, expected behavior, and reviewer notes in a versioned format. Run setup and cleanup predictably so a previous test cannot leave state that makes the next one pass by accident.
Report the result without hiding risk
After a run, separate failures into categories the team can act on: wrong retrieval, unsupported answer, unexpected tool call, missing approval, or recovery failure. Link each failed case to a sanitized trace. A single aggregate score is convenient, but it can hide a serious permission failure among many easy passes.
Compare the same cases before and after a change, and have a person inspect cases where the expected result is ambiguous. If a model-based grader disagrees with a human reviewer, investigate the disagreement rather than treating either output as automatically correct.
A small set is a starting point, not proof of a production success rate. Grow it from failures, review the ground truth when business rules change, and keep a holdout set that does not drive day-to-day prompt edits.
Sources and further reading
Documentation reviewed November 9, 2026. Verify current model support, limits and pricing before implementation.