Valid JSON can still contain the wrong answer
Suppose your application extracts a tender deadline. The model returns a date in exactly the format your schema requires. Your parser is happy, but the date belongs to a different document.
A schema test alone will miss that error. An extraction evaluation should check three things: whether the result has the expected shape, whether each value is correct, and whether the source actually supports it. Then decide what happens when any of those checks fail.
Define one task and its ground truth
Start with a narrow output such as a supplier name, a submission deadline, a currency, and a source reference. Ask someone who understands the documents to label a set of examples. Record unknown values explicitly instead of guessing them.
Separate development cases from a held-out test set. If you tune a prompt against every example, the result can look better on paper than it performs on a new document. Expand the set when a real failure reveals a gap, while removing or replacing sensitive material before sharing it.
- Include complete documents and documents with required information missing.
- Add conflicting dates, multiple currencies, and ambiguous references.
- Include scanned or poorly extracted text if users will actually submit it.
- Add instructions embedded in a document that the extractor must treat as source data.
Separate schema checks from business checks
Structured Outputs can constrain a response to a supported schema. OpenAI's guide distinguishes a structured response from function calling, which is used when the model requests an application tool. The supported schema subset and model availability can change, so check the current guide for the API you use.
Even after a result passes schema validation, apply business rules in your own code. A valid date string can still be impossible for your workflow, and a valid identifier can still belong to a different account. Preserve the source passage so a reviewer can resolve a disagreement.
Compare fields without hiding missing values
The following TypeScript example compares a few fields after your application has already validated the shape of both records. It is deliberately small: it does not call a model, verify the source passage, or replace authorization and business rules.
type Extraction = {
supplier: string | null;
deadline: string | null; // normalized date, or null if unknown
currency: string | null;
};
type Field = keyof Extraction;
const fields: Field[] = ["supplier", "deadline", "currency"];
export function compareFields(expected: Extraction, actual: Extraction) {
return fields.map((field) => ({
field,
expected: expected[field],
actual: actual[field],
matches: expected[field] === actual[field],
}));
}
// Keep null distinct from an invented or incorrectly extracted value.
const expected = { supplier: "Northwind", deadline: null, currency: "EUR" };
const actual = { supplier: "Northwind", deadline: "2026-12-01", currency: "EUR" };
console.table(compareFields(expected, actual));Count refusals and incomplete responses separately
Not every response arrives as a usable object. OpenAI's Structured Outputs guide describes refusals and incomplete responses, including responses cut short by a token limit. Treat those as separate outcomes rather than converting them silently into empty fields or retrying without a limit.
Record the reason for failure, model and prompt versions, schema version, and the final review decision. Keep private source documents and secrets out of logs. If a retry is appropriate, give it a bounded policy and preserve the original outcome for analysis.
Choose measurements that match the job
Count field matches and source-supported values for each document. Also track missing-value handling, refusal and incomplete rates, review workload, and the rate at which reviewers correct a field. A single percentage can hide a serious failure on the one field that triggers payment or a deadline.
Use exact comparisons for deterministic fields where normalization is well defined. For subjective judgments, a model-based grader can help triage results, but compare it with human review before relying on it. OpenAI's evaluation documentation describes datasets and graders; check the current product status when choosing a platform, because evaluation interfaces can change.
Sources and further reading
Documentation reviewed November 9, 2026. Verify current model support, limits and pricing before implementation.