This guide was prepared with AI assistance using the linked documentation. Code examples are illustrative; verify them against your environment and current API documentation before use.
Define what must improve
Before changing a model or prompt, pick a task and an outcome you can check. For a support assistant, that could be whether an answer cites the right policy and whether an agent can correct it before it is sent.
Save the current behavior as a baseline. Without it, a successful demo cannot tell you whether the new version is better, worse, or simply different.
During a pilot, give reviewers a way to report why they rejected an output. Separate a missing source, incorrect extraction, slow response, and permission problem. These categories lead to different fixes, and counting them together makes it harder to decide whether the next release is ready.
Version the full decision path
A release is more than a model name. Record the prompt, tool definitions, schema, retrieval configuration, corpus version, application code, and model identifier. Any of them can change the result.
Keep the old configuration available for rollback. Do not silently reuse old evaluation results after changing the document parser or a tool's permissions.
Store the release record where operators can find it during an incident. Include the collection or retrieval version as well as the application version; otherwise a team may roll back code while leaving the changed source documents in place. Never put secrets or raw customer documents in the release record.
Build a reviewable test set
Start with synthetic examples that include normal requests, missing evidence, conflicting sources, authorization denial, tool failure, and instructions hidden in a document. For extraction, include expected values and the supporting passage. For generated answers, define what an evaluator should check.
Split examples by workflow or document type when their failure modes differ. A total score can hide a serious failure in a small but important group.
For each evaluation case, describe the input, expected outcome, evidence to inspect, and an owner for adjudicating disagreements. Include the cases where refusing or escalating is the correct behavior. A test that rewards an answer for every request can encourage unsupported answers when a document is missing.
Decide what triggers a stop
Agree on acceptance thresholds before comparing versions. Include task quality, evidence support, unauthorized actions, p95 completion time, cost, and the rate of cases escalated to a person.
Use the thresholds your own users and risk profile require. A borrowed benchmark or a single overall score cannot tell you whether a financial, medical, or other high-impact workflow is safe to operate.
Add an explicit stop condition for an unauthorized action or cross-tenant disclosure. Aggregate accuracy is not a reason to continue a pilot after a serious access-control failure. For less severe regressions, define who can decide whether to pause, fix, or keep the rollout limited.
Release in stages
Begin with internal or explicitly selected pilot users. Give reviewers the source and the proposed output, and make it clear when a result is incomplete. Expand access only after checking failures and the operational load created by review.
Keep tool permissions narrow. A configuration change that makes an agent more capable can also let it act on more data or systems; review that separately from a wording change.
- Pilot the smallest complete workflow.
- Inspect failure categories, not only averages.
- Make human escalation and cancellation easy to find.
- Document who can stop or roll back the release.
Re-run and rollback
After the release, compare the same workload and time window with the baseline. Watch for changes in the source documents and user mix. Save the failures that reveal a new behavior as regression cases, using synthetic or appropriately protected data.
If the new configuration breaches a stop condition, restore the earlier version and check that the rollback actually took effect. Re-test the affected workflow before expanding again.
Test rollback before relying on it. Restore the prior configuration in a non-production environment and verify the behavior on a small set of known cases. For any actions already taken by the new version, decide whether a separate reconciliation process is needed; changing the model configuration does not undo a message, ticket, or database write.
Example: replace a document collection
Run the same policy questions against the current and proposed collections. Compare whether each retrieves the current source, whether the answer follows that source, and whether unauthorized users can see a result.
Include outdated and missing policies. The expected behavior may be to ask for clarification or involve a reviewer rather than produce an unsupported answer. Record the configuration, expected results, failures, pilot audience, and rollback owner.
Sources and further reading
Documentation reviewed November 9, 2026. Verify current model support, limits and pricing before implementation.