Human reviewGuide6 min

Create a human review loop for generated recommendations

A product pattern for letting AI accelerate decisions while keeping accountable people in control.

Real workflow example

An account team receives AI-generated renewal-risk summaries from support tickets and product usage. Some are accurate; others miss context only the account manager knows.

Render each recommendation as a draft. The reviewer can approve it, edit the summary, reject it with a reason, or request more evidence. Downstream CRM updates run only after approval.

The AI speeds up analysis, but the accountable human keeps control of customer-facing and revenue-impacting records.

Implementation approach

This guide is anchored in OpenAI Responses API guide. Use the official API behavior as the boundary, then design the surrounding product state so the feature can be reviewed, retried, and improved.

Mark each generated recommendation as draft, approved, rejected, or needs evidence.
Expose source material and model reasoning summary next to the editable fields.
Let reviewers correct the output before approval instead of forcing a full rerun.
Store reviewer decisions as training and evaluation examples for future changes.
Send downstream notifications only after approval.

Workflow state shape

type human_review_loop_ai_recommendations_workflow_state = {
  sourceId: string;
  status: "draft" | "needs_review" | "approved" | "blocked";
  evidence: Array<{ source: string; summary: string }>;
  nextAction: string;
};

Field notes

Review loops work when the reviewer sees inputs, output, evidence, and consequences on one screen.
The system should record why people overrode or accepted the recommendation.
Review data becomes the best source for prompt and workflow improvement.

Mistakes to avoid

Do not make review a side-channel in Slack or email.
Do not lose the original model output after a human edits it.
Do not optimize only for approval rate; correction and rejection reasons matter more.

Ready checklist

Draft state exists
Reviewer can edit before approve
Override reason captured
Downstream action gated
Review metrics visible

Practical note

The edit diff is valuable. It shows what the model systematically misses.

Use this as an implementation constraint, not just advice. The interface, server code, and validation path should make the same behavior true.

Apply this to a build