AI Observability: Trace a Request Without Logging the Conversation

A single model timing is not enough

A customer reports that an answer took twelve seconds. The model may have been slow, but retrieval, a tool, a queue, or your own rendering path could have used most of the time. A trace lets you connect those steps to one request.

Start with the user-visible task and add spans for the components you own. Preserve the distinction between a successful HTTP response and a task that actually solved the user's problem.

Agree on a small set of fields before instrumenting every component. Start with task and attempt identifiers, timing, outcome, and configuration version. Inspect a sample export in a test environment, then ask whether each additional field is necessary to investigate an actual failure. Recheck the list after a new tool or exporter is introduced.

Choose the spans

A useful first trace can include request validation, document retrieval, model generation, each tool invocation, and response delivery. Attach a generated trace identifier to the support record so an operator can find the right execution without searching by prompt text.

Use the same generated task identifier across the browser request, backend job, and tool execution where your architecture permits it. A trace should answer which step failed and whether any later action ran. If a task is retried, preserve the relationship between attempts so a dashboard does not mistake them for separate successful tasks.

• Record start, end, and outcome for each step.
• Track timeout and retry attempts as distinct events.
• Measure time to first useful output and total completion separately.
• Connect a task failure to its underlying error class.

Keep attributes low-risk

OpenTelemetry's generative AI attribute registry flags several content-bearing attributes as potentially sensitive. Do not copy every prompt, response, tool argument, or retrieved passage into a trace by default. Decide which fields are needed, who can see them, and when they expire.

Use a pseudonymous identifier only when it is needed for the operational question and supported by your privacy policy. A hash alone does not automatically make data anonymous.

Define an allowlist for telemetry fields and review it with the people who operate the system. Model identifiers, duration, outcome, and token counts may be useful for a performance investigation; document titles, customer identifiers, and tool arguments may require additional access controls or omission altogether. The right choice depends on the data and purpose.

• OpenTelemetry generative AI attributes ↗

Record timing without a provider SDK

The example below illustrates how to record elapsed time for a local operation. It is not a complete tracing implementation: a real application needs a tracing SDK, span context propagation, export configuration, and a privacy review.

async function measure<T>(name: string, run: () => Promise<T>) {
  const start = performance.now();
  try {
    const result = await run();
    console.info({ operation: name, outcome: "ok", ms: performance.now() - start });
    return result;
  } catch (error) {
    console.info({ operation: name, outcome: "error", ms: performance.now() - start });
    throw error;
  }
}

Connect quality and cost to the trace

For each completed task, record available token usage and tool-call counts alongside latency. Record a quality outcome when an evaluation or reviewer can supply one. A low-latency answer that cites the wrong document is still a failed task.

Do not calculate a cost from a price copied months ago. Store the price version and use the applicable rate for the model and processing mode. Compare results over the same time window and workload.

Create separate views for request volume, failure rate, latency distribution, and evaluation outcomes. Avoid treating an HTTP 200 as evidence that an answer was correct. When reviewers label a case, connect their decision to an appropriately protected task record rather than adding the full conversation to a general metrics dashboard.

Turn traces into a debugging loop

Reproduce a slow request with synthetic data. Compare the slow trace with a normal one, isolate the dominant span, change one factor, and rerun the same workload. Check p50 and p95, not a single observation.

Review access to the telemetry system and its retention policy. A debugging tool should not become an uncontrolled archive of customer conversations.

For an incident drill, use a synthetic document and force one retrieval timeout. Confirm that the trace identifies the failing step, the user receives an honest incomplete status, and the operator can find the record without access to a production customer document. Review both the data collected and who can open it.

• OpenAI production best practices ↗

Example: investigate a slow answer

Suppose retrieval takes most of the total time in a trace. Check the query, collection, network path, and access filter before changing models. If generation dominates, investigate input size, output requirements, and unnecessary work.

These are hypotheses, not universal fixes. Rerun the measurement under a comparable workload, then confirm that evidence retrieval and answer quality have not worsened.