Your AI feature takes eight seconds to answer. Retrieval or a tool call may account for much of that time. Without timings for each step, you cannot tell.
Add a request identifier and measure the whole path before changing the architecture. Look at typical and slow requests, such as p50 and p95, rather than relying on one fast demo.
Use prompt caching
Prompt caching reuses work when requests share a prefix. Put stable instructions and tool definitions together; place variable customer or request data later.
A hit is not guaranteed. Request content, tool definitions, settings, model, routing, and retention policies all matter. OpenAI documents different cache settings for newer and earlier models, so check the guide for the model you deploy.
Caching does not replace data isolation. Check that each user and tenant is authorized to see the data sent in a request, and review the provider's retention options before using sensitive material.
Record cache-read and cache-write usage where available. Compare total cost and end-to-end latency under real traffic instead of assuming a fixed discount.
Measure search and long tasks
Hosted web search can retrieve sources for questions that depend on current information. Background mode can support longer research tasks when the API and model support it. The model, tools, and workload all affect time and cost.
For long tasks, show progress and define cancellation, timeout, and retry behavior. A spinner alone is not enough to operate the workflow.
File search adds a retrieval step before the answer. Restrict the corpus to documents the user can access, organise the files, and measure relevance.
Tool definitions also consume context. Deferred tool discovery and programmatic tool calling may help in supported workflows. A smaller allowlist is often a simpler place to start.
Budget for the whole workflow
Count the whole task: input and output usage, tool calls, storage, retries, and human review. Set per-user and per-tenant limits, timeouts, and circuit breakers.
Measure quality next to cost. A cheap request that frequently needs manual correction may cost more overall.
Measure quality alongside cost
Build a regression set with expected outcomes, adversarial inputs, and known edge cases. Measure task success, evidence quality, schema validity, time to first useful output, completion time, and cost.
Compare changes with a baseline and record the model and prompt versions. Keep changes reversible. One synthetic benchmark is not a production service-level objective.
Use the worksheet below to record the environment, workload, traffic, latency percentiles, retention choice, and acceptance thresholds. Revisit the figures when provider capabilities, pricing, or model behavior change.
Sources and further reading
Documentation reviewed November 9, 2026. Verify current model support, limits and pricing before implementation.