This guide was prepared with AI assistance using the linked documentation. Code examples are illustrative; verify them against your environment and current API documentation before use.
A cheap call can be an expensive task
A request that costs little but frequently needs a second model call or a human correction may cost more than a reliable alternative. Calculate at the level your customer cares about: a completed, accepted task.
Do not compare cost without holding quality and workload reasonably constant. A model that answers more quickly by skipping evidence may not be an acceptable substitute.
Choose the unit of work before calculating a rate. A completed document, a resolved support question, and an individual model call are different denominators. Put the unit next to the number so a reader cannot mistake a low cost per call for a low cost per successful customer outcome.
Gather the cost inputs
Record the provider usage returned for each attempt, the processing mode, tool calls, and the model and price version used. Include retrieval, storage, network services, and review work where they materially affect the workflow.
Provider prices and cache behavior can change. Use the applicable published rates and your account terms at the time of analysis rather than copying a fixed price into application logic.
A useful record joins usage to the application task rather than only to a raw API request. Capture the attempt number, model, processing mode, and task outcome. Separate provider-reported usage from costs estimated by your own team so a later audit can see which numbers came from which source.
Use a transparent calculation
The following formula is an accounting template. It does not use real prices and does not estimate a particular model. Use consistent currency and time windows, and state what you excluded.
If the period contains no accepted tasks, do not divide by zero or present a misleading average. Report the total spend and zero accepted tasks instead. Likewise, a task that is still awaiting human review should not be counted as accepted merely because a response was generated.
- Count all attempts, including failures and retries.
- Use accepted tasks from the same period and cohort as costs.
- Report the denominator and the quality criteria.
- Show uncertainty when review or infrastructure costs are estimated.
Total workflow cost = model usage + tool charges + infrastructure
+ retries + review cost
Cost per accepted task = total workflow cost / accepted tasksInclude caching and asynchronous work
Separate cached and uncached input where the provider's usage information supports it, and apply the correct rates. A cache hit depends on the model and matching context; it is not guaranteed. Prompt caching does not remove the need to account for rate limits.
Batch processing can suit independent work that can wait, while a background response handles a different kind of long task. Compare the applicable service terms and the cost of meeting your own deadline.
Run a fair comparison
Use the same test cases, expected evidence, acceptance criteria, and traffic assumptions for each configuration. Record the model, prompt, schema, corpus, and tool versions. If one configuration needs more human corrections, include that work instead of counting only the initial request.
Report latency and quality with the cost. A cheaper configuration that fails the required quality threshold is not a valid winner for that task.
Use a fixed evaluation set for an initial comparison, then test representative traffic before drawing a production conclusion. If a model handles one document type well but consistently fails another, report the groups separately. A blended average can conceal a failure for the customers who depend on the weaker path.
Set limits and review them
Define per-user or per-tenant budgets, output limits, timeouts, and a response when a budget is reached. Check whether the application can fail safely when a tool call or model request is unavailable.
Review the numbers after changing models, prices, prompts, or usage patterns. Keep a cost owner and an escalation path so a growing bill does not wait until month-end to be noticed.
Reconcile estimates with actual provider invoices or usage reports when those are available. Investigate discrepancies caused by retries, tool usage, rounding, or work performed outside the measured application path. Update the worksheet when pricing or your accounting assumptions change; keep historical calculations tied to the rates used at the time.
Example: include review and retries
Suppose reviewers accept only some document extractions without correction. Add the cost of model attempts, relevant tool calls, infrastructure, and the review work you chose to include. Divide by accepted tasks from the same set and time window.
Define accepted before comparing configurations. A valid response might require a correct deadline and its supporting passage, not just parseable JSON. If infrastructure or review costs are estimates, say so.
Sources and further reading
Documentation reviewed November 9, 2026. Verify current model support, limits and pricing before implementation.