This guide was prepared with AI assistance using the linked documentation. Code examples are illustrative; verify them against your environment and current API documentation before use.
A long conversation is not a database
A user changes the delivery address halfway through a conversation. Later, the assistant acts on the earlier address because it relied on stale context. Keeping every message is not enough: the application must know which facts are current and which actions still need authorization.
Separate conversation history from authoritative application state. Fetch the current record before a consequential action and ask for confirmation when the workflow requires it.
Choose how state is carried forward
OpenAI documents several approaches: manually providing prior messages, continuing from a previous response, or using the Conversations API with the Responses API for a durable conversation object. Choose based on the control, storage, and lifecycle your application needs.
Each approach has retention and failure implications. Review the provider's current documentation and your own data policies before deciding what to keep.
Make context limits visible in your design
A model's context window is shared by the input and output, with reasoning tokens relevant for supported reasoning models. Large histories, documents, and tool outputs compete for that space. An overlong request may lose information or produce a truncated output, depending on the workflow.
Measure token usage for representative conversations. Decide which information must remain available and how the application behaves when it cannot fit. Do not assume a summary preserves every detail.
Compact with an audit trail
Compaction or summarization can reduce context, but it can also lose a qualifier or an earlier decision. Keep authoritative records outside the summary. Record the summary version and retain source references where your policy permits.
The sketch below separates a summary from current state. It is a data-model example, not an implementation of OpenAI's compaction endpoint.
type ConversationContext = {
summary: string;
sourceMessageIds: string[];
currentRecordId: string;
};
// Load the current record and permissions from your application before a write.Keep stable context stable
Prompt caching can reuse matching prefixes on supported models. Stable instructions and tool definitions are better candidates for reuse than data that changes every turn. Cache behavior, retention, and supported settings differ by model; consult the current provider guide.
Caching does not replace tenant isolation. Limit which records enter the prompt and ensure cache accounting and retention meet your requirements.
Test what happens after many turns
Construct a long synthetic conversation with a corrected fact, a permission change, a tool failure, and a request that depends on an earlier source. Compare the assistant's answer and tool calls before and after compaction.
A good test checks that the latest authoritative value wins, a revoked permission is respected, and the assistant can say when context is insufficient. Track these cases whenever you change the model, summary strategy, or tool definitions.
Walk through a corrected address
A user first gives an address, then corrects the apartment number in a later turn. The assistant summarizes the conversation and eventually proposes a delivery action. A useful summary must not cause the old address to replace the new one.
Keep the current address in the authoritative application record, with the change associated with the appropriate user and workflow. Before the delivery action, the server reads that record again, checks the user's current permission, and presents the destination for confirmation when the workflow requires it.
In a regression test, make the earlier message and later correction unambiguous. Run the same scenario with and without your summary or compaction step. Check the action arguments, not only the assistant's final wording.
Decide what the application must retain
List the information needed for the next step: the user's request, approved choices, source references, open questions, and identifiers for authoritative records. A summary can help orient the model, while the application remains responsible for retrieving current data and enforcing permissions.
Review what should happen when a summary omits a constraint or a previous response cannot be retrieved. A clear request for the missing information is safer than guessing. Give the user a way to correct state, and record which version of a summary or record was used when investigating an error.
Token budgets are workload-specific. Measure representative long conversations, including tool output and the expected response, before choosing thresholds. If you use provider compaction or caching controls, verify the options for the selected API and model and test data-retention settings with your organization's policy.
Sources and further reading
Documentation reviewed November 9, 2026. Verify current model support, limits and pricing before implementation.