Context windowsbound each generation.
Context windows:what the AI can use for one answer.
A context window is the model’s finite input-and-output budget for one generation. Instructions, conversation, retrieved evidence, tool definitions, and the response all consume it.For each answer, an AI product supplies a limited amount of information. Instructions, recent conversation, useful documents, and the new answer all need room.
- BOUNDEDLIMITEDmeasured in tokensonly so much fits at once
- SHAREDinput + output need roomthe question and answer share it
- NOT MEMORYpersistence lives elsewhereolder details can fall away
“Remember the decision from twelve turns ago?”
The model can use only the context assembled for this generation.The AI can use only the information supplied for this answer.
A chat transcript may look continuous in the interface. Underneath, an application or API service still decides what material becomes the model’s input for this turn.A chat may look like one endless conversation, but the product still chooses which parts are placed in front of the AI for each new answer.
- 01ASSEMBLEGATHER
The application assembles the input.The app chooses what the AI receives.
System instructions, tool definitions, retrieved material, prior turns, and the latest request are serialized into model input.It gathers instructions, useful documents, recent conversation, and the latest question.
- 02COUNTFIT
Every item consumes capacity.Every item takes up space.
Text—and, depending on the model, other modalities—becomes tokens. The total must stay within the model’s supported limits.Words, pictures, and the reply all use some of the available space.
- 03GENERATEANSWER
The response needs reserved capacity.The answer needs space too.
A full input budget can leave too little space for the output you actually wanted. Reserve room before sending the request.If the desk is completely full, there may not be enough room left for a useful reply.
The model cannot use information that the application did not include.
Input and output share one limit.
This fictional 30-unit window uses a sensible packing policy: keep instructions, relevant evidence, and the latest request; use remaining room for recent history. Increase the reply budget and see which older items no longer fit.
Kickoff decision is out. The model cannot use omitted material unless the application retrieves, summarizes, or re-inserts it.
This is a packing policy, not universal model behavior. Some APIs reject oversized input; some can truncate according to configuration; many applications manage history themselves. Measure the serialized request and know which policy you chose.
More context does not guarantee better context.
Capacity answers “can these tokens be present?” It does not guarantee that every detail will influence the answer equally—or that irrelevant material is harmless.
Choose context for the current question.
Select evidence for the current question. Remove duplicate or stale material. Put instructions where they are unambiguous. Preserve exact facts when exactness matters.
- Capacity is a hard boundary.
- Relevance is a selection problem.
- Attention quality is something you test, not assume.
Compaction saves space by discarding detail.
Compaction turns a long stretch of history into a shorter representation. It can preserve the main state of a conversation, but it cannot be assumed to preserve exact wording, chronology, or every edge case.
09:12 USER Let us use a queue, but only after the beta.
09:14 ASSISTANT Proposed an event-driven worker.
09:18 USER Important: EU jobs must stay in-region.
09:21 TOOL Current worker timeout is 30 seconds.
09:24 USER Keep the synchronous path for tiny exports.
After beta, route large exports to a regional queue. Keep small exports synchronous. Respect EU residency and the current 30-second worker timeout.
USEFUL, NOT LOSSLESS- goal
- constraints
- decision
- open question
- exact wording
- chronology
- minority view
- raw evidence
Assemble the smallest context that reliably supports the task.Give the AI the information this answer needs.
The goal is to include the instructions and evidence needed for this generation while reserving output capacity and excluding stale or irrelevant material.More is not automatically better. Give the AI the smallest useful set of instructions, evidence, and recent conversation it needs for this answer.
- 01
Budget the reply first.
Subtract expected output and tool overhead before filling the input. A request that barely fits may have nowhere useful to go.
- 02
Retrieve for the question.
Choose a small set of relevant source material instead of pouring the archive into the window because it technically fits.
- 03
Keep recent turns verbatim.
They carry immediate intent and references. Compact older history, then retain raw records elsewhere for recovery.
- 04
Count the full serialized request.
System prompts, tool schemas, examples, images, and application scaffolding all consume capacity.
- 05
Test what survives pressure.
Run long-conversation and crowded-context evals. Check whether the right evidence is present and whether the answer actually uses it.
window_limit
− reply_reserve
− tool / modality overhead
──────────────────────────
= usable input budget
input = [
instructions,
relevant_evidence,
compacted_history,
recent_turns,
current_request
]
assert tokens(input) <= budgetTokenization and multimodal accounting vary. Use the provider’s tokenizer or counting endpoint where available.
- Keep the instructions.
- Add only the evidence that helps this question.
- Summarize older conversation.
- Keep recent turns and the current request.
Text, images, tool descriptions, and the answer itself all use space—even when the chat screen does not make that obvious.
The request is rejected.
Some APIs return an error when the serialized input exceeds the allowed limit. This is annoying, but at least it is visible.
The wrong material survives.
A truncation or home-grown packing policy can remove the one fact that mattered. Log what entered the model, not just what the chat UI displayed.
Four common misconceptions.
“The chat remembers everything above.”
The interface may preserve a transcript. The model receives only the context assembled for the current generation.
UI ≠ INPUT“Oldest messages always drop automatically.”
There is no universal policy. An API may reject, truncate, compact, or rely on your application to manage the budget.
CHECK CONFIG“If it fits, the model will use it.”
Presence is necessary, not sufficient. Retrieval quality, placement, clarity, and model behavior still need evaluation.
TEST IT“A summary is basically lossless memory.”
A summary is a lossy edit. Keep exact records elsewhere and rehydrate details when the task demands them.
KEEP SOURCEKeep durable records outside the prompt. Assemble only the relevant context for this generation, and reserve room for the response.