The model has a desk.Not a filing cabinet.
A context window is the bounded material available for one act of generation. Instructions, conversation, retrieved evidence, and the answer all compete for the same finite stage.
- BOUNDEDmeasured in tokens
- SHAREDinput + output need room
- NOT MEMORYpersistence lives elsewhere
“Remember the decision from twelve turns ago?”
Context is the current cut, not the archive.
A chat transcript may look continuous in the interface. Underneath, an application or API service still decides what material becomes the model’s input for this turn.
- 01ASSEMBLE
The app assembles a cut.
System instructions, tool definitions, retrieved material, prior turns, and the latest request are serialized into model input.
- 02COUNT
Tokens occupy the reel.
Text—and, depending on the model, other modalities—becomes tokens. The total must stay within the model’s supported limits.
- 03GENERATE
The answer needs headroom.
A full input budget can leave too little space for the output you actually wanted. Reserve room before sending the request.
The model cannot consult material that never made this cut.
Make room for the ending.
This fictional 30-unit window uses a sensible packing policy: keep instructions, relevant evidence, and the latest request; use remaining room for recent history. Increase the reply budget and watch the old scenes leave the reel.
Kickoff decision is out. The model cannot use omitted material unless the application retrieves, summarizes, or re-inserts it.
This is a packing policy, not universal model behavior. Some APIs reject oversized input; some can truncate according to configuration; many applications manage history themselves. Measure the serialized request and know which policy you chose.
Fits is not the same as matters.
Capacity answers “can these tokens be present?” It does not guarantee that every detail will influence the answer equally—or that irrelevant material is harmless.
Context engineering is editorial work.
Select evidence for the current question. Remove duplicate or stale material. Put instructions where they are unambiguous. Preserve exact facts when exactness matters.
- Capacity is a hard boundary.
- Relevance is a selection problem.
- Attention quality is something you test, not assume.
A synopsis buys space by selling detail.
Compaction turns a long stretch of history into a shorter representation. It can carry the plot forward. It cannot be assumed to preserve every line, hesitation, or edge case.
09:12 USER Let us use a queue, but only after the beta.
09:14 ASSISTANT Proposed an event-driven worker.
09:18 USER Important: EU jobs must stay in-region.
09:21 TOOL Current worker timeout is 30 seconds.
09:24 USER Keep the synchronous path for tiny exports.
After beta, route large exports to a regional queue. Keep small exports synchronous. Respect EU residency and the current 30-second worker timeout.
USEFUL, NOT LOSSLESS- goal
- constraints
- decision
- open question
- exact wording
- chronology
- minority view
- raw evidence
Practical context engineering.
The job is not “stuff in as much as possible.” The job is to assemble the smallest reliable cut that contains the instructions and evidence needed for this turn.
- 01
Budget the reply first.
Subtract expected output and tool overhead before filling the input. A request that barely fits may have nowhere useful to go.
- 02
Retrieve for the question.
Choose a small set of relevant source material instead of pouring the archive into the window because it technically fits.
- 03
Keep recent turns verbatim.
They carry immediate intent and references. Compact older history, then retain raw records elsewhere for recovery.
- 04
Treat instructions as cargo.
System prompts, tool schemas, examples, images, and application scaffolding all consume capacity. Count the serialized reality.
- 05
Test what survives pressure.
Run long-conversation and crowded-context evals. Check whether the right evidence is present and whether the answer actually uses it.
window_limit
− reply_reserve
− tool / modality overhead
──────────────────────────
= usable input budget
input = [
instructions,
relevant_evidence,
compacted_history,
recent_turns,
current_request
]
assert tokens(input) <= budgetTokenization and multimodal accounting vary. Use the provider’s tokenizer or counting endpoint where available.
The request is rejected.
Some APIs return an error when the serialized input exceeds the allowed limit. This is annoying, but at least it is visible.
The wrong material survives.
A truncation or home-grown packing policy can remove the one fact that mattered. Log what entered the model, not just what the chat UI displayed.
Four myths to leave on the floor.
“The chat remembers everything above.”
The interface may preserve a transcript. The model receives only the context assembled for the current generation.
UI ≠ INPUT“Oldest messages always drop automatically.”
There is no universal policy. An API may reject, truncate, compact, or rely on your application to manage the budget.
CHECK CONFIG“If it fits, the model will use it.”
Presence is necessary, not sufficient. Retrieval quality, placement, clarity, and model behavior still need evaluation.
TEST IT“A summary is basically lossless memory.”
A summary is a lossy edit. Keep exact records elsewhere and rehydrate details when the task demands them.
KEEP SOURCEKeep the archive outside. Assemble the right cut inside. Leave room for the ending.