A context window is the maximum amount of text, measured in tokens, that an LLM can hold and reason over in a single call, spanning the system prompt, the conversation so far, any retrieved documents, and the model's own response.
What actually fills up a context window in production?
Rarely the user’s message alone. A typical call stacks a system prompt, conversation history, tool definitions, and, in a RAG system, a batch of retrieved passages, all competing for the same budget measured in tokens. A window that looks generous on paper fills fast once retrieval, tools, and history are all stacked into the same call, and every extra token costs money whether or not it helps the answer.
Why is a bigger context window not automatically better?
Length is a ceiling on what a model can hold, not a guarantee it reasons well across all of it: models measurably lose track of facts placed in the middle of a very long context, a pattern often called the lost-in-the-middle effect. This is the practical case for retrieval and deliberate context management over simply pasting in everything available, the discipline covered in context engineering: curate what actually goes in, rather than trusting a large window to sort it out.
Frequently asked questions
Does a bigger context window mean a smarter model?
No. It means the model can hold more text at once, not that it reasons better over all of it. Models reliably lose track of details buried in the middle of a very long context, so a bigger window raises the ceiling without raising the floor.
What happens when I exceed the context window?
The call fails or the system truncates older content to fit, usually dropping the earliest messages first. That is why long agent runs need a deliberate strategy for what to keep, not just a bigger window to postpone the problem.