What is a context window, and does a million tokens matter?

A model's context window is the maximum amount of text it can consider in a single request: your prompt, any documents you include, the conversation so far, and the answer it writes. Everything outside the window does not exist to the model.

How big is big

In 2023 a typical window was 4,000 to 8,000 tokens, roughly ten pages. Today the flagship tiers from OpenAI, Anthropic and Google all advertise around a million tokens, which is roughly 1,500 pages, or a mid-sized codebase, or every email you sent last year.

What a bigger window buys you

• Whole-document questions. Drop in a 300-page contract and ask what changed from the last version, with no retrieval pipeline.
• Long conversations. An agent can keep its full working history instead of summarising and forgetting.
• Many examples. You can show the model hundreds of worked examples in the prompt instead of fine-tuning.

What it costs

Every token in the window is billed as input on every request. A conversation that carries a 500,000-token document costs 500,000 input tokens each turn unless you use prompt caching, and some vendors charge a higher rate above a threshold (see the pricing tracker). Latency also rises with context length.

Where it still breaks

Models do not read a million tokens with uniform attention. Independent tests repeatedly show accuracy drops for facts buried in the middle of very long inputs, and reasoning across many distant parts of a document is harder than reading any one of them. For large, frequently queried corpora, retrieval (RAG) that selects the relevant pieces is still cheaper and often more accurate than stuffing the window.

Rule of thumb

Use the big window when the whole document matters and you ask few questions of it. Use retrieval when the corpus is large, changes often, or you will query it thousands of times.