What is a context window? LLM tokens, 1M limits and recall

Key takeaways

• A context window is the maximum number of tokens a large language model (LLM) can work with in one request. That count covers your prompt, documents, chat history, tool definitions, any hidden reasoning, and the reply itself.
• Current flagship models from Anthropic, OpenAI and Google all offer roughly 1 million tokens. Output is capped much lower: 128,000 tokens per synchronous request for Claude and GPT-6 models, and 65,536 by default for Gemini.
• One token is about four characters of English text. Anthropic estimates that 1 million tokens hold roughly 555,000 words on its current tokenizer.
• A bigger window does not guarantee better recall. The 2023 "Lost in the Middle" study found that models use information at the start and end of a long prompt much better than information in the middle.
• For large or changing document sets, retrieval (RAG), summarisation or compaction usually beats pasting everything into the prompt, both on accuracy and on cost.

What is a context window in an LLM?

What is a context window? It is an LLM's working memory: the maximum number of tokens it can read and write in one request, including your prompt, history and the reply. Flagship models from Anthropic, OpenAI and Google now offer roughly 1 million tokens.

The context window is not the model's training data. Training data shapes what a model knows in general. The context window holds what it can see right now: your instructions, pasted files, earlier turns of the conversation, results returned by tools, and the answer it is producing.

When people say a model has a "1M context window", they mean the total budget for one request is about one million tokens. Once that budget is used up, the request either fails or the output stops early.

What is a token, and how are tokens counted?

Models do not read words or characters directly. A tokenizer splits text into tokens, which are short chunks such as whole words, word pieces, punctuation or spaces. OpenAI's open-source tokenizer library, tiktoken, describes itself as "a fast BPE tokeniser" and says each token averages about 4 bytes in practice. Google gives a similar rule for Gemini: about 4 characters per token, or 60 to 80 English words per 100 tokens.

Each vendor has its own tokenizer, so the same text produces different counts on different models. Anthropic notes that its newer tokenizer, introduced with Claude Opus 4.7, fits roughly 555,000 words into 1 million tokens. Earlier Claude models fit about 750,000. Code, non-English text and unusual formatting usually need more tokens per word than plain English prose.

Text is not the only thing that uses tokens. Images, audio and video are converted into tokens too. Google's token guide gives these figures for Gemini:

Source: Google AI for Developers, "Understand and count tokens", accessed Oct. 23, 2026.

At those rates, an hour of video comes to roughly 950,000 tokens (263 × 3,600), nearly an entire 1M window. Google puts 1 million tokens of text at about 50,000 lines of code or around eight average-length English novels.

What counts toward the context window?

Everything in the request counts, not just the message you type. Anthropic lists the system prompt, every message in the conversation (including tool results, images and documents), tool definitions, and the model's output for that turn, including its extended thinking.

• Conversation history. In a chat over the API, each new turn resends every earlier turn. A long conversation fills the window even if each message is short.
• Reasoning tokens. Reasoning models think before they answer. OpenAI says these hidden tokens "still occupy space in the model's context window" and are billed as output. It recommends reserving at least 25,000 tokens for reasoning and output when you start experimenting.
• Cached prompts. Prompt caching lowers the price of a repeated prefix, but Anthropic notes that cached tokens still take up the window. Caching changes what you pay, not how much room is left.
• Tool definitions. Agents with dozens of tools can use thousands of tokens on tool descriptions before the user has asked anything.

Input limit vs output limit

The headline context figure is a combined budget. Each model also has a separate, much smaller cap on how many tokens it can generate in one reply. Vendors handle the split in different ways:

• OpenAI GPT-6 Astra has a 1,050,000-token context window with a maximum of 922,000 input tokens and 128,000 output tokens. The two limits add up to exactly the window.
• Google Gemini 4 Argon has a "unified" 1,048,576-token window that input, thinking and output all share. Output defaults to 65,536 tokens, and the "long decode" mode allows up to the full window.
• Gemini 3.8 Flash uses separate budgets instead: 1,048,576 input tokens and 65,536 output tokens.
• Claude models generate up to 128,000 output tokens per synchronous request. On Claude 4.5 and later models, the API accepts a request even if input plus max_tokens exceeds the window. If generation then hits the limit, it stops with stop_reason: "model_context_window_exceeded". If the input alone is too long, every Claude model returns a 400 error ("prompt is too long").

Context window sizes of major models (October 2026)

Data as of Oct. 23, 2026, from the official model pages of Anthropic, OpenAI and Google. These are API limits. Consumer chat apps such as ChatGPT, Claude.ai and the Gemini app can use different limits depending on the plan, and Anthropic notes that chat interfaces may drop the oldest turns on a rolling basis. For a head-to-head on price and benchmarks, see our Gemini 4 Argon vs GPT-6 Astra and Claude Opus 5.5 comparison.

Why long context still loses information

A model that accepts 1 million tokens does not use all of them equally well. Anthropic's documentation says that "as token count grows, accuracy and recall degrade," and calls this context rot.

The "Lost in the Middle" finding

The best-known evidence is a 2023 paper by Nelson F. Liu, Percy Liang and colleagues, accepted for publication in Transactions of the Association for Computational Linguistics. They tested models on multi-document question answering and key-value retrieval while moving the relevant passage around the prompt. Performance was "often highest when relevant information occurs at the beginning or end of the input context." It "significantly degrades" when the answer sits in the middle, even for models built for long context.

Advertised size vs effective size

A 2024 benchmark called RULER, by Cheng-Ping Hsieh and colleagues and presented at COLM 2024, tested 17 long-context models that all claimed windows of 32,000 tokens or more. The authors found that "only half of them" kept satisfactory performance at 32K. Almost all of them scored near-perfectly on the simple "needle in a haystack" test, which hides one fact in filler text. The lesson is that single-needle results overstate how well a model reasons over a full window.

Google makes the same point in its long-context guide. Its models can reach about 99% on a single-needle query, but accuracy falls when a prompt contains several facts that must all be found. Google adds a cost caveat as well: you pay the input-token price for the whole context every time you send that query.

What a full context window costs

API pricing is per token, and the whole history is resent on every turn, so a long context gets expensive fast. Take Claude Opus 5.5, listed at $4 per million input tokens. A 500,000-token document costs about $2.00 in input for each question asked about it. Twenty follow-up questions resend that document twenty times, which adds up to about $40 in input alone.

Prompt caching changes that maths. Anthropic bills cache reads on Opus 5.5 at 5% of the base input price, about $0.10 per question for the same document, plus a one-time charge for writing the cache. Some models also charge more for very long prompts: OpenAI prices GPT-6 Astra requests above 272,000 input tokens at twice the input rate and 1.5 times the output rate for the whole request. Our guide on how to reduce LLM API costs covers caching, batching and routing in detail.

Latency rises too. Google notes that longer queries take longer to process, so a huge prompt makes every reply slower.

When to use RAG, summarisation or long context

Practical rules that follow from the vendor guidance and the research above:

1. Put the question last. Google advises placing the query at the end of the prompt, after the context, especially for long inputs.
2. Send only what the model needs. Google and Anthropic both warn against padding the window. Curating what goes in matters as much as the size limit.
3. Count before you send. Anthropic offers a token-counting API, Gemini has a count-tokens method, and tiktoken counts tokens locally for OpenAI models.
4. Leave room for the answer. Budget output and reasoning tokens explicitly. OpenAI's 25,000-token reserve is a sensible starting point for reasoning models.
5. Compact long sessions. Anthropic offers server-side compaction in beta on Claude 4.6 and later models. It summarises earlier turns automatically so a conversation can continue past the limit.
6. Test it. Build a small evaluation set with answers placed at different positions and depths. Our guide to evaluating an LLM app in Python shows how.

If you are building retrieval yourself, our step-by-step RAG app tutorial with Ollama and Chroma runs entirely on a local machine. Our prompt engineering guide covers how to structure long prompts with clear sections and tags.

What it means

Flagship models from Anthropic, OpenAI and Google now accept about a million tokens per request, which removes many hard limits. A whole codebase, a contract bundle or hours of transcripts can now fit in one request. The research is consistent, though: capacity is not comprehension. Models still find facts at the edges of a prompt more reliably than in the middle, and every extra token adds cost and latency.

For most applications, the best results come from treating the context window as scarce. Retrieve or summarise what matters, put the question at the end, reserve room for the answer, and measure recall on your own data instead of relying on the advertised number.

Sources

• Anthropic: Context windows
• Anthropic: Models overview
• OpenAI: Models
• OpenAI: GPT-6 Astra model page
• OpenAI: Reasoning models guide
• OpenAI: tiktoken on GitHub
• Google: Gemini API long context
• Google: Understand and count tokens
• Google: Gemini 4 Argon model page
• Google: Gemini 3.8 Flash model page
• Liu et al., Lost in the Middle: How Language Models Use Long Contexts (arXiv:2307.03172)
• Hsieh et al., RULER: What's the Real Context Size of Your Long-Context Language Models? (arXiv:2404.06654)