Last week I pasted an entire service — 41 files, most of a mid-sized repo — into a single prompt and asked for a bug hypothesis. A million-token window swallowed it without complaint. The answer came back confident, well-structured, and pointed at the wrong file. When I trimmed the same request down to the four files that actually mattered plus a one-paragraph description of the failure, the model found it on the first try.
That gap is the whole story of long context. The window got dramatically bigger. The model's attention did not scale with it. Making the most of a million tokens is not a capacity problem — it's a curation problem, and the teams treating it like free storage are quietly making their outputs worse.
The window is not the working memory
The number that expanded is the input limit. The thing that didn't is what Anthropic's engineering team calls the model's attention budget. Their write-up on context engineering puts it plainly: "as the number of tokens in the context window increases, the model's ability to accurately recall information from that context decreases." They call the effect context rot.
There's an architectural reason, not just an empirical one. A transformer computes relationships between every pair of tokens in the context — roughly n² of them. Double the input and you don't double the difficulty of finding the relevant signal; you quadruple the field of noise the model has to see past. Every token you add competes with every other token for a fixed pool of attention. Past a point, each additional token has diminishing marginal returns, and eventually negative ones.
So the framing that actually helps is the one Anthropic recommends: aim for the smallest possible set of high-signal tokens that maximize the likelihood of your desired outcome. Not the shortest prompt — a terse prompt that omits the constraint the model needed is worse than a longer one that includes it. The goal is signal density, not brevity.
Where you put things changes the answer
Placement is the cheapest win available, and most people get it backwards. Anthropic's long-context guidance is specific: when you're working with large inputs — they draw the line around 20,000 tokens — put the longform data at the top of your prompt, above your query, your instructions, and your examples.
The effect size is not subtle. Per their docs, moving the query to the end, after the documents, "can improve response quality by up to 30 percent in tests, especially with complex, multidocument inputs." That's a free 30% for cutting and pasting your question to the bottom of the prompt.
The second half is structure. Wrap each document in tags with real metadata so the model can tell your sources apart instead of blending them into one undifferentiated wall:
<documents>
<document index="1">
<source>annual_report_2023.pdf</source>
<document_content>
{{ANNUAL_REPORT}}
</document_content>
</document>
<document index="2">
<source>competitor_analysis_q2.xlsx</source>
<document_content>
{{COMPETITOR_ANALYSIS}}
</document_content>
</document>
</documents>
Using only the documents above, summarize the three metrics
where our position changed most against the competitor.
The tags aren't decoration. When ten documents arrive as one stream of text, "the model attributed a figure to the wrong source" is a real and common failure. An explicit <source> per document gives every fact a return address.
Make it quote before it reasons
The single most reliable technique I've adopted is quote-grounding, and it directly counters context rot. Before the model does the real task, make it pull the relevant passages out first:
Find quotes from the documents that are relevant to the question and place them in <quotes> tags. Then, using only those quotes, write your answer.
This does two things. It forces a retrieval pass over the full input while the whole thing is still in view, so the model commits to specific evidence instead of pattern-matching a plausible-sounding answer from a hazy overall impression. And it gives you an audit trail: if the quotes are wrong, you catch the reasoning error before it reaches production, instead of debugging a confident hallucination downstream.
Prefer just-in-time over dump-everything
The instinct with a huge window is to pre-load — pull the whole database, every log line, the full history — so the model "has everything it needs." Anthropic's agent guidance argues for the opposite: keep lightweight identifiers in context and let the agent load data at runtime when a step actually calls for it. Their own Claude Code does this, querying for the specific slice it needs rather than ingesting the dataset up front. The context stays small and every token in it is there because the current step earned its place.
For anything long-running, three patterns keep the window from silently filling with sludge:
- Compaction — periodically summarize the conversation so far and restart from the compressed version, so old, resolved detail stops consuming attention.
- Structured note-taking — have the agent write durable state to an external file it can re-read, rather than trusting it to survive in-context across dozens of turns.
- Sub-agents — hand a focused task to a specialized agent that returns only its distilled conclusion, keeping the raw intermediate work out of the main thread.
Every one of these is a way of spending the attention budget deliberately instead of letting it drain.
The takeaway
Treat the context window like a working set, not an archive. Before your next long prompt, run three checks: cut everything that doesn't change the answer, move your instructions and question to the very bottom, and tag each source so facts keep their provenance. If the task is big enough to need retrieval, load context when a step needs it instead of front-loading everything you might use.
The million-token window is worth having — not because you should fill it, but because it gives you room to be selective without running out. The engineers getting the most out of it are the ones putting the least in.
Sources: Effective context engineering for AI agents — Anthropic, Long context prompting tips — Claude Platform Docs