Context window management compared
Context windows keep getting bigger (a million tokens is normal now), but two problems don’t go away just because the window got bigger. Sending more tokens still costs more and takes longer, and a model doesn’t necessarily use everything in a long prompt equally well. A well known finding, lost in the middle, showed that a fact sitting in the middle of a long context gets used less reliably than the same fact sitting near the start or end, even though every token is technically right there the whole time. So most real systems don’t just widen the window and stop thinking about it. They decide what actually goes in. I built four of those decisions side by side and measured what each one actually costs and gets wrong.
Read the code (dashboard not deployed yet).
The setup
I built a controlled test instead of using a naturally long document, because a naturally long document doesn’t let you control where the important fact sits or how long the surrounding text is. Take a real short paragraph, question, and answer from SQuAD, the “needle”, and bury it inside a pile of other, unrelated SQuAD paragraphs, the “haystack”, at a controlled length and a controlled position. This is the same construction the original lost-in-the-middle paper uses.
Every needle got tested at 3 lengths (short, ~500 words; medium, ~2,000; long, ~6,000) and 3 positions (start, middle, end of the haystack), 9 documents per needle, 8 needles, 72 documents total, each run through all 4 techniques below. Everything ran on the same local model, llama3.1:8b through Ollama, so any difference comes from what a technique put in front of the model, not from a different model answering.
flowchart LR
A["1 needle: paragraph + question + answer"] --> C{"Build haystack document"}
B["Unrelated SQuAD paragraphs"] --> C
C --> D["3 lengths x 3 positions"]
D --> E{"4 techniques"}
E --> F["llama3.1:8b via Ollama"]
F --> G["Predicted answer + cost"]
Four ways to decide what stays
- Full Context: every word goes in, no decisions made. The baseline.
- Sliding Window: keep only the last 1,500 words, drop everything older, the same idea as a chat app that only resends the last few turns.
- Retrieval Selection: embed every paragraph, keep only the 3 most similar to the question.
- Hierarchical Summarization: process the document in ~1,200-word chunks, folding each one into a running summary as it goes, and answer from that summary plus the last chunk kept word for word. This is the same idea behind Claude’s own context compaction feature: don’t drop older content, compress it.
What actually happened
| Technique | Got the answer right (F1) | Mean tokens per call | Mean latency |
|---|---|---|---|
| Full Context | 79.5% | 3,947 | 32.7s |
| Retrieval Selection | 76.4% | 596 | 3.5s |
| Sliding Window | 52.6% | 1,676 | 12.5s |
| Hierarchical Summarization | 50.2% | 4,628 | 74.1s |
Full Context won, but Retrieval Selection came close at roughly a sixth of the tokens and a ninth of the latency, and at the longest documents it actually won outright (77.7% vs. 75.6%), the one case where dumping everything in should hurt the most.
The other two techniques scored much lower overall, but not because they’re uniformly worse. Breaking the results out by length and position (both averaged over 8 needles apiece) tells a much sharper story than the two headline numbers do:
Sliding Window fails as a hard cutoff, not a gradual slide. Score exactly 0% whenever the needle falls outside its 1,500-word window, and a flat 73% whenever it falls inside, at every length. No in-between. That’s exactly what you’d expect from a technique that makes no judgment about relevance at all, only about recency, and it’s worth knowing that going in if you’re about to reach for this approach for anything longer than your window: it doesn’t degrade politely, it just stops working the moment the thing you need scrolls out of range.
Hierarchical Summarization failed completely, 16 documents out of 16, whenever the needle landed in the middle of a chunk that got folded into the running summary, at both medium and long lengths. Not “mostly wrong”, not “partial credit”, every single one of the 8 needles said “Insufficient information” or something equally unhelpful. When the same technique’s needle sat right at the start of a chunk, or survived in the untouched final chunk, it did fine. This isn’t a new discovery: recursive summarization losing detail is well documented, and so is lost-in-the-middle. What’s genuinely interesting is how cleanly the two effects compound here, once, into a total, deterministic-looking failure, rather than a moderate accuracy hit.
One honest caveat: Full Context’s own position breakdown (middle scored best, end scored worst) doesn’t reproduce the classic lost-in-the-middle shape. I checked the actual wrong answers before writing that down, and it’s driven by 2 or 3 specific hard questions failing at the edges, not a uniform pattern across all 8 needles. Eight needles per cell just isn’t enough to say anything definitive about Full Context’s position sensitivity either way. Sliding Window’s and Hierarchical Summarization’s failures, by contrast, reproduce perfectly every time, because they come from how each technique is built, not from the model’s moment-to-moment attention.
A real bug worth mentioning
Ollama’s own model listing reports llama3.1:8b supports up to 131,072 tokens of context. The server process it actually launches to handle a request defaults to a runtime window of only 4,096 tokens unless you explicitly ask for more. My “long” documents run to about 8,000 tokens once the prompt is included, so my first attempt at this run silently overflowed that default and hung. The fix was one line, explicitly requesting num_ctx: 16384 on every call, but it’s a good reminder that a model card’s rated context length and what a local server actually serves you are two different numbers, and the failure mode is silent unless you go looking for it.
Browse the code, which has every technique’s exact prompts, the full length x position grid, and a document explorer to compare all 4 techniques on the same document side by side.