Needle in a Haystack

The needle in a haystack test is a widely used evaluation of a language model's long-context recall. A single out-of-place statement — the needle — is inserted into a much larger body of unrelated filler text — the haystack — and the model is asked a question whose answer depends only on that needle. By repeating the test across many context lengths and many insertion depths, the evaluation produces a heatmap showing where in its advertised context window a model reliably retrieves information and where it quietly fails.

Origin and method

The test was popularized by Greg Kamradt, who placed a deliberately incongruous sentence at depths ranging from the very top to the very bottom of documents of increasing length, then plotted retrieval accuracy as a two-dimensional grid: context length on one axis, needle depth on the other. The methodology is disarmingly simple, which is its virtue. Early runs on frontier models revealed distinct dead zones — regions of length and depth where the model simply could not surface the planted fact — and those results echoed the academic lost-in-the-middle finding that models retrieve information from the beginning and end of a long context far more reliably than from its interior. A model marketed with an enormous context window could, in practice, be blind to material buried at the 40 percent mark of a long document.

What a clean heatmap does and does not prove

The test's appeal is that it is cheap, intuitive, reproducible, and tied to a real failure mode. Its limitation is that retrieving one distinctive, semantically isolated sentence is close to the easiest possible long-context task. A model can post a perfect needle heatmap and still fail at what long context is actually for: synthesizing themes across a whole document, reasoning over multiple scattered facts at once, tracking entities through a long narrative, or noticing that two distant sections contradict each other. Harder descendants of the test exist for exactly this reason — multiple needles that must be aggregated, needles that paraphrase rather than repeat the question's wording, and adversarial haystacks full of near-miss distractors. Treat a clean single-needle result as a necessary baseline, not as proof that a long context window is fully usable end to end.

Running it yourself

For sovereign deployments the test earns its keep as a bench instrument. Published heatmaps describe a vendor's model served on the vendor's stack; your local reality — a quantized open-weight model, your inference server's rotary-embedding scaling settings, your actual KV-cache limits — can differ substantially, and quantization or context-extension tricks sometimes degrade deep-context recall in ways that never show up in a marketing table. The recipe is straightforward: take documents representative of your real workload, plant facts at controlled depths, sweep the lengths you actually intend to use, and score the retrievals. An afternoon of this tells you the honest operating envelope of your pipeline before you trust it with a 200-page contract or a season of maintenance logs. That is the same verify-don't-trust reflex that puts a block explorer next to every Bitcoin payment: measure the system yourself, on your own hardware, before you rely on it. See lost in the middle for the positional bias this test exposes.

Interpret results with the workload in mind, too. A model that recalls perfectly at 32,000 tokens but degrades at 100,000 is not broken — it may be exactly right for a pipeline that chunks documents to 20,000 tokens with retrieval, and wrong for one that stuffs whole manuals into a single prompt. The heatmap is a map of an operating envelope, not a verdict; the engineering decision is whether your use case stays inside it. Cheap, honest measurement beats vendor benchmarks every time, and this test remains the cheapest honest measurement long-context AI has.