Long Context Window

A long context window refers to a language model whose maximum input length stretches far beyond the few-thousand-token windows of early transformers — commonly 128K or 200K tokens on modern models, with some advertising a million or more. This lets the model ingest entire books, multi-file codebases, or months of conversation history in a single pass, instead of relying on external retrieval to feed it small chunks at a time. For self-hosters, it is one of the most misunderstood specifications on a model card, because the advertised number and the usable number are rarely the same thing.

Length is not comprehension

An advertised window is an upper bound on what a model can read, not a guarantee of what it can use. Models routinely attend unevenly across very long inputs, weighting the beginning and end more heavily than the middle — the widely observed “lost in the middle” pattern. A model rated for a million tokens may retrieve a fact planted at token 500,000 far less reliably than one planted at token 500. Serious evaluations therefore measure long-context behavior directly, with needle-in-a-haystack retrieval tests and multi-step reasoning tasks spread across the input, rather than trusting the specification. The practical rule: treat the headline number as marketing until you have tested the depths you actually plan to use.

What long windows cost

Long context is expensive in two distinct ways. Compute: self-attention scales quadratically with sequence length unless mitigated, so prompt processing time balloons as the input grows — you feel this as a long pause before the first token appears. Memory: the KV cache — the stored keys and values for every processed token — grows linearly with context length and can rival or exceed the size of the model weights themselves at extreme lengths. Techniques such as sliding-window and sparse attention, grouped-query attention, cache quantization, and context compression exist precisely to bend these curves. On a self-hosted rig, the window you can realistically serve is bounded by your VRAM after weights and overhead, not by the model card — runners like llama.cpp let you set the allocated context explicitly, and setting it larger than you need wastes memory you could have spent on a bigger or less-quantized model.

Long context versus retrieval

A long window and retrieval-augmented generation solve overlapping problems and pair well. Stuffing everything into context is simple and preserves cross-document reasoning, but costs compute on every request and degrades in the middle. Retrieval keeps prompts short and cheap, but only surfaces what the search step finds. For a sovereign operator building a private knowledge assistant, the pragmatic pattern is both: retrieve the relevant slice, then rely on a comfortably sized window to hold the conversation, the instructions, and the retrieved material with room to spare.

See the context window entry for the underlying concept, and tokens per second for how long prompts show up as latency on your own hardware.