Position Interpolation

Position interpolation extends the usable context length of a model built on rotary position embeddings (RoPE) by squeezing longer position indices back into the range the model was trained on, rather than letting them run off the end. Introduced by Chen and colleagues in 2023, it lets RoPE-based models reach much larger windows — extending a model to tens of thousands of tokens — with only a short fine-tuning pass instead of an expensive full retrain from scratch.

The trick at the heart of it is almost cheeky in its simplicity. Rather than laboriously teach a model to understand positions it has never once encountered, you lie to it gently — you take a token that genuinely sits at position twenty thousand and present it as if it sat at position two thousand, safely back inside familiar territory. The model, none the wiser, applies exactly the positional reasoning it already learned during training. A short fine-tune then smooths over the slightly denser spacing this creates, and a context window that would otherwise have shattered on out-of-range positions instead extends cleanly and cheaply. It is a neat reminder that a great deal of practical machine learning is really about finding ways to keep a capable model operating inside the world it was actually trained on.

The problem it solves

If you simply feed positions beyond the trained range, the rotary embeddings extrapolate into angular territory the model has never seen, producing wildly large attention scores that wreck the self-attention mechanism and send output quality off a cliff. Linear position interpolation instead down-scales every position so that, for example, position twenty thousand is presented to the model as if it were position two thousand, keeping every value inside the familiar range the model was actually trained to interpret. The model then needs only light adaptation to read this denser positional grid, because nothing it sees is out-of-distribution — the positions are merely closer together than before.

NTK-aware scaling

A refinement called NTK-aware scaling improves on uniform interpolation by scaling each rotary frequency differently rather than squeezing them all equally. High-frequency dimensions, which encode fine local order between neighbouring tokens, are stretched less, while low-frequency dimensions, which encode coarse long-range position, are stretched more. This preserves the model's ability to tell adjacent tokens apart — something uniform scaling tends to blur, because it compresses the fine-grained signal along with everything else — and it underpins later schemes such as YaRN. Together these methods are why so many released models advertise windows far larger than their base training length.

The catch at the extremes

Extending a window is not the same as using it well. An interpolated model can attend across its full advertised range, but its precision often thins out toward the far end, where the positional signal has been compressed the most. In practice this means the model may reliably use the first large fraction of its window while growing hazier in the last stretch — technically in-range, practically less trustworthy. Knowing how a model earned its long window therefore matters: a natively long-trained model and an interpolated one can share the same headline number and behave very differently in the tail.

What it means for self-hosting

For a self-hosting operator, the lesson is not to take the advertised context length on faith but to test it. A needle in a haystack run, which buries a specific fact deep in a long context and checks whether the model can retrieve it, quickly reveals where usable attention actually ends for your workload. Position interpolation also pairs with KV cache quantization and the attention sink as part of the machinery that makes long, local context affordable; see long context window for the broader capability it unlocks.