Position Interpolation

Position interpolation extends the usable context length of a model built on rotary position embeddings by squeezing longer position indices back into the range the model was trained on, rather than letting them run off the end. Introduced by Chen and colleagues in 2023, it lets RoPE-based models reach much larger windows, such as extending a model to tens of thousands of tokens, with only a short fine-tuning pass.

The problem it solves

If you simply feed positions beyond the trained range, the rotary embeddings extrapolate into territory the model has never seen, producing wildly large attention scores that wreck the self-attention mechanism. Linear position interpolation instead down-scales every position so that, for example, position twenty thousand is presented as if it were position two thousand, keeping all values inside the familiar range. The model then needs only light adaptation to read the denser positional grid.

NTK-aware scaling

A refinement called NTK-aware scaling improves on uniform interpolation by scaling each rotary frequency differently: high-frequency dimensions, which encode fine local order, are stretched less, while low-frequency dimensions are stretched more. This preserves the model's ability to distinguish nearby tokens, which uniform scaling tends to blur, and underpins later schemes like YaRN. Together these methods are why so many models ship with windows far larger than their base training length.

For a self-hosting operator, knowing whether a model's long window comes from native training or from interpolation matters, because interpolated models can lose fidelity at the extreme end of their advertised range. Test it with a needle in a haystack run, and see long context window for the broader capability.