EAGLE Decoding

EAGLE decoding (Extrapolation Algorithm for Greater Language-model Efficiency) is a speculative-decoding technique that accelerates large language model inference without changing the model's output distribution — the responses are provably the same as ordinary decoding, just delivered faster. Standard autoregressive generation produces one token per forward pass, and each pass is dominated by streaming the model's weights through memory, leaving the GPU's arithmetic units idle. Speculative decoding attacks that waste: draft several candidate tokens cheaply, then verify them all in a single forward pass of the full model, keeping the longest prefix the model agrees with. Every accepted draft token is a full forward pass you didn't pay for.

The feature-level insight

Earlier speculative schemes used a separate small draft model, which had to be trained, loaded, and kept behaviorally close to the target — a real operational burden. EAGLE's core insight, from the original 2024 paper, is that drafting works better one level down: instead of predicting the next token, predict the target model's next feature vector — the hidden state at the second-to-top layer — which evolves far more regularly than discrete token sequences. EAGLE runs a lightweight draft head autoregressively at this feature level, feeding in the sampled token from one step ahead to resolve the uncertainty that sampling introduces. Because the head reuses the target model's own internal representations rather than running a separate network, it is tiny, cheap to train, and produces drafts the verifier accepts at high rates — and the accept/reject rule guarantees losslessness.

EAGLE-2 and EAGLE-3

The line has iterated quickly. EAGLE-2 observed that acceptance rates depend heavily on context — some continuations are predictable, others aren't — and introduced dynamic draft trees: instead of a fixed speculation pattern, the draft head's own confidence scores shape which branches of candidate continuations get expanded, spending the speculation budget where it is most likely to pay. EAGLE-3 changed the recipe again: it abandoned feature prediction as the training objective in favor of direct token prediction, fused hidden states drawn from low, middle, and high layers of the target model rather than one layer, and added a training-time simulation of multi-step drafting so the head learns under the same conditions it will face at inference. Reported speedups reach roughly 6x over vanilla decoding, with outputs identical to the unaccelerated model.

What it costs

EAGLE's price of admission is a trained draft head per target model — small, but model-specific, so you either train one or use a model for which the community already publishes heads. Speculation also shines brightest at small batch sizes, where decoding is most memory-bound; under heavy batching the GPU is already busier, acceptance overhead matters more, and the net gain shrinks. That makes EAGLE-style methods best matched to the interactive, low-concurrency regime — which happens to be exactly how most self-hosted deployments run. Memory overhead, by contrast, is modest — the draft head amounts to a fraction of a single transformer layer — so the technique fits on the same card as the model it accelerates.

Why self-hosters should care

For an operator serving a model from a single GPU, EAGLE is one of the highest-impact levers available: multiplying tokens per second severalfold, with zero quality loss, on the same silicon, is the difference between a sluggish assistant and a genuinely usable one. Major open-source inference engines have adopted EAGLE-family speculation, putting it within reach of anyone running local models — sovereignty over your AI stack includes squeezing full performance from hardware you own. Compare the simpler, training-free alternative in our N-gram speculation entry — a weaker but zero-setup form of the same idea — and see throughput-optimized serving for how speculative decoding interacts with batching when a box serves many users at once.