Chain-of-Thought Decoding

Chain-of-Thought decoding is a method for drawing reasoning out of a pre-trained language model by changing how it decodes rather than how it is prompted. Standard greedy decoding always takes the single most probable next token and often jumps straight to an answer. CoT decoding instead inspects the top-k candidates at the first decoding step; some of those alternative branches naturally unfold into a step-by-step reasoning chain. It was described by Wang and Zhou of Google DeepMind and presented at NeurIPS 2024.

Reasoning without prompt engineering

The key finding is that the capacity for chain-of-thought reasoning is already latent in a base model and does not require phrases like "let's think step by step." By branching on the first token and examining each path, you recover reasoning chains that greedy decoding would have skipped. The authors also observed a useful signal: when a decoding path contains a genuine reasoning chain, the model tends to be more confident in the final answer token. That confidence gap can be used to automatically select the best branch.

Practical relevance

For operators running their own models, CoT decoding means better reasoning can sometimes come from the decoder configuration rather than from longer prompts or larger models. It does cost extra computation because multiple branches are explored, placing it among inference-time techniques that trade compute for accuracy.

This sits alongside the broader family of test-time compute methods and contrasts with a purpose-built reasoning model, which is trained to emit long reasoning by default.