Chain-of-Thought Decoding

Chain-of-Thought decoding is a method for drawing reasoning out of a pre-trained language model by changing how it decodes rather than how it is prompted. Standard greedy decoding always takes the single most probable next token and often jumps straight to an answer. CoT decoding instead inspects the top-k candidate tokens at the first decoding step and lets each one unfold into its own continuation; some of those alternative branches naturally develop into a step-by-step reasoning chain that greedy decoding would never have visited. The method was described by Wang and Zhou of Google DeepMind and presented at NeurIPS 2024.

Reasoning Without Prompt Engineering

The key finding is that the capacity for chain-of-thought reasoning is already latent in a base model and does not require phrases like "let's think step by step." That reframes what techniques such as chain-of-thought prompting actually do: they do not teach the model to reason, they steer decoding toward reasoning paths the model already contains. CoT decoding reaches the same paths mechanically — by branching on the first token and examining each continuation, you recover reasoning chains without touching the prompt at all. This matters most for base models that have not been instruction-tuned, where prompting tricks are least reliable and where the gap between greedy output and the model's best latent path is widest.

The Confidence Signal

Exploring multiple branches raises an obvious question: which one do you trust? The authors observed a usable answer in the model's own probabilities. When a decoding path contains a genuine reasoning chain, the model tends to be markedly more confident in its final answer tokens — the probability gap between the top answer token and the runner-up widens. Ranking branches by that answer-token confidence, rather than by total sequence probability, is what turns the branch set into a selection procedure. The result behaves like an unsupervised sibling of self-consistency: where self-consistency samples many chains and takes a majority vote over final answers, CoT decoding branches systematically and lets calibrated confidence pick the winner, no vote required.

Costs and Practical Relevance

Nothing is free: exploring k branches costs roughly k times the generation compute of a single greedy pass, placing CoT decoding squarely in the family of test-time compute methods that trade inference cycles for accuracy. For operators running models on their own hardware, that trade is often attractive. Extra decoding passes on a model you already host are billed in electricity, not API invoices, and better reasoning from the decoder configuration can substitute for a larger model you cannot fit in VRAM. The technique does require access to token-level probabilities and control over the decoding loop — trivial with local inference stacks, impossible through most hosted chat interfaces, which is itself a quiet argument for owning your inference.

Where It Sits in the Toolbox

CoT decoding occupies a useful middle rung. Below it sits plain greedy decoding — cheapest, weakest on multi-step problems. Alongside it sit prompting strategies and sampling-plus-voting schemes. Above it sits the purpose-built reasoning model, trained with reinforcement learning to emit long deliberation by default, at correspondingly higher token cost per answer. The deeper lesson of the paper travels well: a model's weights contain more capability than its default decoding reveals, and sometimes the cheapest upgrade is not a bigger model or a cleverer prompt, but a better way of listening to the model you already have.

If you want to try it, the recipe is short: run a base or lightly-tuned model locally, branch on the top ten first tokens, generate each continuation, and rank by the confidence margin on the answer tokens. On arithmetic and commonsense benchmarks the original paper reports large gains over greedy decoding with no prompt changes at all — a reminder that decoding strategy is a first-class dial, not an afterthought.