Chain-of-Thought Prompting

Chain-of-thought (CoT) prompting is a technique that improves a language model's reasoning by encouraging it to generate intermediate steps before committing to a final answer. Introduced by Wei and colleagues at Google in 2022, it dramatically raised performance on tasks like arithmetic word problems, logical reasoning, and multi-hop question answering — problems where a direct answer from a single forward pass routinely fails.

How it works

Instead of asking for an answer directly, you either instruct the model to reason before answering — the zero-shot variant, whose canonical trigger phrase "let's think step by step" was documented by Kojima and colleagues the same year — or you supply few-shot examples in which the reasoning is written out longhand, and the model imitates the pattern. The mechanics reward the verbosity: because each generated token conditions all subsequent tokens, spelling out intermediate results gives the model both more computation (every reasoning token is another forward pass) and a scaffold to follow, so partial results are written down rather than juggled implicitly. Errors that are common under direct prompting — dropped carries, skipped constraints, conflated entities — fall away when each step is externalized where the model can attend to it.

An emergent behavior

A notable finding of the original work is that chain-of-thought gains are largely emergent with scale: they appear in large models — roughly the hundred-billion-parameter class — and do little or nothing for small ones, which tend to produce fluent-sounding but incoherent reasoning chains. Refinements followed quickly. Self-consistency samples several independent reasoning chains and takes a majority vote on the final answer, buying accuracy with compute. And the current generation of "reasoning" models internalizes the pattern during training, producing extended deliberation automatically without being asked. The prompting technique nonetheless remains a free, universally available lever for any capable model — including the open-weight models you run yourself.

Practical use on local models

For a self-hoster the trade-offs are tangible. Reasoning tokens are generated at ordinary inference speed and occupy the context window, so a long chain costs seconds and kilobytes on hardware you can hear working; for a batch job triaging miner kernel logs, you might reserve CoT for the entries a fast direct pass flags as ambiguous. Model choice matters more than usual, since small local models benefit least from the technique — though modern mid-size open models distilled from larger reasoners do markedly better than their size suggests. As a form of in-context learning, CoT costs nothing to try: add the instruction, compare results, keep what wins.

In pipelines, pair the technique with an explicit answer delimiter. Instruct the model to reason freely and then emit its conclusion after a fixed marker or inside a structured field, so downstream code parses only the final answer and never mistakes an intermediate musing for the result. The pattern also separates audiences cleanly: the chain is for the operator reviewing quality, the extracted answer is for the system consuming it, and nothing forces you to show end users the deliberation at all. Log the chains anyway — when a batch job misclassifies something at 3 a.m., the stored reasoning is the closest thing you get to asking the model what it was thinking.

Legibility, not proof

A quiet virtue of chain-of-thought is that it makes a model's logic legible: you can read the steps and catch the mistake, which pure answer-only output never allows. But a stated rationale is not a guarantee of a correct or faithful one — models can produce plausible-looking reasoning that does not reflect how the answer was actually reached, and verbose intermediate steps do not prevent a confidently delivered hallucination. Treat the chain as you would a junior tech's worksheet at the repair bench: valuable for review, never a substitute for checking the result.