Self-Consistency

Self-consistency is a prompting technique, introduced by Wang and colleagues in 2022, that improves reasoning reliability by sampling multiple independent reasoning paths and then taking a majority vote on the final answer. Rather than accepting the single answer a model produces on its first attempt, self-consistency runs the model several times, lets it reason through the problem in different ways, and selects the answer that appears most often across those runs.

The intuition

When a model reasons step by step, the first path it takes is not necessarily the best one. Complex problems often admit several valid routes to the same correct answer. If you sample a diverse set of reasoning chains and many of them converge on the same conclusion, you can be more confident that conclusion is right. Wrong answers, by contrast, tend to scatter — different mistakes lead to different incorrect results, so no single wrong answer dominates the vote. The mechanism is the same one that makes averaging independent measurements reduce noise: errors decorrelate, truth accumulates. It only works, though, when the answers can be compared cleanly — a number, a category, a short final verdict — which is why the technique shines on math, logic, and classification and blurs on open-ended prose, where no two runs produce comparable outputs to vote on.

How it is used

Self-consistency layers on top of step-by-step reasoning prompts. You enable sampling with a nonzero temperature (so each run varies), generate several complete reasoning traces, extract the final answer from each, and pick the most frequent one. It has been shown to improve greedy step-by-step prompting by a large margin across arithmetic, commonsense, and symbolic reasoning benchmarks. Implementation is refreshingly boring: a loop, an answer-extraction step, and a counter. Pairing it with structured output — forcing each run to end in a parseable answer field — makes the extraction reliable, and an even/odd sample count with a tie-break rule closes the last gap. Diminishing returns arrive quickly; small sample counts capture most of the gain, and you can stop early when the first few runs already agree.

The economics of buying reliability

The trade-off is cost: producing N answers takes roughly N times the compute. For sovereign Bitcoiners running local models, self-consistency is a tunable knob — spend more inference to raise reliability on questions that matter, and spend nothing extra on the ones that don't. On self-hosted hardware the marginal cost is electricity and time rather than a per-token bill, which changes the calculus in your favor: a local rig idling overnight can afford ten samples of a question that a metered API would make you think twice about. It also partially converts a weaker model into a stronger answer — a small local model voting five times can rival a larger model's single shot on tasks with checkable answers, which is exactly the trade a RAM-constrained operator wants available.

Where it fits in the toolbox

Self-consistency is the simplest member of a family of test-time-compute techniques: it explores reasoning paths independently and aggregates at the end, while exploratory methods like tree-of-thought branch and prune mid-reasoning at higher orchestration cost. It composes cleanly with careful prompt engineering and few-shot prompting, since better individual runs make for a better electorate. Treat it as your first upgrade when a local model is almost reliable enough: no fine-tuning, no new model, just more votes.

Two refinements are worth knowing. The vote margin is free information: five out of five runs agreeing means something different from three out of five, so treat lopsided votes as high confidence and narrow ones as a flag for human review or more samples. And because runs are independent, they parallelize perfectly — batch them through your inference server and the wall-clock cost shrinks even though the compute does not.

Cost the multiple samples in the inference cost calculator.