Chinchilla-Optimal Training

Chinchilla-optimal training refers to the compute-optimal recipe established by DeepMind's 2022 study "Training Compute-Optimal Large Language Models." Its central finding: for a fixed compute budget, model size and the number of training tokens should be scaled in roughly equal proportion — about 20 training tokens for every model parameter. Earlier practice had built ever-larger models trained on comparatively little data; the study showed those models were systematically undertrained.

The Chinchilla result

To prove the point, the authors trained a 70-billion-parameter model named Chinchilla on 1.4 trillion tokens — far more data per parameter than its predecessors. Despite being four times smaller than the 280-billion-parameter Gopher, Chinchilla outperformed it, along with GPT-3 (175B) and other larger models, across a broad range of benchmarks. The lesson was that a smaller model fed more data can beat a bigger model starved of it, at equal training cost.

Why it reshaped model design

Chinchilla-optimality corrected the recipe implied by earlier scaling work, shifting the field toward training smaller, data-richer models. This has a practical upside for everyone downstream: a compute-optimal smaller model is cheaper to run at inference time and more feasible to self-host than a bloated, undertrained giant. The principle is not a hard ceiling — models are now often trained well past the 20-tokens-per-parameter point specifically to make inference cheaper — but it remains the reference point for thinking about the data-versus-size trade-off.

This recipe is a refinement of the broader scaling laws, and the models it produces are the foundation models that downstream applications build on.