BF16 (bfloat16)

BF16, or brain floating-point, is a 16-bit numeric format developed by the Google Brain team for training neural networks. It allocates one sign bit, eight exponent bits, and seven mantissa bits. Crucially, that eight-bit exponent gives BF16 the same dynamic range as 32-bit FP32, but with far less precision. The design reflects a hard-won insight: neural networks are far more sensitive to the size of the exponent (range) than to the size of the mantissa (precision).

Range over precision

Older 16-bit formats such as IEEE FP16 use a 5-bit exponent and 10-bit mantissa. That extra precision sounds appealing, but the narrow exponent causes gradients to underflow to zero, forcing techniques like loss scaling to compensate. Because BF16 preserves FP32's exponent, values that would vanish in FP16 survive untouched, so models can often drop into BF16 without rescaling the loss or retuning hyperparameters.

Why sovereign builders care

For anyone running training or fine-tuning on their own hardware rather than renting cloud GPUs, BF16 roughly halves memory and bandwidth for weights, activations, and gradients. That means larger models fit on the same card, and a single workstation goes further. Modern accelerators implement BF16 directly in their matrix-multiply units, so the speedup is real silicon, not just storage savings. The trade is reduced numeric precision per value, which most deep networks tolerate well because their statistics average over many parameters.

BF16 is one of several reduced-precision formats in the training toolkit. To see how it compares to even more aggressive formats and how the pieces fit together, read about FP8 and the role of an FP32 master weights copy during optimizer updates.