GPTQ

GPTQ (Generative Pre-trained Transformer Quantization) is a one-shot, post-training quantization method that compresses large language model weights down to 3 or 4 bits each while keeping accuracy close to the full-precision baseline. Introduced in a 2022 paper by Frantar et al., it was among the first techniques to reliably push LLMs below 8 bits, making it practical to run multi-billion-parameter models on a single consumer GPU. For sovereign Bitcoiners running local inference, that was the unlock: smaller weights mean a capable model fits in the VRAM you already own, with no cloud dependency and no usage meter.

How it works

GPTQ is a weight-only method: it quantizes the stored weights but leaves activations in higher precision at runtime. It processes each layer's weight matrix column by column, and after rounding each column it updates the remaining un-quantized columns to compensate for the error just introduced. That compensation uses approximate second-order (Hessian) information derived from a small calibration dataset, so quantization error is actively cancelled as the algorithm sweeps across the matrix rather than being left to accumulate. The insight is that rounding decisions are not independent — an error committed in one column can be partially absorbed by the columns still awaiting quantization — and exploiting that dependency is what lets GPTQ hold quality at bit-widths where naive round-to-nearest falls apart. Because the whole procedure is one-shot and post-training, there is no retraining, no gradient computation over the full model, and no need for the original training data; the original work reported quantizing a 175-billion-parameter model in roughly four GPU-hours.

What to expect in practice

A 4-bit GPTQ model occupies roughly a quarter of the memory of its FP16 parent plus modest overhead for scales and metadata, which in practical terms moves whole model classes down a hardware tier: models that demanded datacenter cards become runnable on enthusiast GPUs, and mid-sized models fit alongside a healthy context window on cards ordinary people own. Quality loss at 4-bit is typically small for general use, though it can surface on tasks sensitive to fine-grained precision; 3-bit pushes further at a real accuracy cost. The calibration set matters at the margin — a model calibrated on generic text may give up a little on specialized domains.

Where it fits

The knobs that matter

Two parameters dominate the quality of any GPTQ artifact you download. Group size controls how many weights share a quantization scale: smaller groups track the weight distribution more faithfully and preserve accuracy at the cost of extra metadata, while very coarse grouping saves memory and gives some of the quality back. Activation ordering changes the sequence in which columns are quantized so the most impactful weights are handled while the most compensation headroom remains, usually worth a measurable perplexity improvement. Community model cards typically state both settings, and two "4-bit GPTQ" files of the same model can behave noticeably differently because of them. The calibration corpus is the third quiet variable — a model quantized against general web text may shed a little capability on code or specialized domains. The practical habit is the same one this site preaches for mining firmware claims: benchmark the artifact you actually run, on the tasks you actually care about, rather than trusting the label.

GPTQ sits in a toolkit of competing approaches: AWQ decides importance from activation statistics rather than Hessian-guided error compensation, while the GGUF-family block-wise schemes used by llama.cpp serve CPU and mixed CPU/GPU inference. GPTQ remains a common choice for GPU-served INT4 models and is broadly supported across inference frameworks, and its low barrier to entry — no retraining, hours not weeks — is why quantized community releases appear within days of new open-weight models. To understand the broader landscape, see our entries on LLM quantization and the closely related AWQ method.

See what fits at 4-bit in the GPU–LLM fit dataset.