bitsandbytes

bitsandbytes is an open-source library that adds low-precision quantization to PyTorch models with minimal code changes. It is best known for enabling 8-bit and 4-bit operation of large language models, and it is the engine behind the popular QLoRA fine-tuning workflow. For sovereign operators, bitsandbytes is often the most direct path to loading a large model that would otherwise overflow available VRAM, because it can quantize weights as the model loads rather than requiring a pre-quantized file.

What it provides

The library exposes drop-in linear layers that store weights at reduced precision. Its 4-bit layer supports two data types: a standard 4-bit float (FP4) and 4-bit NormalFloat (NF4), the latter tuned for the normal distribution typical of model weights. Under the hood it uses blockwise k-bit quantization, where weights are divided into blocks that each get their own scaling factor, limiting the impact of outliers. Computation is performed in a higher-precision type such as FP16 or BF16, with weights de-quantized on the fly, which introduces some runtime overhead in exchange for the memory savings.

Where it fits in the stack

bitsandbytes is widely integrated into Hugging Face Transformers, so loading a model in 4-bit or 8-bit is often a single configuration flag. This makes it a common entry point for self-hosters who want to experiment with quantization before moving to format-specific tooling for deployment.

For the data type that made 4-bit fine-tuning practical see NF4, and for the broader topic see LLM quantization.