{ "meta": { "title": "D-Central — LLM Quantization Format Comparison", "description": "Runtime support, real hardware requirements, calibration needs, maintenance status and best-use situation for 7 LLM quantization formats — GGUF, MLX, EXL2/EXL3, GPTQ, AWQ, FP8 and bitsandbytes NF4.", "generated": "2026-07-20T17:33:18+00:00", "as_of": "2026-07-20", "version": "1.0", "license": "https://creativecommons.org/licenses/by/4.0/", "license_name": "CC BY 4.0", "source": "https://d-central.tech/fr/llm-quantization-formats/", "method": "Each format was researched from its own project documentation and repository (llama.cpp / ggml-org, Apple MLX, turboderp ExLlamaV2 and V3, GPTQModel, AutoAWQ and the MIT Han Lab implementation, vLLM, HuggingFace transformers, bitsandbytes) plus the original papers, then handed to an independent adversarial fact-checker instructed to assume the row was wrong. Reviewers were told to attack hardware-support and maintenance-status claims hardest, because that is where public content is most often stale, and to DELETE any performance figure that could not be attributed to a named source rather than soften it.", "provenance": "Project primary documentation and original papers. Quantitative claims that survived are attributed to whoever measured them, with hardware and date stated.", "disclaimer": "This lane moves fast: runtimes gain and drop format support, and projects get archived. Values are as of the stated date — check the project repo before committing to a toolchain. Where a claim could not be verified, the row says so instead of guessing." }, "formats": [ { "name": "GGUF", "full_name": "GGUF, the single-file model container used by GGML-based executors. The spec defines it plainly: \"GGUF is a file format for storing models for inference with GGML and executors based on GGML,\" and adds \"It is a successor file format to GGML, GGMF and GGJT, and is designed to be unambiguous by containing all the information needed to load a model.\" The spec itself neither expands the acronym nor names an author — the widely repeated gloss \"GGML Universal File\" does not appear in it, and should not be treated as official. Authorship attribution comes from HuggingFace's hub documentation instead: \"GGUF was developed by @ggerganov who is also the developer of llama.cpp.\" It is now maintained under the ggml-org umbrella alongside llama.cpp. Unlike safetensors, a GGUF file carries the tensors AND the metadata — architecture, hyperparameters, vocabulary and tokenizer — in one file, which is why a .gguf download is self-contained and needs no companion config.", "bit_widths": "llama.cpp's README claims \"1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use\". In practice the container holds: K-quants (Q2_K through Q6_K, exposed by llama-quantize with _XS/_S/_M/_L file-type variants); I-quants (IQ1_S, IQ1_M, IQ2_XXS/XS/S/M, IQ3_XXS/XS/S/M, IQ4_XS, IQ4_NL); ternary TQ1_0/TQ2_0; MXFP4 (4-bit Microscaling Block Floating Point); legacy round-to-nearest types; unquantized F16/BF16/F32/F64; and fixed-width integer types I8/I16/I32/I64. HuggingFace's table groups SIX types under \"Legacy types\" with the note \"Legacy quantization method (not used widely as of today)\": Q8_0, Q8_1, Q5_0, Q5_1, Q4_0 and Q4_1. Two caveats on that label. Q8_0 remains in routine use as the near-lossless option and is still offered by llama-quantize, so its \"legacy\" tag reflects the RTN lineage rather than obsolescence — it is the 4/5-bit RTN types that K-quants and I-quants genuinely displaced. Q8_1 and Q8_K are intermediate-computation types, not practical weight-storage choices; HF describes Q8_K as \"Only used for quantizing intermediate results.\" MXFP4 is often described as having been added for GPT-OSS-class models; that motivation is not stated in the cited primary docs, which record only the type's addition. Per-weight sizing of each type is D-Central's separate GGUF-internals reference — not repeated here.", "runtimes": "The widest runtime support of any quantization format. Native and first-class: llama.cpp (llama-cli / llama-server), Ollama (its import docs take `FROM /path/to/file.gguf` in a Modelfile directly), LM Studio, GPT4All, and the many llama.cpp wrappers and bindings. HuggingFace Hub's GGUF page lists exactly llama.cpp, LM Studio, GPT4All and Ollama as the open-source tools. Two important asterisks. (1) HuggingFace transformers CAN load a GGUF via `from_pretrained(..., gguf_file=...)`, but its docs state \"Transformers supports loading models stored in the GGUF format for further training or finetuning. The GGUF checkpoint is dequantized to fp32 where the full model weights are available and compatible with PyTorch\" — that path is for training, NOT for running the model quantized, so it saves nothing at inference time. (2) vLLM's GGUF support has moved out of the main tree: its docs state \"GGUF support has migrated to OOT vllm-gguf-plugin,\" and warn it \"is highly experimental and under-optimized at the moment, it might be incompatible with other features.\" Treat GGUF as a local/single-user format, not a vLLM serving format.", "hardware": "The broadest hardware reach of any format, by a wide margin — this is GGUF's real moat. llama.cpp's own backend table lists: Metal (Apple Silicon), BLAS (All), BLIS (All), SYCL (Intel GPU), OpenVINO [In Progress] (Intel CPUs, GPUs, and NPUs), MUSA (Moore Threads GPU), CUDA (Nvidia GPU), HIP (AMD GPU), ZenDNN (AMD CPU), Vulkan (GPU), CANN (Ascend NPU), OpenCL (Adreno GPU), IBM zDNN (IBM Z & LinuxONE), WebGPU (All), RPC (All), Hexagon [In Progress] (Snapdragon), and VirtGPU (VirtGPU APIR). Note the two [In Progress] markers — OpenVINO and Hexagon are not finished backends. It also runs pure CPU with no GPU at all. Vulkan is the generic-GPU path and in practice the usual route for AMD and Intel Arc cards on Windows without a ROCm stack; that deployment advice is practitioner guidance, not a claim made in the backend table itself. Critically, this is the only format in this comparison with real CPU+GPU hybrid execution: the README advertises \"CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity,\" driven by `-ngl, --gpu-layers, --n-gpu-layers N` (\"max. number of layers to store in VRAM, either an exact number, 'auto', or 'all' (default: auto)\"). For MoE models there are dedicated escape hatches: `--cpu-moe` (\"keep all Mixture of Experts (MoE) weights in the CPU\") and `-ncmoe, --n-cpu-moe N` (\"keep the Mixture of Experts (MoE) weights of the first N layers in the CPU\"), which is how a single consumer GPU runs a large MoE by parking rarely-hot expert weights in system RAM.", "calibration": "Both — and the distinction matters when you pick a file. Plain K-quants (Q4_K_M, Q5_K_M, Q6_K and similar), along with the legacy RTN types such as Q8_0, are DATA-FREE: `llama-quantize` converts an F16/BF16 GGUF directly, with no dataset. I-quants (and optionally K-quants) use an importance matrix produced by the separate `llama-imatrix` tool, documented as \"Compute an importance matrix for a model and given text dataset. Can be used during quantization to enhance the quality of the quantized models.\" It requires a calibration corpus via the mandatory `-f | --file` argument (docs reference `wiki.train.raw` and `calibration-data.txt` as examples); the result is passed to `llama-quantize --imatrix`. For the very low-bit types the imatrix is not optional — llama.cpp's `llama-quant.cpp` has a `tensor_requires_imatrix()` guard that hard-fails with \" ERROR: this quantization requires an importance matrix!\" naming the \"offending tensor\" and \"target type\". The gated types are IQ1_S, IQ1_M, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, plus Q2_K tensors that are part of a Q2_K_S file — the code comment reads \"as a general rule, the k-type quantizations don't require imatrix data. the only exception is Q2_K tensors that are part of a Q2_K_S file.\" A second path logs \"Missing importance matrix for tensor %s in a very low-bit quantization\" and \"The result will be garbage, so bailing out\". Practical consequence for a downloader: two files with identical names can differ in quality depending on whether the uploader used an imatrix and what corpus they calibrated on — check the model card.", "quality_speed": "Stated qualitatively, because no verified head-to-head numbers are cited here. llama.cpp's quantize README frames the tradeoff itself: quantization \"may introduce some accuracy loss which is usually measured in Perplexity (ppl) and/or Kullback-Leibler Divergence (kld)\" and says this \"can be minimized by using a suitable imatrix file\" — i.e. at a given bits-per-weight, an imatrix-calibrated file is expected to lose less than an uncalibrated one, which is the whole reason the I-quant family exists. Speed is dominated by placement: fully offloaded to VRAM is fast, and the moment layers spill to system RAM via `-ngl` you are bound by host memory bandwidth and throughput drops sharply. It is widely reported in the community that I-quants cost more compute per token at decode than K-quants of the same file size, so a plain K-quant can win on tokens/sec on a compute-starved device; that relationship is NOT documented in any of the primary sources cited here and should be treated as unverified folklore until benchmarked on your own hardware. GGUF's decode is fundamentally single-stream-oriented and does not batch-serve concurrent users well, which is consistent with vLLM classifying its own GGUF path as \"highly experimental and under-optimized.\"", "strengths": "Runs on essentially anything — NVIDIA, AMD, Intel, Apple Silicon, Ascend, Moore Threads, IBM Z, Snapdragon, or plain CPU. One self-contained file with tokenizer and metadata baked in, mmap-loadable, no config juggling. Unique ability to run a model bigger than your VRAM by splitting layers, or MoE expert weights specifically, between GPU and system RAM — the single most useful property for a self-hoster with one consumer card. Largest selection of pre-quantized weights on HuggingFace (filter `library=gguf`), the widest quality ladder (1.5-bit through 8-bit plus F16/BF16/F32), and low-friction tooling: HF's `ggml-org/gguf-my-repo` Space will convert/quantize a model for you. It is the default currency of Ollama and LM Studio, so it is what a novice will actually end up with.", "limitations": "Not a serving format: throughput under concurrency is poor relative to vLLM-class stacks, and vLLM's GGUF support was demoted out-of-tree to an experimental plugin. transformers only loads it by dequantizing to fp32, so there is no \"finetune directly at 4-bit from my GGUF\" path. New model architectures need explicit support in `convert_hf_to_gguf.py` before any GGUF can exist, so day-one coverage of a brand-new release lags safetensors. Tokenizer conversion is a known rough edge — vLLM's docs recommend the base model's tokenizer instead because \"the tokenizer conversion from GGUF is time-consuming and unstable, especially for some models with large vocab size.\" Quality is not reproducible from the filename alone: whether an imatrix was used, and on what corpus, is uploader-dependent. And the 4/5-bit legacy RTN types (Q4_0, Q4_1, Q5_0, Q5_1) still circulate on old repos despite HuggingFace labelling them \"not used widely as of today\" — prefer a K-quant or I-quant at the same size. Do not extend that avoidance to Q8_0, which shares the legacy label but remains the standard near-lossless choice.", "best_for": "The default answer for almost every self-hoster with one or two consumer GPUs, and the only sane answer if your GPU is AMD or Intel, if you are on Windows without a working ROCm stack (use the Vulkan build), or if you have no GPU at all. It is specifically the right pick when the model does NOT fit in your VRAM — nothing else lets you keep, say, 40 of 60 layers on the card and the rest in system RAM, or park MoE experts on the CPU with `--n-cpu-moe`. It is the wrong pick if you are serving many concurrent users from a datacenter NVIDIA card; use a vLLM-native format for that.", "status": "Actively maintained and clearly dominant for local single-user inference — llama.cpp remains one of the most active projects in the space, and new quant families (MXFP4, ternary TQ1_0/TQ2_0) are still landing. Internal to the format, the 4- and 5-bit legacy round-to-nearest types (Q4_0, Q4_1, Q5_0, Q5_1) are superseded in practice by K-quants and I-quants; HuggingFace's table files them, together with Q8_0 and Q8_1, under \"Legacy types\" with the note \"not used widely as of today,\" though Q8_0 remains in active use as the near-lossless tier. On the SERVER side momentum has gone the other way: vLLM moved GGUF out of tree into a separate experimental plugin.", "confidence": "high", "sources": [ "https://github.com/ggml-org/ggml/blob/master/docs/gguf.md", "https://github.com/ggml-org/llama.cpp", "https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md", "https://github.com/ggml-org/llama.cpp/blob/master/tools/imatrix/README.md", "https://raw.githubusercontent.com/ggml-org/llama.cpp/master/src/llama-quant.cpp", "https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md", "https://huggingface.co/docs/hub/en/gguf", "https://huggingface.co/docs/transformers/en/gguf", "https://docs.vllm.ai/en/latest/features/quantization/gguf.html", "https://github.com/vllm-project/vllm-gguf-plugin", "https://docs.ollama.com/import" ], "headline": "Widest hardware reach; only true CPU+GPU split", "sweet_spot": "Local, single user", "health": "healthy", "slug": "gguf" }, { "name": "MLX", "full_name": "MLX — an array framework from Apple machine learning research. Apple does not officially expand the acronym; its own docs describe it as \"a NumPy-like array framework designed for efficient and flexible machine learning on Apple silicon, brought to you by Apple machine learning research.\" Important framing for a hardware audience: MLX is a FRAMEWORK (a PyTorch/NumPy analogue), not primarily a file format. There is no \".mlx\" file you download the way you download a .gguf; an \"MLX model\" is a converted model repo produced by `mlx_lm.convert` and published to HuggingFace, most often under the community org `mlx-community`. Its defining architectural feature is unified memory: \"Arrays in MLX live in shared memory. Operations on MLX arrays can be performed on any of the supported device type without performing data copies.\" Supported device types are the CPU and GPU.", "bit_widths": "`mlx.core.quantize` documents four modes. The default \"affine\" mode supports 2, 3, 4, 5, 6, or 8 bits (default 4) with group sizes of 32, 64 (default), or 128 elements, returning quantized weights plus scales and biases; per the docs it computes α = max_i w_i, β = min_i w_i, s = (α − β)/(2^b − 1), ŵ_i = round((w_i − β)/s), storing s and β as the returned scales and biases. Three microscaling float modes are also documented, and these return no biases: \"mxfp4\" (E2M1 elements, group size 32, E8M0 shared scale), \"mxfp8\" (E4M3 elements, group 32, E8M0 scale), and \"nvfp4\" (4-bit fp4 elements, group 16, E4M3 scale). The docs state: \"The elements are quantized to 4-bit or 8-bit precision floating-point values: E2M1 for 'fp4' and E4M3 for 'fp8'. There is a shared 8-bit scale per group.\" Inputs must have two or more dimensions with the last dimension divisible by group_size. In the wild the community publishes overwhelmingly 4-bit and 8-bit affine variants (mlx-lm's default generation/chat model is `mlx-community/Llama-3.2-3B-Instruct-4bit`). Group size is the second lever most people ignore: a 4-bit/group-32 file is larger and more faithful than 4-bit/group-128.", "runtimes": "A small, Apple-centric set. `mlx-lm` is the reference runtime — Python package and CLI (`mlx_lm.generate`, `mlx_lm.convert`) with HuggingFace Hub integration, plus `mlx_lm.server`, an HTTP server whose API is \"intended to be similar to the OpenAI chat API\"; the docs caution that it \"is not recommended for production as it only implements basic security checks.\" `mlx-vlm` covers vision-language models. LM Studio ships an MLX engine (`lmstudio-ai/mlx-engine`), which \"LM Studio 0.3.4 and newer for Mac ships pre-bundled with,\" supporting text generation, vision models including \"Llama-3.2-Vision, Pixtral, Qwen2-VL, and Llava-v1.6,\" structured output via Outlines, and speculative decoding with draft models — so a Mac user gets MLX through a GUI without touching Python. `mlx-examples` covers LoRA finetuning, Stable Diffusion and Whisper. Notably absent: Ollama, whose import documentation covers only Safetensors and GGUF; MLX appears there solely as one of the fine-tuning tools that can emit Safetensors adapters, never as a format Ollama loads. Also absent: vLLM, TGI, ExLlama. Traffic runs one way — mlx-examples includes a GGUF loader for MLX, but there is no MLX-into-other-runtime path.", "hardware": "This is the field most write-ups get wrong in BOTH directions, so state it precisely. The default Mac path is hard-confined: Apple silicon only, \"macOS >= 14.0\", and \"native Python >= 3.10\" (arm64, not Rosetta) — Intel Macs are not supported, and neither is an eGPU or any discrete card. mlx-lm's \"Large Models\" path adds \"This requires macOS 15.0 or higher to work,\" and you may need to raise the wired-memory ceiling via `sudo sysctl iogpu.wired_limit_mb=N`. BUT the common claim that \"MLX is Apple-only\" is stale: MLX's official install docs ship a CUDA backend for Linux (requiring \"Nvidia architecture >= SM 7.5\", \"Nvidia driver >= 550.54.14\" — >= 580 for CUDA 13 — \"CUDA toolkit >= 12.0\", and \"Linux distribution with glibc >= 2.35\"), plus a CPU-only Linux package needing glibc >= 2.35 and Python >= 3.10. So the FRAMEWORK genuinely runs on NVIDIA/Linux and Linux CPU. What has not moved is the ecosystem: the model zoo and the GUI (LM Studio's MLX engine requires \"macOS 14.0 (Sonoma) or greater\") remain Mac-bound. No AMD/ROCm backend, no Vulkan backend, and no Windows package is documented in the install options.", "calibration": "Both paths exist, and the default is data-free. `mlx_lm.convert --model -q` performs plain round-to-nearest affine quantization with no dataset — fast, deterministic, runs on the same Mac you'll infer on. (HuggingFace's MLX page still shows the older `--hf-path` spelling; upstream mlx-lm documents `--model`.) For higher quality, mlx-lm documents four LEARNED quantization methods that DO require calibration data: AWQ, which \"scales and clips the weights prior to quantization\" (32 samples by default); DWQ / distilled weight quantization, which \"fine-tunes non-quantized parameters (including quantization scales and biases) using the non-quantized model as a teacher\" (1,024 samples by default); GPTQ, which \"finds quantized weights which minimize the squared error of each layer's output given the provided input\"; and dynamic quantization, which \"estimates the sensitivity of a model's outputs to each layer and uses a higher precision for layers which have higher sensitivity\" — i.e. mixed precision by sensitivity. Default sample counts for GPTQ and dynamic are not stated in the docs. The DWQ memory-reduction advice (\"Distill from an 8-bit model instead of a 16-bit model\", \"Using --max-seq-length 512 reduces the memory and still gets good results\", \"Use a smaller batch size, e.g. --batch-size 1\") signals it is the heavy one. Rule of thumb: a repo name ending in `-4bit` is almost certainly the data-free RTN convert; `-dwq`, `-awq` or `-gptq` suffixes indicate a calibrated run.", "quality_speed": "No benchmark numbers are asserted here because none were verified from a primary source. Structurally: MLX kernels are written for Apple silicon's unified memory and Metal, so arrays live in shared memory and operations run on CPU or GPU \"without performing data copies\" — there is no host-to-device transfer and no VRAM/RAM split to manage. That is the mechanism behind the widely-repeated claim that MLX outruns llama.cpp's Metal backend on the same Mac; the claim is plausible and commonly reported in community benchmarks, but D-Central should not publish a speed figure without measuring it in-house. On quality, mlx-lm's own learned-quant documentation is the honest framing: the default `-q` convert is plain RTN, and AWQ/DWQ/GPTQ/dynamic exist specifically because RTN leaves quality on the table at low bit-widths — so a 4-bit RTN MLX file is not automatically comparable to an imatrix-calibrated 4-bit GGUF. Match like with like when comparing.", "strengths": "Native fit for Apple silicon: unified memory means a 64GB or 128GB Mac can hold models that would need multiple consumer NVIDIA cards, with no offload gymnastics and no `-ngl` tuning. First-party — Apple machine learning research maintains it, so Metal support tracks Apple's own hardware. Genuinely full-stack: unlike every other format here it does training as well as inference, so LoRA-style finetuning of a quantized model runs on the same laptop that serves it, and the learned-quant toolkit (AWQ/DWQ/GPTQ/dynamic) is built in rather than bolted on. Fine-grained quality control via bits AND group_size, plus modern microscaling formats (mxfp4/mxfp8/nvfp4). Real distributed support: `mx.distributed` documents MPI, Ring (TCP, \"Always available and usually faster than MPI\"), JACCL (\"Low latency communication with RDMA over thunderbolt. Necessary for things like tensor parallelism\") and NCCL (\"The backend of choice for CUDA environments\"), and mlx-lm lists \"Distributed inference and fine-tuning with mx.distributed\" — so multi-Mac inference is a documented path, not a hack. Accessible through LM Studio's GUI, so a novice Mac owner does not need Python.", "limitations": "Ecosystem confinement is the real cost. The practical model zoo and GUI tooling are macOS-on-Apple-silicon only: Intel Macs excluded, no AMD/ROCm, no Vulkan, no documented Windows package; the Linux CUDA and CPU wheels exist but the model zoo and tooling are not built around them. Far fewer pre-converted models than GGUF, and slower coverage of new architectures. macOS >= 14.0 minimum (>= 15.0 for the large-model path), and you can hit the wired-memory ceiling and need to raise it manually with `sysctl`. No CPU/GPU hybrid offload story — you either fit in unified memory or you don't, so a 16GB Mac has a much harder wall than a 12GB GPU running GGUF with layer spill. Not a batch-serving stack: no vLLM/TGI integration, and `mlx_lm.server` is explicitly \"not recommended for production as it only implements basic security checks.\" Distributed inference exists but carries topology constraints — the ring backend connects nodes so that \"rank 1 can only communicate with rank 0 and rank 2,\" and JACCL \"supports only fully connected topologies. Namely, there needs to be a thunderbolt cable connecting all pairs of Macs directly.\" And Ollama, the most common on-ramp for beginners, does not load MLX at all.", "best_for": "The right pick if you own an Apple silicon Mac — especially a high-unified-memory M-series (32GB+) — and want the fastest local inference that hardware can give, or want to finetune on the same machine. Choose MLX over GGUF on a Mac when the model comfortably fits in unified memory and you want the Apple-native path; stay on GGUF on a Mac when the specific model you want has no MLX conversion, when you need a quant type MLX doesn't offer, or when you want one file that also works on your Linux box. Despite the CUDA backend, it is not yet a practical choice on Windows or a non-Apple GPU, because the model ecosystem and tooling there are thin.", "status": "Actively maintained and expanding — Apple ships it, `mlx-lm` was split into its own repository, LM Studio bundles an MLX engine for Mac from version 0.3.4, and the framework has grown beyond Apple silicon with official CUDA and CPU-only Linux packages plus an NCCL distributed backend. It is the youngest and fastest-moving format in this comparison, and the only one with a first-party hardware vendor behind it. Nothing here is in decline; the honest caveat is reach, not momentum.", "confidence": "high", "sources": [ "https://ml-explore.github.io/mlx/build/html/index.html", "https://ml-explore.github.io/mlx/build/html/install.html", "https://ml-explore.github.io/mlx/build/html/python/_autosummary/mlx.core.quantize.html", "https://ml-explore.github.io/mlx/build/html/usage/distributed.html", "https://github.com/ml-explore/mlx", "https://github.com/ml-explore/mlx-lm", "https://raw.githubusercontent.com/ml-explore/mlx-lm/main/mlx_lm/LEARNED_QUANTS.md", "https://raw.githubusercontent.com/ml-explore/mlx-lm/main/mlx_lm/SERVER.md", "https://huggingface.co/docs/hub/en/mlx", "https://github.com/lmstudio-ai/mlx-engine", "https://docs.ollama.com/import" ], "headline": "Apple Silicon native, unified memory", "sweet_spot": "Mac owners", "health": "healthy", "slug": "mlx" }, { "name": "EXL2 / EXL3", "full_name": "ExLlamaV2 quantization format (EXL2) and its successor EXL3, both by turboderp (turboderp-org). EXL2 is the quant format produced by the ExLlamaV2 library; EXL3 is produced by ExLlamaV3 and is described in its README as \"a streamlined variant of QTIP from Cornell RelaxML.\"", "bit_widths": "Fractional average bitrate is the distinguishing feature of the lineage. The ExLlamaV2 README states EXL2 \"is based on the same optimization method as GPTQ and supports 2, 3, 4, 5, 6 and 8-bit quantization\" and that \"the format allows for mixing quantization levels within a model to achieve any average bitrate between 2 and 8 bits per weight.\" This is why repos are labelled 4.65bpw, 5.0bpw, 6.5bpw — you pick a bitrate that fills your card's VRAM instead of jumping between fixed tiers. EXL3 keeps fractional bitrates: convert.py's -b/--bits is documented only as \"Target average number of bits per weight\" with no stated minimum or maximum, and the docs show values like 3.75 and 4.00. The output layer is set separately via -hb/--head_bits, which \"must be an integer from 1 to 8, default is 6.\" An -hq flag raises the bitrate on selected layers (attention, shared-expert), noted as adding roughly 0.05-0.10 bpw on MoE models. The ExLlamaV3 README states Llama-3.1-70B-EXL3 is \"coherent at 1.6 bpw\" — the author's own assessment, not an independently verified floor.", "runtimes": "Narrow, and narrowing further for EXL2 specifically. EXL3 loads in ExLlamaV3 itself; TabbyAPI, its official OpenAI-compatible server, which lists Exl3 as \"Highly recommended\"; text-generation-webui, where ExLlamaV3 is one of five listed backends (\"llama.cpp, ik_llama.cpp, Transformers, ExLlamaV3, and TensorRT-LLM\"); and via an HF Transformers plugin listed among ExLlamaV3's headline features. EXL2 is increasingly orphaned: TabbyAPI's README states \"ExLlamaV2 models are no longer supported in the `main` branch. The last commit with ExLlamav2 support is preserved on the `exl2-checkpoint` branch,\" and ExLlamaV2 no longer appears in text-generation-webui's backend list at all. Neither format appears anywhere in vLLM's supported-quantization documentation, which covers AutoAWQ, BitsAndBytes, GPTQModel, LLM Compressor, TorchAO, AMD Quark and others but no ExLlama entry. The ExLlamaV3 README frames vLLM support as future possibility, not present fact: \"It will be possible to extend EXL3 support to other frameworks like HF Transformers and vLLM.\" Ollama and LM Studio do not load either format.", "hardware": "NVIDIA CUDA, effectively exclusively — the single most important practical fact and the one most often stated wrongly. ExLlamaV3 requires PyTorch with CUDA 12.4 or later, and its to-do list contains exactly one item: \"ROCm support.\" EXL3 is therefore not an AMD option today. On GPU generation, the README is specific rather than exclusionary: \"The Marlin-inspired GEMM kernel achieves roughly memory-bound latency under optimal conditions (4bpw, RTX 4090), though it still needs some work to achieve the same efficiency on Ampere GPUs and to remain memory-bound at lower bitrates.\" Ampere runs, it is simply less optimal than Ada. TabbyAPI separately notes it \"supports parallel batching using paged attention for Nvidia Ampere GPUs and higher,\" so Ampere is the practical floor for the full serving feature set. Multi-GPU is genuinely supported, not a workaround: ExLlamaV3 lists \"flexible tensor-parallel and expert-parallel inference for consumer hardware\" as a headline feature. On Windows, triton-windows is recommended — without it \"many things will work suboptimally.\" ExLlamaV2 was more AMD-tolerant at the build level; its setup.py branches on `if torch and torch_version.hip:` to add `-DHIPBLAS_USE_HIP_HALF`, so a source build under ROCm was possible, but that project is archived and whether such builds still work in practice was not verified here. No CPU inference. No CPU offload of layers in the llama.cpp sense. No Apple Silicon, Metal or MLX path. On a Mac or an AMD card, these formats are not for you.", "calibration": "Both require a calibration/quantization pass; neither is data-free. EXL2 works by \"quantizing each matrix multiple times, measuring the quantization error (with respect to the chosen calibration data) for each of a number of possible settings, per layer,\" then choosing \"a combination that minimizes the maximum quantization error over the entire model while meeting a target average bitrate\" — thorough but slow. EXL3 is deliberately cheaper: conversion \"requires only an input model (in HF format) and a target bitrate,\" computing Hessians on the fly with a fused Viterbi kernel in a single step. The README puts this at minutes for smaller models and a couple of hours for 70B+ on a single RTX 4090 or equivalent. Calibration data shape is tunable via -cr/--cal_rows and -cc/--cal_cols, with no ranges documented. In practice almost nobody quantizes their own — you download someone else's bpw variant.", "quality_speed": "Qualitative only; the published figures are the author's own. EXL2's design goal is to spend bits where they matter — mixed per-layer precision under a global bitrate budget — and to run fast on consumer NVIDIA hardware with weights resident in VRAM. EXL3, as a QTIP-derived trellis method, is the stronger of the two at low bitrate, and the ExLlamaV3 repo publishes its own bpw-vs-perplexity and VRAM curves. The README's headline claim, quoted in full because its conditions are load-bearing: \"Llama-3.1-70B-EXL3 is coherent at 1.6 bpw. With the output layer quantized to 3 bpw and a 4096-token cache, inference is possible in under 16 GB of VRAM.\" Repeating the 16 GB figure without the 3 bpw head and 4096-token cache qualifiers misstates it. These are the author's measurements published in the repo, not independently reproduced. No perplexity deltas or tokens/sec figures should be quoted without pointing at that chart. The README also positions EXL3 on conversion economics, citing AQLM at around \"720 GPU-hours on an A100 server, costing $850 US\" for a 70B — the pitch being comparable quality at hobbyist conversion cost.", "strengths": "Fractional bitrate is genuinely useful: target 4.65bpw to fill a 24 GB card exactly rather than leaving 3 GB unused or spilling over. EXL3 is not single-GPU-bound — tensor-parallel and expert-parallel inference for consumer multi-card setups is a headline feature, alongside continuous dynamic batching, speculative decoding, multimodal support and LoRA support. Cache quantization is a real advantage for long context on a small card: ExLlamaV3 lists \"2-8 bit cache quantization,\" while ExLlamaV2 offers Q4 cache (its README notes Q4 \"is supported and performs better anyway\" than FP8). EXL3 conversion is dramatically cheaper than EXL2's, and EXL3 models \"largely retain the original file structure (unlike EXL2 which renames some tensors in its quest to turn every model into a Llama variant)\" — the structural change that makes broader framework support plausible, and that already yields an HF Transformers plugin.", "limitations": "NVIDIA-only; no ROCm for EXL3, no CPU inference, no CPU offload, no Apple Silicon. If the model does not fit in VRAM across your cards there is no graceful fallback — the opposite of GGUF's design. Ampere kernels are acknowledged as not yet as efficient as Ada. On Windows, running without triton-windows means \"many things will work suboptimally.\" The ecosystem is one maintainer plus a small orbit of frontends: no vLLM, Ollama or LM Studio support, and far fewer prequantized repos on HuggingFace than GGUF. ExLlamaV3 warns directly: \"Expect that some things may be a little broken at first.\" EXL2 additionally carries migration risk — the project is archived and its principal serving frontend has already dropped it from main.", "best_for": "A self-hoster with one or more NVIDIA cards (Ada or newer ideally, Ampere workable) who wants maximum quality per gigabyte of VRAM and is willing to run TabbyAPI or text-generation-webui rather than Ollama. Use EXL3, not EXL2, for anything new — particularly when squeezing a 70B-class model onto 24-48 GB, or spreading a large or MoE model across consumer cards via tensor/expert parallelism. Skip entirely on a Mac, on AMD, or if you need to offload layers to system RAM.", "status": "EXL2: superseded and archived. The ExLlamaV2 README carries the notice \"This project is archived for now. Development continues on ExLlamaV3,\" and the repo is read-only. TabbyAPI has removed it from main (legacy support preserved on the exl2-checkpoint branch) and text-generation-webui no longer lists an ExLlamaV2 backend. Treat EXL2 files as legacy — still loadable via pinned or legacy branches, but not where new work goes. EXL3: actively maintained and the live successor, with release 1.1.0 dated 18 July 2026, though still self-described as early. Its to-do list is short and specific — ROCm support is the only item on it. Net: the EXL lineage is alive and developing, but it is a smaller and far more NVIDIA-bound ecosystem than GGUF, and advice to download an EXL2 quant in 2026 is working from stale information.", "confidence": "high", "sources": [ "https://github.com/turboderp-org/exllamav2", "https://raw.githubusercontent.com/turboderp-org/exllamav2/master/README.md", "https://raw.githubusercontent.com/turboderp-org/exllamav2/master/setup.py", "https://github.com/turboderp-org/exllamav3", "https://raw.githubusercontent.com/turboderp-org/exllamav3/master/README.md", "https://raw.githubusercontent.com/turboderp-org/exllamav3/master/doc/convert.md", "https://raw.githubusercontent.com/theroyallab/tabbyAPI/main/README.md", "https://raw.githubusercontent.com/oobabooga/text-generation-webui/main/README.md", "https://docs.vllm.ai/en/latest/features/quantization/" ], "headline": "Fractional bitrate; fastest single-stream on NVIDIA", "sweet_spot": "One NVIDIA GPU, model fits", "health": "mixed", "slug": "exllama" }, { "name": "GPTQ", "full_name": "GPTQ — \"GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers,\" Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh (IST Austria / ETH Zurich), arXiv October 2022, ICLR 2023. Originally tooled by AutoGPTQ; that project is archived and the maintained implementation is now GPTQModel by ModelCloud.", "bit_widths": "Fixed integer bit-widths, not fractional — typically 4-bit, with 3-bit and 8-bit also supported (GPTQModel and HuggingFace's GPTQConfig both document bits=3, 4, and 8). The paper's abstract claims reduction \"down to 3 or 4 bits per weight, with negligible accuracy degradation,\" and states the method \"can still provide reasonable accuracy in the extreme quantization regime, in which weights are quantized to 2-bit or even ternary quantization levels.\" The other real knob is group size (commonly 128, 64, or 32) plus act-order/desc_act: smaller groups cost a little more size and buy accuracy. Weights are stored int4 and, per HuggingFace's docs, \"restored to fp16 on the fly during inference\" in a fused dequant kernel. Practically you choose from a handful of published combinations (4-bit-128g-actorder is the default you will meet) rather than dialing an arbitrary bitrate.", "runtimes": "GPTQ's real advantage — it is the most widely loadable GPU-native format. HuggingFace Transformers (via GPTQConfig, backed by gptqmodel), Optimum, PEFT, vLLM, SGLang, and Text Generation Inference all load GPTQ checkpoints; GPTQModel's own README advertises \"native integration with HF Transformers, Optimum, and Peft\" plus \"vLLM and SGLang inference integration for quantized models.\" Note the desktop shift: text-generation-webui's current loader list is \"llama.cpp, ik_llama.cpp, Transformers, ExLlamaV3, and TensorRT-LLM\" — there is no dedicated GPTQ/AutoGPTQ loader any more (nor an ExLlamaV2 one), so on that frontend GPTQ models arrive through the Transformers backend with gptqmodel installed rather than a first-class loader. Ollama and LM Studio do not load GPTQ; they are GGUF-centric.", "hardware": "Wider than the common \"GPTQ is NVIDIA-only\" claim, but narrower than vendor summaries imply, and the answer differs by runtime. GPTQModel's README lists NVIDIA CUDA (Turing / sm_75 and newer), AMD ROCm — specified as \"7900XT+, ROCm 6.2+\", so RDNA3-class and newer, not all ROCm cards — Intel XPU (Arc, Datacenter Max), Huawei Ascend NPU (Ascend 910B via torch-npu/CANN), Apple Silicon (M1+, Metal GPU and CPU), and Intel/AMD CPU with AVX/AMX, across Linux, macOS, and Windows 11. HuggingFace independently repeats that GPTQModel has \"broader support for ... platforms (Linux, macOS, Windows 11), and hardware (AMD ROCm, Apple Silicon, Intel/AMD CPUs, and Intel Datacenter Max/Arc GPUs).\" Critical caveat that is widely misreported: that library-level breadth does not carry into every serving stack. vLLM's own quantization support matrix marks GPTQ supported on Volta, Turing, Ampere, Ada, Hopper, Intel GPU, and x86 CPU, but NOT supported on AMD GPU or Arm CPU — so \"run GPTQ on ROCm\" is a GPTQModel/Transformers path, not a vLLM one. The fastest kernels are also NVIDIA-specific: HuggingFace describes Marlin as \"a 4-bit only CUDA GPTQ kernel, highly optimized for the NVIDIA A100 GPU (Ampere) architecture,\" inference-only, and Marlin's own repo requires \"compute capability >= 8.0 (Ampere or Ada, Marlin is not yet optimized for Hopper)\"; Marlin also does not support GPTQModel's asymmetric quantization. Apple Silicon support is documented as Metal GPU plus CPU inference, but no first-party benchmark establishes it as a fast serving path, and MLX or GGUF remains the conventional choice there.", "calibration": "Yes — GPTQ is calibration-dependent by construction. It is a one-shot, weight-only post-training method using approximate second-order (Hessian) information, and you must supply a calibration dataset; HuggingFace's GPTQConfig takes a `dataset` argument and notes \"it is highly recommended to use the same dataset from the GPTQ paper\" (c4, wikitext2, etc.). This makes quantization a slow, data-dependent GPU job: the paper's abstract reports quantizing a 175B-parameter model \"in approximately four GPU hours,\" and HuggingFace warns it takes \"~5 minutes to quantize the facebook/opt-350m model on a free-tier Google Colab GPU, but it'll take ~4 hours to quantize a 175B parameter model on a NVIDIA A100.\" Practical consequences for a downloader: quality varies with whose calibration set was used, calibration can bias the model toward the calibration domain, and you should prefer a well-known quantizer's upload over a random one.", "quality_speed": "At 4-bit, GPTQ is a solid, well-understood baseline rather than the current accuracy leader. The original paper reports end-to-end inference speedups of 3.25x on A100 and 4.5x on A6000 versus FP16, measured by the authors on OPT/BLOOM-class models — 2022-2023 figures on datacenter cards, not what a consumer GPU will show today. Marlin's repo claims \"close to ideal (4x) speedups\" on favorably-shaped matrices and \"essentially ideal speedups at all batchsizes, enabling the maximum possible 3.87x speedup up to batchsizes around 16-32\" for its FP16xINT4 kernel; those are the authors' own measurements on Ampere/Ada. HuggingFace notes GPTQModel's asymmetric quantization \"can potentially lower quantization errors compared to symmetric quantization\" versus legacy AutoGPTQ output. The widely-repeated claim that GPTQ degrades faster than AWQ or EXL3 below 4 bits is not verified here against any named benchmark harness and should be treated as community impression, not measurement; what is documented is that 3-bit and 2-bit GPTQ never saw wide adoption while 4-bit remains standard. Do not quote perplexity deltas without naming the harness — comparisons across GPTQ builds are confounded by group size, act-order, and calibration set.", "strengths": "Ubiquity and interoperability: the closest thing to a lingua franca for GPU-native 4-bit weights, loadable by Transformers, vLLM, SGLang, and TGI alike, which makes it the safe pick when a model must run in more than one place. Mature, well-studied, enormous back-catalogue on HuggingFace. Genuine multi-vendor hardware support at the library level through GPTQModel, including ROCm, Intel XPU, Ascend NPU, Apple Silicon, and CPU. Strong batched/server throughput on NVIDIA with Marlin. Being a plain int4 weight format with fused dequant, it slots cleanly into production serving stacks in a way EXL2/EXL3 do not.", "limitations": "No fractional bitrate — you cannot tune the file to exactly fill your VRAM the way EXL2/EXL3 let you. No meaningful CPU-offload story for a model that does not fit; GPTQ assumes the weights live on the accelerator, so a self-hoster with a small card and a big model is better served by GGUF. Calibration dependence adds a quality lottery and makes DIY quantization slow. The multi-vendor story is runtime-dependent and easy to over-read: vLLM's matrix excludes AMD GPU and Arm CPU for GPTQ, and Marlin is CUDA-only, Ampere/Ada, 4-bit-only, inference-only, and does not cover asymmetric quants. There is also a real toolchain trap — HuggingFace states GPTQModel \"is not backward compatible with legacy AutoGPTQ checkpoints,\" so old AutoGPTQ-era files and new ones are not fully interchangeable.", "best_for": "Serving a 4-bit model that has to load in more than one runtime, or serving concurrent requests through vLLM/SGLang/TGI on NVIDIA hardware — GPTQ's throughput and ecosystem reach are strongest in batch/server settings. For a single self-hoster on one consumer GPU who just wants the best quality that fits, EXL3 (NVIDIA) or GGUF (everything else) is usually the better download. Reach for GPTQ when the model you want only exists in that format, or when your serving stack demands it. If your target is an AMD GPU under vLLM specifically, check the support matrix first — GPTQ is not covered there.", "status": "The method is alive and standard; the original tooling is dead. AutoGPTQ was archived by its owner on 11 April 2025 and is read-only, its last release (v0.7.1) dating to March 2024, and the repo banner reads \"AutoGPTQ is unmaintained - we suggest using GPTQModel for bug fixes and new models support.\" HuggingFace states flatly: \"AutoGPTQ is no longer supported in Transformers. Install GPT-QModel instead,\" and describes GPTQModel as \"the actively maintained backend for GPTQ in Transformers,\" originally forked from AutoGPTQ but since diverged. GPTQModel is on the v7.2.0 release line as of July 2026 and says it \"has fully supplanted AutoGPTQ and AutoAWQ for HF Transformers/Optimum/Peft integration.\" So: if content tells you to `pip install auto-gptq`, it is stale — the package is `gptqmodel`. Meanwhile GPTQ's centre of gravity has shifted from hobbyist desktop use (where GGUF won on CPU-offload and EXL3 on VRAM efficiency, and where text-generation-webui has dropped the dedicated loader) toward server-side serving, and vLLM now points new users at LLM Compressor, \"a library for optimizing models for deployment with vLLM that supports FP8, INT8, INT4, and other quantization formats.\"", "confidence": "medium", "sources": [ "https://arxiv.org/abs/2210.17323", "https://github.com/AutoGPTQ/AutoGPTQ", "https://github.com/ModelCloud/GPTQModel", "https://github.com/ModelCloud/GPTQModel/releases", "https://huggingface.co/docs/transformers/main/en/quantization/gptq", "https://docs.vllm.ai/en/latest/features/quantization/", "https://raw.githubusercontent.com/vllm-project/vllm/main/docs/features/quantization/README.md", "https://raw.githubusercontent.com/oobabooga/text-generation-webui/main/README.md", "https://github.com/IST-DASLab/marlin" ], "headline": "Method standard and widely loaded; original tooling dead", "sweet_spot": "Server serving on Ampere", "health": "mixed", "slug": "gptq" }, { "name": "AWQ", "full_name": "Activation-aware Weight Quantization. Algorithm from MIT Han Lab (Lin, Tang, Tang, et al., arXiv:2306.00978), winner of the MLSys 2024 Best Paper Award as \"AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration\". The reference packaging tool most people mean by \"AWQ\" was AutoAWQ by Casper Hansen; the original research implementation is mit-han-lab/llm-awq.", "bit_widths": "4-bit weight-only (INT4) in practice. AutoAWQ's documented config is `w_bit: 4`, typically group_size 128 with zero-point, and it exposes no other width. Two kernel variants exist for the same 4-bit weights: GEMM, which AutoAWQ describes as \"much faster than FP16 at batch sizes below 8 (good with large contexts)\", and GEMV, described as \"20% faster than GEMM, only batch size 1 (not good for large context)\". Unlike GGUF or EXL2 there is no ladder of bit-widths to pick from — you take 4-bit or you take a different format. The one exception is MLX-LM's separate implementation, which exposes a `--bits` flag defaulting to 4; its docs do not enumerate which other widths are accepted. Activations stay FP16, so AWQ is a W4A16 scheme.", "runtimes": "vLLM (first-class, and the main reason the format exists in the wild), SGLang, HuggingFace TGI, HuggingFace transformers (loading only — see AwqConfig, which also exposes an ExLlamaV2 kernel backend via `version=\"exllama\"`), TensorRT-LLM, and — newer and widely missed — MLX-LM on Apple Silicon via `mlx_lm.awq`, documented in mlx-lm's LEARNED_QUANTS.md alongside DWQ, GPTQ and dynamic quantization. Transformers can also load checkpoints produced by the original llm-awq. NOT loadable by llama.cpp/Ollama/LM Studio, which want GGUF. Producing new AWQ checkpoints has moved to llm-compressor (vLLM project), which lists AWQ among its algorithms, since AutoAWQ itself is archived.", "hardware": "NVIDIA CUDA is the primary target: AutoAWQ documents compute capability 7.5+ (Turing and later) and CUDA 11.8+. vLLM's quantization support matrix lists AWQ as supported on Turing, Ampere, Ada, Hopper, Intel GPU and x86 CPU. Two exclusions in that matrix matter. First, AWQ is marked unsupported on Volta while GPTQ is supported, so a V100 is a real exclusion. Second, and more often gotten wrong: vLLM marks AWQ as NOT supported on AMD GPU. AMD is therefore not a vLLM path — the working AMD route is HuggingFace transformers with the ExLlamaV2 kernel backend, which HF's docs state is supported on AMD GPUs, plus AutoAWQ's documented Triton-compatible ROCm builds. Treat AMD as a reduced-capability side door, not parity. Intel CPU and GPU are supported through intel_extension_for_pytorch (Torch 2.4+) or the Intel XPU Triton backend, and vLLM lists x86 CPU as a supported AWQ target. Apple Silicon: yes, but only through MLX-LM's own AWQ implementation, not through the CUDA-kernel ecosystem — an AWQ repo on HuggingFace is not automatically usable on a Mac. Note the distinction between CPU execution, which exists, and GGUF-style CPU offload, which does not: you cannot spill layers to system RAM, so the model must fit the accelerator's memory.", "calibration": "Required. AWQ needs a calibration dataset — it collects activation statistics offline to identify the roughly 1% of salient weight channels, then scales those channels via an equivalent transformation before quantizing. The paper is explicit that AWQ \"does not rely on any backpropagation or reconstruction\" (which is what distinguishes it from GPTQ's error-compensation approach) and argues this is why it generalizes across domains and modalities without overfitting the calibration set. Practically: quantizing is a slow, data-dependent, GPU-bound job. MLX-LM's implementation exposes the knob directly with `--num-samples` (default 32, documented as \"using more samples can lead to better results but takes longer\") and `--n-grid` for search granularity. Almost nobody quantizes their own — you download someone else's AWQ repo.", "quality_speed": "At 4-bit weight-only, AWQ is a well-regarded quality point. The paper claims AWQ outperforms prior work on language modeling and domain-specific benchmarks (coding and math), achieves strong results on instruction-tuned LMs, and was first to do 4-bit multi-modal LMs — these are the authors' own claims measured in the paper. The honest framing for a self-hoster: AWQ is one fixed 4-bit rung, and its advantage shows up under server-style batching rather than in single-stream token generation. Because activations stay FP16, the win is memory bandwidth on weights, not arithmetic. HuggingFace's transformers AWQ page publishes a measured fused-vs-unfused table for TheBloke/Mistral-7B-OpenOrca-AWQ at batch_size=1: decode throughput of roughly 31–38 tokens/s unfused versus roughly 80–106 tokens/s fused across prefill/decode lengths from 32 to 2048. That is HF's own measurement on their hardware and a fused-vs-unfused comparison within AWQ — it is not a cross-format benchmark. AutoAWQ's project-side claims (3x speedup and 3x memory reduction versus FP16; 1.6x–2.5x from fused modules) and the paper's \">3x speedup over the Huggingface FP16 implementation on both desktop and mobile GPUs\" via TinyChat were all measured by the authors, not independently.", "strengths": "Genuinely good accuracy retention at 4-bit for a weight-only method, with a large pre-quantized catalogue on HuggingFace (AutoAWQ cited 7,000+ models; vLLM's docs cite 6,500+). Its real strength is being a first-class citizen in vLLM/SGLang/TGI, so it drops straight into an OpenAI-compatible serving stack with continuous batching and tensor parallelism — things GGUF and EXL2 do not do nearly as well. Quantizing is inference-only statistics collection with no backpropagation, making it far cheaper than any training-based method. Reach beyond CUDA exists — Intel GPU and x86 CPU in vLLM, AMD via ExLlamaV2 kernels in transformers, Apple Silicon via MLX-LM — but that reach is uneven and mostly outside the vLLM serving path that makes AWQ worth choosing in the first place.", "limitations": "The big one: AutoAWQ is officially deprecated and was archived read-only on 11 May 2025, and vLLM's own AWQ documentation page opens by stating the library is deprecated and that the functionality has been adopted into llm-compressor. The format still loads everywhere; the reference tool gets no fixes and pinned old dependencies (last tested at Torch 2.6.0 / Transformers 4.51.3, and HF warns that installing autoawq downgrades Transformers to 4.47.1, requiring a reinstall afterwards). Beyond that: only 4-bit, so no quality dial; calibration required; no CPU offload, so it is useless if the model does not fit your accelerator memory; no Volta support; no AWQ on AMD GPU in vLLM specifically; and per HF's docs, fused modules cannot be combined with other optimizations such as FlashAttention2.", "best_for": "Choose AWQ when you are running vLLM or SGLang to serve a model to more than one concurrent user or an app backend, on an NVIDIA card, and you want 4-bit memory savings without leaving the CUDA serving ecosystem. If you are a single person chatting with one model on one GPU, GGUF (bit-width choice, CPU offload, runs anywhere) or EXL2/EXL3 (better single-stream speed on NVIDIA) are the more sensible picks. On AMD, the loss of the vLLM path makes AWQ a poor default. On a Mac, reach for MLX or GGUF first.", "status": "Split verdict, and worth stating plainly. The FORMAT is alive and served daily by vLLM, SGLang and TGI. The TOOLCHAIN has been retired: AutoAWQ is archived, and new production is expected to happen in llm-compressor, which is actively maintained (v0.12.0, June 2026) and lists AWQ alongside GPTQ, SmoothQuant, AutoRound and rotation-based methods. Momentum among people producing new server checkpoints has shifted toward FP8 and llm-compressor's W4A16, and on newer NVIDIA hardware toward the NVFP4/MXFP4 microscaling formats that llm-compressor now targets. Among single-GPU self-hosters AWQ has clearly lost ground to GGUF and to ExLlama's own formats. Not dead, but no longer where the energy is.", "confidence": "high", "sources": [ "https://arxiv.org/abs/2306.00978", "https://mlsys.org/virtual/2024/awards_detail", "https://github.com/casper-hansen/AutoAWQ", "https://github.com/mit-han-lab/llm-awq", "https://docs.vllm.ai/en/latest/features/quantization/auto_awq.html", "https://docs.vllm.ai/en/latest/features/quantization/", "https://huggingface.co/docs/transformers/main/en/quantization/awq", "https://github.com/vllm-project/llm-compressor", "https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/LEARNED_QUANTS.md" ], "headline": "4-bit only; served daily by vLLM, reference tool archived", "sweet_spot": "Server serving on Ampere", "health": "mixed", "slug": "awq" }, { "name": "FP8", "full_name": "8-bit floating point for inference, in the two OCP/industry variants E4M3 and E5M2. The specification \"FP8 Formats for Deep Learning\" (arXiv:2209.05433) was jointly authored by NVIDIA, Arm and Intel, published in an open, license-free format to encourage broad adoption, and submitted to IEEE for standardization. Unlike GGUF/EXL2/AWQ this is not one project's file format — it is a numeric type that many toolchains emit and many runtimes consume.", "bit_widths": "8-bit, in two layouts. E4M3: 1 sign + 4 exponent + 3 mantissa bits, max magnitude 448 — more precision, less range, used for weights and activations in forward-pass inference. Per the spec, E4M3 deliberately departs from IEEE 754 by not representing infinities and using a single mantissa bit-pattern for NaN, which is what buys it the extra range. E5M2: 1 sign + 5 exponent + 2 mantissa bits, max magnitude 57344, and it does follow IEEE 754 conventions for special values — more range, less precision, used where dynamic range matters such as gradients in training. For inference you are almost always looking at E4M3. Common deployment schemes: FP8 W8A8 (weights and activations both FP8) and FP8 weight-only W8A16. Blockwise/fine-grained variants exist too — HuggingFace transformers quantizes weights per 2D 128x128 block with activations quantized per group per token (group size 128 by default, matching the weight input channel), the DeepSeek-V3/R1 style. DeepSeek-V4-style checkpoints extend this by storing FP8 weight scales in packed float8_e8m0fnu (UE8M0) format rather than float32. Related but distinct 4-bit microscaling formats (NVFP4, MXFP4) are emerging alongside FP8 on Blackwell.", "runtimes": "vLLM (mature FP8 support, including online dynamic quantization at load time via quantization=\"fp8\"), SGLang, NVIDIA TensorRT-LLM, HuggingFace transformers (FineGrainedFP8Config), TGI. Producers: llm-compressor (FP8 W8A8), NVIDIA Model Optimizer, AMD Quark, TorchAO, Intel Neural Compressor. Note that FP8 support in SGLang, TensorRT-LLM and TGI is widely reported but was not confirmed against those projects' own documentation for this entry; the vLLM and transformers paths were. Not a llama.cpp/Ollama/LM Studio inference format — the desktop GGUF stack does not execute FP8 as a compute dtype, and an FP8 checkpoint must be converted (dequantized to BF16/FP16, then requantized to a GGUF type) before that stack will run it. This is the single most common point of confusion for self-hosters.", "hardware": "Native FP8 tensor-core math requires NVIDIA compute capability >= 8.9 per vLLM's documentation — that is Ada Lovelace (8.9) and Hopper (9.0), plus Blackwell (SM100+), which HuggingFace documents as an FP8 DeepGEMM target. In consumer terms: RTX 4090 / 4080 / L40S / L4 (Ada) DO have native FP8; RTX 50-series (Blackwell) does, and adds FP4. Below that threshold there is a real fallback, and it extends further back than commonly stated: vLLM supports FP8 weight-only W8A16 on compute capability >= 7.5, so Turing AND Ampere (RTX 2080 Ti, RTX 3090, A100) can load an FP8 checkpoint and run it through FP8 Marlin kernels as W8A16 — you get the memory saving and none of the FP8 arithmetic speedup. Volta (7.0) and older are genuinely out. AMD: vLLM's support matrix marks FP8 W8A8 as supported on AMD GPU, served via ROCm with AMD Quark as the producing toolkit; Quark's documentation names AMD Instinct MI325, MI300 and MI250, i.e. datacenter-class silicon in practice, not consumer Radeon. Apple Silicon has NO FP8 tensor hardware and MLX is not an FP8 inference path; Mac users should ignore this format. CPU inference: no — vLLM's support matrix does not mark FP8 W8A8 as supported on either the x86 CPU or ARM CPU column. A stricter sub-case: HuggingFace's fine-grained (blockwise) FP8 path documents Compute Capability >= 9, i.e. H100 and up, and its DeepGEMM fast path needs Hopper SM90+ or Blackwell SM100+ plus a full CUDA toolkit with nvcc (12.3+ on Hopper, 12.9+ on Blackwell) — a runtime-only CUDA install silently falls back to Triton.", "calibration": "Optional — and this is FP8's quiet advantage over AWQ/GPTQ. The scheme llm-compressor and vLLM recommend, FP8_DYNAMIC, uses static per-channel scales on weights plus dynamic per-token scales on activations. vLLM describes it as simple RTN quantization and states plainly that because activations are quantized dynamically, no calibration data is needed for this flow. vLLM also supports online dynamic quantization: point it at an FP16/BF16 model and it converts to FP8 at load time with no dataset and no offline step, though vLLM notes latency improvements are limited in this mode because activation scales are computed per forward pass rather than statically. Static per-tensor activation scaling DOES need a calibration pass — AMD Quark's documented FP8 flow, for instance, uses an observer_method=min_max spec and passes a calibration dataloader to quantize_model. So: you can reach FP8 in minutes without data, or spend a calibration run to squeeze out a bit more.", "quality_speed": "FP8 is generally considered close to lossless relative to FP16/BF16 for inference, and far more forgiving than any 4-bit scheme, because you keep a real exponent field rather than integer buckets. vLLM's own FP8 documentation claims a 2x reduction in model memory requirements and up to a 1.6x improvement in throughput with minimal impact on accuracy — that is vLLM's published claim on their hardware and workloads, and should be cited as such rather than treated as a universal number. HuggingFace separately states its DeepGEMM kernels are 3-6x faster than the Triton fallback for fine-grained FP8, which is a kernel-vs-kernel comparison inside transformers, not an FP8-vs-FP16 figure. No independent cross-format accuracy benchmark is cited here because none was verified against a named primary source. The tradeoff to communicate qualitatively: FP8 buys throughput and accuracy safety, not footprint. 8 bits halves an FP16 model; 4-bit formats quarter it. Throughput gains are largest under batching, where FP8 tensor cores are actually saturated.", "strengths": "Near-lossless quality compared to 4-bit alternatives, so it is the safe choice when output fidelity is the priority. Data-free quantization paths mean no calibration corpus and no slow offline job — FP8_DYNAMIC and vLLM's online conversion both need zero data. It accelerates compute, not just memory traffic: genuine FP8 tensor-core math on Ada/Hopper/Blackwell, which weight-only INT4 schemes cannot offer since they dequantize to FP16 to multiply. It is a vendor-neutral published specification (NVIDIA/Arm/Intel, open and license-free, submitted to IEEE) rather than one repo's format, and it has broad producer tooling (llm-compressor, NVIDIA ModelOpt, AMD Quark, TorchAO, Intel Neural Compressor). It is increasingly the default serving format for new open-weight releases. It also degrades gracefully: hardware without FP8 math can still load the checkpoint as W8A16 rather than failing outright.", "limitations": "Hardware-gated in a way no other format on this list is: pre-Ada NVIDIA cards get no FP8 arithmetic at all, so a 3090 owner reading FP8 throughput benchmarks will not reproduce them even though the model loads. Only 8 bits of compression, so a 70B model still will not fit a single 24GB card — for VRAM-constrained self-hosters this is the wrong tool entirely. No CPU inference, no Apple Silicon, and no native llama.cpp/Ollama/LM Studio execution. Consumer AMD is effectively unsupported; the supported AMD path is Instinct-class datacenter silicon on ROCm. The blockwise/fine-grained variants raise the bar further to H100-class (CC >= 9), and their fast kernel path additionally demands a full CUDA toolkit, which is an easy way to silently land on the slow Triton fallback. Static-activation FP8 reintroduces the calibration step that dynamic FP8 avoids.", "best_for": "Right pick when you have an Ada or newer NVIDIA GPU (RTX 4090, RTX 50-series, L40S, H100) with enough VRAM to hold the model at 8 bits, and you are serving via vLLM or SGLang to multiple concurrent requests or an application backend where throughput and answer fidelity both matter. Also the correct choice when you specifically do not trust 4-bit for your task — code generation, long-context reasoning, structured extraction. It is the wrong pick if you are trying to squeeze a large model onto one consumer card, if you are on a Mac, if you are running on CPU, or if your GPU is Ampere or older — in those cases go GGUF. One nuance: if you already have an FP8 checkpoint and an Ampere or Turing card, running it as W8A16 through vLLM's FP8 Marlin path is a legitimate option for the memory saving alone.", "status": "Actively maintained and ascending — the clearest growth story in this comparison, but hardware-gated, so its relevance to any given reader depends entirely on which GPU they own. It has become a default for server-side deployment of new open-weight models, with deep, actively-developed support in vLLM and transformers, and it is still gaining features rather than stalling: HuggingFace's FP8 path has added Blackwell SM100 DeepGEMM kernels and UE8M0 packed-scale support for DeepSeek-V4-style checkpoints. The forward-looking caveat: 4-bit microscaling formats (NVFP4, MXFP4) are arriving on Blackwell and may take over some weight-storage duty, so FP8 is best described as the current server-side default rather than a permanent endpoint. The strength of that displacement trend was not verified against a primary source and should be treated as directional, not settled.", "confidence": "high", "sources": [ "https://arxiv.org/abs/2209.05433", "https://developer.nvidia.com/blog/nvidia-arm-and-intel-publish-fp8-specification-for-standardization-as-an-interchange-format-for-ai/", "https://docs.vllm.ai/en/latest/features/quantization/llm_compressor/fp8/", "https://docs.vllm.ai/en/latest/features/quantization/", "https://docs.vllm.ai/en/latest/features/quantization/quark.html", "https://huggingface.co/docs/transformers/main/en/quantization/finegrained_fp8", "https://github.com/vllm-project/llm-compressor" ], "headline": "Near-lossless 8-bit float, native on Ada/Hopper+", "sweet_spot": "Serving on modern datacenter GPUs", "health": "healthy", "slug": "fp8" }, { "name": "bitsandbytes (NF4 / INT8)", "full_name": "bitsandbytes — LLM.int8() 8-bit quantization and 4-bit NormalFloat (NF4)/FP4 from the QLoRA paper. Created by Tim Dettmers; LLM.int8() by Dettmers, Lewis, Belkada & Zettlemoyer (arXiv 2208.07339, NeurIPS 2022), NF4/QLoRA by Dettmers, Pagnoni, Holtzman & Zettlemoyer (arXiv 2305.14314, May 2023). Now maintained by the bitsandbytes-foundation organization under the MIT license, with sponsorship from Hugging Face and Intel. Current stable release 0.49.2 (published 2026-02-16); continuous preview wheels are rebuilt from main on every commit. Requires Python >= 3.10 and PyTorch >= 2.4 per the install guide (the PyPI metadata pins torch >= 2.3, < 3); Windows ARM64 requires Python >= 3.12.", "bit_widths": "Two families. 8-bit: LLM.int8() — vector-wise INT8 with separate normalization constants for each inner product, plus a mixed-precision decomposition that isolates outlier feature dimensions into a 16-bit matmul while, per the paper, more than 99.9% of values are multiplied in 8-bit. 4-bit: two quant types selected via bnb_4bit_quant_type — 'nf4' (4-bit NormalFloat, described in the QLoRA paper as information-theoretically optimal for normally distributed weights, and the type HF recommends for training 4-bit base models) and 'fp4'. Optional nested/double quantization (bnb_4bit_use_double_quant=True) quantizes the quantization constants themselves for an additional 0.4 bits/parameter saving (HF docs). Weights are stored in 4 or 8 bits but the matmul compute runs in higher precision, set by bnb_4bit_compute_dtype — this defaults to float32, and HF recommends explicitly setting bfloat16 or float16 to speed computation up. No sub-4-bit option exists. Separately the library ships 8-bit optimizers, which are a training feature rather than a weight format.", "runtimes": "Hugging Face transformers is the primary consumer — pass BitsAndBytesConfig(load_in_4bit=True) or (load_in_8bit=True) to from_pretrained(). Also: PEFT (the canonical QLoRA path: prepare_model_for_kbit_training + LoraConfig, with LoftQ initialization available for 4-bit; PEFT also supports bnb under VeRA, AdaLoRA and (IA)³), Accelerate (device_map, CPU offload), and vLLM (--quantization bitsandbytes, which handles both in-flight quantization and pre-quantized bnb checkpoints detected from the model config). Because it swaps torch.nn.Linear for Linear4bit/Linear8bitLt, it works on any Accelerate-compatible model in any modality — HF's own wording is \"any model in any modality, as long as it supports Accelerate and contains torch.nn.Linear layers\" — so vision and multimodal models are included, unlike most LLM-only formats. NOT loadable by llama.cpp, Ollama, LM Studio, ExLlamaV2/V3, or MLX. It is a PyTorch/Python library, not a portable file format with a standalone C++ loader.", "hardware": "NVIDIA CUDA is the only first-class, fast path, and it reaches back further than people assume: NF4/FP4 and 8-bit optimizers need only Compute Capability 6.0+ (Pascal, GTX 10-series/P100), while LLM.int8() needs CC 7.5+ (Turing, RTX 20-series/T4). Caveat on Pascal: sm60 cubins ship only in the CUDA 11.8–12.6 wheel builds; the CUDA 13.0 builds start at sm75. Wheels are built against CUDA Toolkit 11.8–13.0 for Linux x86-64, Linux aarch64 and Windows x86-64 (Linux glibc >= 2.24). Jetson/L4T is not covered by the aarch64 wheels and needs a source build with an explicit compute capability, or a third-party Jetson build. CPU is an officially supported backend with prebuilt wheels and full feature coverage: Linux x86-64 (AVX2 minimum, AVX512F/BF16 optimized), Linux aarch64, Windows x86-64 (AVX2), Windows ARM64 (NEON, Python >= 3.12), and macOS 14+ arm64 — functional, not a performance path. Intel XPU is officially supported via SYCL+Triton kernels (Linux and Windows x86-64, oneAPI 2025.1.3, PyTorch >= 2.6.0, glibc >= 2.34), covering Data Center, Arc and Battlemage. Intel Gaudi HPU (Gaudi2/Gaudi3, Linux x86-64, Gaudi v1.21 + PyTorch 2.6.0) is partial, and the split is specific: the repo's support table marks LLM.int8() supported, QLoRA partial, and 8-bit optimizers NOT supported. AMD ROCm is labelled Preview in the install guide, but within that preview all features are claimed for both CDNA and consumer RDNA; prebuilt Linux x86-64 wheels exist for ROCm 6.4.4, 7.0.2, 7.1.1, 7.2.4 and 7.14.0 (CDNA gfx90a/gfx942/gfx950, plus gfx908 only on 7.14.0; RDNA gfx1100–1103, gfx1150–1153, gfx1200/1201, with gfx1030–1036 added only on 7.14.0), plus Windows RDNA builds on ROCm 7.2.1 and 7.14.0. The HF transformers quantization overview table still marks bnb's ROCm column amber. Apple Silicon is weak but not absent: the macOS arm64 CPU backend is fully supported, while the repo's table marks Metal/MPS with a slow marker for LLM.int8() and QLoRA and work-in-progress for 8-bit optimizers, and the HF overview marks Metal amber. Mac owners wanting speed should use MLX or GGUF. Note a live doc conflict: the transformers bitsandbytes page's hardware section lists only NVIDIA/Intel XPU/Gaudi/CPU and omits ROCm, while the bitsandbytes install guide documents ROCm preview wheels — trust the library's own install guide.", "calibration": "Data-free. No calibration dataset, no activation statistics, no separate quantization run. This is bitsandbytes' defining property and the reason it exists alongside the others: quantization happens on-the-fly at load time, layer by layer, as the ordinary fp16/bf16 checkpoint streams in. vLLM's docs put it plainly — \"BitsAndBytes eliminates the need for calibrating the quantized model with input data.\" GPTQ and AWQ by contrast both require a calibration corpus and a quantization pass. The practical cost of bnb's approach is bandwidth, not time: you download the FULL-precision weights every time, so a 70B model is a ~140 GB download even though it occupies roughly a quarter of that in VRAM at 4-bit. Serialization IS supported in current versions — save_pretrained() and push_to_hub() work for both 4-bit and 8-bit models (HF notes this requires recent transformers and bitsandbytes), the HF overview table marks bnb serializable, and dequantize() can restore a loaded model to original precision at some quality cost. Widely-circulated claims that bitsandbytes cannot save 4-bit weights trace back to 2023 write-ups and are stale.", "quality_speed": "Quality claims come from the original papers and are strong: LLM.int8() reports inference in LLMs with up to 175B parameters \"without any performance degradation\" via its outlier decomposition, and QLoRA reports that 4-bit NF4 finetuning preserves \"full 16-bit finetuning task performance.\" Speed is the weak axis, and the only measured head-to-head from a primary vendor source is Hugging Face's comparison blog of 12 September 2023 (Llama-2-7b/13b on A100-SXM4-80GB, T4 and Titan RTX). In that post GPTQ ran forward passes at 29.663 tokens/sec vs bitsandbytes' 19.23 at batch size 1 on A100, and 167.68 vs 140.38 at batch size 16; on generation HF wrote that with batch_size=4 and use_cache=True GPTQ \"is twice as fast\"; bitsandbytes was FASTER than GPTQ for adapter fine-tuning on A100; and both methods cost roughly 1 point of average benchmark score on the 7B model. That measurement is nearly three years old and both libraries have moved since, so treat it as directional, not current — and note the same post's claim that bnb lacks 4-bit serialization is now obsolete. The structural reason for the speed gap still holds: 4-bit bnb is weight-only compression with higher-precision compute, so every matmul pays a dequantization cost and no fused low-bit kernel does the actual arithmetic — you buy memory, not throughput. LLM.int8() 8-bit is slower still because the outlier path forces a second 16-bit matmul; HF's docs note that setting llm_int8_threshold=0.0 \"significantly speeds up inference at the potential cost of some accuracy loss,\" which is a tacit admission that the accuracy mechanism is what costs the speed. No current tokens/sec or perplexity figures are quoted here because no current primary-source measurements could be found.", "strengths": "Zero friction: point it at any normal Hugging Face repo and it runs today with two lines of config — no waiting for someone to publish a quant, no calibration run, no separate toolchain. It is the only format in this comparison that works on essentially any model architecture built from nn.Linear layers, including vision, multimodal and diffusion models that are second-class or unsupported in GGUF/EXL2/AWQ. It is the canonical and best-supported path for QLoRA fine-tuning on a single consumer GPU (the paper's headline result: a 65B model finetuned on one 48GB GPU while preserving 16-bit finetuning task performance), and LoRA adapters trained this way merge back into the base model cleanly. It runs on genuinely old NVIDIA hardware (Pascal for NF4, on the CUDA 11.8–12.6 builds). It supports CPU offload of layers that don't fit, 8-bit optimizer states for training, nested quantization for an extra 0.4 bits/param, and torch.compile. And unusually for this list it is an actively broadening project — a real CPU backend, Intel XPU and Gaudi, ROCm preview, macOS arm64 and Windows ARM64 wheels have all landed since 2025.", "limitations": "Slower inference than any purpose-built inference format at the same bit width — this is the whole reason GGUF, EXL2 and AWQ exist. Requires the full PyTorch/transformers Python stack; there is no lightweight standalone binary, so it is not what you install on a mini-PC to run a chatbot. You download full-precision weights every time, which is brutal on large models. Apple Silicon GPU (Metal/MPS) is marked slow in the project's own support table and is not a serious option. AMD ROCm is Preview-grade, not the sure thing GGUF-on-Vulkan is, and the transformers docs don't even list it. On Intel Gaudi the gap is 8-bit optimizers (unsupported) with QLoRA only partial. Transformers warns that \"8 and 4-bit training is only supported for training extra parameters\" — you train adapters on a quantized base, not the base itself. There is no sub-4-bit option, so it cannot rescue a model that only fits at 3 bits. And 8-bit LLM.int8() specifically is now a legacy choice for pure inference: it is the slowest option in the library and offers little that 4-bit NF4 or a proper serving format doesn't do better.", "best_for": "Fine-tuning. If you want to train a LoRA/QLoRA adapter on a model that won't fit in your VRAM at fp16, bitsandbytes NF4 via transformers + PEFT is the answer and essentially the only mature one. Its second real use is as the day-zero fallback for inference: when a new model drops and nobody has published a GGUF/EXL2/AWQ yet, in-flight 4-bit bnb runs it immediately, in transformers or under vLLM. Third, it is the pragmatic choice for quantizing vision/multimodal/diffusion models that the LLM-specific formats don't handle. The sensible division of labour is to fine-tune with bitsandbytes, merge the adapter, then re-quantize the merged model into a purpose-built serving format for deployment.", "status": "Actively maintained, but its ROLE has narrowed sharply — a different thing from decline, and most write-ups conflate the two. The project itself is healthier than it has been: it moved from Tim Dettmers' personal repository to the bitsandbytes-foundation organization with Hugging Face and Intel sponsorship, shipped 0.49.0 through 0.49.2 between December 2025 and February 2026, rebuilds preview wheels from main continuously, and is still adding accelerator backends (Intel XPU and Gaudi, AMD ROCm in preview, a real CPU backend, macOS arm64 and Windows ARM64 wheels). What has genuinely declined is its position as an inference format. In 2023, load_in_4bit=True was how most people ran LLMs locally; that traffic has since moved to GGUF (llama.cpp/Ollama/LM Studio), EXL2/EXL3 for single-GPU speed, and AWQ/GPTQ/FP8/compressed-tensors under vLLM for serving. bitsandbytes now survives — and dominates — as the fine-tuning substrate rather than the runtime. LLM.int8() 8-bit in particular should be treated as legacy for inference: a landmark 2022 result that is now rarely the right pick.", "confidence": "high", "sources": [ "https://github.com/bitsandbytes-foundation/bitsandbytes — README accelerator support table (per-platform LLM.int8()/QLoRA/8-bit-optimizer status, including the Gaudi and Metal/MPS rows), MIT license, maintainer and Hugging Face + Intel sponsorship", "https://huggingface.co/docs/bitsandbytes/main/en/installation — authoritative hardware matrix: CUDA compute-capability requirements and per-toolkit sm target lists, Jetson/L4T caveat, CPU wheel matrix (AVX2/NEON/macOS arm64), Intel XPU and Gaudi requirements, AMD ROCm (Preview) version and gfx target tables, Python/PyTorch minimums", "https://huggingface.co/docs/transformers/main/en/quantization/bitsandbytes — BitsAndBytesConfig, NF4/FP4, nested quantization, compute dtype default, offloading, llm_int8_threshold, serialization/push_to_hub for 4-bit and 8-bit, dequantize(), extra-parameters-only training warning, hardware section that omits ROCm", "https://huggingface.co/docs/transformers/main/en/quantization/overview — cross-method support matrix; bitsandbytes row: on-the-fly green, CPU green, CUDA green, ROCm amber, Metal amber, Intel GPU green, torch.compile green, 4/8 bits, PEFT green, serializable green", "https://arxiv.org/abs/2208.07339 — LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (Dettmers, Lewis, Belkada, Zettlemoyer, NeurIPS 2022) — 175B without performance degradation, vector-wise quantization, mixed-precision outlier decomposition, >99.9% of values in 8-bit", "https://arxiv.org/abs/2305.14314 — QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers, Pagnoni, Holtzman, Zettlemoyer, May 2023) — NF4, double quantization, paged optimizers, 65B on a single 48GB GPU preserving 16-bit finetuning task performance", "https://huggingface.co/docs/peft/main/en/developer_guides/quantization — QLoRA setup, prepare_model_for_kbit_training, LoftQ (bnb 4-bit only), bnb support under VeRA, AdaLoRA and (IA)³", "https://docs.vllm.ai/en/latest/features/quantization/bnb.html — vLLM bitsandbytes support, --quantization bitsandbytes, in-flight vs pre-quantized checkpoints, the no-calibration statement; no deprecation notice", "https://huggingface.co/blog/4bit-transformers-bitsandbytes — NF4 vs FP4, the 0.4 bits/param double-quant saving, compute dtype", "https://huggingface.co/blog/overview-quantization-transformers — the 12 Sept 2023 measured bnb-vs-GPTQ comparison (Llama-2-7b/13b on A100-SXM4-80GB, T4, Titan RTX); source of every speed figure quoted above; its 4-bit-serialization claim is now stale", "https://pypi.org/pypi/bitsandbytes/json — current release 0.49.2, requires_python >= 3.10, torch >= 2.3 < 3", "https://api.github.com/repos/bitsandbytes-foundation/bitsandbytes/releases — release cadence and dates: 0.48.2 (Oct 2025), 0.49.0 (Dec 2025), 0.49.1 (Jan 2026), 0.49.2 (16 Feb 2026), continuous-release_main rebuilt Jul 2026" ], "headline": "Quantizes at load time; the fine-tuning answer", "sweet_spot": "QLoRA fine-tuning", "health": "healthy", "slug": "bnb" } ], "guide": { "scenarios": [ { "situation": "I have a Mac (Apple Silicon)", "pick": "MLX first, GGUF as fallback.", "why": "MLX is Apple's own framework and is built for unified memory; run it via mlx-lm or LM Studio's MLX runtime. Use GGUF (llama.cpp/Ollama) when the model you want has no MLX conversion — GGUF's model coverage is far wider and its Metal backend is mature. Do NOT use bitsandbytes, EXL2, AWQ or GPTQ on a Mac; the first is CPU-only there and the rest are CUDA-first." }, { "situation": "I have one NVIDIA consumer GPU and the model FITS in VRAM", "pick": "ExLlamaV3 (EXL3) via TabbyAPI", "why": "EXL3 is the fastest single-user, single-GPU path on NVIDIA (note EXL2 and ExLlamaV2 are archived — use ExLlamaV3) and lets you dial the bitrate to exactly fill your card. If you'd rather not fight with a Python stack, GGUF fully offloaded to GPU is the easy button and gives up less than people claim." }, { "situation": "I have one NVIDIA consumer GPU and the model does NOT fit", "pick": "GGUF.", "why": "GGUF is the only format that splits a model gracefully across GPU and system RAM. Push as many layers onto the card as fit (`-ngl`) and let the rest run on CPU. Every other format here is all-or-nothing." }, { "situation": "I have an AMD card", "pick": "GGUF, via llama.cpp's Vulkan backend.", "why": "Vulkan works on essentially any modern Radeon with no ROCm install at all, which is the single biggest practical advantage in the whole matrix. ROCm/HIP builds are faster if you're on a supported card and willing to set it up. bitsandbytes ROCm exists but is officially preview; AWQ/GPTQ on AMD are possible but fiddly." }, { "situation": "I have an Intel Arc GPU", "pick": "GGUF, via llama.cpp's SYCL or Vulkan backend.", "why": "bitsandbytes now has a real Intel XPU backend too, but only reach for it if you're fine-tuning." }, { "situation": "I only have CPU + RAM", "pick": "GGUF. There is no second choice.", "why": "llama.cpp was born a CPU inference engine and remains the only one of these formats designed for it. Expect to live at 7B-14B; buy RAM bandwidth, not cores." }, { "situation": "I want to serve many concurrent users", "pick": "vLLM with FP8, or AWQ/GPTQ 4-bit.", "why": "If your GPUs are Ada/Hopper/Blackwell (RTX 40/50, L40S, H100), FP8 is the pick — near-lossless with native tensor-core support. On Ampere (A100, RTX 30) use AWQ or GPTQ 4-bit. Do not serve concurrent traffic with GGUF or EXL2 — they're optimized for single-stream latency and fall apart under batching — and do not serve with bitsandbytes at all." }, { "situation": "I want to fine-tune", "pick": "bitsandbytes NF4 (QLoRA), via transformers + PEFT.", "why": "This is the one situation where bitsandbytes is unambiguously the right answer, and it isn't close. Set `load_in_4bit=True`, `bnb_4bit_quant_type=\"nf4\"`, `bnb_4bit_use_double_quant=True`, `bnb_4bit_compute_dtype=torch.bfloat16`. Requires an NVIDIA card in practice. When you're done: merge the adapter, then re-quantize the merged model into GGUF or AWQ for actual serving — that two-stage split is what Hugging Face itself recommends, and it's the correct mental model. bitsandbytes is a training tool that can do inference, not an inference format." }, { "situation": "The model came out this morning and no quants exist yet", "pick": "bitsandbytes in-flight 4-bit.", "why": "It's the only format that requires nobody to have prepared a file first. Accept that you're downloading full-precision weights and running slowly; swap to GGUF or EXL3 once the community publishes." }, { "situation": "It's a vision, multimodal or diffusion model", "pick": "bitsandbytes.", "why": "It quantizes any `nn.Linear`, so it doesn't care about architecture. The LLM-specific formats mostly do. (GGUF has partial VLM support and is worth checking first.)" }, { "situation": "I want to publish quants for other people", "pick": "GGUF.", "why": "Widest runtime reach by a large margin." } ], "donts": [ "Don't use 8-bit LLM.int8() for inference in 2026. It was a landmark in 2022; it is now the slowest option in its own library and a 4-bit NF4 or a proper serving-format quant beats it on both axes.", "Don't pick a format by bit-count alone. A well-made 4-bit quant of a bigger model is usually a better buy than 8-bit of a smaller one — but check a quality reference for the specific model rather than assuming.", "Don't trust hardware-support claims in blog posts older than a year. This is the field that goes stale fastest — bitsandbytes gained a CPU backend, Intel XPU support and ROCm preview wheels after most of the \"which quant format\" articles on the web were written." ] } }