ExLlamaV2

ExLlamaV2 is an open-source inference library built for running large language models fast on consumer-class NVIDIA GPUs. It is the successor to the original ExLlama and is optimized specifically for generation from quantized models, making it a natural fit for a sovereign operator who wants strong single-GPU performance from a desktop card rather than data-center hardware. Where general frameworks aim for breadth, ExLlamaV2 aims for one thing — tokens per second from a quantized model on the GPU you already own — and it is very good at it.

The EXL2 format

ExLlamaV2 introduced the EXL2 quantization format, which builds on the same underlying method as GPTQ but removes its rigidity. EXL2 supports 2, 3, 4, 5, 6, and 8-bit quantization and — its signature feature — allows different bit rates to be mixed within a single model, chosen per layer according to measured sensitivity during quantization. This lets a model target an arbitrary average bits-per-weight value: rather than choosing between "4-bit fits, 5-bit doesn't," an operator can request, say, 4.65 bits per weight and fill their card to the last usable megabyte of VRAM. Quality degrades gradually as the average drops instead of falling off a cliff at fixed steps, so the trade-off between model fidelity and memory becomes a dial rather than a switch.

Performance and serving

For 4-bit-class quantized models on a single modern consumer GPU, ExLlamaV2 is among the fastest options available, with custom CUDA kernels doing the heavy lifting. Later versions added paged attention via Flash Attention, dynamic batching, and key-value cache deduplication, along with quantized KV-cache options that stretch how much conversation context fits in memory. The library can be embedded directly in Python, but it is commonly served behind TabbyAPI, an OpenAI-compatible server, so chat front-ends and agent frameworks can talk to a locally hosted EXL2 model exactly the way they would talk to a remote API — a pattern that keeps the whole stack swappable and vendor-free.

Where it fits, and where it doesn't

Quantizing a model yourself is a routine, documented process: the converter runs a calibration pass over sample text to measure per-layer sensitivity, then allocates the bit budget you requested. The choice of calibration data matters at aggressive bit rates — calibrating on text unlike your actual usage can skew which layers get squeezed — so operators targeting a specific domain feed it representative material. Multi-GPU operation is supported by splitting layers across cards, letting two mid-range GPUs serve a model neither could hold alone, and the successor project ExLlamaV3 continues the same philosophy with a revised format. The community publishes EXL2 quants of popular models at several average bit rates, so checking for an existing quant before converting is always the first move — and comparing two published quants of the same model at different average bit rates is the fastest way to feel where your own quality threshold sits.

The honest trade-offs: EXL2 models must fit entirely in GPU memory — there is no CPU-offload escape hatch, which is precisely the flexibility that makes GGUF and llama.cpp the better choice when a model is bigger than your card. It is NVIDIA/CUDA-centric (with partial AMD support via ROCm), so Apple-silicon and CPU-only operators look elsewhere. And the EXL2 ecosystem is smaller than GGUF's: fewer pre-quantized models are published, so you may need to quantize a model yourself, which takes a calibration pass and some patience. The practical rule of thumb for a self-hoster: if the model fits fully on your GPU and you want maximum interactive speed, ExLlamaV2 is a top contender; if it doesn't fit, or you value run-anywhere portability, take the GGUF path. The underlying technique both roads share is covered under model quantization.