Adapter (LLM)

An adapter is a small, trainable neural module inserted between the layers of a large pretrained model to adapt it to a new task without retraining the whole network. First introduced for transformers by Houlsby et al. in 2019, adapters keep the original model weights frozen and learn only the inserted modules, making them a foundational parameter-efficient fine-tuning technique — the family of methods that turned "customize a language model" from a datacenter project into something an individual can do.

The bottleneck architecture

A classic adapter uses a bottleneck design: it projects the layer's high-dimensional representation down to a much smaller dimension, applies a non-linear activation, then projects back up to the original size, with a residual connection around the whole block. Because the down- and up-projection matrices are tiny relative to the model's own layers, an adapter adds only a small fraction of trainable parameters. The residual connection is the safety net: initialized near zero, the adapter starts as an approximate no-op, so training begins from the pretrained model's full competence and learns only the delta the new task requires. Houlsby's work reported matching full fine-tuning accuracy on standard benchmarks while training a small fraction of the parameters — the result that opened the door to the whole PEFT research line.

Why adapters are useful

Adapters are modular and composable. You can train a separate adapter for each task or domain and swap them in and out at inference time over a single shared base model, which is far cheaper than maintaining a full fine-tuned copy per task: one base model on disk, a shelf of megabyte-scale adapters beside it. Follow-up methods extended the idea — AdapterFusion learns to combine several task adapters, and later variants shrank the modules further. The trade-off worth knowing is a small inference cost: because bottleneck adapters add sequential computation inside each layer, they add latency that weight-merging methods avoid.

Adapters versus LoRA

The other famous PEFT approach, LoRA, shares the philosophy — freeze the base, train something small — but differs in mechanics: LoRA learns low-rank updates to existing weight matrices rather than inserting new modules into the forward pass, and its updates can be merged into the base weights after training, leaving zero inference overhead. That is why LoRA and its quantized descendant QLoRA dominate today's open-weight fine-tuning. "Adapter" survives both as the specific bottleneck technique and, loosely, as the umbrella word for any small trained attachment — LoRA files shipped for local models are routinely called adapters.

Why self-hosters care

For anyone running models on their own hardware, adapters are the practical answer to a real problem: you cannot afford to store or retrain a full model per specialty, and you should not have to. A home-lab machine can hold one strong open-weight base and cheap adapters for each domain that matters — mining-fleet diagnostics, contract drafting, code review — activating expertise on demand. All of it trains and runs locally, so the data that teaches the model never leaves your custody. Adapters sit inside the transformer blocks of the underlying model rather than altering its pretrained weights, and alongside fine-tuning proper they form the customization layer of a sovereign AI stack.

The serving side is where modularity pays off: local inference servers can load a base model once and switch or stack adapters per request, so one machine hosts many specializations without duplicating weights in memory. When a specialization stabilizes, merging its adapter into the base weights produces a single deployable model with no runtime seam. Treat adapters like configuration under version control — small, diffable artifacts that describe how your model differs from stock — and the whole customization layer stays auditable.