vLLM

vLLM is an open-source library for fast LLM inference and serving, originally developed at UC Berkeley's Sky Computing Lab. Where lightweight runtimes target a single user on one machine, vLLM is built for throughput: serving many simultaneous requests efficiently, which makes it the tool of choice when self-hosting a model for a team, an application, or a small "Hashcenter" that combines mining and compute.

Key techniques

vLLM's signature innovation is PagedAttention, which manages the attention key-value cache in non-contiguous memory blocks, much like virtual memory paging in an operating system. This dramatically reduces wasted GPU memory. It pairs this with continuous batching, which adds and removes requests from the running batch dynamically rather than waiting for fixed groups, alongside prefix caching, chunked prefill, and speculative decoding.

Hardware and compatibility

vLLM supports NVIDIA and AMD GPUs, x86/ARM/PowerPC CPUs, and specialized accelerators, and runs 200+ model architectures. It exposes an OpenAI-compatible API server, so existing client code can target a private vLLM deployment with minimal change. That compatibility is the recurring theme of sovereign AI tooling: keep the familiar interface, remove the dependence on someone else's servers.

For single-user local runtimes, compare llama.cpp and Ollama; for the model registry many deployments pull from, see the Hugging Face Hub.

Find serving stacks in the sovereign self-hosting catalog.