Model Serving

Model serving is the infrastructure layer that takes a trained machine-learning model and makes it answer prediction requests in production. Where training is a one-time or periodic batch job, serving is a long-running service: it loads the model into memory, accepts incoming input, runs a forward pass, and returns a result — ideally within tight latency bounds and without falling over under load. Training makes the model; serving is how anyone ever actually uses it.

What a serving layer handles

A serious model-serving stack does far more than wrap a model in a web handler. It validates input against an expected schema, supports real-time and batch invocation modes, manages model loading and versioned rollouts so an update does not drop traffic, and scales with request volume. For large language models specifically, the hard problems are memory and throughput: a served LLM must hold its weights plus a growing key-value cache for every active conversation in VRAM, so serving frameworks earn their keep with continuous batching (interleaving many users' token generation on one GPU), paged attention-cache management, and quantized weights that shrink the memory bill. Engines such as vLLM, NVIDIA Triton, and KServe target this space at datacenter scale; llama.cpp and Ollama bring the same serving role — load model, expose endpoint, answer requests — down to a single workstation.

Serving versus the broader pipeline

It is easy to conflate serving with MLOps as a whole, but serving is just the runtime piece. MLOps also covers training pipelines, governance, the model registry that tracks which artifact is deployed where, and the ongoing monitoring that catches quality drift after deployment. Serving is where a model finally earns its keep, and it is the layer most exposed to real-world latency, availability, and cost constraints — which is why it tends to be engineered and measured more like a database than like a research artifact. The interface it exposes, typically an HTTP API, is the inference endpoint the rest of your software talks to.

The sovereign angle

For sovereign operators, self-hosted serving is the whole point of running open-weight models: the model and the hardware are both yours, so there is no per-token metering, no rate limit imposed from outside, no upstream provider that can deprecate, alter, or log the endpoint your tools depend on. A home-lab serving setup can be as simple as Ollama on a GPU workstation answering requests from your LAN — the same machine-room thinking a miner applies to a hashboard applies here: dedicated hardware, a known workload, and full-stack ownership. The trade is that capacity planning, updates, and uptime become your craft rather than someone else's SLA. Size the model to your VRAM honestly, keep inference latency measured rather than assumed, and treat the endpoint like any other service you expose: bound to the LAN unless you have a reason and a plan.

The metrics that matter

Serving quality is measured in a small set of numbers worth knowing even for a single-user home lab. Time-to-first-token is the pause before a response begins and is dominated by prompt processing, so it grows with input length. Tokens per second is the generation rate that makes output feel fluid or sluggish. Concurrency is how many requests the endpoint sustains before both numbers degrade, and on a GPU it is bounded by memory: each active conversation's attention cache competes for the same VRAM as the weights. Measuring these three under your real workload — long prompts, your actual model, simultaneous requests if you share the box — turns "the model feels slow" into an engineering statement with a fix attached: quantize further, shrink the context, batch smarter, or buy memory.

To learn how requests actually reach a served model, read our entry on the inference endpoint; for the ecosystem around it, see MLOps.