Definition
Model serving is the infrastructure layer that takes a trained machine learning model and makes it answer prediction requests in production. Where training is a one-time (or periodic) batch job, serving is a long-running service: it loads the model into memory, accepts incoming input, runs a forward pass, and returns a result — ideally within tens of milliseconds and without falling over under load.
What a serving layer handles
A serious model-serving stack does far more than wrap a model in a web handler. It manages input validation against an expected schema, supports both real-time and batch invocation modes, and autoscales compute based on request volume. Frameworks such as KServe, NVIDIA Triton, vLLM, or MLflow's built-in server handle GPU placement, request batching, and model loading so the same hardware serves more requests per second.
Serving vs. the broader pipeline
It is easy to conflate serving with MLOps as a whole, but serving is just the runtime piece. MLOps also covers training, governance, the model registry, and ongoing monitoring. Serving is where a model finally earns its keep, and it is the layer most exposed to real-world latency and availability requirements.
For sovereign operators, self-hosted serving means the model and the hardware are both yours — no per-token metering, no upstream provider that can deprecate or alter the endpoint. To learn how requests actually reach a served model, read our entry on the inference endpoint.
In Simple Terms
Model serving is the infrastructure layer that takes a trained machine learning model and makes it answer prediction requests in production. Where training is a…