Model Monitoring

Model monitoring is the continuous process of observing a deployed machine-learning model's behavior and performance in production. A model that scored well in testing can quietly decay once it faces real, shifting data, so monitoring exists to catch that degradation before it costs anything: you track predefined metrics, alert when they cross thresholds, and investigate before users notice. For a self-hosted deployment the stakes are personal — you are the operator, so you own the dashboards, the alerts, and the 2 a.m. response. Nobody else is watching your box.

What gets monitored

Monitoring spans three broad categories. Performance metrics measure prediction quality — accuracy, precision, recall for classifiers; task-specific scores or user feedback signals for generative models — though these are only computable when some form of ground truth eventually arrives, which in many real systems it does slowly or never. Operational metrics measure system health: inference latency per request, tokens per second, throughput, queue depth, GPU temperature, and VRAM utilization. These are always available and are usually your first warning — a memory leak or a thermal throttle shows up here hours before it shows up anywhere else. Data quality metrics check that incoming inputs still match the expected schema and statistical distribution: a spike in missing fields, out-of-range values, or unfamiliar token patterns very often signals a broken upstream pipeline rather than a model fault, and catching it at the input boundary saves a confusing hunt downstream.

Drift: the slow killer

The most important thing monitoring surfaces is drift — the gradual change in input data or in the input-output relationship that erodes accuracy over time. The world moves: vocabulary shifts, user behavior changes, hardware in the field gets replaced, prices and specs go stale. A model trained on last year's distribution serves this year's queries a little worse each month, with no single failure to point at. Statistical drift detectors compare live input distributions against a training-time reference and alert on divergence, giving you a trigger for retraining or fine-tuning before quality visibly sags. Without a drift signal, the usual discovery mechanism is a user complaint — the most expensive monitoring system ever devised.

Monitoring versus observability

Monitoring is typically framed as a subset of observability. Monitoring tells you that a metric crossed a threshold; observability is the property that lets you work out why — whether an accuracy drop traces to a data shift, a new user segment, a dependency upgrade, or a silent pipeline bug. In practice that means keeping enough context to reconstruct incidents: structured logs of requests and responses (with retention and privacy rules you choose, which is precisely the advantage of self-hosting), versioned records of which model and configuration served which request, and traces that connect a bad output back through the stack that produced it.

Monitoring a local LLM stack

The principles port directly to a homelab running open models through a server such as Ollama or llama.cpp. Watch operational basics first: GPU temperature and memory headroom, context-length pressure, latency percentiles, and error rates. Log prompts and completions if your privacy posture allows, sample them for quality review, and record model versions so a regression after swapping checkpoints is attributable in minutes instead of days. Anyone who has run ASICs already knows this discipline — you watch hashrate, temperature, and reject rate because silent degradation is expensive. Model monitoring is the same craft applied to inference: it closes the loop in MLOps, turning a static deployment into a system you can actually trust over time.