Inference Endpoint

An inference endpoint is the network-addressable interface through which a deployed model receives input and returns predictions. In practice it is usually a REST or gRPC API exposed by the model serving layer, secured with authentication and access control, and engineered for predictable latency and throughput. Every "AI feature" you have ever used ultimately terminates at one of these: a URL, a request schema, and a model on the other side.

Anatomy of an endpoint

A request carries the input payload — a prompt, an image, a feature vector — and the endpoint returns the model's output. Behind that simple contract sits real machinery: the endpoint validates the request against an expected schema, authenticates the caller, routes the work to a model instance (possibly one of many replicas behind a load balancer), and may batch concurrent requests together to keep the GPU efficiently fed. For language models, endpoints commonly stream tokens back as they are generated rather than waiting for the full completion. Endpoints can be real-time, answering interactive requests with low latency, or asynchronous, accepting large batch jobs and returning results later. Capacity planning revolves around two numbers in tension — time-to-first-token and total throughput — because aggressive batching improves one at the expense of the other.

The endpoint is the trust boundary

The endpoint is where your AI service meets the outside world, which makes it the natural enforcement point for security and governance. Rate limiting, API keys or tokens, input sanitization, output filtering, and request logging all live here. It is also, bluntly, where your data goes. Every prompt sent to a third-party endpoint — and prompts routinely contain source code, financial details, health questions, business plans — is transmitted to someone else's infrastructure under someone else's retention policy. The endpoint you do not control is a confessional with an unknown priest.

Running your own

This is why self-hosting matters, and why it has become genuinely practical. Tools like Ollama and llama.cpp expose a local HTTP endpoint — typically OpenAI-compatible, so existing clients work unchanged — serving an open-weight model from your own hardware. A quantized model on a consumer GPU or even a well-equipped CPU box gives you an endpoint where prompts never leave the LAN, nothing is logged unless you log it, and no provider can change the model, the price, or the terms underneath you. If you expose that endpoint beyond localhost, treat it like any other service you host: put it behind a reverse proxy with TLS and authentication, firewall it from untrusted segments, and remember that an unauthenticated LLM endpoint on an open port is free compute for the whole internet.

Operating it like infrastructure

An endpoint is a living service, not a one-time deployment. The surrounding MLOps discipline applies at any scale: monitor latency and error rates, version the model behind the URL so clients are insulated from swaps, and roll new versions out gradually via canary deployment rather than cutting everyone over at once. For a sovereign operator the payoff is the same one that motivates running your own Bitcoin node: the interface everyone else rents, you own — and owning the endpoint means owning the trust boundary outright.

When evaluating any endpoint — rented or self-hosted — a short checklist covers most of what matters: What is logged, and for how long? Is the model behind the URL versioned and pinned, or can it change silently? What are the rate limits and timeout behavior under load? How is authentication handled, and can keys be rotated without downtime? Commercial providers answer these in policy documents you must trust; on your own endpoint you answer them in configuration you can verify. That verifiability, more than cost, is the sovereign argument: an endpoint is a promise about data handling, and the only promises you can audit are the ones running on your own machine.