What Is AI Inference? Training, Fine-Tuning and Model Serving Explained

Training, fine-tuning and inference are different jobs

A user can improve a prompt, attach a document or continue a conversation without training the model. The application is changing the information available for that inference call. Fine-tuning is a separate process that deliberately adjusts parameters. The distinction is important for cost, governance and deletion: a document in a controlled retrieval store can be updated independently, while information encoded through training is not managed like a conventional record.

A production inference service has three core layers

Hugging Face describes production inference in terms of model weights and artifacts, an inference engine, and production infrastructure. Business deployments add identity, data handling, interfaces, monitoring and support around those layers.

What happens during an LLM request

The application assembles instructions, user text, conversation history, retrieved passages and tool descriptions into an input. A tokenizer converts that input into token identifiers. The runtime performs a prompt-processing stage, often called prefill, then generates tokens sequentially during decoding. A cache can retain intermediate attention calculations so the model does not recompute the entire preceding sequence for every new token.

The user experiences a single answer, but operators must plan for input length, generated length, concurrent users, time to first token, generation rate and failure recovery. A machine that feels fast for one interactive user may not deliver the same experience when a department sends simultaneous requests. This is why production sizing starts with a workload, not merely a model name.

Batch inference and interactive inference have different priorities

Interactive chat usually values low perceived latency and streaming output. Batch work—classifying a document collection overnight, for example—may value total throughput and cost per completed record. A coding assistant may need long context and tool integration, while a short text classifier may run effectively on much smaller hardware.

These differences affect model choice, batching, replicas, queueing, memory and the acceptable time to recover from an idle or failed service. There is no single “best inference server” independent of operating constraints. A credible design records the workload, expected concurrency, data sensitivity and service objective before purchasing hardware.

Where inference runs is part of the data path

An API call sends inputs to provider-controlled infrastructure. A Canadian-hosted service can execute the same kind of request on compute located in Canada. An on-premises deployment can execute it inside the organization’s network. A local workstation may serve one user or a small team. The underlying weights might be identical across all four arrangements.

Location alone does not describe control. A proper map also identifies the operator, parent organization, administrators, subprocessors, logs, backups, embeddings, retrieval store, monitoring and remote-support paths. Canadian residency can be valuable without resolving every sovereignty question. On-premises execution can reduce external exposure without automatically establishing security or compliance.

What “Canadian AI inference” should mean

For practical procurement, Canadian inference should mean that execution occurs inside a deliberately documented Canadian control boundary. That includes knowing where compute runs, who can operate it, which contractual terms apply and whether data leaves through another layer. It should not be confused with the nationality of the model publisher.

A Canadian organization can run a commercially usable open-weight model created elsewhere on Canadian infrastructure. It can also run a Canadian-developed model where that model fits the task. Intelligence sovereignty comes from the ability to operate, evaluate, replace and recover the capability—not from attaching a flag to an opaque endpoint.

Inference does not need to be all-local or all-cloud

A staged architecture can route sensitive or routine work to controlled infrastructure and retain an approved external model for tasks that genuinely require it. The important controls are explicit routing, user visibility, evaluation and a tested fallback. A hybrid design is not sovereign merely because a local model is present; it is useful when the organization knows which requests go where and can continue important work if one path disappears.

The same principle applies to migration. Start with a representative workflow, compare outputs, record latency and resource use, and document failure handling. Only then expand. A measured transition is usually safer than a company-wide switch based on a benchmark or political headline.

Questions to ask before deploying inference

• What exact work will the endpoint perform, and how will quality be evaluated?
• Which data enters prompts, retrieval, logs and traces?
• How many simultaneous users or automated jobs must it support?
• What model licence and version will be approved and recorded?
• Who administers the runtime, hardware, updates and access?
• What happens when the model, server, network or vendor is unavailable?

Frequently asked questions

Does inference train the model on my prompt?

Ordinary inference does not update the base weights. A service may separately retain or use data under its product settings and terms, so verify those controls rather than inferring them from the technical definition.

Is inference only for generative AI?

No. Classification, computer vision, embeddings, forecasting and other machine-learning tasks also use inference. Generative inference produces content rather than only a label or score.

Does local inference mean no network is involved?

Not necessarily. “Local” may mean the model runs on one machine while other users reach it over a LAN. Tools, updates or hybrid features may still use external networks. Trace the complete system.

Primary sources reviewed August 24, 2026: Google Machine Learning Glossary; Hugging Face, About Inference Endpoints; Google, Fine-tuning, distillation and prompt engineering; NIST AI 600-1, Generative AI Profile.