Local AI Hardware Guide: Which GPU and Server for Every Model Size in Canada

The short answer: choose local AI hardware by working backwards from the service you need to operate. Start with the model files and precision you will actually deploy, add memory for context and concurrent requests, confirm that the runtime supports the processor architecture and accelerator, and only then compare machines. For Canadian buyers, D-Central’s current paths are a custom discrete-GPU inference rig, an AMD Strix Halo 128 GB system at $7,349 CAD, and an NVIDIA DGX Spark at $9,449 CAD. Larger or more demanding deployments belong in a designed on-premises AI environment, not in a desktop shopping cart.

This is the architecture hub: it explains how model memory, throughput objectives, software support, power, networking, and operating responsibility shape the hardware envelope. If you already know the workload and want a merchant-oriented shortlist, use Which Local AI Computer Should You Buy in Canada?. If your decision has narrowed to two 128 GB desktops, use the focused DGX Spark vs AMD Strix Halo comparison. Neither page replaces the planning process below.

A decision framework for local inference hardware

A useful purchase brief has five inputs: target model artefact, context policy, expected concurrency, runtime, and operational owner. “We want to run an LLM” is not yet a hardware requirement. “We want a quantized model served to six internal users through an authenticated interface, with no prompts leaving the office” is much closer. The final sizing still needs a representative model file and a measured pilot, but the table below identifies the right class of system to investigate.

Workload situation	Hardware envelope to evaluate	Why it can fit	What can invalidate the choice	Next D-Central route
One developer, compact quantized models, experimentation	Existing capable computer or a modest discrete GPU	Lowest acquisition friction; proves the workflow before a dedicated purchase	Model file, context cache, or required runtime exceeds available memory	Install Ollama in 10 minutes
Models that fit a discrete GPU, with latency or serving as the priority	Custom GPU inference rig	Dedicated accelerator memory and a configurable tower platform	Weights or cache spill outside GPU memory; power, noise, or expansion assumptions are wrong	Request a rig configuration
Large memory capacity, x86 host compatibility, ROCm-ready workflow	AMD Strix Halo 128 GB	A large unified-memory pool in an x86 desktop class	The exact model/runtime/operator path is unsupported, or shared-memory configuration is not prepared	AMD Strix Halo product page
Large memory capacity, NVIDIA’s CUDA stack, DGX OS and Arm64 are acceptable	NVIDIA DGX Spark	128 GB coherent unified memory and an integrated NVIDIA software platform	An x86-only dependency, unsupported container, or service-level target discovered after purchase	NVIDIA DGX Spark product page
Departmental service, high concurrency, multiple accelerators, strict availability, or models beyond a desktop envelope	Designed workstation, server, or rack deployment	Compute, storage, network, access control, monitoring, backup, and recovery can be engineered together	Desktop assumptions are carried into a production service without an operator or capacity test	AI sovereignty consultation

The important distinction is capacity versus service performance. A model that can be loaded is not automatically a model that meets an interactive latency target, supports several users, or remains stable at a long context length. This guide does not publish a universal tokens-per-second number because model build, quantization, prompt length, runtime version, sampler settings, batching, thermals, and client load can all change the result. Validate the exact service you plan to operate.

Step 1: define the workload before choosing a machine

Write down the exact model repository and file, licence, quantization or precision, maximum accepted prompt size, desired response length, and number of simultaneous requests. If retrieval-augmented generation is part of the design, include the embedding model, reranker, vector store, document ingestion process, and storage growth. If speech, vision, or image generation shares the machine, include those components too. The language model is only one resident in the memory and compute budget.

Separate interactive use from batch work. A private chat assistant can tolerate a queue that would be unacceptable for an application API. A nightly document job may value total completion time rather than first-token latency. A developer workstation can be restarted for a driver update; a departmental knowledge service needs a maintenance window, monitoring, and a rollback path. Those are different systems even if they load the same weights.

Finally, identify non-negotiable software. An internal package compiled only for x86-64 is a material constraint on an Arm64 host. A workflow built around CUDA is not automatically portable to ROCm. A container tag that starts today is not a complete lifecycle plan. Record versions and test the model, runtime, drivers, extensions, and API clients as a unit.

Step 2: turn model artefacts into a memory envelope

Parameter count is a starting clue, not a purchase specification. At a simplified level, weight storage is parameter count multiplied by bytes per stored weight: roughly two bytes at 16-bit, one byte at 8-bit, and one-half byte at 4-bit before metadata and implementation overhead. Quantized file formats add scales and other structures, so the actual downloadable file is better evidence than mental arithmetic. Use the exact artefact size when it exists.

Then budget beyond the weights. The runtime needs working memory. The key-value cache grows with context and active sequences. Longer prompts and more simultaneous requests can therefore make a configuration fail even when the weight file fits. The operating system, display, container runtime, model server, web interface, retrieval components, and any second model also consume memory. Unified memory is shared; its headline capacity is not a promise that every byte can be assigned to weights.

Use the local LLM VRAM calculator to form an initial estimate, then retain meaningful headroom and run a pilot with representative prompts. Do not size to a screenshot showing an idle loaded model. Measure the longest allowed context, expected simultaneous requests, real document retrieval, and failure behaviour. If the service swaps, unloads a model, resets under load, or misses the response target, revise the envelope.

Step 3: understand the four practical hardware paths

Reuse an existing machine for discovery

The cheapest proof of concept often runs on equipment already owned. This is where a team validates model quality, prompt patterns, retrieval design, and user demand. Keep the pilot honest: record its hardware, model hash, quantization, runtime version, context limit, and test prompts. The output is not “local AI works”; it is a reproducible workload description that can be taken to the dedicated-hardware decision.

This path is appropriate when downtime is acceptable and no sensitive production corpus has been introduced. It is not a substitute for access controls, encrypted backups, patch management, or an accountable system owner. Follow the first-boot Ollama and Open WebUI guide for a simple evaluation path.

Use a custom discrete-GPU rig when the workload fits the accelerator

A discrete-GPU tower can be a strong fit when the target weights, cache, and runtime remain inside accelerator memory and the project benefits from a configurable chassis. CPU, system RAM, storage, network, power supply, cooling, and GPU selection can be matched to the brief rather than accepted as an appliance bundle. The trade-off is that the GPU memory boundary is real. Loading part of the workload outside that fast pool changes the behaviour and must be tested rather than assumed.

D-Central treats this as a quoted configuration because the useful bill of materials depends on the model and operations plan. The Canadian GPU inference rig guide explains the category; the custom GPU inference rig page is the commercial route. No universal component list or performance claim is implied by the product name.

Use AMD Strix Halo when x86 plus a large unified pool fits the stack

AMD’s official specification for the Ryzen AI Max+ 395 lists a 16-core, 32-thread x86 processor, Radeon 8060S graphics with 40 graphics cores, support for up to 128 GB of LPDDR5X-8000 memory on a 256-bit interface, and a configurable 45–120 W TDP range. Verify these processor facts on the official AMD product page. A system vendor’s chassis, cooling, firmware, storage, ports, and power policy remain separate from the processor specification.

The attraction for local inference is capacity combined with x86 compatibility, not a promise of equivalent behaviour to a discrete GPU. AMD’s current ROCm compatibility matrix lists supported operating systems and hardware, including Radeon 8060S entries; read the matrix for the exact ROCm release you plan to install. AMD also documents Strix Halo shared-memory configuration, including Linux GPU virtual-memory and GTT considerations. Those steps make operating-system preparation part of deployment.

D-Central’s 128 GB Strix Halo plan is $7,349 CAD. Its product page is the source for the offered configuration, while AMD documentation is the source for silicon and software-platform facts. For a platform-by-platform workflow, read the AMD Strix Halo ROCm LLM guide.

Use DGX Spark when NVIDIA’s integrated stack and Arm64 fit

NVIDIA’s official DGX Spark hardware documentation specifies the GB10 system, a 20-core Arm CPU, 128 GB of unified LPDDR5X memory at 273 GB/s, NVMe storage options, 10 GbE, ConnectX-7, Wi-Fi 7, and a supplied 240 W power adapter. NVIDIA states that the coherent memory pool allows the CPU and GPU to share model data. That is a capacity and architecture fact, not a guarantee for any particular model’s speed or user count.

The DGX Spark system overview describes DGX OS, CUDA libraries, Docker, NVIDIA Container Runtime, and access to NGC. The official porting guide also makes the Arm64 architecture and Ubuntu-based DGX OS explicit. Audit binaries, Python wheels, extensions, and containers for Arm64 support before committing the service to this platform.

D-Central’s DGX Spark plan is $9,449 CAD. Use the DGX Spark setup guide for Canada for the deployment path and Ollama on DGX Spark for that runtime specifically. The DGX Spark product page is the merchant route.

Move to a designed server or rack when the service exceeds a desktop

Multiple accelerators or nodes introduce topology, networking, scheduling, failure domains, power density, cooling, remote administration, and data-protection questions. Adding nominal memory across boxes does not prove that a model or runtime can partition work usefully. NVIDIA’s DGX Spark clustering guide, for example, specifies ConnectX-7 cabling and host network configuration; clustering is an operated system, not a passive sum of two memory numbers.

A production design should begin with a measured workload and availability objective. D-Central can scope an on-premises deployment, but the result is a quoted engineering engagement, not a fictional universal cluster SKU. Procurement should follow architecture approval, site review, and responsibility assignment.

Step 4: match the runtime to the platform and service

Ollama, llama.cpp, and vLLM solve overlapping but different operational problems. Ollama offers an accessible local model workflow. llama.cpp provides a flexible, portable ecosystem for GGUF inference and exposes many tuning controls. vLLM is oriented toward serving and batching. Format, architecture, accelerator back end, and release support matter more than a generic “best runtime” label. See Ollama vs vLLM vs llama.cpp for the dedicated decision.

NVIDIA publishes a DGX Spark llama.cpp build recipe with a CUDA build and an OpenAI-compatible server example. NVIDIA also provides a DGX Spark vLLM playbook. AMD publishes a Ryzen vLLM Docker workflow that identifies its supported architecture and versioned build process. These are stronger starting points than assuming one command behaves identically across CUDA, ROCm, x86, and Arm64.

Choose a version baseline and preserve it. Save container digests or lock files, model hashes, launch flags, context limits, prompt templates, and API configuration. Test an update in staging before changing the working machine. The CUDA vs ROCm local-inference guide explains the ecosystem boundary in more detail.

Ownership and operations boundaries

Layer	D-Central can supply or scope	Customer ownership that must be assigned
Hardware	Selected system or quoted configuration, initial fit review, and agreed setup scope	Site readiness, physical security, asset management, acceptable-use decision
System software	Agreed baseline installation and documented handoff where included	Patch cadence, credentials, change approval, monitoring, incident response
Models and data	Technical deployment assistance for agreed artefacts	Licence review, data authority, retention, access policy, output validation
Application service	Integration or consulting when separately scoped	User support, service objectives, backups, disaster recovery, security review

Buying a computer transfers neither governance nor compliance. Decide who can upload documents, download models, modify system prompts, expose an API, approve upgrades, rotate credentials, restore backups, and shut down a compromised service. Keep administrative access separate from ordinary user access. Log enough to operate and investigate the system, but do not create a new sensitive prompt archive by accident.

Also define the handoff boundary before purchase. “Configured” could mean hardware validation, operating-system baseline, a model server, a private web interface, network integration, or ongoing management. Put the exact deliverables, acceptance test, and post-handoff owner in writing. This guide does not make stock, lead-time, warranty, or performance commitments.

Canadian digital sovereignty: what local hardware changes

Local inference can keep prompts, retrieved documents, embeddings, and outputs inside infrastructure controlled by the organization. It can reduce dependency on a remote API and make data flows easier to inspect. It does not by itself satisfy Quebec Law 25, PIPEDA, a contractual security schedule, or a sector-specific obligation. Governance, lawful authority, minimization, access control, retention, vendor relationships, and incident procedures still matter.

Map every external dependency: model downloads, update servers, telemetry, identity provider, remote support, backups, monitoring, and any fallback API. An application is not fully local if a plugin silently sends content elsewhere. Read the Canadian AI inference hub, the CLOUD Act and Canadian AI explainer, and the Quebec Law 25 on-premise LLM guide as distinct governance layers. Obtain legal advice for your actual obligations.

A practical purchase sequence

1. Prove model usefulness. Test the exact artefact on available hardware or a controlled evaluation system.
2. Freeze the workload brief. Record runtime, precision, model size, maximum context, simultaneous demand, auxiliary services, and data classification.
3. Calculate a capacity envelope. Include weights, cache, runtime overhead, the operating system, and headroom; do not equate “loads once” with “operates reliably.”
4. Eliminate incompatible architectures. Verify CUDA or ROCm support, x86-64 or Arm64 dependencies, operating system, container images, and required extensions against official documentation.
5. Run an acceptance test. Use representative prompts and concurrency, then observe memory, errors, queueing, thermals, and recovery. Keep the results with the configuration record.
6. Assign an operator. Approve patching, backups, access, logging, incident handling, and upgrade policy before live data arrives.

At the commercial stage, compare the three D-Central purchasing paths. If both 128 GB desktops survive the architecture screen, the Spark-vs-Strix decision page isolates their platform trade-offs without redoing this full sizing exercise.

Frequently asked questions

How much memory does a local LLM need?

Begin with the exact model artefact rather than parameter count alone. Add memory for runtime structures, the key-value cache, context, concurrent requests, the operating system, and any embedding or reranking models. Quantization reduces weight storage but does not remove the rest of the budget. Leave headroom and validate the maximum permitted workload.

Is 128 GB of unified memory the same as 128 GB of GPU VRAM?

No. Unified memory and discrete accelerator memory have different architectures, bandwidth characteristics, allocation rules, and software paths. A large unified pool may admit models that do not fit on a smaller discrete GPU, but admission alone does not establish service performance. Read VRAM vs unified memory before comparing capacity labels.

Should I buy DGX Spark or AMD Strix Halo?

Choose based on the software and operating model. DGX Spark provides NVIDIA’s CUDA/DGX environment on Arm64; Strix Halo provides an x86 host and AMD’s ROCm path. Confirm every required binary, model runtime, container, and extension, then test the service. The detailed DGX Spark vs Strix Halo comparison covers this narrower decision.

When is a custom GPU inference rig the better fit?

It is a strong candidate when the workload fits discrete accelerator memory, the preferred software stack supports the selected GPU, and a configurable tower matches the site’s power, cooling, noise, storage, and expansion needs. It should be quoted against the workload rather than treated as one universal bill of materials.

Can I size hardware from parameter count alone?

No. Parameter count does not specify precision, quantization metadata, architecture behaviour, cache, context, batch, concurrency, or auxiliary services. Use the real model file and runtime, then measure the service under representative load. For mixture-of-experts models, total and active parameter descriptions require especially careful interpretation.

Do I need Ollama, llama.cpp, or vLLM?

You need a runtime that supports the exact model format, accelerator back end, processor architecture, operating system, and service pattern. Ollama is accessible for local workflows, llama.cpp is flexible for GGUF deployments, and vLLM targets model serving. None is universally best, and support changes by release.

Does running an LLM locally make it compliant in Canada?

No. It can reduce external data transfers and give the organization more direct control, but compliance depends on purpose, authority, safeguards, retention, access, vendor relationships, and documented governance. Treat local placement as one architectural control, not a legal conclusion.

What should I test before accepting a local AI system?

Test the approved model and runtime at the maximum allowed context and representative simultaneous demand. Include retrieval, authentication, API clients, restart behaviour, logs, backup restoration, update rollback, and loss of network access. Record model hashes, software versions, launch settings, resource use, errors, and whether the agreed service objective was met.

Ready to route the decision? Use the Canadian local AI computer buyer guide for merchant fit, browse the sovereign AI hardware category, or request an architecture and deployment consultation for workloads that exceed a straightforward desktop purchase.