What Actually Fits: 24 GB GPU vs 128 GB Unified vs Dual-GPU

This page is the memory-architecture and capacity-planning spoke. It does not rank complete computers or promise that named models fit a memory tier. Use the local AI hardware guide for the full workload-to-system architecture, the Canadian local AI computer buyer guide for merchant selection, or DGX Spark vs Strix Halo for the focused two-platform comparison.

Start with a memory budget, not a parameter headline

A local inference service uses memory for more than model weights. The operating system and display reserve memory. The inference engine allocates metadata, compute buffers, graph workspaces, temporary tensors, and kernels. The active context uses a key-value cache. Parallel requests can require additional cache and scheduling state. Embedding models, rerankers, vision projectors, speech models, a web interface, and a vector database may share the same host. Updates can also change allocation behaviour.

A useful generic planning equation is:

Required usable pool ≈ loaded weights + KV cache + runtime/workspaces + auxiliary models and services + operating reserve + safety headroom.

This is a checklist expressed as arithmetic, not a universal estimator. Each term must come from the exact model artefact and runtime configuration. On a discrete GPU, some terms remain in host RAM while latency-sensitive tensors and caches occupy VRAM. On a unified-memory system, CPU and GPU allocations draw from the same physical capacity even when tools report them through different interfaces. On multiple GPUs, placement rules decide which terms are sharded, duplicated, or concentrated on one device.

Capacity worksheet for a local LLM deployment

Worksheet field	Evidence to collect	Why it changes memory	Acceptance record
Exact model artefact	Repository, filename, revision and SHA-256	Parameter count does not specify stored precision, quantization metadata, tied weights or auxiliary files	Actual file size and load-time tensor allocation log
Runtime and backend	Ollama, llama.cpp, vLLM or other engine; CUDA, ROCm/HIP, Vulkan or CPU; exact version/build	Backends allocate different workspaces and support different cache formats and placement rules	Build flags, package/container digest and detected devices
Maximum context	Accepted input plus output token policy	KV-cache demand generally grows with stored sequence length, subject to model architecture and runtime strategy	Measured allocation at the maximum permitted request
Concurrency	Maximum simultaneous active sequences, not total named users	Each active sequence needs cache and scheduler capacity; prefix sharing and batching do not make demand disappear	Measured allocation and queue behaviour under representative load
Auxiliary workloads	Embeddings, reranking, vision, speech, speculative models, adapters, database and UI	Additional weights, caches, media buffers and processes share the device or host	Full application test, not an isolated language-model run
Reserved capacity	OS, display, services, monitoring, update process and recovery margin	Installed capacity is not fully allocatable to inference	Observed idle baseline plus an explicit safety margin
Multi-device placement	Split mode, per-device allocation, interconnect, peer access and communication library	Some data is sharded; other state can be replicated or unevenly placed	Per-device logs and a restartable configuration

Complete the worksheet twice: once from documentation and again from observation. The documented estimate decides whether a pilot is reasonable. The measured record decides whether the system is accepted. Keep both with the model and software hashes so the result can be reproduced after an update.

Estimating weights without inventing a fit table

For a simple dense model, raw weight storage can be approximated as parameter count multiplied by storage bits per weight, divided by eight. A hypothetical 4-bit representation therefore starts near one-half byte per stored parameter. That arithmetic is deliberately generic. Real quantized artefacts include scales, metadata, tensors stored at other precisions, and format overhead. Some architectures contain shared or tied parameters; mixture-of-experts descriptions distinguish total from active parameters; multimodal packages add components. The actual artefact and load log supersede the formula.

Quantization changes more than file size. The runtime must implement that quantization type on the selected backend. Kernels may expand, stage, or transform tensors at load or execution time. Quantizing the model weights does not automatically quantize the KV cache, activations, embeddings, or every auxiliary model. It can also affect output quality, so capacity approval and application-quality approval are separate tests.

Use the local LLM VRAM calculator for a first-pass worksheet and the GPU-to-model compatibility tool to narrow candidates. Neither replaces the runtime log. Do not turn a calculated value into “this model fits” until the complete service passes the maximum-context and concurrency test.

What 24 GB of discrete VRAM actually means

A discrete GPU places accelerator memory on the graphics card, separate from system RAM. This gives the accelerator a dedicated local pool with a board-specific capacity and bandwidth. The useful planning question is not whether the card says 24 GB; it is how much remains allocatable after the display, driver, runtime, workspaces, cache, and any other GPU processes are accounted for.

If the required device allocations fit with headroom, a discrete GPU can keep the active workload in its local memory. If they do not, behaviour depends on the engine: it may refuse to load, reduce offload, place layers in system memory, move data across the host interface, or fail later as context grows. “The model started” is therefore weaker evidence than “the full accepted workload stayed within its intended placement.”

Host RAM does not simply extend 24 GB of VRAM into one equally capable pool. Offload can be useful, and unified virtual addressing can simplify access, but the physical memory locations and transfer path remain operationally important. Do not describe a 24 GB card with 64 GB of host RAM as an 88 GB GPU.

A quoted custom GPU inference rig is the D-Central route when the model and service fit a named discrete-GPU configuration. The Canadian GPU inference rig guide covers the system-level trade-offs. Component choice follows the worksheet; this page does not prescribe a universal card or bill of materials.

What 128 GB of unified or shared memory means

In an integrated CPU-GPU design, “unified” or “shared” describes a physical memory architecture, not a 128 GB bucket reserved for model weights. The CPU, GPU, operating system, display, runtime, and applications draw from the installed system memory. The benefit is that the accelerator can address a much larger physical pool than a common discrete card without treating all data as a separate copy across a conventional CPU-to-GPU boundary. The cost and behaviour depend on the platform’s bandwidth, memory controller, allocation model, and software.

NVIDIA’s DGX Spark hardware documentation specifies 128 GB of unified LPDDR5X system memory on a 256-bit interface and publishes a 273 GB/s memory-bandwidth figure. That is a vendor hardware specification, not a measured tokens-per-second promise. The DGX Spark porting guide explains that its Arm system-on-chip lets the CPU and GPU share physical memory through a dynamic unified-memory architecture. The system still needs capacity for DGX OS, display reservation, runtime, cache, and applications.

AMD’s Ryzen AI Max+ 395 specification lists a 256-bit LPDDR5X interface and support for up to 128 GB of system memory. The system manufacturer determines the installed amount and implementation. AMD’s Strix Halo optimization guide describes GPU virtual memory, GTT mappings, and Linux settings that can affect large GPU-accessible allocations. Installed capacity and application-mappable capacity are not interchangeable claims.

The commercial routes are the NVIDIA DGX Spark at $9,449 CAD and the AMD Strix Halo 128 GB plan at $7,349 CAD. Choose between their software and host architectures using CUDA vs ROCm vs Vulkan, not from the shared 128 GB label alone. Platform setup lives in the DGX Spark setup guide and Strix Halo ROCm LLM guide.

Capacity and bandwidth answer different questions

Capacity asks whether the required allocations can remain in the intended memory envelope. Bandwidth describes how quickly a memory subsystem can transfer data under specified conditions. Both matter, but neither by itself predicts application performance. Compute capability, kernels, quantization, cache behaviour, prompt phase, output phase, batch, concurrency, clocks, thermals, and data movement also affect the service.

A smaller high-bandwidth pool can be attractive when the complete workload fits. A larger shared pool can admit a workload that cannot stay on the smaller device. That does not make the larger pool universally faster, and it does not make the smaller pool useless when offload is available. Evaluate fit first, then measure the accepted service on the supported runtime. D-Central does not publish an invented conversion between memory bandwidth and output tokens.

Use consistent units when comparing specifications. Manufacturers may publish effective transfer rates, interface width, or aggregate bandwidth. Even a correctly calculated theoretical bandwidth is not an application benchmark. Cite the exact board or system specification and avoid transferring it to a different vendor chassis or power profile.

KV cache, context length, and concurrency

Transformer inference retains key and value state so the model can attend to prior tokens without recomputing everything. A generic planning form for a conventional attention cache is proportional to layers × stored key/value heads × head dimension × cached tokens × bytes per cache element × active sequences, with a factor for both keys and values. This is not universal: grouped-query or multi-query attention, sliding-window attention, hybrid or state-space layers, cache quantization, prefix reuse, speculative decoding, and runtime block allocation change the result.

The important operational relationship is durable: longer accepted contexts and more simultaneous active sequences can materially increase memory demand. Named user count is not concurrency. Twenty employees who ask occasional questions may generate fewer active sequences than one batch application. Define the queueing policy, maximum context, maximum output, and maximum simultaneous work.

Current vLLM engine documentation exposes per-GPU KV-cache bytes and a per-instance GPU-memory-utilization control, plus cache data type and offload configuration. Those options demonstrate why a vLLM capacity plan must include the serving configuration, not just model weights. Similarly, the current llama.cpp multi-GPU guide states that context size and parallel slots affect cache pressure and documents selectable cache types.

Do not maximize context simply because a model advertises it. Set a business-justified ceiling, test retrieval quality within it, and reject or summarize oversized requests predictably. A controlled context policy preserves capacity for concurrent users and makes the acceptance test reproducible.

Why two GPUs are not one unified pool

Each discrete GPU owns a separate memory device. A runtime can split model layers or tensors across GPUs, but it must decide placement and communicate between devices. Some state can be distributed; some can be duplicated; some remains on a main device. Unequal cards can leave capacity stranded when the split is poorly balanced. The slowest link or an unsupported operation can invalidate a paper sum.

The maintained llama.cpp documentation currently distinguishes single-device, layer split, and experimental tensor split modes. It explains that layer mode assigns contiguous layers and their cache to devices, while tensor mode splits weights and cache and requires more cross-GPU reduction. It also lists architecture, cache-type, peer-access, and communication-library caveats. These are current project details, not guarantees for every future release or backend.

vLLM likewise exposes tensor-, pipeline-, data-, and context-parallel controls in its engine arguments. Selecting a parallel size does not prove that the model, GPU topology, network, container, or operating objective supports it. Multi-node deployment adds network interfaces, addressing, process coordination, fault domains, monitoring, and recovery.

Also distinguish sharding from replication. Sharding one model across two GPUs can expand the model envelope, subject to support and overhead. Running a complete copy on each GPU can serve independent work and add resilience, but each replica must fit one GPU. Adding the two capacities does not enlarge a single replica. For requirements beyond a workstation, scope an on-premises AI deployment.

Memory verification checklist

1. Identify the physical pools. Record installed system RAM, each GPU’s local memory, shared-memory architecture, display reservation, and any firmware carve-out.
2. Pin the artefacts. Save model filenames and hashes, quantization, tokenizer, auxiliary projectors or adapters, runtime version, backend, driver, container digest, and launch flags.
3. Observe idle allocation. Measure the operating system, display, services, monitoring, web UI, and database before loading a model.
4. Read the load log. Confirm the intended devices, weight placement, buffer sizes, CPU offload, per-GPU split, and available headroom. Investigate silent fallback.
5. Exercise maximum context. Use the accepted input and output limits, including retrieval documents and tool payloads. Watch allocation during prefill and generation.
6. Exercise concurrency. Run representative simultaneous sequences and queue behaviour. Include side models and application services.
7. Test failure boundaries. Confirm the service rejects oversized work cleanly, survives repeated loads, and recovers after an out-of-memory or worker failure.
8. Test restart and rollback. Reboot, restore the known configuration, reload models, and roll back a software update without relying on an undocumented shell history.
9. Keep headroom. Accept the service only with an explicit margin for routine variation and operations; do not size to the largest one-time successful allocation.

The runtime comparison helps choose the serving layer after the memory envelope is understood. Changing that runtime reopens the worksheet because cache allocation, model format, offload, multi-GPU placement, and concurrency controls can change.

Frequently asked questions

Can a 24 GB GPU run a model file larger than 24 GB?

Some runtimes can place part of a model in host memory or offload only selected layers to the GPU. That may allow the model to start, but it does not convert system RAM into local GPU memory or establish acceptable performance. Validate placement, context, concurrency, and the complete service before treating offload as a solution.

Is 128 GB of unified memory equivalent to 128 GB of VRAM?

No. It is a shared physical system pool used by CPU, GPU, operating system, display, runtime, and applications. It can provide a larger GPU-addressable envelope than a common discrete card, but architecture, bandwidth, allocation rules, software support, and usable headroom differ from dedicated VRAM.

Do two 24 GB GPUs provide 48 GB for one LLM?

Only when the runtime and model can distribute the required weights, cache, and operations across both devices. The pools remain separate, communication has a cost, and some allocations can be duplicated or concentrated. Verify per-device use and supported split mode; otherwise each model replica must fit one 24 GB device.

How does quantization affect memory?

Lower stored precision usually reduces weight-file size, but metadata and mixed-precision tensors add overhead. The selected runtime and backend must implement the quantization, and weight quantization does not automatically reduce cache or every workspace. Test both capacity and output quality with the exact artefact.

Why did a model load but fail on a long prompt?

The initial load proves only that the starting allocations fit. KV-cache and temporary-workspace demand can grow during prompt processing and generation. A longer context, larger output, parallel sequence, multimodal input, or auxiliary service can consume the remaining headroom and trigger failure.

How much memory should I leave free?

There is no universal percentage. Establish an observed idle baseline, measure the maximum accepted workload, include every co-resident service, and add an explicit operating margin appropriate to update and failure behaviour. The accepted margin belongs in the configuration record and should be retested after changes.

Should I choose discrete VRAM or 128 GB shared memory?

Choose after completing the worksheet. A discrete rig is a strong candidate when the whole intended accelerator workload fits with headroom and the backend supports the selected GPU. A 128 GB shared system is a candidate when capacity is the binding constraint and its CUDA/Arm64 or ROCm/Vulkan/x86 software path is acceptable. Compare actual machines only after those requirements are fixed.

Next step: route a validated memory envelope to the Canadian local AI computer shortlist. D-Central offers a quoted discrete-GPU rig, DGX Spark at $9,449 CAD, and an AMD Strix Halo 128 GB plan at $7,349 CAD. No product link overrides the requirement to test the exact model, runtime, context, and concurrency.