Ollama on DGX Spark: Install, Run 70B Models & Speed Tuning

Last updated: 24 August 2026 · Tested stack: NVIDIA DGX Spark (GB10, 128GB LPDDR5X, DGX OS) ??? Ollama 0.6+, llama.cpp backend, CUDA 12.8 · Pricing: DGX Spark $9,449 CAD / Strix Halo 128GB $7,349 CAD (D-Central, configured in Laval) · Sources: NVIDIA DGX Spark, NVIDIA DGX docs, Ollama, llama.cpp, StorageReview 6 July 2026.

DGX Spark is a CUDA box with 128GB of unified LPDDR5X and a 20-core Grace Arm CPU. Ollama is the fastest way to turn that pool into a usable local LLM server: one daemon, one CLI, one OpenAI-compatible API on localhost:11434. If you can ssh into DGX OS and curl, you can be serving Llama 3.3 70B in under five minutes. This guide shows the Arm-specific install, the exact ollama run lines for 70B-class models, and the 128GB tuning that keeps a 32K context comfortable.

New to local AI sizing? Start with DGX Spark Canada (hub), which models fit on 128GB (24-model Q4/Q8 table), and DGX Spark benchmarks (methodology + StorageReview directional). Comparing desks? Spark vs Strix Halo and Strix Halo Canada.

Why Ollama fits GB10 so well

Ollama wraps llama.cpp with a model registry, a background server, and an OpenAI-compatible endpoint. On DGX Spark that matters because:

• Unified memory means no VRAM cliff. A 70B model at Q4 (~42 GB) plus a 32K KV cache (~10???14 GB) still leaves roughly 70 GB free on the 128GB pool. On a 24GB discrete GPU the same load would page over PCIe at ~32 GB/s. GB10 addresses the whole 128GB uniformly via C2C NVLink ??? no manual offload layers.
• Arm64 is first-class in Ollama/llama.cpp in 2026. Both ship Arm64 binaries. You do not need Rosetta or an x86 shim.
• GGUF is the lingua franca. Every open-weight family (Llama 3.3, Qwen2.5/3, Gemma 3, Phi-4, Mistral) publishes GGUF quantizations that Ollama consumes directly.
• API is immediate. Once ollama serve is running, /api/chat and /v1/chat/completions are live for Open WebUI, n8n, or your app.

Alternatives exist for a reason. vLLM on DGX Spark wins on multi-user throughput via continuous batching; llama.cpp on DGX Spark wins on ultra-long 128K context. Ollama wins on simplicity ??? one user, one model, fastest time-to-first-token for solo and small-team use. See the head-to-head at vLLM vs Ollama on DGX Spark.

GGUF sizes are measured Q4_K_M footprints; add ~3???6% for Ollama wrapper. KV cache at 32K adds ~8???14 GB depending on GQA and heads. All fit on 128GB unified; see the full 24-model table at model fit guide.

Step-by-step: install Ollama on DGX OS (Arm64)

DGX OS is Ubuntu-based with CUDA and drivers pre-installed. You only add Ollama. Verified 24 Aug 2026.

1. SSH in and confirm CUDA

ssh <user>@<dgx-spark-ip>
nvidia-smi                 # should show GB10, driver + CUDA 12.8
uname -m                   # expect: aarch64
df -h                      # confirm 4TB NVMe mount

2. Install Ollama (Arm64)

Ollama’s install script detects aarch64 automatically. No x86 flag needed.

curl -fsSL https://ollama.com/install.sh | sh
ollama --version
systemctl status ollama    # Ollama installs as a systemd service on DGX OS

If systemctl is not available in your DGX OS image, run manually: ollama serve & and add it to your login profile.

3. Pull and run your first model

ollama pull llama3.2:3b
ollama run llama3.2:3b "Explain C2C NVLink in one paragraph."

First pull is network-bound (model store at ~/.ollama on the 4TB NVMe). Subsequent loads are from local SSD.

4. Expose the API on your LAN

By default Ollama binds 127.0.0.1:11434. To let Open WebUI or teammates reach it:

# /etc/systemd/system/ollama.service.d/override.conf
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_ORIGINS=*"

sudo systemctl daemon-reload
sudo systemctl restart ollama
curl http://<dgx-spark-ip>:11434/api/tags   # should list your models

Place DGX Spark behind your LAN firewall or Cloudflare Tunnel ??? never expose :11434 raw to the public internet.

5. Update Ollama

curl -fsSL https://ollama.com/install.sh | sh
ollama --version

Running 70B models: llama3.3:70b-q4 and siblings

With Ollama installed, the 70B workflow is one pull + one run. The 128GB pool means you do not need to shard or offload.

Llama 3.3 70B (Q4_K_M)

ollama pull llama3.3:70b
ollama run llama3.3:70b
# Or explicitly quantized variant if your registry publishes it:
# ollama pull llama3.3:70b-instruct-q4_K_M

At Q4_K_M the GGUF is ~42 GB. Ollama will load it fully into the unified pool and keep it resident until evicted. Prompt at 32K still leaves ~70 GB headroom for KV and system ??? this is the everyday workhorse on DGX Spark.

Qwen2.5 32B, Qwen3 MoE, Gemma 3 27B

ollama pull qwen2.5:32b
ollama pull qwen3:30b-a3b
ollama pull gemma3:27b
ollama run qwen2.5:32b "Write a Python LRU cache with tests."

Switching models without restart

ollama list
ollama ps
ollama run gemma3:27b   # Ollama hot-swaps; previous model unloads if memory pressure demands

Using a custom GGUF (any quant)

# Create a Modelfile in the GGUF directory
# Modelfile:
# FROM ./Meta-Llama-3.3-70B-Instruct-Q4_K_M.gguf
# PARAMETER num_ctx 32768
# PARAMETER temperature 0.7

ollama create llama33-70b-q4 -f ./Modelfile
ollama run llama33-70b-q4

Custom GGUFs let you pin exact quantizations from the fit guide without waiting for a registry tag.

128GB tuning: parallel, context, keep-alive

Ollama exposes runtime knobs as environment variables and per-model PARAMETER lines. With 128GB unified you tune for comfort, not survival.

# Example: serve 70B with 32K context, keep warm for an hour
# In Modelfile:
# PARAMETER num_ctx 32768
# PARAMETER keep_alive 1h

# Or at runtime:
OLLAMA_NUM_PARALLEL=4 OLLAMA_KEEP_ALIVE=1h ollama serve

# Per-request override:
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.3:70b",
  "messages": [{"role":"user","content":"Summarize this 20k-word contract"}],
  "options": {"num_ctx": 32768, "temperature": 0.2}
}'

Context tax, plainly: every extra 32K of context adds roughly 8???14 GB of KV cache at 16-bit, depending on layers and GQA. On 128GB that is affordable up to 64???96K alongside a 70B Q4 model. Past 128K, prefer llama.cpp with explicit context offload, which gives finer control over KV quantization and placement.

Speed: what to expect without inventing numbers

We do not publish invented tokens/s. Two attributions you can verify:

• StorageReview (6 July 2026) found DGX Spark held a roughly 2?????4?? throughput lead over Strix Halo in most vLLM sweeps, stretching further on prefill-heavy workloads. Ollama/llama.cpp were directionally the same ??? same memory ceiling, faster CUDA kernels on GB10.
• Our lab methodology is pending. D-Central will publish same-model, same-quant, same-prompt tok/s across Spark vs Halo vs 5090/4090 at DGX Spark benchmarks with full methodology (prompt set, context, batch, temp=0, median of 3 runs). Until then, treat any tok/s not tied to a named review as unmeasured.

What you can count on structurally: 70B Q4 on unified 128GB is not swapping over PCIe, so latency is smooth rather than cliffed; and GB10’s Blackwell Tensor Cores with NVFP4 give headroom for quantized serving that a 24GB discrete card simply cannot load the weights for. See also Spark vs Mac Studio M4 Ultra and Spark vs RTX 5090/4090 for the cross-platform framing.

Troubleshooting on Arm

Ollama binary not found / exec format error

You fetched an x86_64 binary onto aarch64. Re-run curl -fsSL https://ollama.com/install.sh | sh directly on the Spark ??? do not copy a binary from a Mac/PC.

Model pull is slow or stalls

70B Q4 is ~40 GB over the wire. Use wired 10 GbE or Wi-Fi 7 near the AP; ollama pull resumes on retry but does not parallelize shards.

Context truncated at 4K/8K

Set PARAMETER num_ctx 32768 in the Modelfile or pass "options": {"num_ctx": 32768} per request. Default varies by model tag.

OOM on 128GB (rare)

You kept two 70B models resident and added 64K context. Set OLLAMA_MAX_LOADED_MODELS=1 and OLLAMA_KEEP_ALIVE=5m, or drop one model to Q3. Check with ollama ps and free -h.

Open WebUI cannot reach Ollama

Bind OLLAMA_HOST=0.0.0.0, open 11434 on DGX OS’s firewall, and point Open WebUI’s Ollama URL to http://<spark-ip>:11434. Full walkthrough: private ChatGPT on DGX Spark and Docker on DGX Spark (ARM).

Frequently asked questions

Does Ollama run on DGX Spark’s Arm CPU?

Yes. Ollama and llama.cpp both ship native aarch64 binaries in 2026. The install script auto-detects Arm ??? no x86 emulation needed. See ollama.com and llama.cpp releases.

Can I run gpt-oss:120b on Ollama + DGX Spark?

Yes, at Q4 (~70 GB) it fits on 128GB unified with ~40 GB headroom at 32K context. Use a Q4_K_M GGUF and set num_ctx 32768. Larger contexts erode headroom quickly ??? see the model fit guide and benchmarks.

How do I keep 70B warm for instant replies?

Set OLLAMA_KEEP_ALIVE=1h (or longer). The model stays resident in unified memory until the timer expires. Pair with OLLAMA_MAX_LOADED_MODELS=1 if you only serve one model.

Ollama vs vLLM vs llama.cpp ??? which should I pick?

Pick Ollama for solo/small-team simplicity, vLLM for multi-user continuous batching, and llama.cpp for maximum 128K+ context control. Comparison: vLLM vs Ollama on DGX Spark.

Where do I buy DGX Spark in Canada, and what about Strix Halo?

D-Central lists DGX Spark at $9,449 CAD configured in Laval and Strix Halo 128GB at $7,349 CAD (saves $2,100) for x86/Windows buyers. Guides: where to buy DGX Spark and where to buy Strix Halo.

Credits: NVIDIA DGX Spark product and hardware docs, Ollama docs, llama.cpp (ggml-org), StorageReview 6 July 2026, ServeTheHome GB10 coverage. DGX Spark $9,449 CAD and Strix Halo $7,349 CAD are D-Central configured prices as of 24 Aug 2026. No tok/s on this page is D-Central-measured; see benchmarks for methodology.