vLLM on DGX Spark: Faster Inference for RAG & Multi-User

Last updated: 24 August 2026 · Stack: DGX Spark GB10, DGX OS, CUDA 12.8, vLLM 0.6+ (Arm64), Python 3.11 · Pricing: DGX Spark $9,449 CAD / Strix Halo 128GB $7,349 CAD · Sources: vLLM docs, NVIDIA DGX Spark, StorageReview 6 July 2026.

Ollama is the quickest path to one model answering one user. vLLM is the path when one model must answer many users at once ??? a support team, a RAG service, or an internal OpenAI-compatible API with concurrent callers. Its two ideas ??? continuous batching and PagedAttention ??? keep the GPU busy between tokens instead of idling between requests. On DGX Spark’s 128GB unified pool, that translates into higher aggregate tokens per second under load, with the same weights on disk.

When vLLM beats Ollama (and when it does not)

Ollama wins when: single user, single model, fastest setup, smallest operational surface, chat-style latency matters most.

vLLM wins when: 2+ concurrent users, RAG pipelines with bursty batch ingest, background embedding + chat on the same box, or you need OpenAI-compatible batching with predictable tail latency.

Mechanically, Ollama serves requests largely serially per model instance; vLLM interleaves decode steps across requests in one continuous batch, and PagedAttention manages KV cache in non-contiguous pages so a long-context request does not fragment the pool and block others. On a 128GB unified box you have room for large KV caches ??? vLLM lets many tenants share that room efficiently.

Installing vLLM on DGX OS (Arm + CUDA)

vLLM publishes Arm64 wheels and a CUDA 12.8 build compatible with GB10 (sm_121 Blackwell). Use the DGX OS Python (3.11) and a venv.

ssh <user>@<dgx-spark-ip>
nvidia-smi && uname -m   # aarch64, CUDA 12.8
python3 --version        # 3.11 on DGX OS
python3 -m venv ~/vllm-env && source ~/vllm-env/bin/activate
pip install --upgrade pip

# Install vLLM (CUDA 12.8 / Blackwell)
pip install vllm --extra-index-url https://download.pytorch.org/whl/cu128
# Verify
python -c "import vllm; print(vllm.__version__)"
python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"

If pip cannot find a prebuilt wheel for your DGX OS point release, build from source per vLLM installation docs ??? expect 20???40 minutes on Grace Arm. Keep MAX_JOBS conservative on 128GB to avoid OOM during compilation.

Docker alternative (Arm64)

If you prefer containers, see Docker on DGX Spark (ARM). vLLM’s linux/arm64 CUDA images run with --gpus all when DGX OS’s NVIDIA Container Toolkit is present. Compose example on that page works for vLLM’s OpenAI server image with a volume for ~/.cache/huggingface.

Serving 70B/32B with continuous batching

Below is the minimal OpenAI-compatible server for a 70B model on DGX Spark. Unified memory means no --tensor-parallel-size is needed for single-box operation ??? the whole model fits without sharding.

# Serve Llama 3.3 70B (Q4 or FP8) ??? OpenAI-compatible on :8000
source ~/vllm-env/bin/activate

python -m vllm.entrypoints.openai.api_server 
  --model meta-llama/Llama-3.3-70B-Instruct 
  --quantization fp8 
  --dtype auto 
  --max-model-len 32768 
  --gpu-memory-utilization 0.90 
  --max-num-seqs 16 
  --enable-auto-tool-choice --tool-call-parser llama3_json

# Test from any LAN host:
curl http://<dgx-spark-ip>:8000/v1/chat/completions 
  -H "Content-Type: application/json" 
  -d '{"model":"meta-llama/Llama-3.3-70B-Instruct","messages":[{"role":"user","content":"Explain PagedAttention in 3 bullets"}]}'

# Smaller, faster iteration model (32B) ??? higher concurrency headroom
python -m vllm.entrypoints.openai.api_server 
  --model Qwen/Qwen2.5-32B-Instruct 
  --max-model-len 32768 
  --gpu-memory-utilization 0.88 
  --max-num-seqs 32

Key flags on 128GB unified

• --gpu-memory-utilization 0.88???0.92 ??? fraction of memory vLLM may use for KV. Leave ~10???15 GB for OS/CUDA and a second small process.
• --max-model-len 32768 ??? cap context to control KV growth. Raise to 65536 only if your workload needs it; each 32K of context adds ~8???14 GB of KV at 16-bit across a large model.
• --max-num-seqs 16???32 ??? concurrent decodes. Higher is not always faster ??? measure tail latency.
• --quantization fp8 ??? uses GB10’s Blackwell FP8/NVFP4 path where the model supports it. For GGUF workflows, serve via llama.cpp or convert to HF FP8; vLLM does not serve GGUF directly.

Benchmark vs Ollama: how we will measure

We do not invent tokens/s. When D-Central’s lab run lands at DGX Spark benchmarks, each row will be same-model, same-quant, same-prompt, same max_model_len, median of 3 runs at temperature 0, with prompt tokens, output tokens, and batch size fixed. Until then:

• Directional cited: StorageReview (6 July 2026) found Spark held a 2?????4?? vLLM throughput lead over Strix Halo in most sweeps, larger on prefill-heavy workloads like gpt-oss-120b ingest. That is a third-party vLLM result, not Ollama vs vLLM on the same Spark ??? but it confirms GB10’s CUDA serving lead on multi-tenant loads.
• Expected shape: solo-user latency is often similar between Ollama and vLLM on the same 70B Q4; aggregate throughput under 4???8 concurrent requests is where vLLM’s continuous batching shows a step change. Measure your concurrency, not just one-shot tok/s.

For the controlled same-hardware runtime comparison, bookmark vLLM vs Ollama on DGX Spark: Benchmarked ??? every number there will carry prompt, batch, and methodology.

128GB tuning for multi-user

• Keep one model warm. Pin the primary model with a high --gpu-memory-utilization and avoid loading a second 70B simultaneously. A 7???8B sidecar is fine.
• Cap max_model_len per workload. A team doing 4K chat does not need 64K KV reservations. Over-reserving wastes the unified pool.
• Monitor with vLLM metrics: --enable-metrics exposes Prometheus at /metrics ??? track num_requests_running, KV cache utilization, and queue time.
• Power budget: Spark is a 240 W system. Under sustained multi-user decode, wall draw sits well under that. For tokens-per-kWh math see watts per million tokens.

vLLM on Strix Halo?

Strix Halo (Ryzen AI Max+ 395) has the same 128GB memory ceiling but a different compute path: RDNA 3.5 + XDNA via ROCm/Vulkan. vLLM’s coverage is CUDA-first ??? ROCm support exists but is not as mature in 2026, and llama.cpp/Vulkan is often the smoother path on Halo for single-user work. If your stack is vLLM + NVIDIA containers, Spark is the less painful box. Comparison: Spark vs Strix Halo and Strix Halo Canada.

Frequently asked questions

Does vLLM run on DGX Spark’s Arm CPU?

Yes, via Arm64 wheels/CUDA 12.8. The install is pip install vllm --extra-index-url https://download.pytorch.org/whl/cu128 on the Spark itself. Verify with torch.cuda.is_available().

When should I choose vLLM over Ollama?

Choose vLLM when 2+ users or RAG pipelines hit the same model concurrently ??? continuous batching keeps GB10 saturated. Choose Ollama for single-user chat where simplicity matters.

Can vLLM serve GGUF Q4 files directly?

No. vLLM serves Hugging Face-format models (FP16/BF16/FP8/FP4). For GGUF Q4, use Ollama or llama.cpp.

How many concurrent users can DGX Spark handle?

Depends on model, context, and latency SLO. 70B Q4 with 32K cap comfortably handles low-double-digit concurrent decodes on 128GB before KV pressure; measure with your prompts. Lab methodology: benchmarks and vLLM vs Ollama.

Do I need two Sparks for vLLM?

No. Single Spark fits 70B/120B-class models without sharding. Two Sparks NVLinked to 256GB (NVLink x2 guide) is for larger models or higher batch headroom, not a requirement.

Credits: vLLM docs, PyTorch CUDA wheels, NVIDIA DGX Spark docs, StorageReview 6 July 2026, ServeTheHome. Prices as of 24 Aug 2026. No D-Central tok/s invented on this page; see benchmarks for methodology.