Running Ollama and vLLM on NVIDIA DGX Spark

This is not a tokens/s bake-off. We will not invent throughput. For dated third-party numbers see ServeTheHome, StorageReview, and NVIDIA’s own Spark materials. Runtime comparison in general: Ollama vs vLLM vs llama.cpp.

Ollama on Spark (the default we install)

Ollama is the fastest path to a local chat loop: one service, GGUF-class models, an OpenAI-compatible API. On Spark it runs natively on CUDA/Arm once the binary matches the OS. First-boot steps that are not NVIDIA-secret:

1. Confirm DGX OS is updated from NVIDIA’s current guide.
2. Install the current Ollama Linux/Arm build from the Ollama project (credit: Ollama).
3. Pull one model you are licensed to run. Check the card. Open-weight ≠ open-source.
4. Bind the API to localhost or a private LAN. Put a UI in front (Open WebUI is the usual one — credit that project).
5. Do not publish port 11434 to the internet.

We do this on the $9,449 configured unit so you are not staring at a DGX login the first evening. HowTo without the appliance: install Ollama in 10 minutes, first boot Ollama + Open WebUI.

When to put vLLM on Spark

vLLM is a CUDA serving engine (credit: the vLLM project / Sky Computing lineage). It earns its keep when you want continuous batching, an OpenAI-compatible server, and you already know the model format it wants. It is not simpler than Ollama. It is not a Windows app. On Spark it is the NVIDIA-shaped production path short of NIM/TensorRT-LLM.

If you need Windows, vLLM on Spark is the wrong conversation. That is Strix Halo 128 GB — $7,349 CAD (ROCm/Vulkan, not CUDA) or a custom NVIDIA GPU tower (quote) where vLLM is at home on x86 CUDA.

What we will not claim

• No tokens/s from our lab.
• No “Ollama is 19× slower than vLLM” as a universal law — published concurrent-user gaps depend on model, GPU, and batching. Read the cited benchmark’s hardware line.
• No hosted SLA. This page is about a box you own.
• Spark is sourced NVIDIA hardware, not a D-Central PCB.

DGX Spark — $9,449 CAD · Spark setup in Canada · Which box