Run Qwen Locally in Canada — Qwen3 Setup Guide (Ollama + llama.cpp)

Qwen3 is the third-generation large-language-model series from Alibaba’s Qwen team. It covers a wide parameter range (0.6 B to 235 B), is released under the Apache 2.0 open-source licence, and is available as pre-quantised GGUF files that load in minutes on consumer hardware. For Canadian businesses and developers — particularly those operating under Quebec Law 25 or handling data that must not transit US cloud infrastructure under the CLOUD Act — running Qwen locally is one of the most practical paths to capable, sovereign AI. This guide gives you the concrete steps.

Credit where it belongs: Qwen3 is built by the Qwen team at Alibaba Group, standing on years of community work in open-weight modelling. Ollama, llama.cpp, LM Studio, and vLLM are independent open-source projects maintained by their respective communities — we are simply documenting how to run their tools.

The Qwen3 model family at a glance (as of June 2026 — verify at huggingface.co/Qwen)

Qwen3 ships two architecture types: dense models (every parameter active per token) and MoE (Mixture-of-Experts, where only a fraction of parameters activate per token, cutting inference cost significantly). All carry Apache 2.0 licences, which permit commercial use without royalties — important for Canadian SMBs wanting to build internal tools.

Dense models — pick by VRAM

MoE models — large capability, modest active-compute

*VRAM estimates are community-reported approximations at Q4_K_M quantisation and vary by KV-cache context length and backend. Always verify against your actual hardware before purchasing. Sources: Will It Run AI, Spheron Blog, Ollama library. Data correct as of June 2026 — check Hugging Face for the latest releases.

Not sure what VRAM you need? Use the local LLM VRAM calculator to model your specific setup.

Method 1: Ollama (recommended for most Canadians)

Ollama wraps llama.cpp into a single binary that manages model downloads, runs an OpenAI-compatible local API, and works on Linux, macOS, and Windows. It is the fastest path to a working Qwen3 instance.

Step 1 — Install Ollama

Linux / macOS:

curl -fsSL https://ollama.com/install.sh | sh

Windows: Download the installer from ollama.com/download and run it. Ollama installs as a background service.

Step 2 — Pull and run a model

# Comfortable on 8–10 GB VRAM — good starting point
ollama run qwen3:8b

# Fits 12 GB VRAM — noticeably stronger reasoning
ollama run qwen3:14b

# For the 30B MoE variant (16+ GB VRAM needed)
ollama run qwen3:30b

The first run downloads the quantised GGUF automatically. Subsequent runs are instant — the model is cached locally.

Step 3 — Set the context window

Ollama’s default context window is 2,048 tokens, which truncates most real documents. Qwen3 supports up to 128K tokens on the dense models. Create a Modelfile to raise the limit:

# Modelfile
FROM qwen3:8b
PARAMETER num_ctx 32768

ollama create qwen3-32k -f Modelfile
ollama run qwen3-32k

Use num_ctx 65536 or 131072 if your GPU has enough VRAM — larger context consumes proportionally more memory.

Step 4 — Use the local API

Ollama exposes an OpenAI-compatible REST endpoint on http://localhost:11434. Any tool that accepts an OpenAI base URL (Open WebUI, Cursor, Continue.dev) can point here instead of a cloud API, with zero data leaving your machine.

curl http://localhost:11434/v1/chat/completions 
  -H "Content-Type: application/json" 
  -d '{
    "model": "qwen3:8b",
    "messages": [{"role": "user", "content": "Explain PIPEDA in plain language."}]
  }'

Method 2: llama.cpp (maximum control)

llama.cpp is the underlying inference engine Ollama uses. Running it directly gives you fine-grained quantisation choices, batching controls, and access to the newest model support before Ollama integrates it. Recommended for developers and IT staff comfortable with the command line.

Option A — Download a pre-made GGUF

The Unsloth team (community maintainers) publishes calibrated GGUFs via Hugging Face / unsloth. Look for tags ending in -GGUF such as unsloth/Qwen3-8B-GGUF. Download the Q4_K_M variant — a good balance between quality and VRAM usage.

# Using huggingface-cli (pip install huggingface_hub)
huggingface-cli download unsloth/Qwen3-8B-GGUF 
  Qwen3-8B-Q4_K_M.gguf 
  --local-dir ./models

Option B — Convert and quantise yourself

# 1. Clone llama.cpp and build
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j

# 2. Convert HF checkpoint to GGUF (BF16 preserves quality)
python convert-hf-to-gguf.py Qwen/Qwen3-8B 
  --outtype bf16 --outfile models/Qwen3-8B-BF16.gguf

# 3. Quantise to Q4_K_M
./build/bin/llama-quantize models/Qwen3-8B-BF16.gguf 
  models/Qwen3-8B-Q4_K_M.gguf Q4_K_M

Run inference

./build/bin/llama-cli 
  -m models/Qwen3-8B-Q4_K_M.gguf 
  -n 512 
  --ctx-size 32768 
  -ngl 99 
  -p "Your prompt here"

The -ngl 99 flag offloads all layers to the GPU. Reduce this number to split between GPU and CPU RAM if your VRAM is limited.

Qwen3-Coder: writing code that stays on your machine

The Qwen team also maintains a coding-specialist branch. As of June 2026, the main local-friendly variant is Qwen3-Coder-30B-A3B (30B total parameters, 3B active per token via MoE — fits a 16–20 GB GPU at Q4) with a 256K-token context window suited to large codebases. A larger Qwen3-Coder-480B-A35B exists but requires multi-GPU or server-class hardware.

For most developers, Qwen3-Coder-30B-A3B is the sweet spot: it handles multi-file refactoring, generates tests, and explains legacy code, all locally. Pull it via:

ollama run qwen3-coder:30b

Verify current tag names at ollama.com/library/qwen3-coder and huggingface.co/Qwen before running — model tags change as new versions release.

For air-gapped environments where even Hugging Face downloads are restricted, see the air-gapped AI coding guide for Canada.

Why running Qwen locally matters for Canadian businesses

Quebec Law 25 and data residency

Quebec’s Loi sur la protection des renseignements personnels dans le secteur privé (Law 25) is fully in force as of 2023, with penalties up to $25 million CAD or 4% of worldwide turnover. The law requires that personal information used in automated decision-making be disclosed to the individual, with a right to request human review. Sending employee data, client records, or health information to a US-based cloud API creates cross-border transfer obligations and audit exposure.

Running Qwen3 on-premise means the model weights and every inference call never leave your infrastructure. There is no third-party sub-processor to disclose, no data-processing agreement to negotiate with a foreign cloud vendor, and no exposure to US CLOUD Act extraterritorial requests. For regulated sectors — legal, healthcare, financial services, government — this is the practical path to compliant AI adoption.

For a deeper breakdown, see the Quebec Law 25 and on-premise LLM compliance guide.

CLOUD Act and federal PIPEDA

Even outside Quebec, Canada’s federal PIPEDA requires meaningful consent for cross-border data flows. The US CLOUD Act (2018) allows US authorities to compel cloud providers to produce data stored anywhere in the world if the provider is subject to US jurisdiction. Running Qwen locally eliminates this vector entirely — no US-incorporated vendor is in your processing chain.

The broader context of Canadian digital sovereignty is covered at sovereign AI Canada and digital sovereignty Canada.

Cost over time

At high query volumes, local inference eventually undercuts API costs. A single RTX 4090 (roughly $2,500–$3,000 CAD, prices vary — check current retail) running Qwen3-14B handles thousands of requests per day with no per-token billing. See the cloud vs local AI TCO comparison for a full cost model. The break-even point depends heavily on query volume and the API you are replacing — run the numbers for your workload before committing to hardware.

Hardware sizing guide for Canadian buyers

Canadian buyers face a narrower GPU market than the US: Newegg Canada, Canada Computers, and Memory Express are the main retail channels. Import duties and exchange rates mean GPU prices are typically 15–25% higher than US list prices (verify current pricing — this varies). Here is how to size your purchase:

For a full breakdown of GPU options for local LLM inference, see the GPU for local LLM comparison.

D-Central can assist with hardware sourcing and Hashcenter-grade deployment for teams needing multi-GPU inference infrastructure — contact us for a quote.

Qwen3 vs other open-weight models: where it fits

The open-weight ecosystem has matured rapidly. Qwen3 is not the only option — Llama 3, Mistral, Phi, and DeepSeek all have strong followings — and the right model depends on your task, hardware, and language requirements. Key considerations:

• Multilingual: Qwen3 has notably strong Chinese, Japanese, Korean, and Arabic performance alongside English and French. For Canadian bilingual (EN/FR) use, all major open-weight families perform well in French; Qwen3’s French is solid but verify on your specific task.
• Reasoning: Qwen3’s larger variants include a “thinking mode” (chain-of-thought token budget) competitive with closed models on reasoning benchmarks. Benchmarks change rapidly — treat published scores as directional.
• Licence: Apache 2.0 is one of the most permissive licences in the space. Llama 3 uses a custom Meta licence with commercial restrictions above 700M monthly users. DeepSeek uses MIT. Always verify the current licence for your specific version before building a product.
• CLOUD Act concern: Meta is a US company; DeepSeek is Chinese. Both Apache 2.0 and MIT allow you to run weights locally regardless of the developer’s jurisdiction — the key is that weights are self-hosted, not API-accessed. Running any open-weight model locally eliminates the cloud-access vector for either government.

See the open-weight AI Canada comparison for a side-by-side across Qwen3, Llama, DeepSeek, Mistral, and Phi. For a broader view including closed alternatives, see ChatGPT alternatives for Canada.

Inference tools beyond Ollama and llama.cpp

Ollama and llama.cpp handle the vast majority of local use cases. For production team deployments, two additional tools are worth knowing:

• LM Studio: A GUI application (Windows/macOS/Linux) that lets non-technical users browse, download, and run GGUF models with a chat interface. No command line required. Good for evaluating models before committing to infrastructure.
• vLLM: A high-throughput inference server designed for serving models to multiple concurrent users. Requires a Linux server with a supported GPU and full-precision (or int8) weights — more resource-intensive than llama.cpp but significantly higher throughput. Best for internal SaaS tooling serving a team.
• Open WebUI: A self-hosted chat interface that connects to an Ollama or OpenAI-compatible backend. Gives your team a ChatGPT-like interface without sending any data to OpenAI.

For a detailed comparison of inference engines, see Ollama vs vLLM vs llama.cpp.

Where the cloud still wins (honest assessment)

Running Qwen locally is not the right answer for every situation. Be honest about the trade-offs:

• The very largest models (235B+) require server-grade hardware — a multi-GPU machine or a purpose-built Hashcenter node. Cloud APIs remain the practical path for most teams at that capability level until hardware prices fall further.
• Initial setup requires technical comfort. Ollama has lowered the bar dramatically, but debugging GPU driver issues, CUDA versions, and quantisation artefacts still requires IT competence. Cloud APIs have zero infrastructure overhead.
• Generation speed on consumer GPUs lags frontier cloud. Qwen3-32B at Q4 on a single RTX 4090 produces roughly 20–40 tokens/second — usable, but slower than a paid API under low load.
• For non-sensitive workloads with no regulatory constraints, cloud APIs may be more economical below a certain query volume. Run the TCO numbers before committing to hardware investment.
• Model updates require manual action. Cloud APIs automatically receive model improvements; local deployments require intentional re-download and re-evaluation.

Complementary local LLM guides on D-Central

• Local LLM Canada — complete overview
• VRAM calculator — size your GPU for any model
• Run DeepSeek locally in Canada
• Open-weight AI Canada comparison
• Ollama vs vLLM vs llama.cpp
• Air-gapped AI coding in Canada
• Quebec Law 25 and on-premise LLM
• Sovereign AI Canada

Frequently asked questions

Is Qwen3 legal to use commercially in Canada?

Yes. All Qwen3 models carry an Apache 2.0 licence, which permits commercial use, modification, and redistribution without royalties. As with any AI system, your obligations around data protection (PIPEDA, Law 25) and consumer-protection disclosure are separate from the model licence and apply regardless of which model you use. Verify the licence for each specific model version at huggingface.co/Qwen.

Do I need an internet connection to run Qwen3 after downloading?

No. Once you have downloaded the model weights (via Ollama pull or huggingface-cli), inference runs entirely on your local hardware with no network access required. This is the basis of the data-sovereignty claim: there is no outbound data path.

Can Qwen3 handle French? Is it suitable for Canadian bilingual work?

Qwen3 performs well in French across general tasks (summarisation, Q&A, translation). It is not specifically fine-tuned for Canadian French or Quebec-specific legal terminology. For regulated bilingual tasks, always evaluate the model on representative samples of your actual use case before deploying to end users.

What is the difference between Qwen3 and Qwen3-Coder?

Qwen3 is the general-purpose model family; Qwen3-Coder is a code-specialist variant further trained on programming tasks, agent workflows, and large-context code understanding. For pure coding tasks (code review, refactoring, test generation), Qwen3-Coder will generally outperform same-size general Qwen3. For mixed tasks (documentation, code + business logic), the general Qwen3 is a better fit.

How does Qwen3 compare to Llama 3 for local use?

Both are strong open-weight families that run well locally under Ollama and llama.cpp. Qwen3 tends to outperform Llama 3 on multilingual and reasoning tasks at equivalent parameter counts in published benchmarks (as of mid-2026) — but benchmarks evolve rapidly and vary by task type. The more important difference for Canadian users is licensing: Qwen3 is Apache 2.0; Llama 3 uses a custom Meta licence. Both run locally and give you data-residency control. See the full open-weight AI Canada comparison.

How much RAM do I need beyond VRAM?

System RAM requirements are separate from GPU VRAM. As a rule of thumb: 16 GB system RAM for models up to 8B; 32 GB for 14B–32B; 64 GB for 30B MoE and above. Models partially offloaded to CPU RAM (using -ngl flags below maximum in llama.cpp) will consume system RAM proportional to the offloaded layers.

Can I run Qwen3 on a CPU alone (no GPU)?

Yes, llama.cpp supports CPU-only inference. The 0.6B and 1.7B models are practical on modern CPUs. Larger models (8B+) become very slow on CPU alone — expect 1–5 tokens per second on a modern x86 CPU versus 20–60+ on a mid-range GPU. For anything above 4B, a GPU is strongly recommended for usable response latency.

What quantisation level should I choose?

Q4_K_M is the community standard for the best quality-to-VRAM trade-off. Q5_K_M is slightly higher quality if you have the VRAM headroom. Q3_K_M is useful when VRAM is tight (e.g., running Qwen3-32B on 24 GB). Q8_0 is near-lossless but doubles VRAM requirements. Most users should start with Q4_K_M and only adjust if quality or VRAM is a specific concern on their workload.

Is D-Central’s DCENT_OS compatible with Qwen3?

DCENT_OS is D-Central Technologies’ GPL-3.0 open-source Bitcoin mining firmware for selected Bitmain Antminer ASIC miners. Guarded artifacts exist only for exact S9 XIL and S19j Pro XIL lanes; install evidence is incomplete and neither is production-ready. Check the exact evidence ledger.