Run Llama Locally in Canada — VRAM, Ollama & llama.cpp Setup Guide

Llama is the open-weight model family from Meta — one of the most important contributions to the open-model ecosystem alongside DeepSeek, Mistral, Qwen, and Phi. Because the weights are publicly downloadable, you can run Llama entirely on your own hardware, with zero data transmitted to any server.

For Canadian businesses and developers, that matters. Quebec’s Law 25 requires that personal data about Quebec residents be stored in Canada (or in jurisdictions with equivalent protection) — not routed through US API endpoints. The US CLOUD Act gives American authorities the power to compel US-headquartered cloud providers to hand over data stored anywhere in the world. Running Llama locally eliminates both exposure vectors entirely.

This guide covers what Llama actually is, the licensing terms you should know before building on it, the VRAM requirements for every current version, and step-by-step setup with Ollama and llama.cpp — two outstanding open-source projects built by the community that make local inference practical for everyone.

We stand on the shoulders of giants: Meta’s research team for releasing these weights, Georgi Gerganov and the llama.cpp contributors for building the inference engine that powers most local setups, and the Ollama team for making it accessible enough for non-researchers. Credit where it is due.

What Llama is — and what the license actually says

Meta released the Llama 4 model family on 5 April 2025. As of mid-2026, the active generations available for local deployment are:

• Llama 4 Scout — 109B total parameters, 17B active per token (Mixture-of-Experts with 16 experts), 10 M-token context window. The most capable model currently runnable on a single high-end workstation.
• Llama 4 Maverick — 400B total parameters, 17B active per token (128 experts). Generalist flagship. Requires enterprise multi-GPU hardware.
• Llama 3.3 70B — Dense 70B parameter model, released November 2024. The highest-quality model that fits on a single 48 GB GPU or dual 24 GB setup.
• Llama 3.3 8B — Dense 8B model. The everyday workhorse for most Canadian developers and small teams. Fits on any 8 GB GPU.
• Llama 3.2 3B / 1B — Ultra-lightweight models for edge, embedded, or laptop deployment.

Verify current releases directly at llama.com and Hugging Face / meta-llama — the model family is actively developed and new versions ship regularly.

Llama 4 Community License — key conditions

Llama 4 is released under the Llama 4 Community License Agreement. It permits commercial use, modification, and redistribution. Read the full license before building a product — the main conditions that affect most Canadian businesses are:

• Commercial use is allowed for organizations with fewer than 700 million monthly active users. Above that threshold, you must request a separate licence from Meta.
• Attribution required: you must prominently display a “Built with Llama” badge on any website or in-app page related to the software, and include the attribution notice: “Llama 4 is licensed under the Llama 4 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.”
• Derivative models must include “Llama” at the beginning of the model name.
• The licence is not an Open Source Initiative (OSI)-approved licence; Meta uses the term “open” to mean publicly available weights, not OSI-open. This distinction matters if your organization has OSI compliance requirements.

Llama 3.x models carry similar (but distinct) licence terms. Always confirm the licence for the specific model version you deploy. This summary is informational — consult legal counsel for compliance questions.

Llama VRAM requirements: which hardware runs which model

All figures below are approximate weight-only footprints at the quantization level shown. Add 10–30% headroom for KV cache and runtime overhead at typical context lengths. Actual memory usage scales with context window size, batch size, and the inference framework’s implementation. Treat these as planning minimums verified against community benchmarks; verify independently before purchasing hardware.

Sources: Llama 3.3 8B Q4 — community consensus 5–6 GB VRAM (LocalLLM.in 2026). Llama 3.3 70B Q4_K_M ~42 GB — (RunAIHome 2026). Llama 4 Scout INT4 ~55 GB full-GPU — (APXML system-requirements guide). Llama 4 Maverick Q4 ~200–230 GB — multiple community benchmarks. All figures are approximate and hardware-implementation-dependent; actual usage varies with context length, batch size, and backend. Verify at Hugging Face / meta-llama before purchasing hardware. Use our VRAM calculator for interactive estimates.

For a full comparison of Llama vs. other open-weight models (Mistral, Qwen, Gemma, Phi) alongside hardware tiers, see our Local LLM Canada guide.

Running Llama with Ollama (recommended starting point)

Ollama is a free, open-source tool built by the Ollama team that wraps llama.cpp into a clean CLI and REST API, handles GGUF model downloads automatically, detects your GPU, and serves a local OpenAI-compatible endpoint at http://localhost:11434. It is the fastest path from zero to running Llama on a Canadian workstation. All credit to the Ollama project contributors.

Step 1 — Install Ollama

Download and install Ollama for your OS from ollama.com/download. macOS and Windows installers are available; Linux installs via:

curl -fsSL https://ollama.com/install.sh | sh

Ollama automatically detects NVIDIA (CUDA), AMD (ROCm), and Apple Silicon (Metal) GPUs. CPU-only inference works without any additional configuration.

Step 2 — Pull and run a Llama model

Open a terminal and pull the model appropriate for your hardware:

# Lightweight — runs on nearly any hardware (CPU or GPU)
ollama run llama3.2:3b

# Everyday developer model — needs 8 GB VRAM
ollama run llama3.3:8b

# High-quality reasoning — needs 48 GB+ unified or dual 24 GB GPUs
ollama run llama3.3:70b

# Llama 4 Scout — needs 24–48 GB VRAM (check your RAM before pulling; 32 GB download)
ollama run llama4:scout

# Check what models you have locally
ollama list

Ollama downloads the Q4_K_M GGUF quantization by default — a good balance of quality and memory efficiency. The first pull is the model download; subsequent runs load from local disk.

Step 3 — Use the REST API

Once a model is running, Ollama exposes an OpenAI-compatible API. You can send requests from any language:

# curl — basic completion
curl http://localhost:11434/api/generate 
  -d '{"model":"llama3.3:8b","prompt":"Explain Law 25 compliance in plain language","stream":false}'

# Python — OpenAI-compatible endpoint
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
response = client.chat.completions.create(
    model="llama3.3:8b",
    messages=[{"role":"user","content":"Draft a privacy notice for a Quebec SaaS product"}]
)
print(response.choices[0].message.content)

This means any tool built against the OpenAI SDK — LangChain, LlamaIndex, Open WebUI, Continue.dev — works with your local Llama instance without code changes. Your data never leaves your machine.

Step 4 — Keep Llama running as a service

On Linux, Ollama installs as a systemd service automatically. On macOS, it runs as a background process. To ensure it auto-starts on your Canadian server or workstation:

# Linux — check service status
systemctl status ollama

# Pull a model without launching interactive chat (for scripted deploys)
ollama pull llama3.3:8b

# Serve on a specific address (e.g., for LAN access in a Hashcenter)
OLLAMA_HOST=0.0.0.0:11434 ollama serve

Running Llama with llama.cpp (maximum control)

llama.cpp, created by Georgi Gerganov and maintained by hundreds of contributors, is the foundational open-source C++ library that makes local LLM inference efficient on consumer hardware. Ollama is built on it. Running llama.cpp directly gives you control over every inference parameter — quantization method, context window, GPU layer count, flash attention, speculative decoding — and is the right choice for production deployments, researchers, and anyone who needs to squeeze maximum throughput from specific hardware.

Step 1 — Build llama.cpp

# Clone the repository
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

# Build with CUDA support (NVIDIA GPUs)
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)

# Build with Metal support (Apple Silicon)
cmake -B build -DGGML_METAL=ON
cmake --build build --config Release -j$(nproc)

# CPU-only (any platform)
cmake -B build
cmake --build build --config Release -j$(nproc)

Step 2 — Download a Llama GGUF model

The community-maintained GGUF quantizations by Bartowski and Unsloth are the most widely used. Download directly from Hugging Face:

# Using the Hugging Face CLI (pip install huggingface-hub)
# Llama 3.3 8B — Q4_K_M (recommended for most systems)
huggingface-cli download bartowski/Llama-3.3-8B-Instruct-GGUF 
  --include "Llama-3.3-8B-Instruct-Q4_K_M.gguf" 
  --local-dir ./models

# Llama 3.3 70B — Q4_K_M
huggingface-cli download bartowski/Llama-3.3-70B-Instruct-GGUF 
  --include "Llama-3.3-70B-Instruct-Q4_K_M.gguf" 
  --local-dir ./models

Step 3 — Run inference

# Basic CLI inference — Llama 3.3 8B, all layers on GPU
./build/bin/llama-cli 
  -m ./models/Llama-3.3-8B-Instruct-Q4_K_M.gguf 
  -p "You are a helpful assistant for Canadian businesses. Explain CLOUD Act risk in 3 bullet points." 
  -n 512 
  --gpu-layers 99 
  --ctx-size 4096

# OpenAI-compatible server (drop-in local API)
./build/bin/llama-server 
  -m ./models/Llama-3.3-8B-Instruct-Q4_K_M.gguf 
  --gpu-layers 99 
  --ctx-size 8192 
  --host 127.0.0.1 
  --port 8080

# Llama 4 Scout — partial GPU offload when VRAM is under 55 GB
# Adjust --gpu-layers to fit available VRAM; rest runs on CPU
./build/bin/llama-server 
  -m ./models/Llama-4-Scout-Q4_K_M.gguf 
  --gpu-layers 30 
  --ctx-size 32768 
  --host 127.0.0.1 
  --port 8080

The --gpu-layers flag controls how many transformer layers are offloaded to GPU. Setting it to 99 pushes everything possible to VRAM. Reducing it lets the model layer-split across GPU and CPU — slower, but allows models larger than your VRAM to run. This is how a single RTX 4090 (24 GB) can run Llama 4 Scout with short contexts.

For a deeper comparison of llama.cpp vs. Ollama vs. vLLM — when to use each, throughput trade-offs, and multi-GPU configurations — see our Ollama vs. vLLM vs. llama.cpp comparison.

The Canadian data sovereignty case for local Llama

Every time you send a prompt to a US-based LLM API, that data crosses an international border, lands on servers subject to US jurisdiction, and is — in principle — accessible to US authorities under the CLOUD Act regardless of where the server is physically located.

For Canadian organizations, running Llama locally is not just a cost optimization. It is a data governance decision with real legal weight:

• Quebec Law 25 (Act 25 / Bill 64): Requires organizations handling personal information about Quebec residents to conduct Privacy Impact Assessments before transferring data outside Quebec, and to ensure equivalent protection. Routing client data through a US LLM API is a transfer that requires assessment and disclosure. A local Llama deployment eliminates the transfer entirely.
• CLOUD Act risk: The US Clarifying Lawful Overseas Use of Data Act (2018) allows US law enforcement to compel US companies to produce data stored anywhere in the world. This applies to AWS, Azure, Google Cloud, OpenAI, and Anthropic. It does not apply to a Llama model running on your own hardware in a Canadian Hashcenter.
• Confidential information: Legal files, patient records, financial data, M&A intelligence, government contracts. If your business works with any of these, the risk profile of an external API is fundamentally different from a local inference server.
• Offline and air-gapped deployments: Llama runs fully offline. Once the weights are downloaded, no network access is needed. This makes it viable for classified environments, legal hold situations, or locations with unreliable connectivity.

See also our Digital Sovereignty Canada guide and Sovereign AI in Canada for the broader infrastructure picture.

Quantization: how to get the best quality from your hardware

Quantization reduces the memory footprint of a model by storing its weights at lower precision. The most useful formats for local Llama deployment:

• Q4_K_M — 4-bit, medium variant. The default in Ollama. Retains approximately 95% of the full-precision model quality at roughly 25% of the memory footprint. Best starting point for almost all hardware.
• Q5_K_M / Q6_K — Higher fidelity at 5–6 bits. Use these if you have VRAM headroom and need better performance on reasoning tasks.
• Q8_0 — Near-lossless quality at ~50% of full-precision memory. Use only if you can fit the model; do not half-load a Q8 model by reducing GPU layers.
• Q2_K / Q3_K_S — Aggressive compression for very tight hardware. Noticeable quality degradation on complex tasks. Use only for testing or edge devices.
• IQ4_XS / IQ3_XXS — Importance-sampled quantization. Better quality-per-bit than the standard K-quants at the same bit width, especially for large models. Community GGUF files increasingly ship these.

A practical rule: if your GPU can fit the Q4_K_M, use it. If you have more VRAM, step up to Q5_K_M. If the model barely fits at Q4 (less than 10% VRAM headroom), step down to Q3_K_M rather than layer-splitting, since layer-splitting across CPU/GPU at that margin will be painfully slow.

For interactive hardware sizing by model and quantization, use our Local LLM VRAM calculator.

Where the cloud still wins (honest limits of local Llama)

Local Llama is not the right choice for every Canadian organization. Be honest with yourself about the trade-offs:

• Frontier model quality: GPT-4o, Claude Opus, and Gemini Ultra still outperform Llama 4 Scout on complex reasoning, coding, and long-document synthesis benchmarks as of mid-2026. The gap is narrowing fast, but it is real.
• Multimodal at scale: Llama 4 Scout supports images, but production-scale vision workloads (document processing, video analysis) benefit from the infrastructure investment and reliability of cloud providers.
• No upfront hardware cost: If you are prototyping or have sporadic LLM usage, API pricing is often cheaper than amortizing GPU hardware over a short timeframe.
• Managed reliability: Cloud providers operate at 99.9%+ uptime SLAs with redundancy you would need to engineer yourself on local hardware.
• Very large context windows at scale: Llama 4 Scout’s 10 M-token context window is technically available locally, but KV cache for that context length at full token length requires enormous VRAM — cloud inference handles this more practically for most workloads.
• Llama 4 Maverick and 70B+ locally: Enterprise-grade local deployment of the largest models requires significant hardware investment and technical expertise. For many organizations, a sovereign private cloud with a trusted Canadian provider is a better answer than self-hosted multi-GPU clusters.

The right answer depends on your data sensitivity, usage volume, team technical capacity, and budget. If you have data that cannot leave Canada, local Llama is often the correct baseline. If your data is not sensitive and you need frontier quality for occasional use, the cloud may still make sense. Most Canadian organizations end up with a hybrid: local for sensitive internal tasks, cloud API for non-sensitive external-facing features.

Frequently asked questions

Can I use Llama commercially in Canada?

Yes. The Llama 4 Community License permits commercial use for organizations with fewer than 700 million monthly active users. You must display a “Built with Llama” badge where required by the licence and retain the attribution notice. Read the full licence before launching a product. This applies to Llama 4 specifically; earlier Llama versions have separate licence agreements.

What is the minimum hardware to run Llama in Canada?

Llama 3.2 1B runs on virtually any hardware, including a Raspberry Pi 5 or basic laptop with CPU inference. For a useful everyday model, Llama 3.3 8B at Q4_K_M needs 8 GB VRAM — an RTX 4060 or M2 MacBook Pro is sufficient. Check our VRAM calculator for your specific GPU.

Does running Llama locally satisfy Quebec Law 25 requirements?

Running Llama on hardware located in Quebec, processing only data about Quebec residents, eliminates the cross-border transfer trigger that Law 25 requires you to assess. However, Law 25 compliance depends on many factors beyond data residency — retention periods, access controls, consent, breach notification procedures. Consult a Quebec privacy lawyer for a compliance assessment specific to your situation.

What is the difference between Llama 4 Scout and Maverick?

Both use Mixture-of-Experts (MoE) architecture with 17 billion active parameters per token. Scout has 16 experts and 109B total parameters — it fits on high-end workstation hardware. Maverick has 128 experts and approximately 400B total parameters — it is a multi-GPU cluster deployment that is not practical on a single workstation. For most Canadian businesses and developers, Scout is the relevant Llama 4 model.

Is Llama 4 truly open source?

Meta releases Llama 4 weights publicly and uses the term “open.” However, the Llama 4 Community License is not approved by the Open Source Initiative (OSI) — it has restrictions (user count threshold, attribution requirements, derivative naming rules) that OSI-approved licences do not. If your organization has OSI open-source compliance requirements, verify with your legal team before deploying. The weights are freely downloadable and the model is open in the practical sense that matters for most local deployments.

How does Ollama compare to llama.cpp for running Llama?

Ollama is built on llama.cpp and adds a user-friendly CLI, automatic GGUF downloads, GPU detection, and an OpenAI-compatible REST API. For getting started quickly, Ollama is the right choice. For production deployments, research, or maximum control over inference parameters (quantization method, GPU layer count, flash attention, speculative decoding), running llama.cpp directly gives you more flexibility. See our Ollama vs. vLLM vs. llama.cpp comparison for a detailed breakdown.

Can I run Llama offline — without internet access?

Yes. Once the model weights are downloaded and Ollama or llama.cpp is installed, Llama runs fully offline. No internet connection is needed for inference. This makes it viable for air-gapped environments, legal hold situations, field deployments in remote Canadian locations, or any setting where internet connectivity cannot be guaranteed.

What is the best Llama model for a small Canadian business?

For most small businesses (1–10 employees, standard workstations), Llama 3.3 8B at Q4_K_M is the practical starting point — good enough for document summarization, drafting, code assistance, and internal Q&A, running fast on a mid-range GPU. If you need higher quality for complex analysis or have more than one concurrent user, Llama 3.3 70B on a 48 GB workstation or M4 Mac delivers significantly better results. Use our Local LLM Canada guide for a fuller comparison across models and use cases.

Running Llama on dedicated Canadian hardware

If your organization needs a purpose-built local AI workstation or a private GPU Hashcenter in Canada — with Llama pre-installed, CUDA configured, and an OpenAI-compatible API endpoint ready for your tools — D-Central Technologies designs and ships this hardware to Canadian customers.

We work with the existing open-source ecosystem: Ollama, llama.cpp, Open WebUI, LangChain, and whatever tools your team already uses. We do not lock you into proprietary inference software. Our consulting practice helps Canadian organizations assess which models fit their hardware, their workload, and their Law 25 obligations.

Get in touch with our AI sovereignty consulting team →

All AI hardware is quoted in Canadian dollars. Contact us for current configurations and lead times.