Run Gemma Locally in Canada: Gemma 3 QAT + Gemma 4 Setup Guide (2026)

What is Gemma and who built it?

Gemma is a family of open-weight language models released by Google DeepMind. Unlike proprietary cloud APIs, Google publishes the model weights publicly, which means you can download them, run inference on your own hardware, and fine-tune them — no API key required, no usage telemetry sent back to Mountain View.

The community has embraced Gemma quickly: quantized GGUF versions appear on Hugging Face within hours of each release, Ollama ships native support on day one, and frameworks like llama.cpp and vLLM integrate the architecture cleanly. Credit is due to Google DeepMind for releasing the weights openly and to the open-source quantization community — particularly the llama.cpp project — for making consumer-GPU inference practical.

For a broader comparison of open-weight models you can run in Canada, see our open-weight AI Canada comparison. If you want to understand why running locally matters for Canadian privacy law, start at local LLMs in Canada: the complete guide.

Gemma 3 vs Gemma 4: which should you run?

As of June 2026, there are two active generations:

Gemma 3 (released March 2025)

Four sizes: 1B, 4B, 12B, 27B. The 4B, 12B, and 27B models are multimodal — they accept both text and images. Context window is 128 K tokens. Released under Google’s custom Gemma Terms of Use (not Apache 2.0), which allows personal, research, and internal business use freely but carries restrictions that enterprise legal teams sometimes flag for commercial product distribution — read the terms before building a product you sell. QAT variants (see below) are available for all four sizes.

Gemma 4 (released April 2, 2026)

Four sizes with a different architecture: E2B (~2.3 B effective parameters, edge-optimized), E4B (~4.5 B, edge), 26B MoE (mixture of experts — only ~3.8 B parameters active per token, 26 B total), and 31B Dense. Context window: 128 K for edge models, 256 K for 26B/31B. All four are natively multimodal (text, images, video, OCR). The key licensing upgrade: Gemma 4 ships under Apache 2.0 — commercial use, fine-tuning, and redistribution permitted with no user-count thresholds or revenue caps. QAT versions were released on June 5, 2026 with approximately 72% VRAM reduction vs BF16 (verify current availability at ai.google.dev).

Rule of thumb: If you need the largest, sharpest model and have 16–24 GB VRAM, Gemma 3 27B QAT or Gemma 4 26B MoE. For a laptop or 8 GB GPU, Gemma 3 12B QAT. For constrained edge hardware, Gemma 3 4B QAT or Gemma 4 E4B.

The QAT advantage: why Google DeepMind’s approach matters

Standard post-training quantization applies compression after the model is trained. It works, but each additional bit reduction degrades output quality measurably. QAT (Quantization-Aware Training) takes a different approach: it simulates low-precision arithmetic during the training process itself, so the model learns to compensate for the rounding errors it will face at inference time.

Google’s published benchmarks show this reduces the perplexity drop by approximately 54% compared to post-training Q4_0 quantization applied to the same base weights. In practical terms, the Gemma 3 12B QAT model at int4 performs closer to the full BF16 12B than a post-training Q4 quantization would.

The result: Gemma 3 QAT int4 needs roughly 25% of the VRAM the BF16 model requires. A model that needed 24 GB now fits in 6.6 GB. That is the difference between needing a workstation GPU and running on a laptop’s RTX 4060.

For a deeper technical explanation of int4, int8, and FP16 tradeoffs, see our AI quantization guide (int4 / int8 / FP16).

VRAM requirements: Gemma 3 (verified from Google DeepMind, as of April 2025)

The following figures are sourced from Google’s official QAT announcement. Numbers reflect idle VRAM for model weights only — leave headroom for the OS, context window, and other processes (typically 1–3 GB extra). Verify current figures at the Google Developers Blog before purchasing hardware.

Model	BF16 (full precision)	QAT int4 (verified)	Fits on
Gemma 3 1B QAT	~2 GB	~0.5 GB	Any GPU, Raspberry Pi 5, Apple Silicon
Gemma 3 4B QAT	~8 GB	~2.6 GB	GTX 1060 6 GB, M1 MacBook Air
Gemma 3 12B QAT	~24 GB	~6.6 GB	RTX 4060 8 GB, RTX 3070 8 GB
Gemma 3 27B QAT	~54 GB	~14.1 GB	RTX 3090 24 GB, RTX 4070 Ti 16 GB

Note: VRAM requirements may vary with Ollama version, batch size, and context length. Running on CPU-only is possible but will be 5–20× slower depending on RAM bandwidth. Apple Silicon unified memory is shared CPU/GPU — an M2 Pro with 16 GB unified memory can run the 12B QAT model comfortably.

Not sure which GPU you have or need? Use our local LLM VRAM calculator to estimate requirements for your specific hardware.

VRAM requirements: Gemma 4 (community estimates, hedged — verify at source)

Gemma 4’s mixture-of-experts architecture changes the math: the 26B MoE model activates only ~3.8 B parameters per token, so inference VRAM is much lower than the total parameter count suggests. The figures below are community estimates as of June 2026 — Gemma 4 is newer and published numbers are still stabilising. Verify against the official Gemma 4 documentation and Ollama release notes before purchasing hardware.

Model	Architecture	License	Est. VRAM (Q4)	Context
Gemma 4 E2B	~2.3B dense, edge	Apache 2.0	~2–3 GB	128K
Gemma 4 E4B	~4.5B dense, edge	Apache 2.0	~3–5 GB	128K
Gemma 4 26B MoE	26B total / 3.8B active	Apache 2.0	~12–15 GB	256K
Gemma 4 31B Dense	31B dense	Apache 2.0	~18–22 GB	256K

Gemma 4 QAT checkpoints (released June 5, 2026) reportedly cut VRAM ~72% vs BF16 baseline. Figures above assume Q4 quantization without QAT; QAT versions will be lower — check Hugging Face for current checkpoint availability.

Hardware guide for Canadian users

Starting out: 8 GB VRAM

An NVIDIA RTX 3070, 3060 Ti, or 4060 with 8 GB VRAM runs Gemma 3 12B QAT cleanly. This is the sweet spot for most Canadians running a home AI assistant, a local coding helper, or a Law 25-compliant document processor. GPU prices in Canada fluctuate significantly — shop used RTX 30-series for cost efficiency.

Serious workloads: 16–24 GB VRAM

An RTX 3090 (24 GB) or RTX 4090 (24 GB) runs Gemma 3 27B QAT or Gemma 4 26B MoE. At this tier you get performance approaching GPT-4o-mini class on many benchmarks, fully offline. A used RTX 3090 is often available in Canadian markets for under $800 CAD — contact us for sourcing guidance if you’re setting up a local AI workstation or Hashcenter.

Apple Silicon

M-series Macs use unified memory, so a 16 GB MacBook Pro M3 can run Gemma 3 12B QAT comfortably. An M3 Max or M4 Pro with 36 GB+ can run Gemma 3 27B QAT. Ollama’s macOS package handles Metal acceleration automatically.

CPU-only fallback

Gemma 3 1B and 4B QAT run on CPU-only machines via llama.cpp. Expect 3–8 tokens/second on a modern multi-core system. Adequate for batch document processing overnight; not practical for interactive use at 27B scale.

Step-by-step: Run Gemma 3 locally with Ollama

Ollama is the fastest path to a working local Gemma setup. It handles model download, CUDA/Metal/CPU backend selection, and serves an OpenAI-compatible REST API on localhost:11434. See our Ollama vs vLLM vs llama.cpp comparison if you need production throughput or multi-user serving instead.

Step 1: Install Ollama

Linux (including Ubuntu/Debian on a Canadian-hosted server or home machine):

curl -fsSL https://ollama.com/install.sh | sh

macOS (Homebrew):

brew install ollama

Windows: Download the installer from ollama.com/download and run it. WSL2 with CUDA is recommended if you have an NVIDIA GPU.

Step 2: Start the Ollama service

ollama serve

On Linux with systemd, Ollama may already be running as a service after install. On macOS, the menu-bar app starts it automatically. Test with: curl http://localhost:11434 — you should see “Ollama is running”.

Step 3: Pull and run a Gemma 3 QAT model

Choose based on your VRAM (see table above). In a new terminal:

4 GB GPU or M1 MacBook — 4B QAT:

ollama run gemma3:4b-it-qat

8 GB GPU — 12B QAT (recommended starting point):

ollama run gemma3:12b-it-qat

16–24 GB GPU — 27B QAT:

ollama run gemma3:27b-it-qat

The first run downloads the model weights (~3–15 GB depending on variant). Subsequent launches are instant from local cache. Type your prompt directly, or press Ctrl+D to exit interactive mode.

Step 4: (Optional) Use the REST API

Ollama exposes an OpenAI-compatible endpoint, so any tool that works with OpenAI’s API — Open WebUI, Continue.dev, Aider — works with your local Gemma:

curl http://localhost:11434/v1/chat/completions 
  -H "Content-Type: application/json" 
  -d '{
    "model": "gemma3:12b-it-qat",
    "messages": [{"role": "user", "content": "Summarize this contract clause for a Canadian SMB."}]
  }'

Running Gemma 4 with Ollama

Gemma 4 support requires Ollama v0.20.0 or later. Commands follow the same pattern:

# Edge model — fits on 4–6 GB VRAM
ollama run gemma4:e4b

# MoE — recommended for 16+ GB VRAM
ollama run gemma4:26b

Verify current Gemma 4 Ollama model tags at ollama.com/library/gemma4 — tag naming conventions may update with new releases.

Why this matters for Canadian businesses: Law 25 and the CLOUD Act

Canada’s Law 25 (Quebec, in force since September 2023) imposes strict requirements on how personal information is stored and processed, including explicit disclosure when data leaves Quebec’s borders and impact assessments for cross-border transfers. Using a US-hosted AI API — even one with a Canadian data-residency option — creates transfer obligations that can be difficult to document and audit.

The US CLOUD Act (2018) gives US law enforcement authority to compel US-headquartered cloud providers to produce data stored anywhere in the world. A Canadian data-residency region on a US cloud provider does not fully insulate you from CLOUD Act reach.

Running Gemma locally means your data never travels. Documents you summarize, contracts you analyse, customer communications you process — none of it leaves your machine. For Quebec SMBs processing personal information under Law 25, or for federally regulated industries with data-residency requirements, local inference is the cleanest compliance path.

See our detailed analysis at digital sovereignty in Canada and sovereign AI Canada. For professional implementation support, see our AI sovereignty consulting services.

Use cases where local Gemma excels

• Document processing: Summarizing contracts, bylaws, or internal policies without uploading sensitive text to external servers
• Private coding assistant: With Continue.dev or Aider pointed at localhost:11434, Gemma 3 12B QAT gives solid code completion on proprietary codebases
• Regulated-industry Q&A: Legal, medical, or financial chat tools where firm policy prohibits external data exposure
• Batch classification: Running thousands of customer support tickets through a local classifier overnight — no API costs, no rate limits
• Multimodal document OCR: Gemma 3 4B+ and all Gemma 4 models accept images — useful for processing scanned forms or invoices locally
• Air-gapped environments: For air-gapped AI coding and security research scenarios in Canada, see our dedicated guide at /air-gapped-ai-coding-canada/

Where the cloud still wins

Local Gemma is not always the right answer. Be honest with yourself about the tradeoffs:

• Raw capability ceiling: GPT-4o, Claude Sonnet, and Gemini 1.5 Pro still outperform Gemma 3 27B on complex multi-step reasoning, especially for novel problems outside the training distribution. If accuracy on hard tasks is the primary metric, cloud APIs remain ahead for now.
• Hardware cost vs API cost: An RTX 3090 capable of running Gemma 27B costs $700–$900 CAD used. If your workload is only a few hundred API calls per month, the break-even on hardware is years away. Run the numbers with our cloud vs local AI TCO calculator.
• Latency at large scale: A single consumer GPU generates 15–40 tokens/second. Cloud APIs serving thousands of GPUs in parallel deliver much higher burst throughput. If you’re building a high-concurrency product, local inference on consumer hardware will bottleneck.
• Maintenance overhead: You are responsible for updates, CUDA drivers, disk space management, and model version control. Cloud APIs abstract all of this.
• Multimodal video at scale: Gemma 4 supports video input, but processing long videos locally is slow on consumer hardware. Cloud inference is more practical for high-volume video workflows.

The Canadian sovereignty case for local AI is real and legally significant — but it should complement a clear-eyed cost-benefit analysis, not replace one. If your specific workflow sits at the intersection of privacy requirements and manageable volume, local Gemma is a strong fit.

Comparing Gemma to other local models in Canada

Gemma is one of several strong open-weight families available to Canadian operators. Key alternatives:

• DeepSeek R1/V3: Strong on reasoning and coding. See our run DeepSeek locally in Canada guide for setup specifics. Chinese origin — evaluate supply-chain considerations for sensitive workloads.
• Llama 3 / Llama 4: Meta’s open-weight family. Apache 2.0-equivalent license for most sizes. Strong community support. Covered in ChatGPT alternatives in Canada.
• Mistral / Mixtral: French-origin, EU data-residency alignment. Strong efficiency at 7B and MoE configurations. See our open-weight AI Canada comparison for a full side-by-side.

No single model wins across all tasks. Running Gemma 3 12B QAT for everyday document work while benchmarking Llama 4 Scout for coding tasks is a reasonable multi-model local strategy.

Frequently asked questions

Is Gemma free to use in Canada?

Gemma 3 is free to download and run locally under Google’s Gemma Terms of Use, which permits personal, research, and internal business use. Commercial product distribution may require closer review of the terms. Gemma 4 (April 2026) is released under the Apache 2.0 license — fully free for commercial use, fine-tuning, and redistribution with no restrictions beyond the standard Apache 2.0 terms. Always check the current license at ai.google.dev/gemma.

What is the minimum GPU to run Gemma locally?

Gemma 3 4B QAT runs on a GPU with as little as 4 GB VRAM (RTX 3050 laptop, GTX 1060 6 GB, etc.) with approximately 2.6 GB model weight. Gemma 3 1B QAT needs under 1 GB VRAM and even runs on CPU-only systems. For genuinely useful results on complex prompts, an 8 GB GPU running Gemma 3 12B QAT is the recommended minimum.

Does running Gemma locally satisfy Quebec’s Law 25?

Running inference locally means personal data never leaves your premises, eliminating cross-border transfer obligations under Law 25 for that processing activity. However, Law 25 compliance involves more than data residency — consent, retention schedules, and privacy policies are separate requirements. This guide is technical, not legal advice. Consult a Quebec privacy lawyer for a compliance opinion specific to your business.

How does Gemma 3 QAT compare to standard Gemma 3 quality?

Google’s published benchmarks show the QAT int4 process reduces perplexity degradation by approximately 54% compared to post-training Q4_0 quantization applied to the same weights. In practice, most users report that Gemma 3 12B QAT is indistinguishable from the full BF16 model on everyday conversational and document tasks. The gap widens on highly technical or nuanced prompts — if quality on those tasks matters, benchmark both before committing.

Can I run Gemma on a Mac?

Yes. Ollama’s macOS package uses Apple Metal for GPU acceleration automatically. An M2 MacBook Pro with 16 GB unified memory runs Gemma 3 12B QAT at roughly 25–40 tokens/second. An M3 Max or M4 Pro with 36 GB+ unified memory handles Gemma 3 27B QAT. No discrete GPU required.

What is the difference between Gemma 4 MoE and the dense models?

Gemma 4 26B MoE (mixture of experts) has 26 B total parameters but only activates approximately 3.8 B per token during inference. This means VRAM usage is much closer to a 4B model than a 26B model at the same quality level. The tradeoff: loading the full 26B parameter set still requires more VRAM than a true 4B dense model, and routing overhead adds some latency. For a 16 GB GPU, the 26B MoE is usually a better choice than the 31B dense model.

Should I use Ollama or llama.cpp directly?

Ollama wraps llama.cpp with a model management CLI, REST API server, and automatic backend selection (CUDA, Metal, CPU). For individuals and small teams, Ollama is faster to set up and maintains the model library cleanly. For production serving — multiple concurrent users, custom batching, or fine-tuned model loading — consider vLLM or a bare llama.cpp server. See our full Ollama vs vLLM vs llama.cpp comparison.

Where can I get help sizing hardware for a local AI deployment in Canada?

Use our VRAM calculator to estimate requirements for specific models and context lengths. For a multi-user or Hashcenter deployment, our AI sovereignty consulting team can design and source a hardware stack sized for your workload — Canada-based, no data leaves your facility.