AI GPU Database: VRAM, TFLOPS, TDP for Local LLM Inference

D-Central’s AI GPU Database is a free, citable reference of 30 GPU and accelerator records — the numbers that actually matter for running a large language model locally: VRAM GB, memory bandwidth, FP16 TFLOPS, INT8 TOPS, TDP, and a plain-English inference tier. Data is sourced directly from NVIDIA, AMD, Apple, and Intel official product pages and datasheets, cross-referenced with TechPowerUp GPU Database. Published under CC BY 4.0; verify at source before making purchasing decisions.

Key insight for local AI: For single-stream LLM token generation, memory bandwidth is the dominant bottleneck — not compute TFLOPS. A card with more GB/s will generate tokens faster even if its FP16 number looks lower. VRAM size determines which models fit without CPU offload. FP16/INT8 TFLOPS matter primarily for batch inference and prompt-processing speed.

GPU and accelerator specifications

ⓘ = estimated value (not directly confirmed from official spec page). ‡ = Apple Silicon caveat applies (unified memory / Neural Engine / TDP not published per chip). See per-row notes and the Methodology page. For machine-readable access, use the REST API.

Frequently asked questions

Which GPU is best for running local LLMs at home?

For most home users, the NVIDIA GeForce RTX 4090 (24 GB, 1,008 GB/s) is the best single-card choice — it has the VRAM to run 34B models at FP16 and the bandwidth to generate tokens quickly. If budget is the constraint, the RTX 3090 offers nearly the same VRAM and bandwidth at a much lower used price. Apple Silicon (M4 Max with 128 GB unified memory) is the only platform that can run 70B models at full FP16 precision without a dedicated GPU costing more than a car.

Why does memory bandwidth matter more than compute TFLOPS for local AI?

Modern LLM inference in single-stream mode is memory-bandwidth-bound, not compute-bound. The GPU must stream the entire model weight from VRAM for every generated token. A card with more GB/s will generate tokens faster even if its FP16 number looks lower. Compute TFLOPS matter more for long-context prompt processing (prefill) and batch inference, where the arithmetic intensity rises.

Can I use an AMD GPU for local AI?

Yes. AMD RDNA 3 GPUs work with ROCm on Linux for frameworks like PyTorch and llama.cpp (ROCm/HIP backend). On Windows, Vulkan and DirectML backends are available via llama.cpp and Ollama. The RX 7900 XTX (24 GB) and RX 7900 XT (20 GB) are competitive with NVIDIA options for inference throughput, with the same AI Accelerator FP16/INT8 matrix performance. ROCm tooling is improving but is not yet as seamless as NVIDIA’s CUDA.

What about Intel Arc GPUs?

Intel Arc GPUs (A770 16 GB, B580 12 GB) are a viable budget option, especially for the 16 GB A770. They use Intel’s XMX matrix engines for FP16/INT8 acceleration and are supported via IPEX-LLM, OpenVINO, and llama.cpp’s SYCL backend. CUDA is not available. Ecosystem maturity is lower than NVIDIA or AMD ROCm.

What does “INT8 TOPS” mean and why is it missing for many cards?

INT8 TOPS (tera-operations per second at 8-bit integer precision) measures how fast a GPU can run quantised model inference. NVIDIA does not publish INT8 TOPS for GeForce consumer cards (it appears only on pro/data-center datasheets). Where shown for consumer NVIDIA cards, the figure is estimated. AMD does publish AI Accelerator INT8 TOPS on its product pages for RDNA 3 — notably, RDNA 3 delivers the same throughput for FP16 and INT8 matrix operations.

What is unified memory (Apple) and how does it compare to GPU VRAM?

Apple Silicon uses a unified memory architecture where the same physical memory pool is shared between CPU, GPU, and Neural Engine with no bus-transfer overhead. “VRAM GB” for Apple entries in this table reflects the maximum unified-memory configuration, not a discrete VRAM pool. This means an M4 Max with 128 GB can dedicate all 128 GB to model weights if needed — something no consumer discrete GPU offers. The trade-off is that memory bandwidth (546 GB/s for M4 Max vs. 1,008 GB/s for RTX 4090) is lower, so token generation on the same model is somewhat slower per watt.

What is the smallest GPU that can run a 13B model?

A 13B parameter model in FP16 requires approximately 26 GB of VRAM (13B × 2 bytes). At Q4 quantisation (~0.5 bytes/parameter), 13B fits in about 7 GB. So: a card with 8 GB VRAM (RTX 4060, RX 7600) can run a 13B model at Q4 quantisation but not at FP16. A 16 GB card (RTX 4080, RX 7900 XTX) handles 13B at FP16 comfortably. A 24 GB card handles 13B at FP16 with room to spare.

How often is this dataset updated?

The dataset was last verified in June 2026 against official manufacturer spec pages. GPU specifications are stable once a card is released, but new models are released regularly. We aim to add new entries and correct any discrepancies as they are identified. Check last_verified in the CSV/JSON and verify key specifications at the manufacturer’s source before purchasing.