Tensor Core

Tensor Core is a specialized execution unit inside modern NVIDIA GPUs, first introduced with the Volta architecture, purpose-built to perform fused matrix multiply-accumulate operations — the single most common computation in deep-learning training and inference. Where a standard GPU core handles general arithmetic one value at a time, a Tensor Core chews through small matrix blocks per clock cycle. The design philosophy will feel familiar to anyone who understands why an ASIC beats a CPU at SHA-256: when one operation dominates a workload, silicon dedicated to exactly that operation wins by orders of magnitude.

Mixed-precision math

Each Tensor Core computes an operation of the form D = A x B + C on small matrices. The trick is mixed precision: the input matrices A and B are supplied in a lower-precision format such as FP16, the multiply produces a full-precision product, and the results are accumulated in higher-precision FP32. This preserves enough numerical accuracy for neural networks while dramatically increasing throughput and halving memory traffic for the inputs. A single Volta V100 packed 640 Tensor Cores for roughly 125 mixed-precision teraflops; successive generations added support for further formats such as BF16, INT8, and FP8, each trading a little precision for more speed. This is the same bargain that quantization makes on the model-weights side — deep learning tolerates reduced precision remarkably well, and the hardware is built to exploit that tolerance.

Why it matters for self-hosted AI

Tensor Cores are the reason a modern GPU can train and serve large models so much faster than older hardware — they are the workhorse behind the headline FLOPS figures vendors advertise. When evaluating a card for AI work, the Tensor-Core generation and its supported precisions matter far more than gaming benchmarks or clock speeds. A card whose Tensor Cores natively support the precision your runtime uses will dramatically outperform one that has to emulate it. For the home operator building a private AI box, this is worth internalizing before spending money: two cards with similar gaming performance can differ enormously in usable AI throughput.

Where the bottleneck really is

For local LLM inference specifically, there is an important caveat: generating text token-by-token is usually limited by how fast weights can be streamed from memory, not by raw matrix arithmetic. In that regime, memory bandwidth and VRAM capacity gate performance, and monstrous Tensor-Core counts sit partially idle. Tensor Cores earn their keep in the compute-heavy phases — processing long prompts, batch serving multiple users, fine-tuning, and training — where the arithmetic genuinely saturates. Sizing a rig therefore means matching hardware to workload: a single-user chat box lives and dies on memory bandwidth, while a machine serving a family or small team gets real value from Tensor-Core throughput.

The precision story also connects directly to the quantized models most home operators actually run. A model quantized to 8-bit or 4-bit weights still gets dequantized or computed through whatever formats the hardware accelerates, so a card whose Tensor Cores natively handle low-precision integer and float formats executes those models with far less overhead. Newer generations also exploit structured sparsity — skipping computations on weights that are zero — squeezing more effective throughput from the same silicon. None of this shows up in a gaming review, which is exactly why AI buyers read spec sheets differently.

For anyone speccing hardware to run a local LLM on their own terms, the practical summary is this: Tensor Cores set the ceiling on compute-bound work, memory sets the ceiling on generation speed and model size, and a balanced machine respects both. Understanding the unit behind the marketing numbers is one more piece of running your own compute instead of renting someone else's.