Definition
A GPU kernel is a function written in a serial style that the GPU executes simultaneously across many threads. In CUDA, a kernel is marked with the __global__ specifier and launched with a grid configuration; each of the thousands of threads runs the same code on a different slice of data. The kernel is the fundamental unit of work a GPU performs, and the efficiency of running AI models locally hinges on how well these kernels are written and scheduled.
The execution model: grids, blocks, and warps
When a kernel launches, its threads are organized into a grid of thread blocks, and each block holds a number of threads. On NVIDIA hardware, threads are scheduled in groups of 32 called warps. All threads in a warp execute the same instruction at once on different data, a design called SIMT (Single Instruction, Multiple Thread). This is what lets a single matrix multiply spread across an entire GPU in one launch.
Why kernels matter for local inference
The speed of running an open-weight model at home is largely a story about kernels. Projects optimize hand-tuned kernels for attention, quantized matrix multiplication, and memory layout because a poorly written kernel can leave most of the GPU idle. Tools like FlashAttention exist precisely because a smarter kernel can cut memory traffic dramatically without changing the math. When a quantization format ships a fast kernel, that format becomes practical for home hardware.
Kernels are dispatched through CUDA and frequently target specialized Tensor Cores. Understanding them clarifies why two models with identical parameter counts can run at very different speeds.
In Simple Terms
A GPU kernel is a function written in a serial style that the GPU executes simultaneously across many threads. In CUDA, a kernel is marked…