TensorRT-LLM

TensorRT-LLM is an open-source library from NVIDIA for optimizing the inference of large language models on NVIDIA GPUs. Released publicly in October 2023, it provides a Python API for defining a model and then building a highly optimized runtime “engine” tailored to a specific GPU. It targets users who already run NVIDIA hardware and want to extract maximum throughput and minimum latency from it, including operators self-hosting models alongside Bitcoin or other compute workloads.

How it works

Rather than interpreting a model graph at run time, TensorRT-LLM ahead-of-time compiles the model into a serialized engine. During this build step it fuses operations, selects custom GPU kernels for attention and matrix multiplication, and applies optimizations such as in-flight batching and paged key-value caching. It supports several decoding strategies, including beam search and speculative decoding, and integrates quantization to reduce model size and increase speed.

Trade-offs and deployment

The compile-ahead approach delivers strong performance but ties the resulting engine to a particular GPU architecture and configuration, so an engine built for one card may need rebuilding for another. TensorRT-LLM is frequently paired with NVIDIA's Triton Inference Server to expose the compiled model as a production endpoint. Because it is NVIDIA-specific, it is not a portable choice for operators on AMD, Apple, or other silicon, where a hardware-agnostic engine is more appropriate.

TensorRT-LLM is one of the more performance-oriented serving paths; contrast it with the cross-platform approach of MLC-LLM and with broadly compatible CPU/GPU runtimes such as llama.cpp.