Tokenizer

A tokenizer is the component that converts raw text into the numeric tokens a language model actually processes. LLMs do not read characters or whole words; they operate on a fixed vocabulary of tokens — usually subword fragments — each mapped to an integer ID. The tokenizer encodes your text into IDs on the way in and decodes the model's output IDs back to text on the way out. Every limit you care about — context window size, API pricing, generation speed — is denominated in tokens, not words, so the tokenizer quietly defines the units of the entire system.

Byte-pair encoding

Most modern LLMs use a variant of byte-pair encoding (BPE). Training a BPE tokenizer starts from individual bytes or characters and iteratively merges the most frequent adjacent pairs into longer units, building a vocabulary (commonly tens of thousands to a couple hundred thousand entries) that balances two goals: common words become single tokens, while rare or novel strings can always be spelled out from smaller pieces. This is why an everyday English word is often one token while an unusual technical term, a URL, or a wallet address shatters into many. Byte-level BPE guarantees that any input can be encoded — there is no such thing as an out-of-vocabulary failure — because in the worst case text decomposes to raw bytes.

Practical consequences

Tokenization drives economics and behavior in ways operators feel daily. Density first: the more text fits per token, the more you squeeze into a fixed context budget and the fewer tokens you pay for — and density is language-dependent, so non-English text, code, and numbers often cost noticeably more tokens per unit of meaning. Behavior second: many famous LLM quirks are tokenizer artifacts. Models miscount letters in words because they never see letters, only tokens; arithmetic suffers when digit strings split inconsistently; and a stray leading space can change tokenization and thus the model's output. When something feels inexplicably brittle, inspecting the token boundaries is often the diagnosis.

Why self-hosters must respect the pairing

A tokenizer is not interchangeable between models. The vocabulary and merge rules are fixed before pretraining, and the model's embedding table is learned against exactly that vocabulary — token 4711 means something to this model only because it always meant the same fragment during training. When you run an open-weight model, the tokenizer ships alongside the weights and must match exactly; pair weights with the wrong tokenizer and you get fluent-looking garbage. Packaged formats like GGUF embed the tokenizer in the model file precisely to make this mistake hard, and runners like llama.cpp read it from there.

The operator's takeaway

Think of the tokenizer as the model's alphabet: invisible when correct, catastrophic when wrong, and the unit of account for everything you budget — memory, latency, and cost. Learning to estimate token counts for your own documents is foundational to sizing a self-hosted setup honestly, from context allocation to tokens-per-second expectations.