Tokenizer Vocabulary

A tokenizer vocabulary is the fixed dictionary of tokens a language model can read and emit. It is the bridge between human text and the integer IDs a neural network actually processes: every string the model sees is first segmented into tokens, and each token is looked up in the vocabulary to retrieve its ID. Nothing outside the vocabulary can be represented directly — it is the model's entire alphabet, frozen at training time and unchangeable afterward without retraining the layers that depend on it.

It is worth pausing on how different this is from how computers traditionally handled text. Classical software treats text as characters and words — units humans chose. A tokenizer's units are chosen by frequency statistics instead: whatever strings appeared often enough in the tokenizer's training corpus earn their own entry, whether that is a common English word, a fragment like -ing, a snippet of Python syntax, or a whole boilerplate phrase. The vocabulary is thus a fossil record of the data the tokenizer was trained on — you can read cultural and linguistic priorities straight out of which strings got dedicated tokens — and the model inherits those priorities in the form of cheaper or costlier processing for different kinds of text.

Size and trade-offs

Vocabulary size is a deliberate design choice with real costs on both sides. Monolingual models have often used 30,000 to 64,000 entries; recent multilingual models run much larger — roughly 128,000 for Llama 3 and around 200,000 for GPT-4-class models. A larger vocabulary keeps common words and multilingual text whole, so a given passage costs fewer tokens: better effective use of the context window and faster generation per unit of text. The price is a much larger embedding matrix and output layer. For small models this is no rounding error — a 128K-entry vocabulary can put a substantial fraction of a compact model's total parameters in the embedding table alone, memory that a self-hoster might prefer spent on layers that reason.

Tokens are not words

Vocabulary entries are subword units, not dictionary words. A single entry can be a character, a word fragment, a whole common word, a punctuation mark, or a raw byte, depending on the merge rules learned during tokenizer training — typically via Byte-Pair Encoding (BPE), which iteratively merges the most frequent character pairs in a training corpus until the target size is reached. Frequent strings earn dedicated tokens; rare ones are spelled out from pieces. This is why token counts rarely match word counts, why the same sentence costs different amounts across model families, and why languages underrepresented in the tokenizer's training data fragment into many more tokens — effectively paying more compute for the same meaning. Byte-level fallback guarantees that any input, even binary or emoji, can be encoded, just inefficiently.

Practical consequences for self-hosters

Three things follow for anyone running models locally. First, the vocabulary must match the model exactly — it is baked into files such as GGUF, and runtimes like llama.cpp read it from the model file itself; a mismatched tokenizer produces garbage, one of the classic silent failures when converting models between formats. Second, token economics vary by model: the same repair manual might fit comfortably in one model's context and overflow another's purely because of vocabulary differences, so "how many tokens is my document" has no model-independent answer. Third, tokenization quirks explain some famous failure modes — models miscounting letters in a word, or stumbling on arithmetic, are partly artifacts of the model perceiving multi-character chunks rather than characters.

Structure tokens and the bigger picture

Every vocabulary also reserves slots for special tokens that mark structure rather than ordinary text — sequence boundaries, chat roles, padding — and templates that misuse them are a common source of local-model misbehavior. The vocabulary is a good reminder that a language model's worldview is shaped before training even begins: the tokenizer decides what the model can perceive as a unit, and everything downstream — cost, speed, multilingual fairness, even spelling ability — inherits that decision.