Chat Template

A chat template is the formatting specification that turns a structured list of messages — each tagged with a role such as system, user, or assistant — into the single flat string of tokens a chat-tuned model actually expects. It is the often-invisible layer between a clean conversation object and the raw prompt fed to the model. Get it wrong and a perfectly capable model produces rambling, confused, or truncated output for reasons that look mysterious until you inspect exactly what was sent.

The frustrating thing about a broken chat template is how healthy everything around it looks. The weights loaded without complaint, the tokenizer ran, the model returned fluent text — there is no error, no crash, just output that is subtly or badly wrong. Newcomers to local inference often blame the model itself and go hunting for a better checkpoint when the real culprit is a thin formatting layer they did not even know existed. Learning to see the template as a distinct, inspectable stage sitting between your clean message list and the raw tokens the model consumes is the single most useful debugging habit for self-hosted chat, because it converts a whole class of baffling quality failures into a quick, mechanical thing you can check and rule out first.

Why every model is different

There is no universal format. The ChatML convention popularized by OpenAI uses explicit role markers, but different model families adopt different control tokens and delimiters — even two models fine-tuned from the same base can disagree on the exact wrapper. Modern toolchains express the template as a Jinja string bundled with the tokenizer, so applying it programmatically reproduces the precise spacing, role markers, and reserved special tokens the model saw during training. That is why you should render prompts through the model's own template rather than hand-assembling strings from memory: the template is part of the model's contract, not a stylistic choice you get to make.

Local-inference pitfalls

For sovereign Bitcoiners running models on their own hardware — the kind of local-first stack a DCENT_OS companion node leans on — the chat template is a common and frustrating failure point. Using one model's format with another's weights silently wrecks quality, because the model is now reading control tokens where it expects content and content where it expects structure. A subtler bug: the template already inserts the structural markers, so if your own code re-adds a beginning-of-sequence or role token during tokenization, you create duplicates that quietly degrade output. When you swap checkpoints, always confirm the template travels with the weights.

System prompts and tools

The template does more than wrap user and assistant turns. It also decides how a system prompt is positioned — some formats give it a dedicated slot, others fold it into the first user turn — and how tool definitions and tool results are framed for models trained to call functions. An agent that ignores its system instructions or mangles a tool call is often not misbehaving at all; it is being fed a prompt whose structure does not match what it learned. Because these conventions vary by family, a tool-calling harness that works with one model can fail silently when pointed at another unless the template is respected.

Getting it right

The practical discipline is short. Load the tokenizer that shipped with the checkpoint, apply its chat template to your message list, and disable any automatic special-token addition so nothing is doubled. Verify by printing the fully rendered string once and reading it with your own eyes — the moment you can see the exact bytes going into the model, template bugs become obvious rather than mystifying. Chat templates are assembled from the model's reserved special tokens, and the role-marker handling depends directly on how the tokenizer vocabulary was defined; treat both as fixed properties of the model you downloaded, not knobs to improvise on.