JSON Mode

JSON mode is a generation setting that forces a language model to return output that parses as valid JSON. It exists because applications need machine-readable data, and free-form text wrapped in prose is fragile to extract. Running a local model behind your own tooling, JSON mode is what lets you treat the model as a structured data source rather than a chatbot, without the brittleness of regex-scraping its answers.

JSON mode vs. strict structured output

There is an important distinction. Basic JSON mode guarantees only that the output is syntactically valid JSON; it does not guarantee that required fields are present or that types are correct, and real-world testing has shown a small but nonzero schema-mismatch rate. Strict schema-bound structured output goes further: by enforcing a supplied JSON schema through constrained decoding, the model literally cannot emit a token that would violate the schema, so every required field appears and every type is correct. When your downstream code does data["amount"] without a null check, that difference is the difference between a pipeline that works and one that fails at 3 a.m.

How it is enforced

Under the hood, the stricter mode is grammar-constrained decoding applied to a JSON grammar derived from your schema. At each step, tokens that would break valid JSON, or violate the schema, are masked before sampling. This moves correctness from a post-hoc validation step to a guarantee baked into generation itself. The remaining work for the operator is designing a schema that is expressive enough to be useful but not so rigid that it degrades the model's reasoning — over-constrained schemas can push a model into technically valid but semantically poor answers, so it often helps to let the model reason in free text first and emit the JSON at the end.

Support in local runtimes

This is one area where self-hosted tooling is genuinely first-class. llama.cpp implements grammar files (GBNF) that can express arbitrary JSON schemas, and Ollama exposes both a simple JSON mode and schema-bound structured outputs through its API. Server frontends for local models generally accept a response_format or schema parameter in the OpenAI-compatible style, so code written for hosted APIs ports over cleanly. Because the enforcement happens in the sampler on your own machine, there is no dependency on a provider implementing the feature correctly — you can read the grammar code yourself, which is exactly the kind of verifiability sovereign operators prize.

Where it earns its keep

JSON mode is the backbone of practical local-AI automation. Extracting structured fields from documents, classifying support tickets into fixed categories, generating configuration files, and powering function calling — where the model must emit a well-formed tool invocation — all depend on output that a program can consume without human cleanup. A concrete example from our world: a script that reads ASIC miner log excerpts and emits {"error_code": ..., "suspected_board": ..., "recommended_action": ...} is only automatable if that structure is guaranteed. For the general mechanism behind the guarantee, see grammar-constrained decoding; for the decoding step it sits on top of, see greedy and sampled decoding — JSON mode constrains which tokens are legal, while the sampler still chooses among the legal ones.

A few operational habits make structured output boring in the best sense. Validate parsed output anyway — enforcement guarantees shape, not sense, and a schema cannot stop a model from filling a valid field with a wrong value. Prefer enums over free strings wherever the answer space is closed, keep schemas shallow, and give fields descriptive names, since the model reads them as hints. Log every schema violation or retry your pipeline performs; a rising retry rate is an early warning that a prompt, model, or quantization change degraded reliability. None of this is glamorous, which is exactly the point — structure is what turns a clever demo into plumbing you can forget about.