Interleaved Image-Text

Interleaved image-text refers to content in which images and text alternate within a single sequence — the way an illustrated blog post, a repair manual, or a step-by-step tutorial mixes pictures and prose. Handling it requires a model to understand each modality in the context of the other and, in generative systems, to produce both in a coherent order rather than treating image and text as separate one-shot tasks. It is the difference between a model that can caption a photo and a model that can write an illustrated document.

Understanding versus generation

On the input side, an interleaved-capable model reads a mixed sequence — several images woven between passages of text — and reasons across all of it jointly, so a question asked at the end can depend on a picture shown at the beginning. This is already demanding: the model must keep visual and textual context in one working memory and resolve references like "the board in the second photo." Most modern vision-language models handle interleaved input reasonably well, because training corpora scraped from the web are naturally interleaved documents.

Output is the harder half. A generative interleaved model must decide, token by token, when to keep writing text and when to emit an image, and the two must stay consistent: the picture has to depict what the surrounding prose describes, and the prose that follows has to acknowledge what the picture showed. Anole, an open-source autoregressive model, generates interleaved image-text natively as a single unified token stream — images are just runs of visual tokens in the same sequence as words. Other architectures bolt modality-specific heads onto a shared backbone and switch between writing words and rendering pixels. The unified-stream approach is conceptually cleaner and composes better with standard transformer machinery, which is why it anchors most any-to-any model designs.

Why coherence is the hard part

Captioning grades a model on one image-text pair in isolation. Interleaved generation grades it on a running narrative: image three must not contradict paragraph two, and paragraph four must not describe details image three failed to render. This couples the modalities far more tightly than any single-shot task, and it is where current open models still visibly strain — drifting character appearance across a sequence of generated images, or text that confidently describes an image element the generator omitted. Evaluation is equally unsettled, since scoring "did this document stay coherent" is much harder than scoring a caption.

The self-hosting angle

For the sovereignty-minded user, interleaved generation is the capability that turns a local model from a chatbot into a documentation engine. A model that can produce an illustrated build guide, an annotated diagnostic walkthrough, or a photo-essay entirely on owned hardware means drafts, images, and the ideas behind them never transit an external service — no upload, no content policy filter, no per-image API bill. The catch is cost: interleaved generation multiplies sequence length, since each image costs hundreds or thousands of visual tokens, so VRAM and throughput planning matter more than for text-only work. Quantized builds and modest image resolutions make it workable on prosumer GPUs today, and the trajectory is clearly toward better.

Training data explains much of the recent progress. Web-scale interleaved corpora such as filtered Common Crawl documents preserve the natural rhythm of human-authored mixed media, and models pretrained on them inherit a sense of when an image belongs, not just what one contains. Caption-pair datasets, by contrast, teach association without sequencing — which is why models trained only on pairs can describe images fluently yet cannot author a document that flows.

Interleaved image-text is a hallmark capability of any-to-any architectures, and it rests on two foundations covered elsewhere in this glossary: representing every image as a stream of visual tokens the model can both read and write, and a backbone trained on genuinely mixed documents rather than paired captions. Watch for it as the dividing line between models that merely see and models that can actually author.