Visual Token

A visual token is the unit by which an image enters a multimodal language model. A vision encoder splits the picture into a grid of patches, encodes each patch, and a projector maps those encodings into vectors that live in the same embedding space as text tokens. From that point on, the language model attends jointly over the interleaved visual and text tokens, treating the image much like a stretch of words — the transformer itself has no idea which tokens began life as pixels.

How many tokens an image becomes

Resolution drives token count. LLaVA-1.5, a common open baseline, encodes a 336×336 image into 576 visual tokens; higher-resolution inputs can balloon into thousands. Many modern open models handle large images by tiling: the image is cut into sub-images, each encoded separately, and the token counts add up fast. A single detailed screenshot can easily consume more of the context window than several pages of text. Visual tokens also carry less structured, more spatially redundant information than language — large regions of sky or blank background each still cost tokens — which is why token compression and pruning are active research areas: many visual tokens are simply unnecessary for a given question.

Why the token budget matters on your own hardware

Every visual token consumes context-window space, key-value cache memory, and attention compute that scales quadratically with sequence length via the attention mechanism. For someone running a multimodal model locally, controlling how many visual tokens an image produces is the single biggest lever on memory use and latency. Downscaling inputs, cropping to the region of interest, and choosing a model with an efficient projector can be the difference between an interactive assistant and one that pauses for tens of seconds per image on a consumer GPU. If you are quantizing the model to fit in VRAM, remember that quantization shrinks weights but does nothing about the sequence-length cost of a token-hungry image.

Practical intuitions for the workshop

On the repair bench, visual tokens are what make "show the model a photo of the hashboard" possible: the burn mark, the lifted component, and the silkscreen labels all arrive as patch encodings the model can reason over alongside your text question. The same mechanics explain the failure modes. Fine detail smaller than a patch — tiny component markings, hairline solder bridges — may simply not survive encoding at low resolution, so the model literally cannot see what you are asking about. Sending a tighter crop at native resolution usually beats sending the whole board and hoping. Treat the visual token budget the way you treat a text prompt: feed the model exactly the evidence it needs, and no more.

Compression and pruning

Because so many visual tokens are redundant, a lot of engineering goes into producing fewer of them. Some architectures pool neighboring patch encodings before they ever reach the language model, trading fine spatial detail for a smaller sequence. Resampler modules go further, condensing hundreds of patch features into a small, fixed number of learned tokens regardless of input resolution — the image costs the same context budget whether it is a thumbnail or a poster. Runtime pruning methods instead rank visual tokens by attention relevance and drop the low scorers mid-inference. For a self-hoster the takeaway is that two models with the same parameter count can differ several-fold in how much context and compute an image costs, so when choosing a multimodal model for local deployment, the tokens-per-image figure deserves as much scrutiny as the benchmark scores — it is the number your VRAM will actually feel.

Visual tokens are produced from an image patch embedding step and routed into the language model by a modality projector.