Early vs Late Fusion

Early fusion and late fusion describe where in a pipeline information from different modalities gets combined. The choice shapes how well a multimodal model captures interactions between, say, an image and its caption, and how robust it is when one input is missing or noisy.

Early (feature-level) fusion

Early fusion, also called feature-level fusion, merges data from all modalities into one joint representation before the main learning model runs. Because features mix at a low level, the model can learn fine-grained cross-modal relationships from the start. The downside is that it can struggle to capture associations that only emerge at higher levels of abstraction, and it tends to be more sensitive to a corrupted or absent modality.

Late (decision-level) fusion

Late fusion runs each modality through its own independent model and combines only the final outputs or scores. This is modular and fault-tolerant: each sub-model operates on its own, so a failed sensor degrades the system gracefully, and it is usually less computationally intensive. The trade-off is that the sub-models never see each other's raw data, so they cannot exploit low-level cross-modal cues. Intermediate fusion sits between the two, mixing representations partway through.

For a self-hosting operator, late fusion's modularity makes it easier to swap or upgrade one modality's model on owned hardware. Modern vision-language systems often use intermediate strategies built on a modality projector or cross-attention fusion rather than pure early or late fusion.