Re-Ranking Cross-Encoder

A re-ranking cross-encoder is the precision stage of a two-stage retrieval pipeline. Where a first-stage retriever quickly fetches a broad candidate set, the cross-encoder takes each query-document pair, feeds both texts together through a transformer, and outputs a single relevance score used to reorder the candidates. Because query and document are processed jointly, the model's self-attention can reason about token-level interactions between them — which terms answer which question words, whether a negation flips the meaning, whether "S19" in the query matches "S19 Pro" in the passage or merely resembles it. That joint reading produces far more accurate relevance judgments than comparing two independently computed vectors ever can.

Why it must be a second stage

The accuracy comes at a structural cost. A cross-encoder produces no reusable embedding: nothing can be precomputed, so it must run a full transformer forward pass for every query-document pair, and its cost scales linearly with the number of pairs scored. Running it across an entire corpus would take minutes per query. The fix is the retrieve-and-rerank pattern: a cheap first stage — dense vectors, BM25, or a hybrid of both — narrows millions of documents down to perhaps the top 50 or 100, and only those survivors are passed to the cross-encoder for careful reordering. You pay for the expensive model dozens of times per query instead of millions, and the first stage's recall ceiling becomes the pipeline's only hard limit: the reranker can promote a good document buried at rank 80, but it cannot rescue one the retriever never fetched.

The accuracy-cost spectrum

It helps to see the three tiers of matching as one spectrum. Bi-encoders embed query and document separately and compare vectors — cheapest, fully precomputable, least precise. Late interaction (ColBERT) keeps per-token vectors and matches them at query time — a middle ground that recovers much of the token-level signal while remaining indexable. The cross-encoder reads both texts jointly — most accurate, nothing precomputable. Mature pipelines often use two or even all three tiers in sequence, each stage cutting the candidate set before the next, more expensive stage runs. The general concept and its variants are covered under reranking.

Where it sits in a self-hosted stack

For a local retrieval system, a small cross-encoder reranker is one of the cheapest quality upgrades available — often worth more than a bigger embedding model or a fancier index. Compact rerankers in the tens of millions of parameters run acceptably on CPU for candidate sets of 50–100, meaning the upgrade costs no GPU memory at all on a box that is already busy hosting your LLM. In a RAG pipeline the effect compounds: the generator's answer quality is gated by what lands in its context window, and a reranker is precisely the tool that decides which passages make the cut. Feeding a language model five carefully reranked passages routinely beats feeding it twenty raw retriever hits — better answers, a smaller context bill, and fewer chances for an irrelevant passage to derail the generation.

Practical notes

Score candidates in batches to amortize overhead, cache scores for repeated queries, and keep the candidate count honest — reranking 1,000 candidates on CPU will not feel interactive, and the quality gain past the top hundred candidates is usually negligible anyway. Watch the model's maximum input length too: long documents must be chunked or truncated before scoring, and where you cut them changes the scores. And evaluate on your own corpus: rerankers are trained on general web and QA data, and a specialized domain (say, ASIC error codes and hashboard schematics) may justify fine-tuning one on a few thousand labeled pairs. Compare the alternatives under dense vs sparse retrieval when budgeting your own pipeline.