Retrieval Evaluation Metrics

Retrieval evaluation metrics quantify how well a search or retrieval system surfaces the right documents for a query. They are essential for tuning a private retrieval pipeline objectively rather than by gut feel, especially when comparing index settings, embedding models, or a hybrid search configuration. Most are reported "@K", meaning they only consider the top K returned results, and they range from 0 to 1 with higher being better. The discipline matters because retrieval failures are silent: a language model will happily answer from bad context, and without measurement you will never know the retriever was the weak link.

Rank-unaware metrics

Precision@K is the fraction of the top K results that are actually relevant; it measures how clean the result list is, which matters when every retrieved chunk consumes context budget. Recall@K is the fraction of all relevant documents that appear in the top K; it measures coverage, which matters when missing the one critical document means a wrong answer. The two pull against each other — retrieving more raises recall and usually lowers precision — and neither cares about ordering within the top K: a relevant document scores the same at rank 1 or rank K.

Rank-aware metrics

When position matters, use rank-aware measures. Mean Reciprocal Rank (MRR) looks only at the first relevant result and scores its reciprocal rank — rank 1 scores 1.0, rank 2 scores 0.5, rank 5 scores 0.2 — averaged over queries. It suits tasks where one right answer is enough, like looking up a specific spec. NDCG (Normalized Discounted Cumulative Gain) rewards placing multiple relevant documents near the top, discounting lower positions logarithmically and supporting graded relevance (highly relevant beats marginally relevant), making it the most comprehensive single measure of ranking quality. MAP (Mean Average Precision) sits between them, averaging precision at each relevant result's position. A common pragmatic pairing: recall@K to verify the answer is retrievable at all, plus NDCG or MRR to verify it arrives near the top.

Building your own evaluation set

All of these require labelled data: queries paired with known-relevant documents. For a private corpus no benchmark exists, so you build a golden set by hand — even 50–100 real queries with judged results is enough to rank configurations reliably. Draw queries from actual usage, include the awkward ones (abbreviations, part numbers, misspellings), and freeze the set so every experiment is comparable. This is bench discipline applied to software: the same instinct that says never trust a repair without measuring it says never trust a retriever without a test set. From there, tuning becomes engineering — swap the embedding model, change chunk sizes, adjust the vector database index, add a reranker, and let the numbers arbitrate.

Why it matters downstream

Strong retrieval scores are the precondition for a trustworthy RAG answer, since a local LLM can only ground its response on what the retriever surfaces — generation quality has a hard ceiling at retrieval quality. For a sovereign operator, these metrics turn the semantic search layer from a black box into an instrumented system you can improve on your own labelled data, on your own hardware, without shipping a single document to an outside evaluator.

A few traps recur. Optimizing a single metric distorts the system — chasing precision@5 alone teaches the retriever to return only easy documents, while chasing recall alone floods the context with noise, so report a small dashboard rather than one number. Evaluating only on queries the system already handles well produces flattering, useless scores; the golden set must include the failures that motivated tuning in the first place. And metrics measure retrieval, not truth: a perfectly ranked list of outdated documents scores 1.0 and still yields wrong answers, so corpus freshness is a separate discipline the numbers will never surface. Finally, re-run the suite whenever anything upstream changes — a new embedding model, different chunking, a reindex — because retrieval regressions are silent by nature, and the entire value of owning your evaluation set is that nothing degrades without leaving a mark.