BLEU and ROUGE

BLEU and ROUGE are two foundational automatic metrics for evaluating generated text against human reference text by counting overlapping word sequences. They predate modern large language models but remain widely reported because they are fast, deterministic, and require no human grader. Both compare a machine output to one or more references and reduce the comparison to a number between 0 and 1, but they were designed for different tasks and emphasize opposite sides of the same coin.

BLEU: precision for translation

BLEU (Bilingual Evaluation Understudy), introduced by Papineni and colleagues in 2002, was built for machine translation. It computes modified n-gram precision, the fraction of n-grams (typically of length one to four) in the candidate that also appear in a reference, then multiplies by a brevity penalty that discourages translations shorter than the reference. BLEU was the first automatic metric to correlate well with human judgments at scale, which cemented its role in translation research.

ROUGE: recall for summarization

ROUGE (Recall-Oriented Understudy for Gisting Evaluation), introduced by Chin-Yew Lin in 2004, was built for summarization. Its common variants are ROUGE-N, the n-gram recall of the system summary against references, and ROUGE-L, which rewards the longest common subsequence and so credits in-order matches without a fixed n. Because summaries should capture reference content, ROUGE emphasizes recall where BLEU emphasizes precision.

Shared limitations

Both metrics only measure surface word overlap, so they penalize correct paraphrases that use different wording and can reward fluent-but-wrong text that happens to share n-grams. For this reason modern evaluation increasingly favors execution-based or model-judged approaches such as HumanEval and MT-Bench, though BLEU and ROUGE remain useful, low-cost baselines.