ROUGE

ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation, a metric that measures how much a model’s generated text overlaps with the reference. It was originally built for evaluating summarisation, but it is used across text generation generally — QA, translation, and so on.

The core idea is simple: how many of the words in the reference show up in the prediction?

The shared formulas: precision, recall, F1

Every ROUGE variant follows the same framework. Only the unit in which overlap is counted changes.

• Recall: how much of the reference did we capture (did we miss anything)
• Precision: what fraction of the prediction was actually in the reference (is there any nonsense)
• F1: the harmonic mean of the two. This is what HuggingFace evaluate reports by default.

The running example

We will compute every ROUGE variant on the example below.

Reference:   "Seoul has about ten million residents and is the capital"
Prediction:  "Seoul is the capital and has about nine million residents"

Tokenised on whitespace:

R = [Seoul, has, about, ten, million, residents, and, is, the, capital]   -> m = 10
P = [Seoul, is, the, capital, and, has, about, nine, million, residents]  -> n = 10

ROUGE-1 (unigram)

Formula

For each unigram $w$, take the smaller of its count in the reference and its count in the prediction, and sum.

Counting

Overlapping tokens: everything but “ten” → $|\text{overlap}| = 9$

Computation

ROUGE-2 (bigram)

Formula

Overlap is counted over pairs of adjacent words (bigrams) instead of unigrams.

Counting

Reference bigrams (9):
  (Seoul, has)         (has, about)        (about, ten)
  (ten, million)       (million, residents) (residents, and)
  (and, is)            (is, the)           (the, capital)

Prediction bigrams (9):
  (Seoul, is)          (is, the)           (the, capital)
  (capital, and)       (and, has)          (has, about)
  (about, nine)        (nine, million)     (million, residents)

Overlapping bigrams:
  (has, about) ✅   (million, residents) ✅   (is, the) ✅   (the, capital) ✅
  everything else ❌

$|\text{overlap}| = 4$

Computation

ROUGE-2 comes out lower than ROUGE-1 because a word only counts once it appears adjacent to the right neighbour. “Seoul” and “and” overlap as unigrams, but their surrounding tokens differ, so they contribute to no bigram at all.

ROUGE-L (longest common subsequence)

Formula

This one uses the length of the LCS (longest common subsequence).

$m$ = reference length, $n$ = prediction length, $\beta = P_{lcs} / R_{lcs}$ (setting it this way makes $F_{lcs}$ identical to $F_1$).

The key point: a subsequence does not have to be contiguous. It can skip, as long as the original order is preserved.

Finding the LCS (the DP table)

R = [Seoul, has, about, ten, million, residents, and, is, the, capital]
P = [Seoul, is, the, capital, and, has, about, nine, million, residents]

DP table $L[i][j]$ = the LCS length of $R[0..i]$ and $P[0..j]$:

$\text{LCS}(R, P) = 5$ → [Seoul, has, about, million, residents]

“and”, “is”, “the” and “capital” all exist on both sides, but in the reference they come after “residents” while in the prediction they come before “has”, so they cannot be part of the LCS at the same time as [has, about, million, residents].

Computation

Here ROUGE-L < ROUGE-1. ROUGE-1 ignores order and counts 9 tokens; ROUGE-L counts only 5 because of the ordering constraint.

An extreme case where ROUGE-1 = 1.0 but ROUGE-L is low

R = [A, B, C]
P = [C, B, A]

ROUGE-1: overlap = {A, B, C} -> 3/3 = 1.0
ROUGE-L: LCS = [A] or [B] or [C] -> 1/3 = 0.33

Every word is present, but the order is completely reversed, so ROUGE-L falls sharply. This is exactly why ROUGE-L exists.

ROUGE-Lsum

Formula

For multi-sentence text, compute the LCS per reference sentence and sum.

When the reference consists of sentences $r_1, r_2, \ldots, r_k$:

Example

Reference:
  r1: "Seoul has about ten million residents"    (6 tokens)
  r2: "Seoul is the capital of South Korea"      (7 tokens)

Prediction:
  "Seoul is the capital and has about nine million residents"

LCS(r1, P) = [Seoul, has, about, million, residents] -> 5
LCS(r2, P) = [Seoul, is, the, capital] -> 4

Recall = (5 + 4) / (6 + 7) = 9/13 ~ 0.6923

Applying ROUGE-L once over the whole text ignores sentence boundaries, so a match in one sentence can get in the way of another sentence’s score. ROUGE-Lsum computes each sentence independently, which is fairer for tasks like summarisation where there are several sentences.

For a single sentence, ROUGE-L = ROUGE-Lsum.

Overall comparison

Generally: ROUGE-1 >= ROUGE-L >= ROUGE-2

The limits of ROUGE

It cannot catch synonyms

Reference:  "glad"
Prediction: "happy"
-> ROUGE = 0 (same meaning, different token)

Writing at length dilutes precision

Reference:  "symphony"                 -> m = 1
Prediction: "Wagner wrote a symphony"  -> n = 4

Recall = 1/1 = 1.0  (caught every reference token)
Precision = 1/4 = 0.25  (75% of the prediction is unnecessary)
F1 = 2 x (1.0 x 0.25) / (1.0 + 0.25) = 0.4

It does not see semantic accuracy

Reference:  "2023"
Prediction: "2024"
-> ROUGE-1 = 0, even though it is essentially almost right

Because of limits like these, practitioners use ROUGE alongside BERTScore (embedding similarity), exact match, and human evaluation rather than on its own.

Computing it in Python

import evaluate

rouge = evaluate.load("rouge")

predictions = ["Seoul is the capital and has about nine million residents"]
references = ["Seoul has about ten million residents and is the capital"]

results = rouge.compute(predictions=predictions, references=references)
print(results)
# {'rouge1': 0.9, 'rouge2': 0.4444, 'rougeL': 0.5, 'rougeLsum': 0.5}