don't score an empty F1 gold reference as zero

[gyanu2507]

Fixes #1353

F1_score maps nltk's None onto 0.0. That's fine when the prediction bag is empty (undefined precision → miss). It's wrong when the gold bag is empty after normalize — that's bad reference data, not a model failure, and np.mean then treats it like a real zero.

Raise ValueError for empty gold. Empty pred still scores 0.0.

[buzzedaldrinx86-creator]

Thanks for picking this up. I filed #1353, and this fixes the case I reported: an empty gold no longer scores as a model miss, and an empty prediction still scores 0.0.

One thing I hit when I ran it. compute() loops over every gold and aggregates with max, so the new raise in compute_one_item fires even when another reference is valid. With two references ["Paris", "???"], a normaliser that strips punctuation and the prediction "Paris", main returns 1.0 and this branch raises ValueError. One unusable reference among several now aborts a sample that used to score correctly.

A small change would keep the fix without that: skip references whose normalised bag of words is empty inside compute(), and raise only when none are left. Something like:

results = []
for gold in golds:
    for pred in predictions:
        item = self.compute_one_item(gold=gold, pred=pred)
        if item is not None:
            results.append(item)
if not results:
    raise ValueError("F1_score: every gold reference is empty after normalization; invalid evaluation data.")
return self.aggregation_function(results)

with compute_one_item returning None for an empty gold instead of raising. A test for the mixed case (golds=["Paris", "???"] → 1.0) would pin it. I tried that sketch on four cases (mixed references → 1.0, all references empty → ValueError, empty prediction → 0.0, partial overlap unchanged) and it behaves as described.

How I checked: I ran the F1_score class from main (6ba40c42) and from this branch's head (0362433) against real nltk 3.10.3, with lighteval's other imports stubbed. I haven't run the full test suite, and I haven't looked at how a ValueError from compute is handled further up. Whether a bad reference should raise or be reported some other way is the maintainers' call; LarryHu0217 raised the aggregation side of that in #1353.

The check was run with an AI coding assistant (Claude Code). The script is short and I can paste it if useful.

[gyanu2507]

compute() now skips a gold whose bag of words is empty after normalization, and only raises if nothing usable is left. ["Paris", "???"] with helm_normalizer scores 1.0 again. Direct compute_one_item still errors on an empty gold, so a lone empty reference does not silently score 0.