Fix within-document blindness in exact matcher and self-matches in embedding search

[ishankgp]

Measured all three duplicate-detection methods against each other on the sample corpus (5 docs, 1,590 sentences). The comparison surfaced two defects, both fixed here, plus a full writeup in docs/FINDINGS.md.

Before / after

Metric	Before	After
Exact pairs	21	143
— within-document	0	122
SimHash pairs (h≤8)	186	186 (unchanged)
Embedding pairs (cos≥0.88)	389	277
Embedding self-match contamination	28.8%	0.0%
Exact coverage of SimHash	11%	77%

Union across methods stays at 289 unique pairs — the fixes removed false positives and surfaced pairs already reachable via SimHash, rather than inventing new ones.

Defect 1 — exact matcher blind to within-document duplicates

It grouped identical sentences, then dropped any group confined to one document before pairing only across documents. 85% under-report: 21 against an actual 143.

The "within-document" label computed for exact pairs was unreachable by construction — dead code indicating the capability was intended.

Defect 2 — embeddings matched sentences to themselves

A self-match guard existed:

for j_idx, sim in zip(I[i][1:], D[i][1:]):   # slice off the top hit

Slicing I[i][0] removes the self-hit only when it ranks first. With byte-identical sentences the tied cosines of 1.0 come back in arbitrary order, so the self-hit often landed at rank ≥1 and survived. 112 of 389 pairs were a sentence paired with itself, which also inflated matched_sentences_pct in doc_metrics.csv.

This is the harder class of bug: correct-looking code that degrades precisely on the input the tool exists to handle.

Fixed by excluding the self-index explicitly rather than positionally.

Other findings in docs/FINDINGS.md

• Parsing dominates latency. read_docx_text is 88% of runtime (7.16s of 8.14s). SimHash costs 902× more per sentence than exact and it still doesn't matter. The optimisation target is the parser.
• The Hamming knob is nearly inert. h=0→2 adds zero pairs; the whole shipped strict→moderate band moves results 6%; the real cliff (4.6×) is at h=20, outside any range a user would try. 82.8% of hits sit at h=0.
• 62.5% of "duplication" is a table artifact. Tables are 46.5% of extracted characters. Strip them and the duplicate rate drops from 9.09% to 5.95%. Not fixed here — listed as open work.
• Embeddings are the only method finding true paraphrase (103 unique pairs), at 9.5× runtime, with a false-positive mode on numeric table rows.
• Four documentation/code mismatches in Algorithms.md.

Limitations

No ground-truth labels exist, so every figure is yield, overlap, or cost — no precision or recall against human judgement is claimed. "Unique contribution" measures novelty, not correctness. Single corpus, n=5, one domain, heavily table-laden. Timing is single-run, no repeats.

The defect findings don't depend on any of that — they're properties of the code.

Verification

python corpus_dedup_runner.py --input_dir docs --out_dir out --use_embeddings

SimHash counts are unchanged at 186/175, confirming the diff is isolated to the two intended paths.

🤖 Generated with Claude Code