Judge every producer's output with every reader on the machine: cmd/judges

[tannevaled]

html2pdf carries, in its nested corpus module, a harness that runs every PDF reader present on the machine over a set of PDFs and says whether they agree. It is generic — render, ops and gotex need exactly it — and this repository is where go-pdfkit is judged by implementations that are not its own. So it lands here as cmd/judges, with the marker-preserving Markdown writer it uses (internal/mdreport) and the two pdf.js node scripts under judges/.

Same flags (-pdfs, -out, -report, -results, -pdfium / PDFIUM_TEST, -nodedir, -timeout), same seven judges (qpdf --check; poppler as the reference via pdfinfo/pdftoppm/pdftotext; mutool; gs; pdfium_test with its UTF-32LE --txt; pdf.js under node; Quartz via sips, page 1, composited over white), same sampling (first/middle/last page at 96 dpi, 400 px thumbnails, grey threshold 48/255), same per-judge worst-Δ vs poppler and pairwise consensus, same JUDGES.md table with the ⚠n / ❌ conventions. The doc comments keep their reasoning.

Split along this repository's seams: a thin main.go over run(args, out, errOut) int, and every shell-out through one runCmd variable (plus lookPath), so the whole harness — every judge, every refusal, the hang — is exercised at 100% on a runner that has none of the binaries. The pure parts (UTF-32LE decoder, pdfium progress filter, grey-thumbnail differ, consensus score, report writer, mdreport) are unit-tested directly.

Changed on purpose

• -pdfs is required rather than defaulting to html2pdf's out/*.pdf,out/bench/*.pdf layout.
• The report heading reads Judged by every reader on this machine — date rather than naming html2pdf.
• A judge that passes -timeout is reported as hung: <tool> did not finish within 3m0s, by tool, rather than as signal: killed or the first warning it printed before it was killed — the deadline is read off the context, as internal/poppler does.
• A results JSON that could not be written is an error (exit 1) rather than a silence.
• One line on stdout at the end naming the report and results paths.
• poppler's ms was 0 on every multi-page document (the re-render on the full sample replaced the verdict without timing it); it is now the time of both runs.
• judges/package.json loses npm-init noise (main, a failing test script, empty keywords/author) and carries the repository's licence; the lock is consistent and npm ci accepts it.

Proof: run over html2pdf's corpus/out/*.pdf, all seven judges present, 41 s

Judges: qpdf, poppler, mupdf, gs, pdfium, pdfjs, quartz. Versions: poppler pdftoppm version 26.04.0; mupdf mutool version 1.28.3; gs 10.07.1; qpdf qpdf version 12.4.1; pdf.js 6.3.289; macOS 26.6.2.

| PDF | Bytes | qpdf | poppler | mupdf | gs | pdfium | pdfjs | quartz | consensus |
|---|---|---|---|---|---|---|---|---|---|
| out/en-wikipedia-org-wiki-Go_programming_language.pdf | 251 KB | ✅ – · – · – | ✅ 13p · 1.000 · ref | ✅ 13p · 1.000 · Δ4.0% (p1) | ✅ – · 1.000 · Δ11.9% (p1) | ✅ 13p · 1.000 · Δ6.2% (p1) | ✅ 13p · 1.000 · Δ2.6% (p1) | ✅ – · – · Δ14.3% (p1 only) | 8.0% (p1) |
| out/en-wikipedia-org-wiki-List_of_countries_by_population_United_Nations.pdf | 126 KB | ✅ – · – · – | ✅ 7p · 1.000 · ref | ✅ 7p · 1.000 · Δ1.9% (p4) | ✅ – · 1.000 · Δ6.2% (p1) | ✅ 7p · 1.000 · Δ3.5% (p1) | ✅ 7p · 1.000 · Δ1.3% (p1) | ✅ – · – · Δ9.0% (p1 only) | 4.8% (p1) |
| out/example-com.pdf | 7 KB | ✅ – · – · – | ✅ 1p · 1.000 · ref | ✅ 1p · 1.000 · Δ0.0% (p1) | ✅ – · 1.000 · Δ0.2% (p1) | ✅ 1p · 1.000 · Δ0.1% (p1) | ✅ 1p · 1.000 · Δ0.0% (p1) | ✅ – · – · Δ0.2% (p1 only) | 0.1% (p1) |
| out/go-dev-blog-subtests.pdf | 61 KB | ✅ – · – · – | ✅ 5p · 1.000 · ref | ✅ 5p · 1.002 · Δ1.3% (p3) | ✅ – · 1.014 · Δ5.1% (p3) | ✅ 5p · 1.014 · Δ2.7% (p3) | ✅ 5p · 1.002 · Δ0.7% (p3) | ✅ – · – · Δ7.2% (p1 only) | 3.6% (p1) |

The figures match html2pdf's own corpus/JUDGES.md for the same documents (e.g. mupdf Δ4.0% p1, gs Δ11.9% p1 on the Go article), and pdfium — which that report lacked — is judged alongside.

🤖 Generated with Claude Code