diff --git a/README.md b/README.md index 1b2b834..1ce7d04 100644 --- a/README.md +++ b/README.md @@ -91,6 +91,92 @@ carry one, and all 22 are soft masks. A filter with nothing comparable left is not reported as the worst thing in the corpus: nothing to compare is not evidence of being wrong. +## Every reader, not one: `judges` + +`compare` and `images` ask poppler how well we read the world's files. `judges` +asks the other direction — how well the world reads **ours** — and asks +**everyone the machine has**. A PDF that is the smallest, or that our own reader +reads back perfectly, is worth nothing if one reader in the field disagrees with +the others about it. That is a claim every producer in `go-pdfkit` — +`html2pdf`, `render`, `ops`, `gotex` — has to be able to make, which is why the +harness lives here and not in the producer that first needed it. + +``` +judges -pdfs 'out/*.pdf,out/bench/*.pdf' -pdfium /path/to/pdfium_test \ + -out out/judges -report JUDGES.md -results judges.json +``` + +| judge | what it is | gives | +|---|---|---| +| `qpdf --check` | structural validator | clean / warnings / errors | +| poppler (`pdfinfo` / `pdftoppm` / `pdftotext`) | the reference the per-judge Δ is taken against | pages, text, renders | +| MuPDF (`mutool`) | independent parser and rasteriser | pages, text, renders | +| Ghostscript (`gs`) | PostScript-lineage interpreter | text, renders | +| pdfium (`pdfium_test`) | Chrome's engine — `-pdfium /path` or `PDFIUM_TEST` | pages, text, renders | +| pdf.js (`judges/pdfjs-*.mjs` under node) | Firefox's engine | pages, text, renders | +| Quartz (`sips`) | macOS ImageIO — Preview's engine | page-1 render | + +A judge whose binary is absent is **skipped with a note, never faked**. Each +cell reads `pages · text ratio · Δworst (page)`: the pages the judge reports, +its extracted text as a ratio of poppler's, and the largest distance of its +renders from poppler's over a sample of pages — the first, the middle and the +last, at 96 dpi, as the share of pixels whose grey level moves by more than 48 +of 255 after both are box-downsampled to 400 px wide — with the page it +happened on. `⚠n` is n lines the judge complained on; `❌` is a judge that +would not process the file, with the first thing it said. + +Four things are decided rather than defaulted. + +**Poppler is the reference, and the `consensus` column is why it is not the +truth.** A distance needs a second point and poppler is the reader every +machine this runs on has; but poppler is one reader, so each sampled page also +carries the mean pairwise distance between *all* judges' renders of it, no +reader privileged. A page every reader draws differently is a page to look at, +whichever one is "right". + +**Text is counted without its whitespace.** Judges disagree wildly on it for +reasons that are not about the file — Ghostscript's `txtwrite` pads lines to +reproduce the column layout, pdfium separates every glyph run — while the +glyphs they *recover* are what the comparison is about. And pdfium's `--txt` is +UTF-32LE with a byte-order mark, four bytes a character, verified with `xxd`: +it is decoded before it is counted, or it reads as four times the text. + +**Quartz is composited over white before it is compared.** `sips` renders a +page on a transparent background, and a transparent pixel converted straight to +grey is black: every Quartz render would read as a 99% mismatch against an +opaque one. `sips` has no page selection either, so Quartz is judged on page 1 +only and its cell says so. + +**A judge's narration is not a warning.** `pdfium_test` says "Processing PDF +file x." and "Processed N pages." on stderr as it goes; those lines are +progress, and are struck before the rest is counted. + +Judge a **control** beside your own output. `html2pdf` judges Chrome's PDFs of +the same pages alongside its own: a judge that disagrees on the control too is +judge noise, and one that disagrees only on ours is a defect. The table is +written above an `` marker, and whatever a reader writes +beneath it — which cells were noise, which were defects, where they were fixed +— survives the next run. + +Every judge runs under `-timeout` (three minutes per judge per file), and one +that does not answer is reported as `hung`, by tool, rather than as the first +warning it printed before it was killed. The reason is the next section's. + +pdf.js is two node scripts under [`judges/`](judges/); they are not Go, and CI +does not build them. Once, on the machine that judges: + +``` +cd judges && npm ci # pdfjs-dist + @napi-rs/canvas, from the lock +``` + +`-nodedir judges` (the default) then finds them, and a machine without them +simply has no pdf.js column. From another repository: + +``` +go run github.com/go-pdfkit/conformance/cmd/judges@latest \ + -pdfs 'out/*.pdf' -nodedir /path/to/conformance/judges -pdfium "$PDFIUM_TEST" +``` + ## The judge can hang, and a hang looks like a slow run `pdfimages -list` **does not return** on @@ -590,4 +676,7 @@ bisection at v0.19.0 or from the chroma defect at v0.20.0. ## How it is checked Exact 100% statement coverage including every error branch, `go vet`, `-race`, -and nine cross-compile targets. Nothing outside the standard library. +and nine cross-compile targets. Nothing outside the standard library. The +readers `judges` shells out to are stood in for under test, so the whole +harness — every judge, every refusal, the hang — is exercised on a runner +that has none of them. diff --git a/cmd/judges/judges.go b/cmd/judges/judges.go new file mode 100644 index 0000000..87a4175 --- /dev/null +++ b/cmd/judges/judges.go @@ -0,0 +1,408 @@ +package main + +import ( + "bytes" + "context" + "errors" + "fmt" + "os" + "os/exec" + "path/filepath" + "regexp" + "sort" + "strconv" + "strings" + "time" +) + +const ( + renderDPI = 96 + pdfiumScale = "1.3333" // 96/72, so pdfium's render matches the others' size + thumbWidth = 400 // renders are compared after downsampling to this width + diffThresh = 48 // a pixel "differs" when its grey level moves by more than this, of 255 +) + +// pageDiff is one judge's render of one page measured against poppler's. +type pageDiff struct { + MeanDiff float64 `json:"mean_diff"` // mean grey Δ, 0..255 + DiffPct float64 `json:"diff_pct"` // share of pixels with |Δ| > diffThresh, in % +} + +// verdict is one judge's reading of one PDF. +type verdict struct { + Judge string `json:"judge"` + Skipped bool `json:"skipped,omitempty"` + OK bool `json:"ok"` + Err string `json:"err,omitempty"` + Warnings int `json:"warnings"` + Pages int `json:"pages"` // 0 when the judge reports none + TextChar int `json:"text_chars"` // -1 when the judge extracts none + Renders map[int]string `json:"renders,omitempty"` + Diffs map[int]pageDiff `json:"diffs,omitempty"` // vs poppler, per sampled page + WorstPct float64 `json:"worst_pct"` + WorstPage int `json:"worst_page"` + Ms int64 `json:"ms"` +} + +// fileResult is every judge's reading of one PDF, and how far they are from +// one another. +type fileResult struct { + File string `json:"file"` + Bytes int64 `json:"bytes"` + SampledPages []int `json:"sampled_pages"` + Verdicts []verdict `json:"verdicts"` + Consensus map[int]float64 `json:"consensus"` // per page: mean pairwise diff% among all judges' renders + ConsensusMax float64 `json:"consensus_max"` + ConsensusPg int `json:"consensus_page"` +} + +// A judge is one reader: whether this machine has it, and how to ask it. +type judge struct { + name string + avail func() bool + run func(ctx context.Context, pdf, outDir string, pages []int) verdict +} + +var ( + pdfiumBin string // pdfium_test, Chrome's engine; empty when it is not built + nodeDir string // directory holding pdfjs-*.mjs and node_modules + judgeTimeout time.Duration // how long one judge may take on one file +) + +// lookPath is a variable so a test can decide which readers this machine has. +var lookPath = exec.LookPath + +func have(bin string) bool { _, err := lookPath(bin); return err == nil } + +// runCmd runs one tool under the judge's deadline and returns what it said on +// each stream. It is a variable so every judge can be exercised on a machine +// that has none of their binaries — the CI runner is one. +// +// It does not go through internal/poppler, because it needs what that +// deliberately leaves out: stderr on its own, since a judge's warning count is +// the number of lines it complained on; and a working directory, since +// pdfium_test writes beside its input and pdf.js resolves node_modules from +// where it is run. What it keeps is poppler.Run's one rule — a deadline that +// passed is read off the CONTEXT and not off the error, because a killed +// process reports a signal, and "signal: killed" in a report cell is a hang +// nobody can tell from a crash. +var runCmd = func(ctx context.Context, dir string, name string, args ...string) (stdout, stderr string, err error) { + cmd := exec.CommandContext(ctx, name, args...) + cmd.Dir = dir + var o, e bytes.Buffer + cmd.Stdout, cmd.Stderr = &o, &e + err = cmd.Run() + if err != nil && ctx.Err() == context.DeadlineExceeded { + err = &hang{tool: name} + } + return o.String(), e.String(), err +} + +// A hang is a tool that did not answer within the judge's bound. +type hang struct{ tool string } + +func (h *hang) Error() string { + return "hung: " + h.tool + " did not finish within " + judgeTimeout.String() +} + +func countLines(s string) int { + s = strings.TrimSpace(s) + if s == "" { + return 0 + } + return strings.Count(s, "\n") + 1 +} + +// textLen counts the non-whitespace runes of an extraction. Whitespace is +// left out because judges disagree wildly on it for reasons that are not +// about the file — Ghostscript's txtwrite pads lines to reproduce column +// layout, pdfium separates every glyph run — while the glyphs they recover +// are what the comparison is about. +func textLen(s string) int { + n := 0 + for _, r := range s { + if r > ' ' { + n++ + } + } + return n +} + +var rePages = regexp.MustCompile(`(?mi)^Pages:\s+(\d+)`) + +func pagesFrom(s string) int { + if m := rePages.FindStringSubmatch(s); m != nil { + n, _ := strconv.Atoi(m[1]) + return n + } + return 0 +} + +// samplePages picks the pages whose renders are compared: the first, the +// middle and the last. Text and page counts cover every page regardless; +// this bounds the render work while still looking past page 1. +func samplePages(n int) []int { + if n <= 1 { + return []int{1} + } + set := map[int]bool{1: true, (n + 1) / 2: true, n: true} + var out []int + for p := range set { + out = append(out, p) + } + sort.Ints(out) + return out +} + +// exitCode is the status a tool ended with, or -1 when it did not end with +// one. +func exitCode(err error) int { + var ee *exec.ExitError + if errors.As(err, &ee) { + return ee.ExitCode() + } + return -1 +} + +// ---- judges ------------------------------------------------------------- + +func judgeQpdf(ctx context.Context, pdf, outDir string, _ []int) verdict { + v := verdict{Judge: "qpdf", TextChar: -1} + out, errs, err := runCmd(ctx, "", "qpdf", "--check", pdf) + all := out + errs + v.Warnings = strings.Count(all, "WARNING") + switch { + case err == nil: + v.OK = true + case exitCode(err) == 3: + v.OK = true // qpdf's "warnings only" + default: + v.Err = firstLine(all, err) + } + return v +} + +// judgePoppler is the reference. It is not privileged as the truth — the +// consensus column exists so that it is not — but every other judge's Δ is +// taken against it, because a distance needs a second point and poppler is +// the reader that is on every machine this runs on. +func judgePoppler(ctx context.Context, pdf, outDir string, pages []int) verdict { + v := verdict{Judge: "poppler", Renders: map[int]string{}} + info, e1, err := runCmd(ctx, "", "pdfinfo", pdf) + if err != nil { + v.Err = firstLine(e1, err) + return v + } + v.Pages = pagesFrom(info) + txt, e2, err := runCmd(ctx, "", "pdftotext", pdf, "-") + if err != nil { + v.Err = firstLine(e2, err) + return v + } + v.TextChar = textLen(txt) + v.Warnings = countLines(e1) + countLines(e2) + for _, p := range pages { + base := filepath.Join(outDir, fmt.Sprintf("poppler_p%d", p)) + ps := strconv.Itoa(p) + _, e3, err := runCmd(ctx, "", "pdftoppm", "-png", "-r", strconv.Itoa(renderDPI), "-f", ps, "-l", ps, "-singlefile", pdf, base) + if err != nil { + v.Err = firstLine(e3, err) + return v + } + v.Renders[p] = base + ".png" + v.Warnings += countLines(e3) + } + v.OK = true + return v +} + +func judgeMupdf(ctx context.Context, pdf, outDir string, pages []int) verdict { + v := verdict{Judge: "mupdf", Renders: map[int]string{}} + info, e0, _ := runCmd(ctx, "", "mutool", "info", pdf) + v.Pages = pagesFrom(info) + txt, e1, err := runCmd(ctx, "", "mutool", "draw", "-q", "-F", "txt", "-o", "-", pdf) + if err != nil { + v.Err = firstLine(e1, err) + return v + } + v.TextChar = textLen(txt) + v.Warnings = countLines(e0) + countLines(e1) + for _, p := range pages { + out := filepath.Join(outDir, fmt.Sprintf("mupdf_p%d.png", p)) + _, e2, err := runCmd(ctx, "", "mutool", "draw", "-q", "-r", strconv.Itoa(renderDPI), "-o", out, pdf, strconv.Itoa(p)) + if err != nil { + v.Err = firstLine(e2, err) + return v + } + v.Renders[p] = out + v.Warnings += countLines(e2) + } + v.OK = true + return v +} + +// judgeGs reports no page count: Ghostscript has no "info" verb, and a count +// read off txtwrite's output would be a count of what it chose to emit. +func judgeGs(ctx context.Context, pdf, outDir string, pages []int) verdict { + v := verdict{Judge: "gs", Renders: map[int]string{}} + txtFile := filepath.Join(outDir, "gs.txt") + o1, e1, err := runCmd(ctx, "", "gs", "-q", "-dNOPAUSE", "-dBATCH", "-dSAFER", "-sDEVICE=txtwrite", "-sOutputFile="+txtFile, pdf) + if err != nil { + v.Err = firstLine(o1+e1, err) + return v + } + if b, err := os.ReadFile(txtFile); err == nil { + v.TextChar = textLen(string(b)) + } + v.Warnings = countLines(o1 + e1) + for _, p := range pages { + out := filepath.Join(outDir, fmt.Sprintf("gs_p%d.png", p)) + ps := strconv.Itoa(p) + o2, e2, err := runCmd(ctx, "", "gs", "-q", "-dNOPAUSE", "-dBATCH", "-dSAFER", "-sDEVICE=png16m", "-r"+strconv.Itoa(renderDPI), + "-dFirstPage="+ps, "-dLastPage="+ps, "-sOutputFile="+out, pdf) + if err != nil { + v.Err = firstLine(o2+e2, err) + return v + } + v.Renders[p] = out + v.Warnings += countLines(o2 + e2) + } + v.OK = true + return v +} + +// pdfiumNoise is what pdfium_test narrates on stderr as it goes ("Processing +// PDF file x.", "Processed N pages.") — progress, not warnings; only anything +// else counts as one. +var pdfiumNoise = regexp.MustCompile(`(?m)^(Processing PDF file .*|Processed \d+ pages\.)\n?`) + +func judgePdfium(ctx context.Context, pdf, outDir string, pages []int) verdict { + v := verdict{Judge: "pdfium", Renders: map[int]string{}} + // pdfium_test writes ..png / .txt beside the input; work on a + // copy in outDir so the corpus tree stays clean. + work := filepath.Join(outDir, "pdfium.pdf") + b, err := os.ReadFile(pdf) + if err != nil { + v.Err = err.Error() + return v + } + if err := os.WriteFile(work, b, 0o644); err != nil { + v.Err = err.Error() + return v + } + defer os.Remove(work) + o1, e1, err := runCmd(ctx, outDir, pdfiumBin, "--txt", "pdfium.pdf") + if err != nil { + v.Err = firstLine(o1+e1, err) + return v + } + txts, _ := filepath.Glob(filepath.Join(outDir, "pdfium.pdf.*.txt")) + v.Pages = len(txts) + total := 0 + for _, t := range txts { + if tb, err := os.ReadFile(t); err == nil { + total += textLen(utf32leToString(tb)) + } + os.Remove(t) + } + v.TextChar = total + v.Warnings = countLines(pdfiumNoise.ReplaceAllString(e1, "")) + for _, p := range pages { + o2, e2, err := runCmd(ctx, outDir, pdfiumBin, "--png", "--scale="+pdfiumScale, "--pages="+strconv.Itoa(p-1), "pdfium.pdf") + if err != nil { + v.Err = firstLine(o2+e2, err) + return v + } + src := filepath.Join(outDir, fmt.Sprintf("pdfium.pdf.%d.png", p-1)) + out := filepath.Join(outDir, fmt.Sprintf("pdfium_p%d.png", p)) + if err := os.Rename(src, out); err != nil { + v.Err = "no render for page " + strconv.Itoa(p) + return v + } + v.Renders[p] = out + v.Warnings += countLines(pdfiumNoise.ReplaceAllString(e2, "")) + } + v.OK = true + return v +} + +// utf32leToString decodes pdfium_test's --txt output — UTF-32LE with a +// byte-order mark (FF FE 00 00; verified with xxd, four bytes per character) +// — so its length is counted in characters like every other judge's, not in +// bytes, which would read as four times the text. +func utf32leToString(b []byte) string { + if len(b) >= 4 && b[0] == 0xFF && b[1] == 0xFE && b[2] == 0 && b[3] == 0 { + b = b[4:] + } + r := make([]rune, 0, len(b)/4) + for i := 0; i+3 < len(b); i += 4 { + r = append(r, rune(uint32(b[i])|uint32(b[i+1])<<8|uint32(b[i+2])<<16|uint32(b[i+3])<<24)) + } + return string(r) +} + +var rePdfjsPages = regexp.MustCompile(`(?m)^pages (\d+)`) + +func judgePdfjs(ctx context.Context, pdf, outDir string, pages []int) verdict { + v := verdict{Judge: "pdfjs", Renders: map[int]string{}} + out, e1, err := runCmd(ctx, nodeDir, "node", "pdfjs-text.mjs", pdf) + if err != nil { + v.Err = firstLine(e1, err) + return v + } + if m := rePdfjsPages.FindStringSubmatch(out); m != nil { + v.Pages, _ = strconv.Atoi(m[1]) + out = out[len(m[0]):] + } + v.TextChar = textLen(out) + v.Warnings = countLines(e1) + for _, p := range pages { + render := filepath.Join(outDir, fmt.Sprintf("pdfjs_p%d.png", p)) + _, e2, err := runCmd(ctx, nodeDir, "node", "pdfjs-render.mjs", pdf, render, strconv.Itoa(p), pdfiumScale) + if err != nil { + v.Err = firstLine(e2, err) + return v + } + v.Renders[p] = render + v.Warnings += countLines(e2) + } + v.OK = true + return v +} + +// judgeQuartz renders page 1 only: sips has no page selection. Its render is +// on a transparent background, which is why greyRows composites over white. +func judgeQuartz(ctx context.Context, pdf, outDir string, _ []int) verdict { + v := verdict{Judge: "quartz", TextChar: -1, Renders: map[int]string{}} + out := filepath.Join(outDir, "quartz_p1.png") + o, e, err := runCmd(ctx, "", "sips", "-s", "format", "png", pdf, "--out", out) + if err != nil { + v.Err = firstLine(o+e, err) + return v + } + v.Renders[1] = out + v.Warnings = strings.Count(o+e, "Error") + strings.Count(o+e, "Warning") + v.OK = true + return v +} + +// firstLine is what a failed judge's cell reads: the first thing the tool +// said, or the error when it said nothing. +func firstLine(s string, err error) string { + // A hang is said as a hang, whatever the tool managed to print before it + // was killed: the stderr of a killed pdftoppm is its warnings so far, and + // the first of those is not why the cell is red. + var h *hang + if errors.As(err, &h) { + return err.Error() + } + for _, l := range strings.Split(strings.TrimSpace(s), "\n") { + if l = strings.TrimSpace(l); l != "" { + if len(l) > 120 { + l = l[:120] + "…" + } + return l + } + } + return err.Error() +} diff --git a/cmd/judges/judges_test.go b/cmd/judges/judges_test.go new file mode 100644 index 0000000..1484a15 --- /dev/null +++ b/cmd/judges/judges_test.go @@ -0,0 +1,498 @@ +package main + +import ( + "context" + "errors" + "fmt" + "image" + "image/color" + "image/png" + "os" + "os/exec" + "path/filepath" + "strconv" + "strings" + "testing" + "time" +) + +// stubTools stands every binary in: lookPath answers for the names in +// present, and runCmd hands each invocation to h instead of a process, so the +// harness runs on a machine that has none of them — the CI runner is one. +func stubTools(t *testing.T, present []string, h func(dir, name string, args []string) (string, string, error)) { + t.Helper() + wasRun, wasLook := runCmd, lookPath + t.Cleanup(func() { runCmd, lookPath = wasRun, wasLook }) + runCmd = func(_ context.Context, dir, name string, args ...string) (string, string, error) { + return h(dir, name, args) + } + lookPath = func(bin string) (string, error) { + for _, p := range present { + if p == bin { + return "/usr/bin/" + bin, nil + } + } + return "", errors.New("not found") + } +} + +// every reader the harness knows. +var everyReader = []string{"qpdf", "pdfinfo", "pdftoppm", "pdftotext", "mutool", "gs", "node", "sips"} + +// writePNG writes a w×h page, white with a black box, so two judges that +// draw the box in different places differ by a share that can be predicted. +func writePNG(t *testing.T, path string, w, h int, box image.Rectangle) { + t.Helper() + img := image.NewGray(image.Rect(0, 0, w, h)) + for y := 0; y < h; y++ { + for x := 0; x < w; x++ { + c := uint8(255) + if image.Pt(x, y).In(box) { + c = 0 + } + img.SetGray(x, y, color.Gray{Y: c}) + } + } + f, err := os.Create(path) + if err != nil { + t.Fatal(err) + } + if err := png.Encode(f, img); err != nil { + t.Fatal(err) + } + f.Close() +} + +// exitErr is a genuine *exec.ExitError carrying code, since one cannot be +// built by hand and the qpdf judge reads the code off it. +func exitErr(t *testing.T, code int) error { + t.Helper() + err := exec.Command("sh", "-c", "exit "+strconv.Itoa(code)).Run() + if err == nil { + t.Fatalf("sh -c 'exit %d' succeeded", code) + } + return err +} + +// utf32le encodes s the way pdfium_test writes its --txt. +func utf32le(s string, bom bool) []byte { + var b []byte + if bom { + b = append(b, 0xFF, 0xFE, 0, 0) + } + for _, r := range s { + b = append(b, byte(r), byte(r>>8), byte(r>>16), byte(r>>24)) + } + return b +} + +func argAfter(args []string, flag string) string { + for i, a := range args { + if a == flag && i+1 < len(args) { + return args[i+1] + } + } + return "" +} + +func argWithPrefix(args []string, prefix string) string { + for _, a := range args { + if strings.HasPrefix(a, prefix) { + return strings.TrimPrefix(a, prefix) + } + } + return "" +} + +// key names an invocation the way the fakes are keyed: the tool, and the verb +// that tells its uses apart. +func key(name string, args []string) string { + base := filepath.Base(name) + switch { + case base == "mutool" && args[0] == "draw": + if argAfter(args, "-F") == "txt" { + return "mutool draw txt" + } + return "mutool draw png" + case base == "gs" && args[0] != "--version": + if argWithPrefix(args, "-sDEVICE=") == "txtwrite" { + return "gs txt" + } + return "gs png" + case base == "node", base == "mutool", strings.HasPrefix(args[0], "-"): + return base + " " + args[0] + } + return base +} + +// aMachine emulates every reader over an n-page document whose text is +// "hello world" on each page. fail names the invocations that refuse, by +// key. Each judge puts its ink in its own place so their renders differ. +func aMachine(t *testing.T, n int, fail map[string]error) func(dir, name string, args []string) (string, string, error) { + t.Helper() + box := func(judge string) image.Rectangle { + off := map[string]int{"poppler": 0, "mupdf": 2, "gs": 20, "pdfium": 4, "pdfjs": 1, "quartz": 40}[judge] + return image.Rect(off, off, off+30, off+30) + } + return func(dir, name string, args []string) (string, string, error) { + k := key(name, args) + if err, ok := fail[k]; ok { + return "", "refused\n", err + } + switch k { + case "qpdf --check": + return "", "", nil + case "qpdf --version": + return "qpdf version 12.4.1\n", "", nil + case "pdfinfo": + return fmt.Sprintf("Title: x\nPages: %d\n", n), "Syntax Warning: something\n", nil + case "pdftotext": + return "hello world\n", "", nil + case "pdftoppm -png": + writePNG(t, args[len(args)-1]+".png", 80, 100, box("poppler")) + return "", "", nil + case "pdftoppm -v": + return "", "pdftoppm version 26.04.0\nCopyright 2005-2026 The Poppler Developers\n", nil + case "mutool info": + return fmt.Sprintf("Pages: %d\n", n), "", nil + case "mutool draw txt": + return "hello world\n", "", nil + case "mutool draw png": + writePNG(t, argAfter(args, "-o"), 80, 100, box("mupdf")) + return "", "", nil + case "mutool -v": + return "", "mutool version 1.28.3\n", nil + case "gs txt": + // txtwrite pads its lines to the column layout; the count must not + // see that. + os.WriteFile(argWithPrefix(args, "-sOutputFile="), []byte("hello world \n"), 0o644) + return "", "", nil + case "gs png": + writePNG(t, argWithPrefix(args, "-sOutputFile="), 80, 100, box("gs")) + return "", "", nil + case "gs --version": + return "10.07.1\n", "", nil + case "pdfium_test --txt": + for i := 0; i < n; i++ { + os.WriteFile(filepath.Join(dir, fmt.Sprintf("pdfium.pdf.%d.txt", i)), utf32le("hello world", true), 0o644) + } + return "", fmt.Sprintf("Processing PDF file pdfium.pdf.\nProcessed %d pages.\n", n), nil + case "pdfium_test --png": + writePNG(t, filepath.Join(dir, "pdfium.pdf."+argWithPrefix(args, "--pages=")+".png"), 80, 100, box("pdfium")) + return "", "Processing PDF file pdfium.pdf.\nProcessed 1 pages.\n", nil + case "node pdfjs-text.mjs": + return fmt.Sprintf("pages %d\nhello world\n", n), "", nil + case "node pdfjs-render.mjs": + writePNG(t, args[2], 80, 100, box("pdfjs")) + return fmt.Sprintf("pages %d\n", n), "", nil + case "sips -s": + writePNG(t, argAfter(args, "--out"), 80, 100, box("quartz")) + return args[3] + "\n " + argAfter(args, "--out") + "\n", "", nil + case "sw_vers -productVersion": + return "26.6.2\n", "", nil + } + t.Fatalf("an invocation nothing expected: %s %v", name, args) + return "", "", nil + } +} + +// aVerdict runs one judge over a two-page document under the machine above. +func aVerdict(t *testing.T, j func(context.Context, string, string, []int) verdict, fail map[string]error) verdict { + t.Helper() + stubTools(t, everyReader, aMachine(t, 2, fail)) + dir := t.TempDir() + pdf := filepath.Join(dir, "doc.pdf") + os.WriteFile(pdf, []byte("%PDF-1.7\n"), 0o644) + pdfiumBin = filepath.Join(dir, "pdfium_test") + nodeDir = dir + return j(context.Background(), pdf, dir, []int{1, 2}) +} + +func TestQpdfReadsItsExitCode(t *testing.T) { + // 0 is clean, 3 is warnings only — which is a pass with a count beside + // it — and anything else is a refusal. + if v := aVerdict(t, judgeQpdf, nil); !v.OK || v.Err != "" || v.TextChar != -1 { + t.Errorf("clean: %+v", v) + } + if v := aVerdict(t, judgeQpdf, map[string]error{"qpdf --check": exitErr(t, 3)}); !v.OK { + t.Errorf("warnings only: %+v", v) + } + v := aVerdict(t, judgeQpdf, map[string]error{"qpdf --check": exitErr(t, 2)}) + if v.OK || v.Err != "refused" { + t.Errorf("errors: %+v", v) + } +} + +func TestQpdfCountsItsWarnings(t *testing.T) { + stubTools(t, everyReader, func(string, string, []string) (string, string, error) { + return "WARNING: a\nWARNING: b\n", "", nil + }) + if v := judgeQpdf(context.Background(), "x.pdf", "", nil); !v.OK || v.Warnings != 2 { + t.Errorf("%+v", v) + } +} + +func TestPopplerIsTheReference(t *testing.T) { + v := aVerdict(t, judgePoppler, nil) + if !v.OK || v.Pages != 2 || v.TextChar != 10 || len(v.Renders) != 2 { + t.Fatalf("%+v", v) + } + if !strings.HasSuffix(v.Renders[2], "poppler_p2.png") { + t.Errorf("page 2 is at %q", v.Renders[2]) + } + // pdfinfo complained on one line. + if v.Warnings != 1 { + t.Errorf("%d warnings", v.Warnings) + } + for _, k := range []string{"pdfinfo", "pdftotext", "pdftoppm -png"} { + v := aVerdict(t, judgePoppler, map[string]error{k: errors.New("boom")}) + if v.OK || v.Err != "refused" { + t.Errorf("%s failing: %+v", k, v) + } + } +} + +func TestMupdfSurvivesAnInfoThatFails(t *testing.T) { + v := aVerdict(t, judgeMupdf, nil) + if !v.OK || v.Pages != 2 || v.TextChar != 10 || len(v.Renders) != 2 || v.Warnings != 0 { + t.Fatalf("%+v", v) + } + // mutool info refusing costs the page count and a warning, not the + // verdict: the text and the renders are what the comparison needs. + v = aVerdict(t, judgeMupdf, map[string]error{"mutool info": errors.New("boom")}) + if !v.OK || v.Pages != 0 || v.Warnings != 1 { + t.Errorf("info failing: %+v", v) + } + for _, k := range []string{"mutool draw txt", "mutool draw png"} { + if v := aVerdict(t, judgeMupdf, map[string]error{k: errors.New("boom")}); v.OK || v.Err != "refused" { + t.Errorf("%s failing: %+v", k, v) + } + } +} + +func TestGhostscriptCountsGlyphsNotPadding(t *testing.T) { + v := aVerdict(t, judgeGs, nil) + if !v.OK || v.Pages != 0 || v.TextChar != 10 || len(v.Renders) != 2 { + t.Fatalf("%+v", v) + } + for _, k := range []string{"gs txt", "gs png"} { + if v := aVerdict(t, judgeGs, map[string]error{k: errors.New("boom")}); v.OK || v.Err != "refused" { + t.Errorf("%s failing: %+v", k, v) + } + } + // txtwrite that succeeds without writing anything is a text of nought, + // not a failure. + machine := aMachine(t, 2, nil) + stubTools(t, everyReader, func(dir, name string, args []string) (string, string, error) { + if key(name, args) == "gs txt" { + return "", "", nil + } + return machine(dir, name, args) + }) + if v := judgeGs(context.Background(), "x.pdf", t.TempDir(), []int{1}); !v.OK || v.TextChar != 0 { + t.Errorf("%+v", v) + } +} + +func TestPdfiumWorksOnACopyAndReadsUTF32(t *testing.T) { + v := aVerdict(t, judgePdfium, nil) + if !v.OK || v.Pages != 2 || v.TextChar != 20 || len(v.Renders) != 2 { + t.Fatalf("%+v", v) + } + // Its progress narration is not a warning. + if v.Warnings != 0 { + t.Errorf("%d warnings from progress lines", v.Warnings) + } + if !strings.HasSuffix(v.Renders[2], "pdfium_p2.png") { + t.Errorf("page 2 is at %q", v.Renders[2]) + } + for _, k := range []string{"pdfium_test --txt", "pdfium_test --png"} { + if v := aVerdict(t, judgePdfium, map[string]error{k: errors.New("boom")}); v.OK || v.Err != "refused" { + t.Errorf("%s failing: %+v", k, v) + } + } +} + +func TestPdfiumAnythingElseOnStderrIsAWarning(t *testing.T) { + machine := aMachine(t, 1, nil) + stubTools(t, everyReader, func(dir, name string, args []string) (string, string, error) { + o, e, err := machine(dir, name, args) + if strings.HasPrefix(key(name, args), "pdfium_test") { + e += "Warning: font not found\n" + } + return o, e, err + }) + dir := t.TempDir() + pdf := filepath.Join(dir, "doc.pdf") + os.WriteFile(pdf, []byte("%PDF"), 0o644) + pdfiumBin = filepath.Join(dir, "pdfium_test") + v := judgePdfium(context.Background(), pdf, dir, []int{1}) + if !v.OK || v.Warnings != 2 { + t.Errorf("%+v", v) + } +} + +func TestPdfiumSaysWhenItCannotWork(t *testing.T) { + stubTools(t, everyReader, aMachine(t, 1, nil)) + dir := t.TempDir() + pdfiumBin = filepath.Join(dir, "pdfium_test") + if v := judgePdfium(context.Background(), filepath.Join(dir, "absent.pdf"), dir, []int{1}); v.OK || v.Err == "" { + t.Errorf("an absent document: %+v", v) + } + pdf := filepath.Join(dir, "doc.pdf") + os.WriteFile(pdf, []byte("%PDF"), 0o644) + if v := judgePdfium(context.Background(), pdf, filepath.Join(pdf, "not-a-dir"), []int{1}); v.OK || v.Err == "" { + t.Errorf("an output directory that is a file: %+v", v) + } + // A page it said it drew but did not is named as missing. + machine := aMachine(t, 1, nil) + stubTools(t, everyReader, func(dir, name string, args []string) (string, string, error) { + if key(name, args) == "pdfium_test --png" { + return "", "", nil + } + return machine(dir, name, args) + }) + if v := judgePdfium(context.Background(), pdf, dir, []int{1}); v.OK || v.Err != "no render for page 1" { + t.Errorf("a render that was not written: %+v", v) + } + // A .txt it cannot read is left out of the count rather than failing it. + stubTools(t, everyReader, machine) + os.MkdirAll(filepath.Join(dir, "pdfium.pdf.9.txt"), 0o755) + if v := judgePdfium(context.Background(), pdf, dir, []int{1}); !v.OK || v.TextChar != 10 { + t.Errorf("an unreadable text: %+v", v) + } +} + +func TestUTF32LEIsCountedInCharacters(t *testing.T) { + // pdfium_test's --txt is UTF-32LE with a BOM: four bytes a character, + // which counted as bytes would read as four times the text. + if got := utf32leToString(utf32le("héllo", true)); got != "héllo" { + t.Errorf("with a BOM: %q", got) + } + if got := utf32leToString(utf32le("hi", false)); got != "hi" { + t.Errorf("without one: %q", got) + } + // A trailing partial character is not a character. + if got := utf32leToString(append(utf32le("a", true), 0x62, 0)); got != "a" { + t.Errorf("with a torn tail: %q", got) + } + if got := utf32leToString(nil); got != "" { + t.Errorf("empty: %q", got) + } +} + +func TestPdfjsReadsItsPageLine(t *testing.T) { + v := aVerdict(t, judgePdfjs, nil) + if !v.OK || v.Pages != 2 || v.TextChar != 10 || len(v.Renders) != 2 { + t.Fatalf("%+v", v) + } + for _, k := range []string{"node pdfjs-text.mjs", "node pdfjs-render.mjs"} { + if v := aVerdict(t, judgePdfjs, map[string]error{k: errors.New("boom")}); v.OK || v.Err != "refused" { + t.Errorf("%s failing: %+v", k, v) + } + } + // Without the page line, the text is all there is. + machine := aMachine(t, 2, nil) + stubTools(t, everyReader, func(dir, name string, args []string) (string, string, error) { + if key(name, args) == "node pdfjs-text.mjs" { + return "hello world\n", "", nil + } + return machine(dir, name, args) + }) + if v := judgePdfjs(context.Background(), "x.pdf", t.TempDir(), []int{1}); !v.OK || v.Pages != 0 || v.TextChar != 10 { + t.Errorf("%+v", v) + } +} + +func TestQuartzRendersPageOneOnly(t *testing.T) { + v := aVerdict(t, judgeQuartz, nil) + if !v.OK || v.TextChar != -1 || len(v.Renders) != 1 || v.Renders[1] == "" { + t.Fatalf("%+v", v) + } + if v := aVerdict(t, judgeQuartz, map[string]error{"sips -s": errors.New("boom")}); v.OK || v.Err != "refused" { + t.Errorf("failing: %+v", v) + } + stubTools(t, everyReader, func(_, _ string, args []string) (string, string, error) { + writePNG(t, argAfter(args, "--out"), 4, 4, image.Rect(0, 0, 1, 1)) + return "Warning: x\nError: y\n", "", nil + }) + if v := judgeQuartz(context.Background(), "x.pdf", t.TempDir(), nil); !v.OK || v.Warnings != 2 { + t.Errorf("%+v", v) + } +} + +func TestRunCmdKeepsTheStreamsApartAndHonoursTheDirectory(t *testing.T) { + dir := t.TempDir() + o, e, err := runCmd(context.Background(), dir, "sh", "-c", "pwd; echo err 1>&2; exit 3") + if exitCode(err) != 3 { + t.Fatalf("exit %d, %v", exitCode(err), err) + } + want, _ := filepath.EvalSymlinks(dir) + if got, _ := filepath.EvalSymlinks(strings.TrimSpace(o)); got != want { + t.Errorf("ran in %q, not %q", got, want) + } + if e != "err\n" { + t.Errorf("stderr %q", e) + } + if exitCode(errors.New("not a process")) != -1 { + t.Error("an error that is not an exit is given a code") + } +} + +func TestAHangIsSaidAsAHangWhateverTheToolPrinted(t *testing.T) { + // The deadline is read off the context, not the error: a killed process + // reports a signal, and "signal: killed" in a cell is a hang nobody can + // tell from a crash. And the tool's stderr so far — its warnings — is not + // why the cell is red. + was := judgeTimeout + defer func() { judgeTimeout = was }() + judgeTimeout = 50 * time.Millisecond + ctx, cancel := context.WithTimeout(context.Background(), judgeTimeout) + defer cancel() + _, _, err := runCmd(ctx, "", "sleep", "30") + var h *hang + if !errors.As(err, &h) { + t.Fatalf("a tool that does not return was not called a hang: %v", err) + } + got := firstLine("Syntax Warning: so far so good\n", err) + if got != "hung: sleep did not finish within 50ms" { + t.Errorf("a hang reads %q", got) + } +} + +func TestFirstLineIsTheFirstThingSaid(t *testing.T) { + if got := firstLine("\n \n the reason \nmore\n", errors.New("x")); got != "the reason" { + t.Errorf("%q", got) + } + long := strings.Repeat("y", 200) + if got := firstLine(long, nil); got != long[:120]+"…" { + t.Errorf("%q", got) + } + if got := firstLine(" \n", errors.New("exit status 1")); got != "exit status 1" { + t.Errorf("nothing said: %q", got) + } +} + +func TestSamplePagesIsFirstMiddleLast(t *testing.T) { + for n, want := range map[int][]int{0: {1}, 1: {1}, 2: {1, 2}, 3: {1, 2, 3}, 10: {1, 5, 10}, 11: {1, 6, 11}} { + if got := samplePages(n); fmt.Sprint(got) != fmt.Sprint(want) { + t.Errorf("%d pages: %v, want %v", n, got, want) + } + } +} + +func TestSmallHelpers(t *testing.T) { + if pagesFrom("Title: x\nPages: 12\n") != 12 || pagesFrom("pages: 3") != 3 || pagesFrom("Pages: many") != 0 { + t.Error("pagesFrom") + } + if countLines("") != 0 || countLines(" \n") != 0 || countLines("a\nb\n") != 2 { + t.Error("countLines") + } + if textLen("hello world\n\tà") != 11 { + t.Errorf("textLen %d", textLen("hello world\n\tà")) + } + stubTools(t, []string{"qpdf"}, nil) + if !have("qpdf") || have("mutool") { + t.Error("have") + } +} diff --git a/cmd/judges/main.go b/cmd/judges/main.go new file mode 100644 index 0000000..a58666a --- /dev/null +++ b/cmd/judges/main.go @@ -0,0 +1,38 @@ +// judges runs every reference PDF reader present on this machine over a set +// of PDFs and reports whether each one opens, paginates, renders and extracts +// text the same way — conformance per channel. A file that is the smallest +// is worth nothing if one reader in the field disagrees with the others about +// it. +// +// judges -pdfs 'out/*.pdf,out/bench/*.pdf' -pdfium /path/to/pdfium_test +// +// Judges, each skipped with a note when its binary is absent, never faked: +// +// qpdf structural check (qpdf --check) — exit 0 clean, 3 warnings, 2 errors +// poppler pdfinfo / pdftoppm / pdftotext — the reference the per-judge Δ is taken against +// mupdf mutool draw (png + txt), mutool info +// gs Ghostscript png16m + txtwrite +// pdfium pdfium_test --png --txt (Chrome's engine; PDFIUM_TEST=/path or -pdfium) +// pdfjs pdf.js under node (judges/pdfjs-*.mjs) — Firefox's engine +// quartz sips (macOS ImageIO/Quartz, Preview's engine) — page 1 render only +// +// For each PDF and judge it records: pages reported, extracted-text length +// as a ratio of poppler's, and — on a sample of pages: the first, the middle +// and the last — how far its render is from poppler's at 96 dpi (share of +// pixels differing noticeably after both are box-downsampled to the same +// width), worst page reported. Because poppler is itself just one reader, a +// per-page consensus is computed too: the mean pairwise distance between all +// judges' renders of that page, so no single reader is privileged. +// +// A producer should judge a control beside its own output — html2pdf judges +// Chrome's PDFs of the same pages — because a judge that disagrees on the +// control too is judge noise, not a defect of ours. +package main + +import "os" + +// osExit is a variable so the tests can reach the exit path without ending the +// test binary. +var osExit = os.Exit + +func main() { osExit(run(os.Args[1:], os.Stdout, os.Stderr)) } diff --git a/cmd/judges/render.go b/cmd/judges/render.go new file mode 100644 index 0000000..904307a --- /dev/null +++ b/cmd/judges/render.go @@ -0,0 +1,189 @@ +package main + +import ( + "fmt" + "image" + "image/png" + "os" +) + +// ---- render comparison ---------------------------------------------------- + +// greyThumb decodes a PNG and returns it box-downsampled to width w as 8-bit +// grey rows. +func greyThumb(path string, w int) ([][]uint8, error) { + f, err := os.Open(path) + if err != nil { + return nil, err + } + defer f.Close() + img, err := png.Decode(f) + if err != nil { + return nil, err + } + return greyRows(img, w) +} + +// greyRows composites an image over white and box-downsamples it to width w, +// as 8-bit grey rows. There is always at least one row and every row is w +// wide, which is what lets diffRows walk two of them without a guard. +func greyRows(img image.Image, w int) ([][]uint8, error) { + b := img.Bounds() + if b.Dx() == 0 || b.Dy() == 0 { + return nil, fmt.Errorf("empty image") + } + h := b.Dy() * w / b.Dx() + if h == 0 { + h = 1 + } + rows := make([][]uint8, h) + for y := 0; y < h; y++ { + rows[y] = make([]uint8, w) + y0, y1 := b.Min.Y+y*b.Dy()/h, b.Min.Y+(y+1)*b.Dy()/h + if y1 <= y0 { + y1 = y0 + 1 + } + for x := 0; x < w; x++ { + x0, x1 := b.Min.X+x*b.Dx()/w, b.Min.X+(x+1)*b.Dx()/w + if x1 <= x0 { + x1 = x0 + 1 + } + var sum, n int + for yy := y0; yy < y1; yy++ { + for xx := x0; xx < x1; xx++ { + // Composite over white first: Quartz (sips) renders a page + // on a transparent background, and a transparent pixel + // converted straight to grey is black — every such render + // would read as a 99% mismatch against an opaque one. + r, g, bl, a := img.At(xx, yy).RGBA() + if a < 0xffff { + r += 0xffff - a + g += 0xffff - a + bl += 0xffff - a + } + grey := (19595*r + 38470*g + 7471*bl + 1<<15) >> 24 // 0..255 + sum += int(grey) + n++ + } + } + rows[y][x] = uint8(sum / n) + } + } + return rows, nil +} + +// thumbCache keeps each render's downsampled grey rows so the pairwise +// consensus doesn't decode the same PNG once per pair. +var thumbCache = map[string][][]uint8{} + +func thumb(path string) ([][]uint8, error) { + if t, ok := thumbCache[path]; ok { + return t, nil + } + t, err := greyThumb(path, thumbWidth) + if err == nil { + thumbCache[path] = t + } + return t, err +} + +// compareRenders returns the mean absolute grey difference and the share of +// pixels differing by more than diffThresh between two renders of the same +// page, over the height both cover. +func compareRenders(a, b string) (mean, pct float64, err error) { + ra, err := thumb(a) + if err != nil { + return 0, 0, err + } + rb, err := thumb(b) + if err != nil { + return 0, 0, err + } + mean, pct = diffRows(ra, rb) + return mean, pct, nil +} + +// diffRows is compareRenders on rows already decoded: the mean absolute +// difference and the share, in %, of pixels whose difference passes +// diffThresh, over the area both cover. +func diffRows(ra, rb [][]uint8) (mean, pct float64) { + h := min(len(ra), len(rb)) + var sum, big, n int + for y := 0; y < h; y++ { + w := min(len(ra[y]), len(rb[y])) + for x := 0; x < w; x++ { + d := int(ra[y][x]) - int(rb[y][x]) + if d < 0 { + d = -d + } + sum += d + if d > diffThresh { + big++ + } + n++ + } + } + if n == 0 { + return 0, 0 + } + return float64(sum) / float64(n), 100 * float64(big) / float64(n) +} + +// score fills each verdict's per-page distance to poppler and its worst +// page, then the per-page consensus: the mean pairwise distance between every +// two judges' renders of that page, poppler included as just one of them. +func score(fr *fileResult) { + var ref *verdict + for i := range fr.Verdicts { + if fr.Verdicts[i].Judge == "poppler" { + ref = &fr.Verdicts[i] + } + } + for i := range fr.Verdicts { + v := &fr.Verdicts[i] + if ref == nil || v == ref || len(v.Renders) == 0 { + continue + } + v.Diffs = map[int]pageDiff{} + for p, path := range v.Renders { + rp, ok := ref.Renders[p] + if !ok { + continue + } + m, pct, err := compareRenders(rp, path) + if err != nil { + v.Err = "compare: " + err.Error() + continue + } + v.Diffs[p] = pageDiff{m, pct} + if pct >= v.WorstPct { + v.WorstPct, v.WorstPage = pct, p + } + } + } + fr.Consensus = map[int]float64{} + for _, p := range fr.SampledPages { + var paths []string + for _, v := range fr.Verdicts { + if path, ok := v.Renders[p]; ok && v.OK { + paths = append(paths, path) + } + } + var sum float64 + var n int + for i := 0; i < len(paths); i++ { + for j := i + 1; j < len(paths); j++ { + if _, pct, err := compareRenders(paths[i], paths[j]); err == nil { + sum += pct + n++ + } + } + } + if n > 0 { + fr.Consensus[p] = sum / float64(n) + if fr.Consensus[p] >= fr.ConsensusMax { + fr.ConsensusMax, fr.ConsensusPg = fr.Consensus[p], p + } + } + } +} diff --git a/cmd/judges/render_test.go b/cmd/judges/render_test.go new file mode 100644 index 0000000..4275507 --- /dev/null +++ b/cmd/judges/render_test.go @@ -0,0 +1,183 @@ +package main + +import ( + "image" + "image/color" + "os" + "path/filepath" + "strings" + "testing" +) + +func TestATransparentPixelIsWhiteNotBlack(t *testing.T) { + // Quartz renders a page on a transparent background, and a transparent + // pixel converted straight to grey is black — every such render would + // read as a 99% mismatch against an opaque one. + img := image.NewNRGBA(image.Rect(0, 0, 2, 1)) + img.SetNRGBA(0, 0, color.NRGBA{0, 0, 0, 0}) + img.SetNRGBA(1, 0, color.NRGBA{0, 0, 0, 255}) + rows, err := greyRows(img, 2) + if err != nil { + t.Fatal(err) + } + if len(rows) != 1 || rows[0][0] != 255 || rows[0][1] != 0 { + t.Errorf("got %v", rows) + } +} + +func TestGreyRowsAlwaysHasARowAndFullWidth(t *testing.T) { + // A page much wider than tall downsamples to no rows at all unless one + // is kept; a page smaller than the thumbnail is sampled up rather than + // dividing by nothing. Both are what lets diffRows walk two results + // without guards. + wide := image.NewGray(image.Rect(0, 0, 1000, 1)) + rows, err := greyRows(wide, 400) + if err != nil || len(rows) != 1 || len(rows[0]) != 400 { + t.Errorf("a 1000×1 page: %d rows, %v", len(rows), err) + } + tiny := image.NewGray(image.Rect(0, 0, 1, 2)) + tiny.SetGray(0, 1, color.Gray{Y: 255}) + rows, err = greyRows(tiny, 4) + if err != nil || len(rows) != 8 || rows[0][3] != 0 || rows[7][0] != 255 { + t.Errorf("a 1×2 page: %d rows, %v", len(rows), err) + } + if _, err := greyRows(image.NewGray(image.Rect(0, 0, 0, 0)), 4); err == nil { + t.Error("an empty image was downsampled") + } +} + +func TestGreyThumbReadsAPNGAndNothingElse(t *testing.T) { + dir := t.TempDir() + if _, err := greyThumb(filepath.Join(dir, "absent.png"), 4); err == nil { + t.Error("an absent file was read") + } + notPNG := filepath.Join(dir, "not.png") + os.WriteFile(notPNG, []byte("%PDF-1.7"), 0o644) + if _, err := greyThumb(notPNG, 4); err == nil { + t.Error("a PDF was decoded as a PNG") + } + p := filepath.Join(dir, "page.png") + writePNG(t, p, 8, 8, image.Rect(0, 0, 4, 8)) + rows, err := greyThumb(p, 2) + if err != nil || len(rows) != 2 || rows[0][0] != 0 || rows[0][1] != 255 { + t.Errorf("got %v, %v", rows, err) + } +} + +func TestThumbsAreDecodedOnce(t *testing.T) { + // The pairwise consensus would otherwise decode the same PNG once per + // pair. + dir := t.TempDir() + p := filepath.Join(dir, "page.png") + writePNG(t, p, 8, 8, image.Rect(0, 0, 8, 8)) + if _, err := thumb(p); err != nil { + t.Fatal(err) + } + os.Remove(p) + if rows, err := thumb(p); err != nil || rows[0][0] != 0 { + t.Errorf("the cache did not answer: %v", err) + } + absent := filepath.Join(dir, "absent.png") + if _, err := thumb(absent); err == nil { + t.Fatal("an absent file was read") + } + if _, ok := thumbCache[absent]; ok { + t.Error("a failure was cached") + } +} + +func TestDiffRowsCountsWhatPassesTheThreshold(t *testing.T) { + a := [][]uint8{{0, 0, 255, 255, 100}} + b := [][]uint8{{0, 255, 255, 0, 140}} + // Differences 0, 255, 0, 255, 40: two of five pass 48, and the mean is + // 550/5. + mean, pct := diffRows(a, b) + if mean != 110 || pct != 40 { + t.Errorf("mean %v pct %v", mean, pct) + } + // Only the area both cover is compared. + if mean, pct := diffRows(a, [][]uint8{{0, 0}, {9, 9}}); mean != 0 || pct != 0 { + t.Errorf("over the overlap: mean %v pct %v", mean, pct) + } + if mean, pct := diffRows(nil, a); mean != 0 || pct != 0 { + t.Errorf("nothing to compare: mean %v pct %v", mean, pct) + } +} + +func TestCompareRendersNamesTheFileItCouldNotRead(t *testing.T) { + dir := t.TempDir() + a, b := filepath.Join(dir, "a.png"), filepath.Join(dir, "b.png") + writePNG(t, a, 100, 100, image.Rect(0, 0, 50, 100)) + writePNG(t, b, 100, 100, image.Rect(0, 0, 100, 100)) + if _, _, err := compareRenders(filepath.Join(dir, "absent.png"), b); err == nil { + t.Error("an absent first render was compared") + } + if _, _, err := compareRenders(a, filepath.Join(dir, "absent.png")); err == nil { + t.Error("an absent second render was compared") + } + // Half of a is white where b is black. + mean, pct, err := compareRenders(a, b) + if err != nil || pct != 50 || mean != 127.5 { + t.Errorf("mean %v pct %v, %v", mean, pct, err) + } +} + +// pages writes a render per page for one judge, with the ink at off. +func pages(t *testing.T, dir, judge string, off int, ps ...int) map[int]string { + t.Helper() + m := map[int]string{} + for _, p := range ps { + m[p] = filepath.Join(dir, judge+"_p"+string(rune('0'+p))+".png") + writePNG(t, m[p], 100, 100, image.Rect(off, 0, off+50, 100)) + } + return m +} + +func TestScoreMeasuresAgainstPopplerAndThenAmongEveryone(t *testing.T) { + dir := t.TempDir() + bad := filepath.Join(dir, "gs_p1.png") + os.WriteFile(bad, []byte("not a png"), 0o644) + fr := fileResult{SampledPages: []int{1, 2}, Verdicts: []verdict{ + {Judge: "qpdf", OK: true}, + {Judge: "poppler", OK: true, Renders: pages(t, dir, "poppler", 0, 1, 2)}, + // A page the reference did not render is not measured. + {Judge: "mupdf", OK: true, Renders: pages(t, dir, "mupdf", 10, 1, 2, 3)}, + // A render that cannot be read is said so, and does not stop the rest. + {Judge: "gs", OK: true, Renders: map[int]string{1: bad}}, + // A judge that failed is left out of the consensus even if it drew. + {Judge: "pdfium", OK: false, Renders: pages(t, dir, "pdfium", 50, 1, 2)}, + {Judge: "quartz", OK: true, Renders: pages(t, dir, "quartz", 20, 1)}, + }} + score(&fr) + mupdf, gs, pdfium, quartz := fr.Verdicts[2], fr.Verdicts[3], fr.Verdicts[4], fr.Verdicts[5] + if len(mupdf.Diffs) != 2 || mupdf.Diffs[1].DiffPct != 20 || mupdf.WorstPct != 20 { + t.Errorf("mupdf: %+v", mupdf) + } + if !strings.HasPrefix(gs.Err, "compare: ") || len(gs.Diffs) != 0 { + t.Errorf("gs: %+v", gs) + } + if pdfium.WorstPct != 100 || quartz.WorstPct != 40 { + t.Errorf("pdfium worst %v, quartz worst %v", pdfium.WorstPct, quartz.WorstPct) + } + // Page 1: poppler, mupdf and quartz can be paired (gs cannot be read, + // pdfium failed): (20 + 40 + 20) / 3. Page 2: poppler and mupdf: 20. + if fr.Consensus[1] != 80.0/3 || fr.Consensus[2] != 20 || fr.ConsensusPg != 1 { + t.Errorf("consensus %v, worst page %d", fr.Consensus, fr.ConsensusPg) + } +} + +func TestScoreWithoutPopplerMeasuresNothingAgainstIt(t *testing.T) { + dir := t.TempDir() + fr := fileResult{SampledPages: []int{1}, Verdicts: []verdict{ + {Judge: "mupdf", OK: true, Renders: pages(t, dir, "mupdf", 10, 1)}, + {Judge: "quartz", OK: true, Renders: pages(t, dir, "quartz", 20, 1)}, + }} + score(&fr) + if len(fr.Verdicts[0].Diffs) != 0 || len(fr.Verdicts[1].Diffs) != 0 { + t.Errorf("a distance to an absent reference: %+v", fr.Verdicts) + } + // But the readers that are there still disagree by a measurable amount. + if fr.Consensus[1] != 20 { + t.Errorf("consensus %v", fr.Consensus) + } +} diff --git a/cmd/judges/report.go b/cmd/judges/report.go new file mode 100644 index 0000000..ecf4505 --- /dev/null +++ b/cmd/judges/report.go @@ -0,0 +1,162 @@ +package main + +import ( + "context" + "encoding/json" + "fmt" + "os" + "path/filepath" + "strconv" + "strings" + "time" + + "github.com/go-pdfkit/conformance/internal/mdreport" +) + +// now is a variable so a test can know what date the report will carry. +var now = time.Now + +// writeReport writes the Markdown table above mdreport's marker, keeping +// whatever a reader wrote beneath it on the last run. +func writeReport(path string, results []fileResult, judges []judge) error { + var b strings.Builder + fmt.Fprintf(&b, "# Judged by every reader on this machine — %s\n\n", now().UTC().Format("2006-01-02")) + b.WriteString("Judges: ") + var names []string + for _, j := range judges { + if j.avail() { + names = append(names, j.name) + } + } + b.WriteString(strings.Join(names, ", ")) + fmt.Fprintf(&b, ". Versions: poppler %s; mupdf %s; gs %s; qpdf %s; pdf.js %s; macOS %s.\n\n", + version("pdftoppm", "-v"), version("mutool", "-v"), version("gs", "--version"), version("qpdf", "--version"), + pdfjsVersion(), version("sw_vers", "-productVersion")) + fmt.Fprintf(&b, "Cell format: `pages · text ratio · Δworst (page)` — pages the judge reports (– if none), "+ + "its extracted-text length as a ratio of poppler's (– if it extracts none), and the largest distance of its "+ + "renders from poppler's over the sampled pages (first, middle, last, at %d dpi: share of pixels whose grey "+ + "level moves by more than %d/255 after both are downsampled to %d px wide), with the page it happened on. "+ + "`consensus` is the mean pairwise distance between all judges' renders of a page, worst page — no reader "+ + "privileged. ⚠n = n warning lines on stderr; ❌ = the judge failed to process the file.\n\n", + renderDPI, diffThresh, thumbWidth) + + b.WriteString("| PDF | Bytes |") + for _, n := range names { + b.WriteString(" " + n + " |") + } + b.WriteString(" consensus |\n|---|---|") + for range names { + b.WriteString("---|") + } + b.WriteString("---|\n") + for _, fr := range results { + fmt.Fprintf(&b, "| %s | %s |", fr.File, fmtBytes(fr.Bytes)) + var popplerText int + for _, v := range fr.Verdicts { + if v.Judge == "poppler" { + popplerText = v.TextChar + } + } + for _, v := range fr.Verdicts { + if v.Skipped { + continue + } + b.WriteString(" " + cell(v, popplerText) + " |") + } + fmt.Fprintf(&b, " %.1f%% (p%d) |\n", fr.ConsensusMax, fr.ConsensusPg) + } + b.WriteString("\n") + return mdreport.Write(path, b.String()) +} + +// cell is one judge's verdict on one file, as the table reads it. +func cell(v verdict, popplerText int) string { + if !v.OK { + return "❌ " + v.Err + } + var parts []string + if v.Pages > 0 { + parts = append(parts, strconv.Itoa(v.Pages)+"p") + } else { + parts = append(parts, "–") + } + switch { + case v.TextChar < 0: + parts = append(parts, "–") + case popplerText > 0: + parts = append(parts, fmt.Sprintf("%.3f", float64(v.TextChar)/float64(popplerText))) + default: + parts = append(parts, strconv.Itoa(v.TextChar)) + } + switch { + case v.Judge == "poppler": + parts = append(parts, "ref") + case len(v.Diffs) > 0 && v.Judge == "quartz": + parts = append(parts, fmt.Sprintf("Δ%.1f%% (p1 only)", v.WorstPct)) + case len(v.Diffs) > 0: + parts = append(parts, fmt.Sprintf("Δ%.1f%% (p%d)", v.WorstPct, v.WorstPage)) + default: + parts = append(parts, "–") + } + s := "✅ " + strings.Join(parts, " · ") + if v.Warnings > 0 { + s += fmt.Sprintf(" ⚠%d", v.Warnings) + } + return s +} + +func fmtBytes(n int64) string { + switch { + case n >= 1e6: + return fmt.Sprintf("%.1f MB", float64(n)/1e6) + case n >= 1e3: + return fmt.Sprintf("%.0f KB", float64(n)/1e3) + } + return fmt.Sprintf("%d B", n) +} + +// version is the first line a tool prints about itself, on whichever stream +// it chose: pdftoppm and mutool answer -v on stderr, gs and qpdf on stdout. +// The judge is half the measurement, so the report says which one it was. +func version(bin string, args ...string) string { + o, e, err := runCmd(context.Background(), "", bin, args...) + out := o + e + if err != nil && out == "" { + return "?" + } + l := strings.TrimSpace(strings.SplitN(out, "\n", 2)[0]) + if len(l) > 60 { + l = l[:60] + } + return l +} + +// pdfjsVersion is read off the installed package, since pdf.js has no binary +// to ask. +func pdfjsVersion() string { + b, err := os.ReadFile(filepath.Join(nodeDir, "node_modules", "pdfjs-dist", "package.json")) + if err != nil { + return "?" + } + var p struct{ Version string } + if json.Unmarshal(b, &p) != nil { + return "?" + } + return p.Version +} + +// writeResults writes every verdict as JSON, for whatever wants to read the +// run back without parsing a table. +func writeResults(path string, results []fileResult) error { + f, err := os.Create(path) + if err != nil { + return err + } + enc := json.NewEncoder(f) + enc.SetIndent("", " ") + if err := enc.Encode(results); err != nil { + f.Close() + return err + } + return f.Close() +} diff --git a/cmd/judges/report_test.go b/cmd/judges/report_test.go new file mode 100644 index 0000000..3120945 --- /dev/null +++ b/cmd/judges/report_test.go @@ -0,0 +1,133 @@ +package main + +import ( + "errors" + "math" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +func TestCellReadsPagesTextAndDistance(t *testing.T) { + for _, tc := range []struct { + name string + v verdict + ref int + want string + }{ + {"a refusal", verdict{Err: "boom"}, 10, "❌ boom"}, + {"the reference", verdict{Judge: "poppler", OK: true, Pages: 3, TextChar: 10}, 10, "✅ 3p · 1.000 · ref"}, + {"a judge with a distance", verdict{Judge: "mupdf", OK: true, Pages: 3, TextChar: 11, WorstPct: 4.25, WorstPage: 2, + Diffs: map[int]pageDiff{2: {}}}, 10, "✅ 3p · 1.100 · Δ4.2% (p2)"}, + {"quartz, which only has page 1", verdict{Judge: "quartz", OK: true, TextChar: -1, WorstPct: 9, WorstPage: 1, + Diffs: map[int]pageDiff{1: {}}}, 10, "✅ – · – · Δ9.0% (p1 only)"}, + {"text where the reference had none", verdict{Judge: "gs", OK: true, TextChar: 7}, 0, "✅ – · 7 · –"}, + {"warnings", verdict{Judge: "qpdf", OK: true, TextChar: -1, Warnings: 2}, 10, "✅ – · – · – ⚠2"}, + } { + if got := cell(tc.v, tc.ref); got != tc.want { + t.Errorf("%s: %q, want %q", tc.name, got, tc.want) + } + } +} + +func TestBytesAreReadable(t *testing.T) { + for n, want := range map[int64]string{7: "7 B", 6887: "7 KB", 1055265: "1.1 MB"} { + if got := fmtBytes(n); got != want { + t.Errorf("%d: %q, want %q", n, got, want) + } + } +} + +func TestVersionIsTheFirstLineOnEitherStream(t *testing.T) { + stubTools(t, nil, func(_, name string, _ []string) (string, string, error) { + switch name { + case "pdftoppm": + return "", "pdftoppm version 26.04.0\nCopyright\n", nil + case "gs": + return strings.Repeat("9", 70) + "\n", "", nil + } + return "", "", errors.New("not found") + }) + if got := version("pdftoppm", "-v"); got != "pdftoppm version 26.04.0" { + t.Errorf("on stderr: %q", got) + } + if got := version("gs", "--version"); len(got) != 60 { + t.Errorf("a long line: %q", got) + } + if got := version("absent"); got != "?" { + t.Errorf("a tool that is not there: %q", got) + } +} + +func TestPdfjsVersionIsReadOffThePackage(t *testing.T) { + was := nodeDir + defer func() { nodeDir = was }() + nodeDir = t.TempDir() + if got := pdfjsVersion(); got != "?" { + t.Errorf("not installed: %q", got) + } + pkg := filepath.Join(nodeDir, "node_modules", "pdfjs-dist", "package.json") + os.MkdirAll(filepath.Dir(pkg), 0o755) + os.WriteFile(pkg, []byte("{"), 0o644) + if got := pdfjsVersion(); got != "?" { + t.Errorf("a package that will not parse: %q", got) + } + os.WriteFile(pkg, []byte(`{"name":"pdfjs-dist","version":"6.3.289"}`), 0o644) + if got := pdfjsVersion(); got != "6.3.289" { + t.Errorf("%q", got) + } +} + +func TestWriteResultsSaysWhenItCannot(t *testing.T) { + dir := t.TempDir() + if err := writeResults(filepath.Join(dir, "no", "such", "judges.json"), nil); err == nil { + t.Error("written into a directory that does not exist") + } + // A value JSON cannot carry — nothing in the harness produces one, but a + // record that could not be encoded must not be reported as written. + if err := writeResults(filepath.Join(dir, "judges.json"), []fileResult{{ConsensusMax: math.NaN()}}); err == nil { + t.Error("NaN was encoded") + } + if err := writeResults(filepath.Join(dir, "judges.json"), []fileResult{{File: "a.pdf"}}); err != nil { + t.Error(err) + } +} + +func TestTheReportNamesItsJudgesAndTheirVersions(t *testing.T) { + stubTools(t, everyReader, aMachine(t, 1, nil)) + wasNow, wasNode := now, nodeDir + defer func() { now, nodeDir = wasNow, wasNode }() + now = func() time.Time { return time.Date(2026, 9, 6, 12, 0, 0, 0, time.UTC) } + nodeDir = t.TempDir() + judges := []judge{ + {name: "qpdf", avail: func() bool { return true }}, + {name: "poppler", avail: func() bool { return true }}, + {name: "pdfium", avail: func() bool { return false }}, + } + results := []fileResult{{File: "a.pdf", Bytes: 6887, ConsensusMax: 1.25, ConsensusPg: 3, Verdicts: []verdict{ + {Judge: "qpdf", OK: true, TextChar: -1}, + {Judge: "poppler", OK: true, Pages: 3, TextChar: 10}, + {Judge: "pdfium", Skipped: true, TextChar: -1}, + }}} + path := filepath.Join(t.TempDir(), "JUDGES.md") + if err := writeReport(path, results, judges); err != nil { + t.Fatal(err) + } + b, _ := os.ReadFile(path) + got := string(b) + for _, want := range []string{ + "# Judged by every reader on this machine — 2026-09-06\n", + "Judges: qpdf, poppler. Versions: poppler pdftoppm version 26.04.0; mupdf mutool version 1.28.3; gs 10.07.1; qpdf qpdf version 12.4.1; pdf.js ?; macOS 26.6.2.\n", + "| PDF | Bytes | qpdf | poppler | consensus |\n|---|---|---|---|---|\n", + "| a.pdf | 7 KB | ✅ – · – · – | ✅ 3p · 1.000 · ref | 1.2% (p3) |\n", + } { + if !strings.Contains(got, want) { + t.Errorf("missing %q in:\n%s", want, got) + } + } + if err := writeReport(filepath.Join(t.TempDir(), "no", "JUDGES.md"), results, judges); err == nil { + t.Error("a report was written into a directory that does not exist") + } +} diff --git a/cmd/judges/run.go b/cmd/judges/run.go new file mode 100644 index 0000000..e69352e --- /dev/null +++ b/cmd/judges/run.go @@ -0,0 +1,130 @@ +package main + +import ( + "context" + "flag" + "fmt" + "io" + "os" + "path/filepath" + "sort" + "strings" + "time" +) + +func envOr(k, d string) string { + if v := os.Getenv(k); v != "" { + return v + } + return d +} + +// run judges every PDF the globs name with every reader the machine has. +func run(args []string, out, errOut io.Writer) int { + fs := flag.NewFlagSet("judges", flag.ContinueOnError) + fs.SetOutput(errOut) + glob := fs.String("pdfs", "", "comma-separated globs of the PDFs to judge") + outDir := fs.String("out", "out/judges", "directory for per-judge renders") + resultsPath := fs.String("results", "judges.json", "JSON results path") + reportPath := fs.String("report", "JUDGES.md", "Markdown report path") + fs.StringVar(&pdfiumBin, "pdfium", envOr("PDFIUM_TEST", ""), "pdfium_test binary (Chrome's engine); PDFIUM_TEST env") + fs.StringVar(&nodeDir, "nodedir", "judges", "directory with pdfjs-*.mjs and node_modules") + fs.DurationVar(&judgeTimeout, "timeout", 180*time.Second, "how long one judge may take on one file before it is called a hang") + if err := fs.Parse(args); err != nil { + return 2 + } + if *glob == "" { + fmt.Fprintln(errOut, "judges: -pdfs is needed") + return 2 + } + + judges := []judge{ + {"qpdf", func() bool { return have("qpdf") }, judgeQpdf}, + {"poppler", func() bool { return have("pdfinfo") && have("pdftoppm") && have("pdftotext") }, judgePoppler}, + {"mupdf", func() bool { return have("mutool") }, judgeMupdf}, + {"gs", func() bool { return have("gs") }, judgeGs}, + {"pdfium", func() bool { _, err := os.Stat(pdfiumBin); return pdfiumBin != "" && err == nil }, judgePdfium}, + {"pdfjs", func() bool { + _, err := os.Stat(filepath.Join(nodeDir, "node_modules", "pdfjs-dist")) + return have("node") && err == nil + }, judgePdfjs}, + {"quartz", func() bool { return have("sips") }, judgeQuartz}, + } + + var files []string + for _, g := range strings.Split(*glob, ",") { + m, _ := filepath.Glob(strings.TrimSpace(g)) + files = append(files, m...) + } + sort.Strings(files) + if len(files) == 0 { + fmt.Fprintln(errOut, "judges: no PDFs matched") + return 1 + } + nodeDir, _ = filepath.Abs(nodeDir) + + var results []fileResult + for _, f := range files { + abs, _ := filepath.Abs(f) + fr := fileResult{File: f} + if st, err := os.Stat(abs); err == nil { + fr.Bytes = st.Size() + } + // Absolute: pdfium and pdf.js run with another working directory. + dir, _ := filepath.Abs(filepath.Join(*outDir, strings.TrimSuffix(filepath.Base(f), ".pdf"))) + os.MkdirAll(dir, 0o755) + fmt.Fprintf(errOut, "%s\n", f) + // Page count for the sample comes from poppler, which runs before the + // renderers; until then only page 1 is known to exist. + pages := []int{1} + for _, j := range judges { + if !j.avail() { + fr.Verdicts = append(fr.Verdicts, verdict{Judge: j.name, Skipped: true, TextChar: -1}) + continue + } + ctx, cancel := context.WithTimeout(context.Background(), judgeTimeout) + t0 := time.Now() + v := j.run(ctx, abs, dir, pages) + v.Ms = time.Since(t0).Milliseconds() + cancel() + if v.Judge == "poppler" && len(pages) == 1 && v.Pages > 1 { + // Re-render poppler on the full sample now that the count is known. + pages = samplePages(v.Pages) + fr.SampledPages = pages + ctx, cancel := context.WithTimeout(context.Background(), judgeTimeout) + v = j.run(ctx, abs, dir, pages) + cancel() + // Both runs are poppler's time; the record used to say 0. + v.Ms = time.Since(t0).Milliseconds() + } + if fr.SampledPages == nil { + fr.SampledPages = pages + } + fmt.Fprintf(errOut, " %-8s ok=%v pages=%d text=%d warn=%d %dms %s\n", + v.Judge, v.OK, v.Pages, v.TextChar, v.Warnings, v.Ms, v.Err) + fr.Verdicts = append(fr.Verdicts, v) + } + score(&fr) + for _, v := range fr.Verdicts { + if len(v.Diffs) > 0 { + fmt.Fprintf(errOut, " %-8s worst Δ%.1f%% on p%d\n", v.Judge, v.WorstPct, v.WorstPage) + } + } + fmt.Fprintf(errOut, " consensus max %.1f%% on p%d (pages %v)\n", fr.ConsensusMax, fr.ConsensusPg, fr.SampledPages) + results = append(results, fr) + } + + // A run whose record could not be written has not been made: the table is + // read by people and the JSON by the next run, and losing either silently + // is a gap nobody can tell from a run that never happened. + if err := writeResults(*resultsPath, results); err != nil { + fmt.Fprintf(errOut, "judges: results: %v\n", err) + return 1 + } + if err := writeReport(*reportPath, results, judges); err != nil { + fmt.Fprintf(errOut, "judges: report: %v\n", err) + return 1 + } + fmt.Fprintf(out, "%d PDFs judged; report %s; results %s\n", len(results), *reportPath, *resultsPath) + return 0 +} diff --git a/cmd/judges/run_test.go b/cmd/judges/run_test.go new file mode 100644 index 0000000..e760d7e --- /dev/null +++ b/cmd/judges/run_test.go @@ -0,0 +1,260 @@ +package main + +import ( + "bytes" + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/go-pdfkit/conformance/internal/mdreport" +) + +// A rig is a directory with a document, a pdfium_test that is a file, and a +// node directory with pdf.js installed — every check the harness makes for a +// judge's presence, satisfied without a single real one. +type rig struct{ dir, pdf, pdfium, node string } + +func aRig(t *testing.T) rig { + t.Helper() + dir := t.TempDir() + r := rig{dir: dir, pdf: filepath.Join(dir, "doc.pdf"), + pdfium: filepath.Join(dir, "pdfium_test"), node: filepath.Join(dir, "judges")} + os.WriteFile(r.pdf, []byte("%PDF-1.7\n"), 0o644) + os.WriteFile(r.pdfium, []byte("#!/bin/sh\n"), 0o755) + pkg := filepath.Join(r.node, "node_modules", "pdfjs-dist") + os.MkdirAll(pkg, 0o755) + os.WriteFile(filepath.Join(pkg, "package.json"), []byte(`{"version":"6.3.289"}`), 0o644) + return r +} + +func (r rig) args(more ...string) []string { + return append([]string{"-pdfs", filepath.Join(r.dir, "*.pdf"), "-out", filepath.Join(r.dir, "out"), + "-report", filepath.Join(r.dir, "JUDGES.md"), "-results", filepath.Join(r.dir, "judges.json"), + "-pdfium", r.pdfium, "-nodedir", r.node}, more...) +} + +func (r rig) report(t *testing.T) string { + t.Helper() + b, err := os.ReadFile(filepath.Join(r.dir, "JUDGES.md")) + if err != nil { + t.Fatal(err) + } + return string(b) +} + +func (r rig) results(t *testing.T) []fileResult { + t.Helper() + b, err := os.ReadFile(filepath.Join(r.dir, "judges.json")) + if err != nil { + t.Fatal(err) + } + var rs []fileResult + if err := json.Unmarshal(b, &rs); err != nil { + t.Fatal(err) + } + return rs +} + +func TestRunAsksEveryReaderTheMachineHas(t *testing.T) { + stubTools(t, everyReader, aMachine(t, 3, nil)) + r := aRig(t) + var out, errOut bytes.Buffer + if code := run(r.args(), &out, &errOut); code != 0 { + t.Fatalf("exit %d: %s", code, errOut.String()) + } + if !strings.Contains(out.String(), "1 PDFs judged") { + t.Errorf("stdout %q", out.String()) + } + got := r.report(t) + for _, want := range []string{ + "Judges: qpdf, poppler, mupdf, gs, pdfium, pdfjs, quartz.", + "pdf.js 6.3.289;", + "| PDF | Bytes | qpdf | poppler | mupdf | gs | pdfium | pdfjs | quartz | consensus |", + "| ✅ – · – · – | ✅ 3p · 1.000 · ref ⚠1 | ✅ 3p · 1.000 · Δ", + "| ✅ – · 1.000 · Δ", // gs: no page count, padding not counted + "| ✅ 3p · 1.000 · Δ", // pdfium, in characters not bytes + "(p1 only) |", // quartz + mdreport.Marker, + } { + if !strings.Contains(got, want) { + t.Errorf("missing %q in:\n%s", want, got) + } + } + rs := r.results(t) + if len(rs) != 1 || len(rs[0].SampledPages) != 3 || rs[0].Bytes != 9 { + t.Fatalf("%+v", rs) + } + // The sample was chosen once poppler said how many pages there are, and + // every renderer after it drew all three. + for _, v := range rs[0].Verdicts { + switch v.Judge { + case "poppler", "mupdf", "gs", "pdfium", "pdfjs": + if len(v.Renders) != 3 || v.Ms < 0 { + t.Errorf("%s drew %d pages", v.Judge, len(v.Renders)) + } + if v.Judge != "poppler" && (v.WorstPct <= 0 || v.WorstPct >= 100) { + t.Errorf("%s is Δ%v from poppler", v.Judge, v.WorstPct) + } + } + } + if rs[0].ConsensusMax <= 0 { + t.Errorf("no consensus: %+v", rs[0]) + } + // The progress went to stderr, judge by judge. + if !strings.Contains(errOut.String(), "poppler ok=true pages=3") || !strings.Contains(errOut.String(), "consensus max") { + t.Errorf("stderr %q", errOut.String()) + } +} + +func TestARerunKeepsTheAnalysis(t *testing.T) { + stubTools(t, everyReader, aMachine(t, 1, nil)) + r := aRig(t) + var out, errOut bytes.Buffer + if code := run(r.args(), &out, &errOut); code != 0 { + t.Fatalf("exit %d: %s", code, errOut.String()) + } + path := filepath.Join(r.dir, "JUDGES.md") + old := r.report(t) + os.WriteFile(path, []byte(strings.Replace(old, mdreport.Placeholder, mdreport.Marker+"\n\nquartz is judge noise here.\n", 1)), 0o644) + if code := run(r.args(), &out, &errOut); code != 0 { + t.Fatalf("exit %d: %s", code, errOut.String()) + } + if got := r.report(t); !strings.Contains(got, "quartz is judge noise here.") || strings.Contains(got, "Analysis pending") { + t.Errorf("the analysis did not survive:\n%s", got) + } +} + +func TestAJudgeThatIsNotThereIsSkippedNotFaked(t *testing.T) { + stubTools(t, nil, func(string, string, []string) (string, string, error) { + return "", "", os.ErrNotExist + }) + r := aRig(t) + var out, errOut bytes.Buffer + if code := run(r.args("-pdfium", "", "-nodedir", filepath.Join(r.dir, "nowhere")), &out, &errOut); code != 0 { + t.Fatalf("exit %d: %s", code, errOut.String()) + } + got := r.report(t) + if !strings.Contains(got, "Judges: . Versions: poppler ?; mupdf ?; gs ?; qpdf ?; pdf.js ?; macOS ?.") { + t.Errorf("the header does not say nobody judged:\n%s", got) + } + if !strings.Contains(got, "| PDF | Bytes | consensus |\n|---|---|---|\n") { + t.Errorf("the table has columns for judges that are not there:\n%s", got) + } + for _, v := range r.results(t)[0].Verdicts { + if !v.Skipped || v.OK { + t.Errorf("%s was not skipped: %+v", v.Judge, v) + } + } +} + +func TestWithoutPopplerThereIsNoReferenceButStillAConsensus(t *testing.T) { + stubTools(t, []string{"mutool", "gs", "sips"}, aMachine(t, 3, nil)) + r := aRig(t) + var out, errOut bytes.Buffer + if code := run(r.args("-pdfium", ""), &out, &errOut); code != 0 { + t.Fatalf("exit %d: %s", code, errOut.String()) + } + rs := r.results(t) + // Nobody said how many pages there are, so only page 1 is sampled. + if len(rs[0].SampledPages) != 1 { + t.Errorf("sampled %v", rs[0].SampledPages) + } + for _, v := range rs[0].Verdicts { + if len(v.Diffs) != 0 { + t.Errorf("%s has a distance to a reference that is not there", v.Judge) + } + } + if rs[0].ConsensusMax <= 0 { + t.Errorf("mupdf, gs and quartz did not disagree: %+v", rs[0]) + } + if !strings.Contains(r.report(t), "| ✅ 3p · 10 · – |") { + t.Errorf("without a reference the text is a count:\n%s", r.report(t)) + } +} + +func TestPdfiumIsFoundThroughTheEnvironment(t *testing.T) { + stubTools(t, nil, aMachine(t, 1, nil)) + r := aRig(t) + t.Setenv("PDFIUM_TEST", r.pdfium) + var out, errOut bytes.Buffer + args := []string{"-pdfs", r.pdf, "-out", filepath.Join(r.dir, "out"), + "-report", filepath.Join(r.dir, "JUDGES.md"), "-results", filepath.Join(r.dir, "judges.json")} + if code := run(args, &out, &errOut); code != 0 { + t.Fatalf("exit %d: %s", code, errOut.String()) + } + if !strings.Contains(r.report(t), "Judges: pdfium.") { + t.Errorf("%s", r.report(t)) + } +} + +func TestADocumentThatVanishedIsStillNamed(t *testing.T) { + // A glob matches a dangling link; the size is nought and every judge + // says what it found, rather than the run stopping on it. + stubTools(t, []string{"qpdf"}, func(string, string, []string) (string, string, error) { + return "", "", os.ErrNotExist + }) + r := aRig(t) + os.Remove(r.pdf) + if err := os.Symlink(filepath.Join(r.dir, "gone", "doc.pdf"), r.pdf); err != nil { + t.Skip(err) + } + var out, errOut bytes.Buffer + if code := run(r.args("-pdfium", ""), &out, &errOut); code != 0 { + t.Fatalf("exit %d: %s", code, errOut.String()) + } + if !strings.Contains(r.report(t), "| 0 B | ❌ file does not exist |") { + t.Errorf("%s", r.report(t)) + } +} + +func TestRunSaysWhatIsWrong(t *testing.T) { + stubTools(t, everyReader, aMachine(t, 1, nil)) + r := aRig(t) + for _, tc := range []struct { + name string + args []string + code int + want string + }{ + {"no documents named", nil, 2, "-pdfs is needed"}, + {"a flag it does not have", []string{"-nonsense"}, 2, "flag provided but not defined"}, + {"a glob nothing matches", []string{"-pdfs", filepath.Join(r.dir, "*.nothing")}, 1, "no PDFs matched"}, + {"results it cannot write", r.args("-results", filepath.Join(r.dir, "no", "judges.json")), 1, "judges: results:"}, + {"a report it cannot write", r.args("-report", filepath.Join(r.dir, "no", "JUDGES.md")), 1, "judges: report:"}, + } { + t.Run(tc.name, func(t *testing.T) { + var out, errOut bytes.Buffer + if code := run(tc.args, &out, &errOut); code != tc.code { + t.Errorf("exit %d, want %d: %s", code, tc.code, errOut.String()) + } + if !strings.Contains(errOut.String(), tc.want) { + t.Errorf("stderr %q, want %q", errOut.String(), tc.want) + } + }) + } +} + +func TestEnvOr(t *testing.T) { + t.Setenv("JUDGES_TEST_X", "") + if envOr("JUDGES_TEST_X", "d") != "d" { + t.Error("unset") + } + t.Setenv("JUDGES_TEST_X", "v") + if envOr("JUDGES_TEST_X", "d") != "v" { + t.Error("set") + } +} + +func TestMainCallsRun(t *testing.T) { + oldExit, oldArgs := osExit, os.Args + defer func() { osExit, os.Args = oldExit, oldArgs }() + got := -1 + osExit = func(code int) { got = code } + os.Args = []string{"judges"} + main() + if got != 2 { + t.Errorf("main exited %d, want 2", got) + } +} diff --git a/internal/mdreport/mdreport.go b/internal/mdreport/mdreport.go new file mode 100644 index 0000000..2c82990 --- /dev/null +++ b/internal/mdreport/mdreport.go @@ -0,0 +1,35 @@ +// Package mdreport writes a generated Markdown report while preserving the +// hand-written analysis a previous run left below a marker line. +// +// The convention is go-webengine/engine's, from its bench REPORT.md, and +// html2pdf's corpus harnesses carried it; it comes here with judges because a +// table nobody wrote under is a table nobody read. Whatever a reader wrote +// beneath the marker — which cell was judge noise, which one was a defect and +// where it was fixed — survives the next run, and a re-run never clobbers it. +package mdreport + +import ( + "os" + "strings" +) + +// Marker separates the regenerated part of a report (above) from the +// preserved, hand-written analysis (below). +const Marker = "" + +// Placeholder is written below the marker on a first run, when there is no +// analysis to preserve yet. +const Placeholder = Marker + "\n\n_Analysis pending — see the table above and out/ for a first look._\n" + +// Write replaces path's content above Marker with generated (which must not +// itself contain the marker) and keeps everything from the marker on, or +// writes Placeholder there when the file did not exist or had no marker. +func Write(path, generated string) error { + preserved := Placeholder + if old, err := os.ReadFile(path); err == nil { + if i := strings.Index(string(old), Marker); i >= 0 { + preserved = string(old)[i:] + } + } + return os.WriteFile(path, []byte(generated+preserved), 0o644) +} diff --git a/internal/mdreport/mdreport_test.go b/internal/mdreport/mdreport_test.go new file mode 100644 index 0000000..3e775dd --- /dev/null +++ b/internal/mdreport/mdreport_test.go @@ -0,0 +1,63 @@ +package mdreport + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +func TestAFirstRunLeavesRoomForTheAnalysis(t *testing.T) { + path := filepath.Join(t.TempDir(), "REPORT.md") + if err := Write(path, "# table\n\n"); err != nil { + t.Fatal(err) + } + got, _ := os.ReadFile(path) + if string(got) != "# table\n\n"+Placeholder { + t.Errorf("got %q", got) + } +} + +func TestARerunKeepsWhatWasWrittenUnderTheMarker(t *testing.T) { + // The table is regenerated; the reader's notes beneath it are the reason + // the file exists, and a run that lost them would have to be run again + // by someone who no longer remembers what they said. + path := filepath.Join(t.TempDir(), "REPORT.md") + old := "# old table\n\n" + Marker + "\n\nquartz is judge noise here: it disagrees on the control too.\n" + if err := os.WriteFile(path, []byte(old), 0o644); err != nil { + t.Fatal(err) + } + if err := Write(path, "# new table\n\n"); err != nil { + t.Fatal(err) + } + got, _ := os.ReadFile(path) + s := string(got) + if strings.Contains(s, "old table") { + t.Errorf("the old table survived: %q", s) + } + if !strings.HasPrefix(s, "# new table\n\n"+Marker) || !strings.Contains(s, "judge noise") { + t.Errorf("the analysis did not: %q", s) + } +} + +func TestAFileWithoutAMarkerIsReplacedWhole(t *testing.T) { + // There is nothing to preserve in it, and a file with no marker is one + // this package did not write. + path := filepath.Join(t.TempDir(), "REPORT.md") + if err := os.WriteFile(path, []byte("someone else's file\n"), 0o644); err != nil { + t.Fatal(err) + } + if err := Write(path, "# table\n\n"); err != nil { + t.Fatal(err) + } + got, _ := os.ReadFile(path) + if string(got) != "# table\n\n"+Placeholder { + t.Errorf("got %q", got) + } +} + +func TestAReportThatCannotBeWrittenSaysSo(t *testing.T) { + if err := Write(filepath.Join(t.TempDir(), "no", "such", "dir", "REPORT.md"), "x"); err == nil { + t.Error("a report was written into a directory that does not exist") + } +} diff --git a/judges/.gitignore b/judges/.gitignore new file mode 100644 index 0000000..c2658d7 --- /dev/null +++ b/judges/.gitignore @@ -0,0 +1 @@ +node_modules/ diff --git a/judges/package-lock.json b/judges/package-lock.json new file mode 100644 index 0000000..80e96fa --- /dev/null +++ b/judges/package-lock.json @@ -0,0 +1,293 @@ +{ + "name": "judges", + "version": "1.0.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "judges", + "version": "1.0.0", + "license": "BSD-3-Clause", + "dependencies": { + "@napi-rs/canvas": "^1.0.8", + "pdfjs-dist": "^6.3.289" + } + }, + "node_modules/@napi-rs/canvas": { + "version": "1.0.8", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas/-/canvas-1.0.8.tgz", + "integrity": "sha512-/SaLcvlqGWdm0HSCWMgHu7cjJiQXfP8/mOY+6dUyV9flQz7sPBBZ+ed2zYtoukojPmxOaL7bm+d/G4GeWWoN7g==", + "license": "MIT", + "workspaces": [ + "e2e/*" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + }, + "optionalDependencies": { + "@napi-rs/canvas-android-arm64": "1.0.8", + "@napi-rs/canvas-darwin-arm64": "1.0.8", + "@napi-rs/canvas-darwin-x64": "1.0.8", + "@napi-rs/canvas-linux-arm-gnueabihf": "1.0.8", + "@napi-rs/canvas-linux-arm64-gnu": "1.0.8", + "@napi-rs/canvas-linux-arm64-musl": "1.0.8", + "@napi-rs/canvas-linux-riscv64-gnu": "1.0.8", + "@napi-rs/canvas-linux-x64-gnu": "1.0.8", + "@napi-rs/canvas-linux-x64-musl": "1.0.8", + "@napi-rs/canvas-win32-arm64-msvc": "1.0.8", + "@napi-rs/canvas-win32-x64-msvc": "1.0.8" + } + }, + "node_modules/@napi-rs/canvas-android-arm64": { + "version": "1.0.8", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas-android-arm64/-/canvas-android-arm64-1.0.8.tgz", + "integrity": "sha512-5+nkh8i3gt6lqS/d2jTZ1xAn6tdgtB4Lf1mW6T0Qm5/rXNwBuV1sAEyLEWan5o9gJPU/GuvHR3rvSeZ+FaGrbw==", + "cpu": [ + "arm64" + ], + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-darwin-arm64": { + "version": "1.0.8", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas-darwin-arm64/-/canvas-darwin-arm64-1.0.8.tgz", + "integrity": "sha512-7jQ47gi+fZ7KJmfc/5rNyy1CYw/cu4kZ0KPIYbo9UUgSdW0bKQJpt+WihEor6s4Lyp7+xc3a+3HeyXmAEbbnPg==", + "cpu": [ + "arm64" + ], + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-darwin-x64": { + "version": "1.0.8", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas-darwin-x64/-/canvas-darwin-x64-1.0.8.tgz", + "integrity": "sha512-rRjDMZs9pIRKGxgijwezplKc1RnJsqUokrA9h88bbTkqQ+7ePj0ZN4ZnZDy8Vu0tXs7KRlI2tQLaK4mx9QlxHg==", + "cpu": [ + "x64" + ], + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-linux-arm-gnueabihf": { + "version": "1.0.8", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas-linux-arm-gnueabihf/-/canvas-linux-arm-gnueabihf-1.0.8.tgz", + "integrity": "sha512-jGcCd+8ra6Q61xKqZeiItujTpp9a9eRLcQ0jW6qYNku+WpupqOPFPY0SrsuSnXFviJwkpKYT9p7QrB4lsf3LNQ==", + "cpu": [ + "arm" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-linux-arm64-gnu": { + "version": "1.0.8", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas-linux-arm64-gnu/-/canvas-linux-arm64-gnu-1.0.8.tgz", + "integrity": "sha512-od6I2Y7kU7i1SwZYG2EKW8rWz6JiedtPpko4WEe1DDsiikrfaotVBCRaUTM5/yeZKaZ92EatoAS+5xG+6uJlYA==", + "cpu": [ + "arm64" + ], + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-linux-arm64-musl": { + "version": "1.0.8", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas-linux-arm64-musl/-/canvas-linux-arm64-musl-1.0.8.tgz", + "integrity": "sha512-yYkPbJDJiWj6N0gASA3CAvRypZmVpJnxU0DQg3aBhneLDQde9TPLKADsQkobNoJUtTT/lj46aWpzT48PDb3Qcg==", + "cpu": [ + "arm64" + ], + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-linux-riscv64-gnu": { + "version": "1.0.8", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas-linux-riscv64-gnu/-/canvas-linux-riscv64-gnu-1.0.8.tgz", + "integrity": "sha512-PB00MSKAp4VwK/xwe6duKxRKmH8UH4GIl1pqHSbxng0jnU9Dr7FwaDypDiqwNFZ774N+8G7mJLGuLtg9NTcQsg==", + "cpu": [ + "riscv64" + ], + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-linux-x64-gnu": { + "version": "1.0.8", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas-linux-x64-gnu/-/canvas-linux-x64-gnu-1.0.8.tgz", + "integrity": "sha512-TWM2XWJoitLiIPCvgJh7SriC+L/T9qkYCVzC66AidsZy0QP1hkKzBzVwshCdcA3q6fIn3yE0ISbq4lMJSy8jFw==", + "cpu": [ + "x64" + ], + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-linux-x64-musl": { + "version": "1.0.8", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas-linux-x64-musl/-/canvas-linux-x64-musl-1.0.8.tgz", + "integrity": "sha512-hb20MxKXXb5IB7AAwN8UHz9WRsa2HmdZfjsDCzjElwJoeV1aotVEwFU4FrFQcYQVzsJQLeaCc/2Qdt/0Q72mMg==", + "cpu": [ + "x64" + ], + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-win32-arm64-msvc": { + "version": "1.0.8", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas-win32-arm64-msvc/-/canvas-win32-arm64-msvc-1.0.8.tgz", + "integrity": "sha512-WwPN08IXE4SkL+FhJyPz/iFnycMAUkbphFIT4cmKLlvbSU0Zfn1R7BGJ3Hqky1S89QUYc0Q4IOScXb/42Re9wQ==", + "cpu": [ + "arm64" + ], + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-win32-x64-msvc": { + "version": "1.0.8", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas-win32-x64-msvc/-/canvas-win32-x64-msvc-1.0.8.tgz", + "integrity": "sha512-XkrVqKb+pxyba7kjy2LJvABFVBTE0DNpEl7MrG4OYUmaWarrXH+t54z/Czj2YxCKtizYTV4mg6phNm3x24qjhQ==", + "cpu": [ + "x64" + ], + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/pdfjs-dist": { + "version": "6.3.289", + "resolved": "https://registry.npmjs.org/pdfjs-dist/-/pdfjs-dist-6.3.289.tgz", + "integrity": "sha512-ZHjSVpDa3D6izMq8/04lvkhkATUmL9px6ChPaXc1k6nU2Mrhlg1/7F0bdUqCwUjw3NsPTfPZsMDUU6ZIcRaeQw==", + "license": "Apache-2.0", + "engines": { + "node": ">=22.13.0 || >=24" + }, + "optionalDependencies": { + "@napi-rs/canvas": "^1.0.0" + } + } + } +} diff --git a/judges/package.json b/judges/package.json new file mode 100644 index 0000000..503cdaa --- /dev/null +++ b/judges/package.json @@ -0,0 +1,12 @@ +{ + "name": "judges", + "version": "1.0.0", + "private": true, + "description": "pdf.js (Firefox's engine) as a judge for cmd/judges: text and render helpers run under node", + "license": "BSD-3-Clause", + "type": "commonjs", + "dependencies": { + "@napi-rs/canvas": "^1.0.8", + "pdfjs-dist": "^6.3.289" + } +} diff --git a/judges/pdfjs-render.mjs b/judges/pdfjs-render.mjs new file mode 100644 index 0000000..d615057 --- /dev/null +++ b/judges/pdfjs-render.mjs @@ -0,0 +1,19 @@ +// Render one page of a PDF to PNG with pdf.js (Firefox's engine) on a native +// canvas — an independent raster judge next to poppler, MuPDF, Ghostscript, +// Quartz and pdfium. Usage: node pdfjs-render.mjs file.pdf out.png [page=1] [scale=1] +import { readFileSync, writeFileSync } from "node:fs"; +import { createCanvas } from "@napi-rs/canvas"; +import { getDocument } from "pdfjs-dist/legacy/build/pdf.mjs"; + +const [, , file, out, pageArg = "1", scaleArg = "1"] = process.argv; +const data = new Uint8Array(readFileSync(file)); +const doc = await getDocument({ data, useSystemFonts: true, disableFontFace: true, verbosity: 0 }).promise; +const page = await doc.getPage(Number(pageArg)); +const vp = page.getViewport({ scale: Number(scaleArg) }); +const canvas = createCanvas(Math.ceil(vp.width), Math.ceil(vp.height)); +const ctx = canvas.getContext("2d"); +ctx.fillStyle = "#fff"; +ctx.fillRect(0, 0, canvas.width, canvas.height); +await page.render({ canvasContext: ctx, viewport: vp }).promise; +writeFileSync(out, canvas.toBuffer("image/png")); +process.stdout.write(`pages ${doc.numPages}\n`); diff --git a/judges/pdfjs-text.mjs b/judges/pdfjs-text.mjs new file mode 100644 index 0000000..6ea0a47 --- /dev/null +++ b/judges/pdfjs-text.mjs @@ -0,0 +1,16 @@ +// Extract the text of every page of a PDF with pdf.js (Firefox's engine) and +// print it — the same signal `pdftotext` gives for poppler, from a second, +// independent parser. Usage: node pdfjs-text.mjs file.pdf +import { readFileSync } from "node:fs"; +import { getDocument } from "pdfjs-dist/legacy/build/pdf.mjs"; + +const data = new Uint8Array(readFileSync(process.argv[2])); +const doc = await getDocument({ data, useSystemFonts: true, disableFontFace: true, verbosity: 0 }).promise; +let out = ""; +for (let i = 1; i <= doc.numPages; i++) { + const page = await doc.getPage(i); + const tc = await page.getTextContent(); + out += tc.items.map((it) => it.str).join(" ") + "\n"; +} +process.stdout.write(`pages ${doc.numPages}\n`); +process.stdout.write(out);