Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions FIDELITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,18 @@ The committed PNGs under `testdata/renders/` back every claim here. Reproduce
them with the commands at the bottom. The measured-vs-Chrome numbers live in
[`bench/REPORT.md`](bench/REPORT.md).

## 2026-09-09 (round 70) — `document.currentScript` was hardcoded to always return null, so a bundler's own base-URL detection came back empty for every chunk load — found live on tailwindcss.com's Turbopack runtime, 10 failing script executions closed with one fix (engine#159)

No fresh issue was filed this round (checked `gh issue list` again). A visual scan of en.wikipedia.org's own montage — still the corpus's worst page by a wide margin — traced its whole-page diffuse vertical offset directly to the SAME already-documented external-content-variability culprit as rounds 63/64: Chrome's fetch this run happened to include the intermittent "Wiki Loves Monuments" promotional banner again, shifting everything below it down by a constant amount; correctly recognised as noise, not chased further. Pivoted to a fresh `Engine.JSLog` sweep (round 60/69's own methodology) instead, which turned up no new panics but did surface a cluster of 10 repeated script-execution failures on tailwindcss.com that hadn't been individually investigated before: `Error: chunk path empty but not in a worker`, once per lazily-loaded Turbopack chunk.

- **Root-caused by reasoning from the error's own phrasing, not by guessing from the minified bundle text**: "chunk path empty but not in a worker" reads as a bundler-runtime assertion — outside a Web Worker, a chunk-loading module ordinarily derives its OWN base path from `document.currentScript.src` (the standard Webpack/Turbopack idiom for locating sibling chunk files relative to the currently-running script), then falls back to a worker-specific mechanism only inside an actual worker. Checked this engine's own binding for `document.currentScript`: it was unconditionally hardcoded to `null`, regardless of whether a script was actually running — so that base-path detection always failed, producing an empty path, which is exactly the condition the runtime's own error message describes.
- **Fixed by actually tracking the currently-executing script**, rather than special-casing this one property: a new `binder.currentScript` field is set to the `<script>` element's own DOM node immediately before `execute()` runs it and cleared immediately after, in the single call site (`runScripts`) that invokes `execute` — matching the real spec precisely (non-null only for a synchronously-running classic script; null between scripts, and null again inside any later callback/timer, which run through a completely different path (`callSafely`) that never touches this field). `document.currentScript`'s own accessor now simply wraps whatever this field holds.
- **A pre-existing test had encoded the old bug as its own expectation**: `TestDocumentMisc`'s inline script asserted `document.currentScript` stringifies to `"null"` while it was itself the one synchronously executing — correct only because the old implementation always returned null, not because that's the real, spec-correct answer. Updated its assertion to check `instanceof HTMLScriptElement` instead of relying on incidental string output, which is both correct and more informative.
- **Verified live**: the real fetched tailwindcss.com page still renders correctly (checked directly, not just via the log count), and a fresh `Engine.JSLog` sweep shows every one of the 10 "chunk path empty" failures gone — nothing else on any of the ten corpus pages changed.
- Added `TestDocumentCurrentScriptIdentifiesTheRightScriptAndClearsAfterwards` (`js/js_test.go`), covering the two properties this fix actually needs to hold beyond "is it non-null": it must identify the RIGHT script when several are present (not just any script on the page), and it must go back to `null` once synchronous execution ends — inside a queued `setTimeout` callback specifically, since a bundler reading this only once per script would silently miscompute for every later chunk if either property held only sometimes. Git-stash-confirmed to fail with the exact predicted `TypeError` (`Cannot read property 'getAttribute' of undefined or null`) when the source fix is reverted. All five gated packages re-checked against their floors (css 99.5%, layout/paint/paginate 100.0%, dom 98.1%) — unaffected, since this fix lives in `js`, not itself coverage-gated.
- **Bench essentially flat on tailwindcss.com (0.7053→0.7038, ordinary noise) — correctly not oversold as a visual win**: the 10 chunks that now load successfully are plausibly deferred/lazy UI enhancements outside the top-of-page region this comparison captures, not core layout-affecting content — checked directly by rendering the real page, which looks the same as before at a glance. The real, demonstrated value is JavaScript execution correctness (ten fewer failed script runs on a real, heavily-trafficked site's actual production bundle), not a fidelity number. Two other pages moved by their now-familiar external-content-variability amounts (en.wikipedia.org and developer.mozilla.org both landed back on their exact round-68 baseline values, +0.0109/−0.0242 respectively from round 69's snapshot) — consistent with the "Wiki Loves Monuments" banner and MDN's own third-party ad slot toggling on/off between fetches, not anything this round's fix touches.
- No fresh issue filed, no PR/round-number collision — predicted #159 confirmed via `gh pr list` both before and after pushing.

## 2026-09-08 (round 69) — `document.importNode` crashed with a nil-pointer dereference when its argument wasn't a real bound node, aborting the whole script — found live on caniuse.com's real ad-network bundle (engine#158)

No fresh issue was filed this round (checked `gh issue list` again), and this round's two initial visual-scan leads both dead-ended cleanly rather than being forced: developer.mozilla.org's masthead icon differences traced to the already-known, cross-repo `oksvg` wordmark defect (the "mdn" logo is drawn as an SVG path, not text — rounds 25/44/64's already-documented rasteriser gap) plus a set of hamburger/sidebar-toggle icons that never appear in the static HTML at all (client-side-rendered, the same class of gap as react.dev's reskin/github.com's module-cap issues); react.dev's own top-of-page difference reconfirmed as the ALREADY-KNOWN reskin gap (round 51 explicitly checked and ruled unrelated to that round's own isEqualNode fix). Neither was chased further, matching this session's "no confirmed caller, don't guess" / "don't force an over-scoped lead" precedents.
Expand Down
26 changes: 15 additions & 11 deletions bench/REPORT.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

<!-- AUTO-GENERATED: this header, the results table and the montage list are regenerated by `go run ./cmd/compare`. The hand-written analysis below the BEGIN ANALYSIS marker is preserved across re-runs. -->

**Generated:** 2026-09-08 19:53 UTC
**Generated:** 2026-09-09 13:53 UTC
**Viewport:** 1024×768, device-scale 1
**Timing:** median of N=5 runs after 1 warmup, wall-clock incl. network fetch
**Chrome:** `/Applications/Google Chrome.app/Contents/MacOS/Google Chrome`
Expand All @@ -11,16 +11,16 @@
<!-- BEGIN RESULTS TABLE -->
| URL | SSIM | pixdiff % | webengine ms | chrome ms | speed× | region | status |
|-----|-----:|----------:|-------------:|----------:|-------:|:------:|:------:|
| example.com/ | 0.954 | 1.5 | 105.1 | 2503.7 | 23.82 | 1024×768 | ok |
| en.wikipedia.org/wiki/Go_(programming_language) | 0.423 | 22.6 | 2523.3 | 4897.1 | 1.94 | 1024×2500 | ok |
| pkg.go.dev/net/http | 0.644 | 33.7 | 6765.9 | 5729.3 | 0.85 | 1024×2500 | ok |
| go.dev/blog/ | 0.684 | 19.0 | 4798.6 | 4158.4 | 0.87 | 1024×1597 | ok |
| react.dev/ | 0.613 | 46.4 | 2765.4 | 3528.6 | 1.28 | 1024×2500 | ok |
| news.ycombinator.com/ | 0.616 | 13.8 | 1328.6 | 3762.3 | 2.83 | 1024×1109 | ok |
| developer.mozilla.org/en-US/docs/Web/CSS | 0.633 | 14.1 | 681.0 | 3972.5 | 5.83 | 1009×2500 | ok |
| github.com/golang/go | 0.544 | 30.2 | 3086.4 | 4071.8 | 1.32 | 1024×2500 | ok |
| tailwindcss.com/ | 0.705 | 13.2 | 4195.9 | 3686.5 | 0.88 | 1024×2500 | ok |
| caniuse.com/ | 0.664 | 18.0 | 3856.5 | 4961.6 | 1.29 | 1024×1737 | ok |
| example.com/ | 0.954 | 1.5 | 50.2 | 2109.5 | 42.03 | 1024×768 | ok |
| en.wikipedia.org/wiki/Go_(programming_language) | 0.434 | 22.1 | 2808.1 | 4395.9 | 1.57 | 1024×2500 | ok |
| pkg.go.dev/net/http | 0.644 | 33.7 | 5105.1 | 4117.8 | 0.81 | 1024×2500 | ok |
| go.dev/blog/ | 0.684 | 19.0 | 4122.4 | 3007.6 | 0.73 | 1024×1597 | ok |
| react.dev/ | 0.613 | 46.4 | 1790.8 | 2550.9 | 1.42 | 1024×2500 | ok |
| news.ycombinator.com/ | 0.606 | 13.9 | 1323.1 | 2588.5 | 1.96 | 1024×1109 | ok |
| developer.mozilla.org/en-US/docs/Web/CSS | 0.609 | 18.4 | 415.3 | 2114.8 | 5.09 | 1009×2500 | ok |
| github.com/golang/go | 0.543 | 30.2 | 3067.3 | 2305.9 | 0.75 | 1024×2500 | ok |
| tailwindcss.com/ | 0.704 | 13.3 | 3106.5 | 2447.4 | 0.79 | 1024×2500 | ok |
| caniuse.com/ | 0.664 | 18.0 | 3585.0 | 3824.3 | 1.07 | 1024×1737 | ok |
<!-- END RESULTS TABLE -->

Speed× is `chrome_ms / webengine_ms`: >1 means webengine is faster.
Expand Down Expand Up @@ -69,6 +69,10 @@ Speed× is `chrome_ms / webengine_ms`: >1 means webengine is faster.

<!-- BEGIN ANALYSIS (hand-written, preserved across re-runs) -->

## Honest analysis — 2026-09-09 (round 70): `document.currentScript` was hardcoded to always return null, so a bundler's own base-URL detection came back empty for every chunk load — found live on tailwindcss.com's Turbopack runtime via a fresh `Engine.JSLog` sweep (engine#159)

**tailwindcss.com: SSIM 0.7053→0.7038, ordinary noise, correctly not reported as a visual win** — the 10 Turbopack chunks that now load successfully (previously all failing with "chunk path empty but not in a worker") are plausibly deferred/lazy UI enhancements outside the top-of-page region this comparison captures; a direct render of the real page looks the same at a glance, before and after. The bug: `document.currentScript` was unconditionally hardcoded to `null`, so the standard Webpack/Turbopack idiom of reading `document.currentScript.src` to compute a chunk-loading base path always got an empty result outside a Worker. Fixed by actually tracking the synchronously-executing `<script>` node (`binder.currentScript`, set immediately before and cleared immediately after each `execute()` call in the one call site that invokes it) rather than special-casing the property — matching the real spec exactly, including going back to null inside a later timer callback. A pre-existing test (`TestDocumentMisc`) had encoded the old always-null behaviour as its own expectation; updated to check `instanceof HTMLScriptElement` instead, which is both correct and more informative than relying on incidental string output. The real, demonstrated value here is JavaScript execution correctness — ten fewer failed script runs on a real production site's actual bundle — not a fidelity number. Two other pages (en.wikipedia.org, developer.mozilla.org) landed back on their exact round-68 baseline SSIM values this run, reconfirming the already-documented external-content-variability pattern (Wikipedia's intermittent promotional banner, MDN's own third-party ad slot) rather than anything this fix touches.

## Honest analysis — 2026-09-08 (round 69): `document.importNode` crashed with a nil-pointer dereference on a non-node argument, aborting the whole script — found live on caniuse.com's own real ad-network bundle via a fresh corpus-wide `Engine.JSLog` sweep (engine#158)

**caniuse.com: SSIM 0.6658→0.6644, essentially unmeasurable noise either way, correctly NOT reported as a visual win** — the crashing script belongs to a third-party Google ad slot (`pagead2.googlesyndication.com`); its actual rendered content is exactly the kind of live, non-deterministic third-party variability this report has repeatedly documented as invisible to (or actively confounding) a pixel/SSIM comparison against one specific Chrome screenshot. The real value is crash-safety in this engine's own DOM binding: `document.importNode(x, deep)` passed its argument straight into `cloneNode`, which dereferenced it unconditionally — any call with a value that doesn't resolve to a real bound node (null, undefined, or an object from an unrelated API) panicked immediately and aborted that whole script, rather than failing just the one call the way every other malformed-argument case in this same binding file already does. Root-caused with a temporary `debug.PrintStack()` (removed before shipping) rather than guessed from the one-line panic message. Fixed by making `cloneNode` itself nil-safe, mirroring `contains`/`isEqualNode`'s own existing "nil in, no match, don't crash" treatment and `createTreeWalker`'s own nil-fallback a few lines above `importNode`'s binding — the fix pattern already existed in the same file, just not applied here. Two other pages moved by larger-than-typical amounts this run (en.wikipedia.org −0.0109, developer.mozilla.org +0.0278), both consistent with this exact corpus's own already-documented external-content variability (Wikipedia's intermittent promotional banner, MDN's own third-party ad slot) rather than anything this fix touches. Two dead-end leads (MDN's masthead — the already-known cross-repo oksvg wordmark defect plus client-rendered-only toggle icons; react.dev's top-of-page difference — the already-known reskin gap) were checked and correctly abandoned before this fix was found, rather than forced.
Expand Down
Binary file modified bench/out/caniuse.com.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified bench/out/developer.mozilla.org_en-US_docs_Web_CSS.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified bench/out/en.wikipedia.org_wiki_Go_programming_language.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified bench/out/github.com_golang_go.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified bench/out/news.ycombinator.com.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified bench/out/tailwindcss.com.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
84 changes: 42 additions & 42 deletions bench/results.json
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
[
{
"url": "https://example.com/",
"webengine_ms": 105.122,
"chrome_ms": 2503.735,
"speed_ratio": 23.817421662449345,
"webengine_ms": 50.19,
"chrome_ms": 2109.522,
"speed_ratio": 42.03072325164376,
"ssim": 0.9541451179596087,
"pixdiff_pct": 1.53350830078125,
"region_w": 1024,
Expand All @@ -14,11 +14,11 @@
},
{
"url": "https://en.wikipedia.org/wiki/Go_(programming_language)",
"webengine_ms": 2523.288,
"chrome_ms": 4897.09,
"speed_ratio": 1.9407574561445227,
"ssim": 0.4228803249878166,
"pixdiff_pct": 22.645546875,
"webengine_ms": 2808.129,
"chrome_ms": 4395.877,
"speed_ratio": 1.5654113468433966,
"ssim": 0.43376860237167725,
"pixdiff_pct": 22.061171875,
"region_w": 1024,
"region_h": 2500,
"montage": "out/en.wikipedia.org_wiki_Go_programming_language.png",
Expand All @@ -27,9 +27,9 @@
},
{
"url": "https://pkg.go.dev/net/http",
"webengine_ms": 6765.882,
"chrome_ms": 5729.271,
"speed_ratio": 0.8467884896603282,
"webengine_ms": 5105.13,
"chrome_ms": 4117.752,
"speed_ratio": 0.8065910172708629,
"ssim": 0.6435426619444631,
"pixdiff_pct": 33.6956640625,
"region_w": 1024,
Expand All @@ -40,9 +40,9 @@
},
{
"url": "https://go.dev/blog/",
"webengine_ms": 4798.577,
"chrome_ms": 4158.418,
"speed_ratio": 0.8665939923439802,
"webengine_ms": 4122.412,
"chrome_ms": 3007.557,
"speed_ratio": 0.7295624503324751,
"ssim": 0.6835407066166453,
"pixdiff_pct": 19.02700864902943,
"region_w": 1024,
Expand All @@ -53,9 +53,9 @@
},
{
"url": "https://react.dev/",
"webengine_ms": 2765.403,
"chrome_ms": 3528.633,
"speed_ratio": 1.2759923237227992,
"webengine_ms": 1790.845,
"chrome_ms": 2550.925,
"speed_ratio": 1.424425341109923,
"ssim": 0.612591251167784,
"pixdiff_pct": 46.423398437500005,
"region_w": 1024,
Expand All @@ -66,11 +66,11 @@
},
{
"url": "https://news.ycombinator.com/",
"webengine_ms": 1328.57,
"chrome_ms": 3762.314,
"speed_ratio": 2.831852292314293,
"ssim": 0.6163217835291052,
"pixdiff_pct": 13.756322559738503,
"webengine_ms": 1323.059,
"chrome_ms": 2588.474,
"speed_ratio": 1.9564312702608124,
"ssim": 0.606299034173214,
"pixdiff_pct": 13.94828885820559,
"region_w": 1024,
"region_h": 1109,
"montage": "out/news.ycombinator.com.png",
Expand All @@ -79,11 +79,11 @@
},
{
"url": "https://developer.mozilla.org/en-US/docs/Web/CSS",
"webengine_ms": 681.047,
"chrome_ms": 3972.53,
"speed_ratio": 5.832974816716027,
"ssim": 0.6330293140259217,
"pixdiff_pct": 14.128642220019822,
"webengine_ms": 415.276,
"chrome_ms": 2114.756,
"speed_ratio": 5.092410830387501,
"ssim": 0.60875508648779,
"pixdiff_pct": 18.369236868186324,
"region_w": 1009,
"region_h": 2500,
"montage": "out/developer.mozilla.org_en-US_docs_Web_CSS.png",
Expand All @@ -92,11 +92,11 @@
},
{
"url": "https://github.com/golang/go",
"webengine_ms": 3086.404,
"chrome_ms": 4071.788,
"speed_ratio": 1.3192660455338965,
"ssim": 0.5436189514353632,
"pixdiff_pct": 30.174140625,
"webengine_ms": 3067.251,
"chrome_ms": 2305.858,
"speed_ratio": 0.7517669730974087,
"ssim": 0.543321694376328,
"pixdiff_pct": 30.1932421875,
"region_w": 1024,
"region_h": 2500,
"montage": "out/github.com_golang_go.png",
Expand All @@ -105,11 +105,11 @@
},
{
"url": "https://tailwindcss.com/",
"webengine_ms": 4195.857,
"chrome_ms": 3686.497,
"speed_ratio": 0.8786040611012244,
"ssim": 0.7052946258793852,
"pixdiff_pct": 13.184648437499998,
"webengine_ms": 3106.514,
"chrome_ms": 2447.395,
"speed_ratio": 0.7878268052228318,
"ssim": 0.703834739996964,
"pixdiff_pct": 13.2530078125,
"region_w": 1024,
"region_h": 2500,
"montage": "out/tailwindcss.com.png",
Expand All @@ -118,11 +118,11 @@
},
{
"url": "https://caniuse.com/",
"webengine_ms": 3856.465,
"chrome_ms": 4961.642,
"speed_ratio": 1.28657773375358,
"ssim": 0.6643652128965394,
"pixdiff_pct": 17.999109455958546,
"webengine_ms": 3584.989,
"chrome_ms": 3824.333,
"speed_ratio": 1.066762826887335,
"ssim": 0.6642873287595739,
"pixdiff_pct": 18.029525133131834,
"region_w": 1024,
"region_h": 1737,
"montage": "out/caniuse.com.png",
Expand Down
10 changes: 9 additions & 1 deletion js/dom.go
Original file line number Diff line number Diff line change
Expand Up @@ -783,7 +783,15 @@ func (b *binder) installDocument() *goja.Object {
d.Set("writeln", func(goja.FunctionCall) goja.Value { return goja.Undefined() })
d.Set("open", func(goja.FunctionCall) goja.Value { return d })
d.Set("close", func(goja.FunctionCall) goja.Value { return goja.Undefined() })
b.accessor(d, "currentScript", func() goja.Value { return goja.Null() }, nil)
// document.currentScript: non-null only while a classic <script> is
// synchronously executing (see binder.currentScript's own doc comment).
// Hardcoded to always-null before this — real bundlers commonly read
// currentScript.src to compute their OWN base URL for later chunk
// loads; always-null made that base URL empty. Found live on
// tailwindcss.com: its Turbopack runtime threw "chunk path empty but
// not in a worker" for every non-worker chunk load, since the base path
// it derives this way came back empty.
b.accessor(d, "currentScript", func() goja.Value { return b.wrap(b.currentScript) }, nil)
b.accessor(d, "activeElement", func() goja.Value { return b.wrap(dom.Find(b.root, "body")) }, nil)
b.accessor(d, "scrollingElement", func() goja.Value { return b.wrap(dom.Find(b.root, "html")) }, nil)
if p := b.protos["Document"]; p != nil {
Expand Down
11 changes: 10 additions & 1 deletion js/js.go
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,12 @@ type binder struct {
// runs only scripts injected since the previous pass (in document order),
// never re-running a script.
executed map[*dom.Node]bool
// currentScript is the <script> element synchronously executing right now
// (nil between scripts, during callbacks/timers, and during module-bundle
// execution — matching the real document.currentScript spec: non-null only
// for a synchronously-running classic script). Set/cleared by runScripts
// around each execute call.
currentScript *dom.Node
}

// Run builds the DOM binding on root (a dom.Document node), sets the JS-enabled
Expand Down Expand Up @@ -139,7 +145,10 @@ func (b *binder) runScripts(res *Result) {
if !ok {
continue
}
if b.execute(src, s.name) {
b.currentScript = s.node
ok = b.execute(src, s.name)
b.currentScript = nil
if ok {
res.ScriptsRun++
} else {
res.ScriptsFailed++
Expand Down
Loading