Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions FIDELITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,18 @@ The committed PNGs under `testdata/renders/` back every claim here. Reproduce
them with the commands at the bottom. The measured-vs-Chrome numbers live in
[`bench/REPORT.md`](bench/REPORT.md).

## 2026-09-09 (round 71) — two real, independently-confirmed gaps in dynamic script loading closed (`onload`/`onerror` never existed on any element; a script appended during timer-draining could be silently stranded, never run) — neither alone closes react.dev's own `ChunkLoadError`, whose remaining root cause is a genuinely deeper, separate simplification flagged for a future round, not attempted here (engine#160)

No fresh issue was filed this round (checked `gh issue list` again). en.wikipedia.org's own diffuse whole-page offset reconfirmed (not rediscovered) as the same external "Wiki Loves Monuments" banner noise from rounds 63/64/70. Continued round 69/70's `Engine.JSLog`-sweep methodology instead of a fresh visual scan, and dug into react.dev's own recurring `ChunkLoadError: Loading chunk 809 failed. (timeout: …)` — present in every prior sweep this session but never individually root-caused, always folded into the page's already-known, much bigger "reskin gap."

- **Root-caused with direct, temporary Go-level instrumentation** (added, used, then fully removed before shipping — confirmed via `git diff`) rather than guessed from the minified bundle: traced the failing chunk's own dynamically-created `<script>` element all the way through — `document.createElement('script')` DOES run, `document.head.appendChild(...)` DOES attach it to the real tree with the correct `src` — but its fetch is NEVER even attempted (zero matching calls into this engine's own script-fetch function). Two separate, confirmed defects surfaced along this trail:
1. **`onload`/`onerror` did not exist as properties on ANY element** — webpack/Turbopack's own classic chunk-loading helper sets both directly on a dynamically-created `<script>` (never via `addEventListener`), so even if the chunk HAD loaded, its own promise would never have settled. Fixed by wiring both generically in `defineElement`, backed by a new `binder.onHandlers` map that lets a later assignment replace (not stack with) an earlier one — the correct IDL-event-handler-attribute semantics — implemented as a registered listener under the hood so `dispatch` needed no separate code path. `runScripts` now also dispatches "load" on a script that fetched-and-ran (regardless of whether its own top-level code threw — a real browser reports that separately via `window`'s error event, not the element's own) and "error" specifically when the FETCH itself fails.
2. **A dynamically-injected `<script>` created inside a timer callback could be silently stranded, never run at all.** `Session.RunPending` decided its "did anything happen" signal from `runScripts` alone, called BEFORE `drainTimers` — but the standard chunk-loading idiom (`document.createElement('script'); s.onload=...; s.src=chunkURL; document.head.appendChild(s)`) runs from EXACTLY such a timer callback, so the check always ran too early to see it. The caller (`dynamic.go`'s settle loop) reads a false "ran" as "nothing left to do" and stops iterating — so the freshly-appended script, sitting correctly in the DOM, was simply never picked up. A first attempt at the fix (calling `runScripts` a second time within the same `RunPending` call) over-corrected: it broke `TestSettleFixpointCap` by running a whole SYNCHRONOUS self-replicating script chain's next TWO generations in one pass instead of one per outer settle iteration. Fixed instead by having `RunPending` check, after `drainTimers`, whether any unexecuted `<script>` now exists (`hasPendingScripts`) — reporting "there is more to do" without actually running it inline, preserving the existing one-generation-per-outer-pass pacing; the NEXT `RunPending` call (from the settle loop's next iteration, exactly as it already does for any other case) is what actually executes it.
- **Honestly, neither fix — even together — closes react.dev's own visible `ChunkLoadError` symptom**, and this was verified directly rather than assumed: re-running the exact same instrumentation after both fixes still shows zero fetch attempts for the chunk. The most likely remaining cause (not fixed this round, flagged as a real, separate, deeper issue): this engine's `setTimeout` binding discards its own delay argument entirely, draining every timer job in pure FIFO order regardless of requested delay — so a bundler's own internal "give up after 120 seconds" safety-net timeout races (and structurally always wins) against the chunk's own near-instant real fetch, an architectural simplification affecting the whole timer model, not a narrow one-round fix.
- Added `TestSessionInjectedScriptOnloadAndOnerrorFire` (covering both outcomes: a script that fetches-and-runs fires `onload`, one whose fetch fails fires `onerror`, and never the wrong one) and `TestSessionRunPendingReportsPendingScriptAppendedDuringDrainTimers` (a focused white-box test exercising the exact "runScripts finds nothing, but this call's own drainTimers appends a script" ordering, since constructing that exact interleaving through real page HTML alone proved impractical). Both git-stash-confirmed to fail with the exact predicted symptoms when their respective mechanism is reverted. `TestSettleFixpointCap` (the test the first, over-corrected fix attempt broke) re-confirmed still green. All five gated packages re-checked against their floors (css 99.5%, layout/paint/paginate 100.0%, dom 98.1%) — unaffected, `js` isn't itself coverage-gated.
- **Bench flat on react.dev (0.6126→0.6126, exactly unchanged) — correctly predicted in advance, not a surprise**: confirmed directly via the same instrumentation that the chunk still never fetches, for the separate, deeper reason above. Other corpus movements this run (en.wikipedia.org, developer.mozilla.org, news.ycombinator.com, tailwindcss.com) are consistent with this harness's own already-documented sources of noise — external promotional/ad-banner presence toggling for the first two, and HN's own live, constantly-rotating front page plus its previously-documented diffuse font-metric spacing for the third (checked directly via a side-by-side crop: same stories, same ranking, no attributable rendering difference) — none plausibly related to this round's script-loading-specific fixes.
- No fresh issue filed, no PR/round-number collision — predicted #160 confirmed via `gh pr list` both before and after pushing.

## 2026-09-09 (round 70) — `document.currentScript` was hardcoded to always return null, so a bundler's own base-URL detection came back empty for every chunk load — found live on tailwindcss.com's Turbopack runtime, 10 failing script executions closed with one fix (engine#159)

No fresh issue was filed this round (checked `gh issue list` again). A visual scan of en.wikipedia.org's own montage — still the corpus's worst page by a wide margin — traced its whole-page diffuse vertical offset directly to the SAME already-documented external-content-variability culprit as rounds 63/64: Chrome's fetch this run happened to include the intermittent "Wiki Loves Monuments" promotional banner again, shifting everything below it down by a constant amount; correctly recognised as noise, not chased further. Pivoted to a fresh `Engine.JSLog` sweep (round 60/69's own methodology) instead, which turned up no new panics but did surface a cluster of 10 repeated script-execution failures on tailwindcss.com that hadn't been individually investigated before: `Error: chunk path empty but not in a worker`, once per lazily-loaded Turbopack chunk.
Expand Down
26 changes: 15 additions & 11 deletions bench/REPORT.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

<!-- AUTO-GENERATED: this header, the results table and the montage list are regenerated by `go run ./cmd/compare`. The hand-written analysis below the BEGIN ANALYSIS marker is preserved across re-runs. -->

**Generated:** 2026-09-09 13:53 UTC
**Generated:** 2026-09-09 20:02 UTC
**Viewport:** 1024×768, device-scale 1
**Timing:** median of N=5 runs after 1 warmup, wall-clock incl. network fetch
**Chrome:** `/Applications/Google Chrome.app/Contents/MacOS/Google Chrome`
Expand All @@ -11,16 +11,16 @@
<!-- BEGIN RESULTS TABLE -->
| URL | SSIM | pixdiff % | webengine ms | chrome ms | speed× | region | status |
|-----|-----:|----------:|-------------:|----------:|-------:|:------:|:------:|
| example.com/ | 0.954 | 1.5 | 50.2 | 2109.5 | 42.03 | 1024×768 | ok |
| en.wikipedia.org/wiki/Go_(programming_language) | 0.434 | 22.1 | 2808.1 | 4395.9 | 1.57 | 1024×2500 | ok |
| pkg.go.dev/net/http | 0.644 | 33.7 | 5105.1 | 4117.8 | 0.81 | 1024×2500 | ok |
| go.dev/blog/ | 0.684 | 19.0 | 4122.4 | 3007.6 | 0.73 | 1024×1597 | ok |
| react.dev/ | 0.613 | 46.4 | 1790.8 | 2550.9 | 1.42 | 1024×2500 | ok |
| news.ycombinator.com/ | 0.606 | 13.9 | 1323.1 | 2588.5 | 1.96 | 1024×1109 | ok |
| developer.mozilla.org/en-US/docs/Web/CSS | 0.609 | 18.4 | 415.3 | 2114.8 | 5.09 | 1009×2500 | ok |
| github.com/golang/go | 0.543 | 30.2 | 3067.3 | 2305.9 | 0.75 | 1024×2500 | ok |
| tailwindcss.com/ | 0.704 | 13.3 | 3106.5 | 2447.4 | 0.79 | 1024×2500 | ok |
| caniuse.com/ | 0.664 | 18.0 | 3585.0 | 3824.3 | 1.07 | 1024×1737 | ok |
| example.com/ | 0.954 | 1.5 | 90.1 | 1956.3 | 21.72 | 1024×768 | ok |
| en.wikipedia.org/wiki/Go_(programming_language) | 0.423 | 22.6 | 3560.5 | 4405.5 | 1.24 | 1024×2500 | ok |
| pkg.go.dev/net/http | 0.644 | 33.7 | 6202.0 | 4410.7 | 0.71 | 1024×2500 | ok |
| go.dev/blog/ | 0.684 | 19.0 | 4654.2 | 2947.5 | 0.63 | 1024×1597 | ok |
| react.dev/ | 0.613 | 46.4 | 2242.8 | 2733.6 | 1.22 | 1024×2500 | ok |
| news.ycombinator.com/ | 0.643 | 12.7 | 1611.6 | 2672.1 | 1.66 | 1024×1109 | ok |
| developer.mozilla.org/en-US/docs/Web/CSS | 0.633 | 14.1 | 609.9 | 2131.0 | 3.49 | 1009×2500 | ok |
| github.com/golang/go | 0.543 | 30.2 | 3025.6 | 2360.5 | 0.78 | 1024×2500 | ok |
| tailwindcss.com/ | 0.708 | 12.8 | 3944.3 | 2586.4 | 0.66 | 1024×2500 | ok |
| caniuse.com/ | 0.664 | 18.1 | 3718.4 | 4036.4 | 1.09 | 1024×1737 | ok |
<!-- END RESULTS TABLE -->

Speed× is `chrome_ms / webengine_ms`: >1 means webengine is faster.
Expand Down Expand Up @@ -69,6 +69,10 @@ Speed× is `chrome_ms / webengine_ms`: >1 means webengine is faster.

<!-- BEGIN ANALYSIS (hand-written, preserved across re-runs) -->

## Honest analysis — 2026-09-09 (round 71): two real, independently-confirmed gaps in dynamic script loading closed (missing `onload`/`onerror`; a script appended during timer-draining could be silently stranded) — neither alone closes react.dev's own `ChunkLoadError`, whose remaining cause is flagged as a separate, deeper issue (engine#160)

**react.dev: SSIM 0.6126→0.6126, exactly unchanged — correctly predicted in advance, not a surprise**: direct instrumentation confirms the chunk-loader's own dynamically-created `<script src="…f809.js">` is still never fetched, even after both fixes. Traced precisely: `document.createElement('script')` and `document.head.appendChild(...)` both run correctly with the right `src`, but two real, separate defects were gating anything further — (1) `onload`/`onerror` did not exist as properties on any element at all, so even a successful chunk load could never have resolved webpack's own loading promise, and (2) `Session.RunPending` decided whether "anything happened" from `runScripts` alone, called BEFORE `drainTimers` — but the standard chunk-loading idiom creates its `<script>` from exactly a timer callback, so a freshly-appended, never-executed script could sit in the DOM while the caller's settle loop read a false "ran" as "nothing left to do" and stopped iterating, never picking it up. Fixed by wiring `onload`/`onerror` generically (backed by a new per-node handler-slot map so a later assignment correctly replaces, not stacks with, an earlier one) and by having `RunPending` check for pending unexecuted scripts after draining timers. An initial version of the second fix over-corrected — running `runScripts` twice within one `RunPending` call broke `TestSettleFixpointCap` by advancing a synchronous self-replicating script chain two generations in one outer pass instead of one — fixed by reporting "pending" without executing inline, preserving the existing per-pass pacing. Even combined, these fixes don't close the visible symptom: the most likely remaining cause is that this engine's `setTimeout` binding ignores its own delay argument, draining every timer job in pure FIFO order — so a bundler's internal "give up after 120s" safety-net timeout structurally races ahead of (and always wins against) the chunk's own near-instant real fetch. That's a genuinely deeper, separate architectural simplification, not attempted this round. Other corpus movements (en.wikipedia.org, developer.mozilla.org, news.ycombinator.com, tailwindcss.com) are consistent with this harness's own already-documented noise sources — external banner/ad-slot toggling for the first two, HN's own live rotating front page for the third (checked directly: identical stories/ranking, no attributable difference) — none plausibly related to this round's script-loading-specific fixes.

## Honest analysis — 2026-09-09 (round 70): `document.currentScript` was hardcoded to always return null, so a bundler's own base-URL detection came back empty for every chunk load — found live on tailwindcss.com's Turbopack runtime via a fresh `Engine.JSLog` sweep (engine#159)

**tailwindcss.com: SSIM 0.7053→0.7038, ordinary noise, correctly not reported as a visual win** — the 10 Turbopack chunks that now load successfully (previously all failing with "chunk path empty but not in a worker") are plausibly deferred/lazy UI enhancements outside the top-of-page region this comparison captures; a direct render of the real page looks the same at a glance, before and after. The bug: `document.currentScript` was unconditionally hardcoded to `null`, so the standard Webpack/Turbopack idiom of reading `document.currentScript.src` to compute a chunk-loading base path always got an empty result outside a Worker. Fixed by actually tracking the synchronously-executing `<script>` node (`binder.currentScript`, set immediately before and cleared immediately after each `execute()` call in the one call site that invokes it) rather than special-casing the property — matching the real spec exactly, including going back to null inside a later timer callback. A pre-existing test (`TestDocumentMisc`) had encoded the old always-null behaviour as its own expectation; updated to check `instanceof HTMLScriptElement` instead, which is both correct and more informative than relying on incidental string output. The real, demonstrated value here is JavaScript execution correctness — ten fewer failed script runs on a real production site's actual bundle — not a fidelity number. Two other pages (en.wikipedia.org, developer.mozilla.org) landed back on their exact round-68 baseline SSIM values this run, reconfirming the already-documented external-content-variability pattern (Wikipedia's intermittent promotional banner, MDN's own third-party ad slot) rather than anything this fix touches.
Expand Down
Binary file modified bench/out/caniuse.com.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified bench/out/developer.mozilla.org_en-US_docs_Web_CSS.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified bench/out/en.wikipedia.org_wiki_Go_programming_language.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified bench/out/github.com_golang_go.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified bench/out/news.ycombinator.com.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified bench/out/react.dev.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified bench/out/tailwindcss.com.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
86 changes: 43 additions & 43 deletions bench/results.json
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
[
{
"url": "https://example.com/",
"webengine_ms": 50.19,
"chrome_ms": 2109.522,
"speed_ratio": 42.03072325164376,
"webengine_ms": 90.076,
"chrome_ms": 1956.303,
"speed_ratio": 21.71836005151206,
"ssim": 0.9541451179596087,
"pixdiff_pct": 1.53350830078125,
"region_w": 1024,
Expand All @@ -14,11 +14,11 @@
},
{
"url": "https://en.wikipedia.org/wiki/Go_(programming_language)",
"webengine_ms": 2808.129,
"chrome_ms": 4395.877,
"speed_ratio": 1.5654113468433966,
"ssim": 0.43376860237167725,
"pixdiff_pct": 22.061171875,
"webengine_ms": 3560.529,
"chrome_ms": 4405.463,
"speed_ratio": 1.2373057486682455,
"ssim": 0.42289887244503654,
"pixdiff_pct": 22.6463671875,
"region_w": 1024,
"region_h": 2500,
"montage": "out/en.wikipedia.org_wiki_Go_programming_language.png",
Expand All @@ -27,9 +27,9 @@
},
{
"url": "https://pkg.go.dev/net/http",
"webengine_ms": 5105.13,
"chrome_ms": 4117.752,
"speed_ratio": 0.8065910172708629,
"webengine_ms": 6201.999,
"chrome_ms": 4410.695,
"speed_ratio": 0.7111731233752214,
"ssim": 0.6435426619444631,
"pixdiff_pct": 33.6956640625,
"region_w": 1024,
Expand All @@ -40,9 +40,9 @@
},
{
"url": "https://go.dev/blog/",
"webengine_ms": 4122.412,
"chrome_ms": 3007.557,
"speed_ratio": 0.7295624503324751,
"webengine_ms": 4654.22,
"chrome_ms": 2947.526,
"speed_ratio": 0.6333018207132451,
"ssim": 0.6835407066166453,
"pixdiff_pct": 19.02700864902943,
"region_w": 1024,
Expand All @@ -53,10 +53,10 @@
},
{
"url": "https://react.dev/",
"webengine_ms": 1790.845,
"chrome_ms": 2550.925,
"speed_ratio": 1.424425341109923,
"ssim": 0.612591251167784,
"webengine_ms": 2242.775,
"chrome_ms": 2733.637,
"speed_ratio": 1.2188636844979992,
"ssim": 0.612597132081433,
"pixdiff_pct": 46.423398437500005,
"region_w": 1024,
"region_h": 2500,
Expand All @@ -66,11 +66,11 @@
},
{
"url": "https://news.ycombinator.com/",
"webengine_ms": 1323.059,
"chrome_ms": 2588.474,
"speed_ratio": 1.9564312702608124,
"ssim": 0.606299034173214,
"pixdiff_pct": 13.94828885820559,
"webengine_ms": 1611.572,
"chrome_ms": 2672.134,
"speed_ratio": 1.6580916024850272,
"ssim": 0.6431851090845011,
"pixdiff_pct": 12.71635834648332,
"region_w": 1024,
"region_h": 1109,
"montage": "out/news.ycombinator.com.png",
Expand All @@ -79,11 +79,11 @@
},
{
"url": "https://developer.mozilla.org/en-US/docs/Web/CSS",
"webengine_ms": 415.276,
"chrome_ms": 2114.756,
"speed_ratio": 5.092410830387501,
"ssim": 0.60875508648779,
"pixdiff_pct": 18.369236868186324,
"webengine_ms": 609.945,
"chrome_ms": 2131.039,
"speed_ratio": 3.4938215740763514,
"ssim": 0.6330293140259217,
"pixdiff_pct": 14.128642220019822,
"region_w": 1009,
"region_h": 2500,
"montage": "out/developer.mozilla.org_en-US_docs_Web_CSS.png",
Expand All @@ -92,11 +92,11 @@
},
{
"url": "https://github.com/golang/go",
"webengine_ms": 3067.251,
"chrome_ms": 2305.858,
"speed_ratio": 0.7517669730974087,
"ssim": 0.543321694376328,
"pixdiff_pct": 30.1932421875,
"webengine_ms": 3025.55,
"chrome_ms": 2360.518,
"speed_ratio": 0.7801946753482838,
"ssim": 0.5434039906712012,
"pixdiff_pct": 30.1891796875,
"region_w": 1024,
"region_h": 2500,
"montage": "out/github.com_golang_go.png",
Expand All @@ -105,11 +105,11 @@
},
{
"url": "https://tailwindcss.com/",
"webengine_ms": 3106.514,
"chrome_ms": 2447.395,
"speed_ratio": 0.7878268052228318,
"ssim": 0.703834739996964,
"pixdiff_pct": 13.2530078125,
"webengine_ms": 3944.334,
"chrome_ms": 2586.437,
"speed_ratio": 0.6557347831091384,
"ssim": 0.70823820369105,
"pixdiff_pct": 12.791015624999998,
"region_w": 1024,
"region_h": 2500,
"montage": "out/tailwindcss.com.png",
Expand All @@ -118,11 +118,11 @@
},
{
"url": "https://caniuse.com/",
"webengine_ms": 3584.989,
"chrome_ms": 3824.333,
"speed_ratio": 1.066762826887335,
"ssim": 0.6642873287595739,
"pixdiff_pct": 18.029525133131834,
"webengine_ms": 3718.367,
"chrome_ms": 4036.445,
"speed_ratio": 1.0855423899792571,
"ssim": 0.6641586604080089,
"pixdiff_pct": 18.097215475676453,
"region_w": 1024,
"region_h": 1737,
"montage": "out/caniuse.com.png",
Expand Down
Loading