Uh oh!
There was an error while loading. Please reload this page.
feat(benchmarks): client-side CPU benchmark scenario suite - #7732
Merged
Conversation
Add 11 client-side CPU benchmark scenarios to benchmarks/client-nav, each implemented identically across react/solid/vue and tracked by CodSpeed through the existing Benchmarks workflow (no CI changes needed): async-pipeline, control-flow, head, history, links, loaders, mount, nested-params, preload, route-tree-scale, search-params Each scenario is an isolated file-based-routing app (route trees generated by @tanstack/router-plugin) built for production and driven in jsdom by a shared harness (scenarios/harness.ts): circular step sequences of real <Link> clicks (plus hover/preload/history-traversal/invalidate steps), each synchronized on the router's onRendered event, with a warm-up lap that asserts every step's observable output. Determinism rules: no wall-clock timers (counted 0ms hops only), staleTime/gcTime pinned to 0 or 1e9, deterministic preload staleness, stationary DOM and history depth. The existing baseline apps and bench names are untouched for CodSpeed continuity; baseline vite configs only gain an explicit root/setupFiles so they resolve identically under the new per-framework aggregate configs. Notable constraints encoded in the scenarios: - Component-level Await/Suspense is excluded from async-pipeline: React 19 throttles every Suspense reveal by ~300ms wall-clock, which is inherently non-deterministic to benchmark. - control-flow/react silences React 19's onCaughtError reporting, which otherwise dominates the measured loop with console I/O. - Vue route components must be defined before their createFileRoute call (bundled var hoisting silently yields component: undefined otherwise). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Important Review skippedToo many files! This PR contains 626 files, which is 326 over the limit of 300. To get a review, narrow the scope: ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (626)
You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Contributor
View your CI Pipeline Execution ↗ for commit f373691
☁️ Nx Cloud last updated this comment at |
Contributor
🚀 Changeset Version PreviewNo changeset entries found. Merging this PR will not cause a version bump for any packages. |
Contributor
Bundle Size Benchmarks
Current gzip tracks all emitted client JS chunks. Initial gzip tracks only the entry/import graph. Trend sparkline is historical current gzip ending with this PR measurement; lower is better. |
Merging this PR will improve performance by 11.51%
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing  |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All benches in @benchmarks/client-nav now start with "client" so they are easy to identify on the CodSpeed dashboard (the baseline already did). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
…lity Fixes from an adversarial review of the scenario suite: - mount: create the browser history explicitly and destroy() it on unmount. The default per-router createBrowserHistory() monkey-patches window.history.pushState/replaceState, chaining one wrapper per router — per-mount cost degraded 1.8ms -> 6ms over 3000 mounts (O(N^2) drift in the cold-start bench). Verified flat (~1.86ms) after the fix. - links: replace the five <MatchRoute> components with setup-scoped useMatchRoute probes in all three apps. vue-router's MatchRoute calls useMatchRoute() inside its render function, leaking one undisposed watcher per render — the Vue bench accumulated ~5 dead subscribers per navigation. Verified flat per-tick cost over 2000 ticks after the fix. - all scenarios: bound the scroll-restoration cache with getScrollRestorationKey: (location) => location.pathname. The default key is a fresh random per-entry location key, so the module-level cache grew one entry per push navigation for the whole run. - harness hover step: dispatch a single mouseover instead of mouseover+mouseenter. Solid/Vue attach intent-preload handlers to BOTH events, so the old double dispatch ran the preload pipeline twice for them but once for React. Verified all three adapters preload exactly once per hover via cached-matches probes. - README: correct the loaders row (the invalidate step means cached routes re-run once per lap), the links row, and document the session-history growth characteristics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
…arams Closes the three coverage gaps identified in the scenario-suite analysis: - new `rewrites` scenario (react/solid/vue): router basepath '/app' composed with a locale input/output rewrite pair — the client analog of the SSR rewrites scenario. Every href build runs the output rewrite and every committed location runs the input rewrite; assertions check both the internal router pathname and the external window pathname. The harness gained an `initialUrl` option since a basepath app cannot start at '/'. - route-tree-scale now builds with `autoCodeSplitting: true`, so its navigations also exercise lazy route-chunk resolution — the default shape of real file-based apps, previously uncovered. - nested-params and route-tree-scale param sets now include characters that need percent-encoding (spaces, `&`, `%`, `+`, unicode) so the segment encode/decode paths run on every navigation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
…t bench
CodSpeed excludes syscall time from simulation measures, and past a
threshold it warns ("cannot be consistently instrumented") and skips the
benchmark; the exclusion is also inconsistent run-to-run, which was the
root cause of the two same-code CI swings (route-tree-scale react 13%,
async-pipeline react 3.6%) and of skipped runs on hop-heavy benches like
async-pipeline solid (579 calls / 12.1ms).
- Replace all counted setTimeout(0) hops with setImmediate hops (harness
timerHop + async-pipeline hopDelay). Both are deterministic event-loop
turns that yield to timers and the React Node scheduler, but a timer
costs ~3-4 syscalls (timerfd + epoll) per hop vs ~1 for an immediate.
Same-machine instrumented before/after: preload solid 39.8ms -> 1.2ms of
syscall time, preload vue 30.1 -> 1.1, async-pipeline solid 23.5 -> 1.1,
async-pipeline vue 17.9 -> 0.9. head/route-tree-scale are unchanged
(their residual syscalls are GC/scheduler, and their measures were
already stable); history's syscalls come from jsdom's internal
timer-queued traversal and are not reachable from bench code (its
measured values are stable regardless).
- mount: 6 ticks per iteration instead of 2. A single mount simulates to
only ~8ms and very short measures amplify allocator/GC quantization;
the bench now measures ~50ms+ of simulated CPU.
All 39 benches pass; full nx typecheck green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
…imers The history benches were skip-warned by CodSpeed for excessive syscall time (5-26ms, highly variable across runs). Root cause: jsdom delivers history traversals through two nested window.setTimeout(0) tasks, and Node clamps zero timers to 1ms of wall time — so while awaiting onRendered after back/forward/go, the event loop blocked in epoll_wait until each timer expired. That blocked wall-time is recorded as syscall time, and its duration depends on host timer behavior, hence the variability. The harness now reroutes zero-delay window.setTimeout calls onto setImmediate for the duration of a benchmark (non-zero delays pass through; clearTimeout handles both). Zero-delay timeouts carry no ordering semantics a check-phase immediate doesn't satisfy, and the suite's conventions already forbid real-delay timers in measured code. Instrumented A/A verification (2 runs per framework): history syscall time react 23-26ms -> 2.5ms, solid 14-23ms -> 1.1ms, vue 11-12ms -> 1.2ms, with bit-identical syscall counts across runs and measured values stable to 0.06%. The benches also stopped measuring sleep: locally an iteration dropped from ~42-49ms to ~5-11ms of actual CPU work. All 39 benches pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Nx Cloud has identified a possible root cause for your failed CI:
We identified the tanstack-start-example-rscs:build failure as a pre-existing environment issue: nitro-nightly attempts to import ./runtime/meta from nitro, but that subpath is not exported in nitro's package.json, indicating a version incompatibility unrelated to this PR's benchmark changes. Our cross-branch analysis confirmed the identical error exists in branch 7735, proving it predates this PR.
No code changes were suggested for this issue.
Trigger a rerun:
🎓 Learn more about Self-Healing CI on nx.dev
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a complete suite of client-side CPU benchmarks to
benchmarks/client-nav: 11 scenarios × react/solid/vue (36 benches), covering the client router's main code paths. They run in CI through CodSpeed via the existing Benchmarks workflow (the matrix picks them up with zero workflow changes) and locally via the usual nx targets.async-pipelinebeforeLoadcontext, parallel nested async loaders — all via counted 0ms hopscontrol-flowredirect(2-hop chain),notFound(), loader errors +errorComponent, boundary reset on recoveryheadhead()evaluation, meta/link dedupe, head DOM updates per navigationhistoryuseBlocker,useCanGoBack/useLocationlinks<Link>s recomputing active state per navigation,activeOptionsvariants,MatchRoute/useMatchRouteloadersloaderDeps-keyed caching,router.invalidate(),useLoaderDataselectorsmountcreateRouter+ first render + initial load + unmount (fresh router per mount)nested-paramsparams.parse/stringify,beforeLoadcontext accumulation, per-level subscriberspreloadpreloadRoute, deterministic preload cache (defaultPreloadStaleTime: 0)route-tree-scalesearch-paramsvalidateSearch,retainSearchParams/stripSearchParams, functional updaters, structural sharing,useSearchselectorsDesign
@tanstack/router-plugin,routeTree.gen.tscommitted), built for production (minify: false) and benched against the built bundle in jsdom — real-user app shape, not synthetic route objects.scenarios/harness.ts) drives circular step sequences of real<Link>clicks (plus hover / programmatic-preload / history-traversal / invalidate steps), each synchronized on the router'sonRenderedevent. Setup runs a warm-up lap asserting every step's observable output (document.title, DOM markers, active-link counts, pathnames) so a scenario that silently breaks fails instead of reporting a fast time.scenarios/<name>/shared.ts) is framework-agnostic; the three apps differ only in the rendering layer, so cross-framework numbers are comparable in shape.staleTime/gcTimepinned to0/1e9,defaultPreloadStaleTime: 0, stationary DOM and session-history depth across laps. Calibrated so each bench'svitest benchrun lands between ~10s and ~16.5s locally.root/absolutesetupFilesso they resolve identically under the new per-framework aggregate configs).Notable findings encoded in the suite
Await,pendingComponent, fallback-less suspension alike) — component-level Suspense is inherently non-deterministic to benchmark, soasync-pipelinemeasures the router's async pipeline instead (documented in the README and the scenario'sshared.ts).control-flow's React app passesonCaughtError: () => {}tocreateRoot— React 19's default caught-error reporting otherwise emits ~23k stderr lines per run inside the measured loop.defineComponentcomponents before thecreateFileRoute()call: after bundling (varhoisting), a later declaration silently becomescomponent: undefinedwith no error.mouseEnter, Solid/Vue via nativemouseenter— the harness dispatches both).MatchRoutedoesn't acceptparams(usages areto-only; params matching covered viauseMatchRoute). Side observation: vue-router'sMatchRoutecallsuseMatchRoute()inside its render function, which may be worth a look separately.Validation
nx run @benchmarks/client-nav:build:<fw>then instrumentedtest:perf:<fw> --excludeTaskDependencies— 12/12 benches under the CodSpeed plugin.pnpm-lock.yamlchange is a hand-added 3-line importer entry for@tanstack/router-plugin(validated bypnpm install --frozen-lockfile); a full re-resolve is currently blocked by thetmp@0.2.7trust-policy issue, which is unrelated to this PR.🤖 Generated with Claude Code