perf(bench): multi-arch relative Rust-vs-FFI dashboard - #78
Merged
Merged
Conversation
- add target-aware benchmark generation and benchmark-relative.json - run CI benchmark matrix for x86_64-gnu, i686-gnu, x86_64-musl - publish relative-first gh-pages dashboard with filters and merged payloads - document relative metrics and matrix coverage in benchmark docs Closes #77
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughAdds a multi-target CI benchmark matrix, per-target normalized Changes
Sequence Diagram(s)sequenceDiagram
participant GHA as GHA (CI)
participant Bench as Bench Script
participant Storage as Artifact Storage
participant Merge as Merge Job / Script
participant GHP as gh-pages
participant Browser as Dashboard
GHA->>Bench: Start matrix run (per-target)
Bench->>Bench: run cargo bench (optional --target), parse timings, compute deltas, emit per-target artifacts
Bench->>Storage: Upload `benchmark-<id>` artifacts
GHA->>Merge: Trigger benchmark-pages job (after matrix / on main)
Merge->>Storage: Download per-target artifacts
Merge->>Merge: Validate reference_band, normalize/backfill targets_meta, merge relative + delta rows, apply retention, sort by generated_at
Merge->>GHP: Publish merged JSON/MD and dashboard HTML to `gh-pages/dev/bench/`
Browser->>GHP: GET merged `benchmark-relative.json` and dashboard HTML
GHP-->>Browser: Serve merged JSON + dashboard HTML
Browser->>Browser: Apply filters, render charts, overlay parity band, update latest-deltas and raw table
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Pull request overview
This PR upgrades the benchmarking pipeline to publish a multi-target (multi-arch) relative Rust-vs-FFI dashboard, making relative deltas the primary artifact for performance parity tracking across targets.
Changes:
- Add
benchmark-relative.jsongeneration and publish it to GitHub Pages as the relative-first dashboard source. - Run benchmarks in CI as an explicit target matrix (
x86_64-gnu,i686-gnu,x86_64-musl) and upload per-target artifacts. - Merge matrix artifacts on
maininto canonicalgh-pages/dev/bench/*payloads and publish a new relative-first dashboard page.
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Links to the new published relative payload and updates benchmark description. |
| BENCHMARKS.md | Documents the new relative payload schema and CI target matrix behavior. |
| .gitignore | Ignores the new generated benchmark-relative.json artifact. |
| .github/workflows/ci.yml | Adds benchmark target matrix and a publish job that merges artifacts + updates gh-pages dashboard payloads. |
| .github/scripts/run-benchmarks.sh | Adds target tagging, prefixes benchmark series with target, and emits benchmark-relative.json. |
| .github/bench-dashboard/index.html | Introduces a new relative-first dashboard UI powered by benchmark-relative.json. |
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/bench-dashboard/index.html:
- Around line 143-158: Add a missing "Target" filter select to the dashboard
markup so users can filter by architecture/target; insert a label with a select
element (id="f-target") alongside the existing selects for
metric/stage/scenario/level/source (the blocks containing <select
id="f-metric">, <select id="f-stage">, <select id="f-scenario">, <select
id="f-level">, <select id="f-source">) and replicate this addition in the other
similar blocks noted (around the sections that match lines 207-233, 299-306,
355-359) so the UI exposes target filtering everywhere the other filters appear.
- Around line 196-197: The chart currently only uses BAND_LOW and BAND_HIGH for
status text; update the chart rendering routine (e.g., the function that
draws/updates the chart—renderChart/drawChart/updateChart) to draw the parity
band visually by mapping BAND_LOW and BAND_HIGH through the chart's y-scale and:
1) draw semi-transparent filled rectangle spanning the plot's x-range between
those y positions, and 2) draw thin reference lines at each boundary; use the
same BAND_LOW/BAND_HIGH constants so the text and visuals stay in sync and
ensure the band is redrawn on subsequent updates/resizes.
In @.github/workflows/ci.yml:
- Around line 240-255: The current logic that builds summary_lines and
delta_md_lines overwrites per-target markdown bodies (benchmark-report.*.md and
benchmark-delta.*.md) and mixes target identifiers; change it to either write
the count-only summary to a new file name (e.g., benchmark-summary.md /
benchmark-delta-summary.md) instead of publishing over the existing reports, or
merge per-target markdown bodies under target headings when assembling the final
report (use payload.target.id consistently as the canonical key for targets
rather than the artifact filename/alias). Update the code paths that use
summary_lines and delta_md_lines to output to the new filenames or to
prepend/append per-target content under headings, and normalize target
identification when collating so multiple artifacts for the same
payload.target.id update the same report row.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: d19e7803-b0a4-496d-a3ec-dcabd6c3627f
📒 Files selected for processing (6)
.github/bench-dashboard/index.html.github/scripts/run-benchmarks.sh.github/workflows/ci.yml.gitignoreBENCHMARKS.mdREADME.md
- use stable short target id in benchmark artifacts and metadata - merge markdown report bodies per target instead of replacing with counts-only summary - add target filter and parity-band rendering in dashboard - load reference band from payload and remove JSON-driven innerHTML rendering
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/bench-dashboard/index.html:
- Around line 249-271: buildChart currently collapses multiple filtered rows
into a single series per target by using id = `${target}` and grouping only by
generated_at, causing later rows to overwrite earlier ones and updateStatus to
report an arbitrary row; fix it by making the grouping unique per benchmark row
(e.g., include the row's unique key/identifier when forming the series id such
as id = `${target}:${row.key ?? row.id}`) so each scenario/level/source becomes
its own dataset/series, and update updateStatus to select the dataset/row using
the same composite identifier logic used in buildChart (reference functions
buildChart and updateStatus and the variables runKey/target/id).
- Around line 120-121: The static parity-band text "0.99..1.05" must be replaced
with the actual delta_low/delta_high values loaded in main(); after parsing
benchmark-relative.json (the values stored in delta_low and delta_high), update
the intro paragraph node (the <p> that currently contains "Parity band is ...")
and the other occurrences noted around lines 447-451 to programmatically inject
`${delta_low}..${delta_high}` (or formatted equivalents) so the page copy
matches the loaded band; implement this update in main() after the JSON is
loaded and the values are available.
In @.github/workflows/ci.yml:
- Around line 237-328: The merge drops payload-level reference_band; capture
reference_band when reading per-target files (in the loop that builds
relative_records where payload is read) into a variable (e.g., reference_band)
if not already set, preferring existing_payload.get("reference_band") when the
gh-pages file exists, and then include that value in merged_relative_payload as
merged_relative_payload["reference_band"] = reference_band so the published
benchmark-relative.json preserves BAND_LOW/BAND_HIGH.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 930a80e4-647f-47ad-89cb-ed22f3a11e1a
📒 Files selected for processing (3)
.github/bench-dashboard/index.html.github/scripts/run-benchmarks.sh.github/workflows/ci.yml
- make dashboard parity-band copy data-driven from payload - group chart/status by target+benchmark key to avoid row overwrite - preserve reference_band when merging benchmark-relative payloads
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/bench-dashboard/index.html:
- Around line 261-273: The destructured variable target in the datasets mapping
is unused; remove the extraction to avoid an unused variable warning. In the map
callback that builds datasets (the arrow function using grouped, seriesId,
timeline, idx), delete the line const [target] = seriesId.split(":", 2); (or
remove the destructuring entirely) and keep using seriesId for the label as-is.
- Around line 194-195: Add Subresource Integrity for the Chart.js CDN script:
fetch the exact file at
"https://cdn.jsdelivr.net/npm/chart.js@4.4.7/dist/chart.umd.min.js" and compute
the sha384 SRI token (use the provided curl + openssl pipeline), then update the
<script> tag that loads Chart.js to include the
attribute and alongside the existing src so the browser
will verify the file before executing it.
In @.github/workflows/ci.yml:
- Around line 285-293: The code silently falls back to filename parsing when
rows[0].get("target") is None; update the block that sets target (around
delta_path, rows, register_target, target_counts) to detect a missing target
field (rows and rows[0] exist but rows[0].get("target") is None) and emit a
warning-level log that includes the delta_path.name and the offending row (or at
least indicates the missing field) before falling back to parsing the filename;
then continue calling register_target(target) and incrementing
target_counts[target]["delta"] as before.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 8bc81feb-dad0-47c4-b37a-4bbff58203cb
📒 Files selected for processing (2)
.github/bench-dashboard/index.html.github/workflows/ci.yml
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/ci.yml:
- Around line 184-196: The CI is disabling both auto-push and save-data-file for
the benchmark action while benchmark-pages only publishes benchmark-report.md,
benchmark-delta.json and benchmark-relative.json, so alerts (comment-on-alert,
fail-on-alert) end up comparing against no data.js baseline; fix by either
having the benchmark-pages job generate and publish data.js after merging
relative records (use benchmark-relative.json to create data.js and write it to
the pages artifact) or change the alerting step to consume
benchmark-relative.json directly (update the alerting logic that relies on
benchmark-data-dir-path/dev/bench and comment-on-alert to read merged
benchmark-relative.json instead of data.js), and ensure auto-push/save-data-file
settings and benchmark-pages outputs are consistent with the chosen approach.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 22676602-8a51-4ebc-90a0-a88d3bc474d5
📒 Files selected for processing (1)
.github/workflows/ci.yml
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/bench-dashboard/index.html:
- Line 493: The assignment to selectors.metric.value =
"throughput_bytes_per_sec" may pick a non-existent metric; update the logic
around the selectors.metric.value assignment to first check the dataset (or the
metrics/options used to populate the metric dropdown) for the presence of
"throughput_bytes_per_sec" and only set it if found, otherwise set
selectors.metric.value to the first available metric option (or a sensible
default) so the dropdown and initial chart always have a valid selection; locate
the code that populates the metric options and the line setting
selectors.metric.value to implement this existence check and fallback.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 685122b5-930b-4902-a771-dfa15aae0ec9
📒 Files selected for processing (2)
.github/bench-dashboard/index.html.github/workflows/ci.yml
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/ci.yml:
- Around line 188-196: The workflow publishes the canonical baseline from the
parallel benchmark job via the save-data-file setting which can race with the
dashboard merge in the benchmark-pages job when cancel-in-progress is enabled;
either move the baseline publication into the benchmark-pages job or disable
cancel-in-progress for pushes to main. Fix by removing the save-data-file
conditional from the parallel benchmark job (the block referencing
save-data-file and matrix.bench.id == 'x86_64-gnu') and adding the baseline
publish logic/condition (ensure it still only runs for matrix.bench.id ==
'x86_64-gnu' and on pushes to main) into the benchmark-pages job so the
dashboard merge and baseline write happen together, or alternatively set
cancel-in-progress: false for push events on main to prevent mid-flight
cancellation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: c8e8a87b-8d34-4446-9df9-82cdfa2abd41
📒 Files selected for processing (1)
.github/workflows/ci.yml
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/scripts/merge-benchmarks.py:
- Around line 78-84: The SystemExit messages raised for inconsistent/missing
targets (involving reference_band, rel_path, target_payload, filename_target,
target) should be made more informative for CI debugging: update the two
SystemExit calls so they include the filename (rel_path.name), the computed
values (e.g., actual band vs expected reference_band and the payload id vs
filename_target), and any relevant context (payload keys) to make failures
traceable; keep using SystemExit but expand the message strings to show the
expected vs actual values and source of the value (payload id or derived
filename_target).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 50460bd2-4f9a-4603-b1a3-719ccd7dc19a
📒 Files selected for processing (2)
.github/bench-dashboard/index.html.github/scripts/merge-benchmarks.py
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/scripts/merge-benchmarks.py:
- Line 113: The current assignment report_bodies[target] =
report_path.read_text().strip() allows empty strings to be treated as present;
change the loading so you treat blank/whitespace-only files as missing or error:
read the file into a variable (e.g., body = report_path.read_text().strip()),
then if not body raise/exit with a clear error or skip setting
report_bodies[target]; update logic that later checks presence (the existing
key-presence check around the multi-target completeness) to rely on truthy
values instead of just keys so empty bodies don't pass; reference report_bodies,
report_path.read_text().strip(), and the later presence check to implement this
behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 0f989158-e8a1-4626-89fc-9906fca37e43
📒 Files selected for processing (2)
.github/scripts/merge-benchmarks.py.github/scripts/run-benchmarks.sh
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/scripts/merge-benchmarks.py:
- Around line 110-113: The loop that reads benchmark markdown files (iterating
report_path, deriving target, calling register_target and setting
report_bodies[target]) currently stores even-empty stripped text; update the
logic so that after reading report_path.read_text().strip() you only set
report_bodies[target] when the stripped content is non-empty (treat blank
artifacts as missing) and ensure the later completeness checks (the code that
checks for each target's report presence) verify truthiness (e.g.,
report_bodies.get(target) is truthy) rather than merely key presence; adjust
references to register_target, report_bodies, and the report_path loop
accordingly so empty files do not pass validation or produce partial reports.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 61a98f7a-db9e-4529-95d0-8a70b2e4f32d
📒 Files selected for processing (2)
.github/scripts/merge-benchmarks.py.github/scripts/run-benchmarks.sh
Merged
This was referenced May 15, 2026
This was referenced Jun 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
benchmark-relative.jsonx86_64-gnu,i686-gnu,x86_64-muslValidation
Closes #77
2026-04-07: Regression Check Relaxation (Runner Variance)
GitHub-hosted runners show high run-to-run CPU variance, so absolute cross-run benchmark comparisons create false positives.
Changes in this issue scope
x86_64-gnu).benchmark-results.jsonregression inputs to a smoke subset:compress,decompressdefault,bettersmall-4k-log-lines,decodecorpus-z000033(or fallbackdecodecorpus-synthetic-1m),low-entropy-1mbenchmark-delta.json,benchmark-relative.json, markdown report) for diagnostics.Rationale
Summary by CodeRabbit
New Features
Documentation