ENH: Add performance benchmark CI builds by thewtex · Pull Request #612 · InsightSoftwareConsortium/ITK

thewtex · 2019-03-21T17:43:50Z

No description provided.

jhlegarreta · 2019-03-25T13:16:49Z

Speechless. Awesome.

kwrobot-v1 · 2019-04-23T22:27:23Z

Errors:

Failed to run the checks: service error.

thewtex · 2019-04-24T15:46:52Z

Errors:

* Failed to run the checks: `service error`.

CC @bradking @mathstuf

mathstuf · 2019-04-24T16:51:05Z

Hmm. Seems we got a timeout on one of our queries.

Do: check

Add .github/workflows/perf-benchmark.yml, an opt-in GHA workflow that runs the ASV-based performance regression harness hosted at InsightSoftwareConsortium/ITKPerformanceBenchmarking on pull requests labelled 'performance' (or via manual dispatch). The workflow: - Computes BASE = merge-base(PR, origin/master) and builds ITK at both BASE and PR HEAD with Module_PerformanceBenchmarking=ON, BUILD_EXAMPLES=ON, BUILD_TESTING=ON (required for the ExternalData fixture fetch), and BUILD_SHARED_LIBS=OFF. The two builds share a ccache so the second build is mostly incremental. - Builds only the benchmark executables plus ITKBenchmarksData, avoiding a full ITK test suite build. - Checks out InsightSoftwareConsortium/ITKPerformanceBenchmarking at the PERF_HARNESS_REF env var (defaults to the asv-native- harness branch while that work is landing upstream; flip to a merged SHA afterwards). - Creates the 'itk-repo' symlink required by the harness's asv.conf.json, installs the itk_perf_shim Python package, and runs 'asv run --set-commit-hash <SHA>' against each build so result labels track ITK history. - 'asv compare --factor=1.10 --split' produces a markdown diff; posted as a sticky PR comment and uploaded as an artifact along with the 'asv publish' HTML tree. Opt-in via the 'performance' label keeps the two-ITK-build cost off the default PR path. No changes to any existing workflow or Azure Pipelines file. Replaces the intent of the long-stalled thewtex/ITK:ci-performance- builds branch (InsightSoftwareConsortium#612), which proposed an Azure Pipelines job targetting the retired macos-11 image and the legacy evaluate-itk-performance.py driver.

hjmjohnson · 2026-04-23T01:17:36Z

@thewtex — the rebase + origin/master → origin/main workaround in .github/workflows/perf-benchmark.yml (commit c4e39a64) got the asv base vs head check green (11m59s). All Azure C++ pipelines pass on the fresh tip c4e39a64; remaining pending are the usual long-runners (Python-wrapping + Pixi-Cxx).

To remove the in-workflow patch permanently:

Merge InsightSoftwareConsortium/ITKPerformanceBenchmarking#113 (2-line fix to asv.conf.json + README-asv.md).
Bump Modules/Remote/PerformanceBenchmarking.remote.cmakeGIT_TAG here to that merge SHA.
Delete the python -c sed block from the Stage ASV harness step.

hjmjohnson · 2026-04-23T02:29:05Z

@thewtex — ITKPerformanceBenchmarking#113 merged (b24324f922). Force-pushed tip 2359970c24 that (1) bumps Modules/Remote/PerformanceBenchmarking.remote.cmake GIT_TAG to the merge SHA and (2) drops the python -c sed-patch from the Stage ASV harness step — the workflow is now back to your original minimal form.

Net diff vs the pre-rebase tip cfa9a3ff0d:

Modules/Remote/PerformanceBenchmarking.remote.cmake +1/-1 (pin bump)
rebased onto current upstream/main (was 11 commits behind, now 0)

Fresh CI kicked off; should settle similarly to the previous run (asv was 11m59s last round).

greptile-apps · 2026-04-23T02:43:38Z

Greptile Summary

This PR adds a new GitHub Actions workflow (perf-benchmark.yml) that runs ASV (airspeed velocity) benchmarks comparing the PR HEAD against its merge-base, and bumps the PerformanceBenchmarking remote module pin to a commit that introduces the required ASV harness.

P1: The workflow_dispatch trigger causes the "Resolve merge-base and HEAD SHAs" step to fail — github.base_ref is empty outside of pull_request events, making git fetch origin "" fatal.
P2: CMakeVersion: "4.0.1" and ExternalDataVersion: 5.4.5 are declared in env but never referenced; if CMake 4.x is genuinely required, an installation step is missing.

Confidence Score: 4/5

Safe to merge after fixing the workflow_dispatch / github.base_ref breakage.

One P1 defect: manually triggered runs fail immediately due to an empty github.base_ref. The cmake pin bump is clean and the overall workflow design is sound.

.github/workflows/perf-benchmark.yml — the workflow_dispatch + github.base_ref interaction and the unused env vars.

Important Files Changed

Filename	Overview
.github/workflows/perf-benchmark.yml	New CI workflow running ASV benchmarks on base vs HEAD; broken for `workflow_dispatch` due to empty `github.base_ref`, and has unused `CMakeVersion`/`ExternalDataVersion` env vars.
Modules/Remote/PerformanceBenchmarking.remote.cmake	Single-line GIT_TAG bump to pin the commit that introduced the ASV harness (`asv.conf.json`) required by the new workflow.

Sequence Diagram

sequenceDiagram
participant GHA as GitHub Actions
participant Base as itk-base (merge-base)
participant Head as itk-head (PR HEAD)
participant ASV as ASV Harness
GHA->>Base: checkout @ merge-base
GHA->>Head: checkout @ PR HEAD (full history)
GHA->>Base: cmake configure + build benchmark targets
GHA->>Head: cmake configure + build benchmark targets
Note over Head: PerformanceBenchmarking module populated with asv.conf.json
GHA->>ASV: pip install harness (from itk-head)
GHA->>ASV: asv machine --yes
GHA->>ASV: asv run --quick @ merge-base SHA
Note over ASV: continue-on-error: true
GHA->>ASV: asv run --quick @ HEAD SHA
GHA->>ASV: asv compare merge-base HEAD
ASV-->>GHA: asv-compare.txt + results artifact

_{Reviews (1): Last reviewed commit: "COMP: Bump PerformanceBenchmarking pin t..." | Re-trigger Greptile}

hjmjohnson · 2026-04-25T17:12:33Z

@thewtex. I have been addressing old issues and cleaning up the backlog, so I went straight to the oldest PR.

I'd like to take this to a final resting state. I'm ok if we just close it, or include it, or make some changes to get it over a finish line (whatever goal-post changing finish line is currently most beneficial).

This was a nice test bed for resolving some other issues, and I think it is now ready for final remarks.

jhlegarreta

Thanks for pushing this @hjmjohnson.

Adds .github/workflows/perf-benchmark.yml — a GitHub Actions workflow that builds ITK twice (merge-base and PR HEAD) with the PerformanceBenchmarking remote module enabled, then drives the ASV harness from InsightSoftwareConsortium/ITKPerformanceBenchmarking#111 against each build to produce an asv compare report. Bumps the PerformanceBenchmarking remote module pin from 7950c1d76095033edbf7601a925cdacc2fe717f9 to 41bf1b9cfbaa1b146e1b3e5d3ad3571b2203ce1e (master HEAD, post-InsightSoftwareConsortium#111). The new pin carries: * asv.conf.json, benchmarks/{core,filtering,segmentation}.py, python/itk_perf_shim/ (the ASV harness). * Hardened examples/Core/itkCopyIterationBenchmark.cxx that no longer crashes on VectorImage under ITK >= 1d87efa (the VLV null-data regression; see InsightSoftwareConsortium#6087 for the parallel ITK-side defensive null-guard). Supersedes the 2019 Azure Pipelines macos-11 approach; macos-11 is retired and the older evaluate-itk-performance.py driver has been replaced by the ASV-native harness. For the very first PR (this one) the merge-base lacks the new pin, so its benchmark run is best-effort and the asv compare step is wrapped with continue-on-error. The workflow becomes fully useful for regression detection after this PR lands.

Advance the GIT_TAG in Modules/Remote/PerformanceBenchmarking.remote.cmake to 63fec3013cccfe7e8b1b88a1d2d889f7e64e4469, the current tip of InsightSoftwareConsortium/ITKPerformanceBenchmarking master, which includes the COMP: Drop unused InputPixelType typedefs cleanup that silences the -Wunused-local-typedef warnings the perf-benchmark CI job emitted on the prior pin.

hjmjohnson · 2026-04-28T02:05:30Z

/azp run ITK.macOS

thewtex force-pushed the ci-performance-builds branch 8 times, most recently from 6a32231 to ce113c5Compare March 22, 2019 21:29

thewtex force-pushed the ci-performance-builds branch 5 times, most recently from 71f2cc5 to 111d88bCompare March 29, 2019 16:55

thewtex force-pushed the ci-performance-builds branch 2 times, most recently from c0f2bbb to 289543aCompare April 8, 2019 22:22

thewtex force-pushed the ci-performance-builds branch 4 times, most recently from fa20324 to 96b6114Compare April 10, 2019 01:52

thewtex marked this pull request as ready for review April 10, 2019 01:53

thewtex force-pushed the ci-performance-builds branch from 96b6114 to b1d0d43Compare April 11, 2019 20:39

thewtex force-pushed the ci-performance-builds branch from b1d0d43 to 9cc56faCompare April 23, 2019 22:26

thewtex force-pushed the ci-performance-builds branch from 9cc56fa to 1e53cfbCompare April 24, 2019 15:50

thewtex force-pushed the ci-performance-builds branch 3 times, most recently from 0948b8f to 698efc6Compare April 25, 2019 13:54

hjmjohnson modified the milestones: ITK 6.0.0, ITK 6.0 Beta 1 Jan 24, 2025

thewtex modified the milestones: ITK 6.0 Beta 1, Backlog -- Will be considered for future releases Oct 27, 2025

hjmjohnson marked this pull request as draft February 15, 2026 21:13

hjmjohnson mentioned this pull request Apr 19, 2026
ENH: Add ASV-over-native-ITK performance benchmark harness InsightSoftwareConsortium/ITKPerformanceBenchmarking#111
Merged
5 tasks

hjmjohnson force-pushed the ci-performance-builds branch from c6b6b60 to fbbbf29Compare April 22, 2026 00:01

github-actionsBot added area:Remotes Issues affecting the Remote module and removed type:Testing Ensure that the purpose of a class is met/the results on a wide set of test cases are correct labels Apr 22, 2026

hjmjohnson force-pushed the ci-performance-builds branch from fbbbf29 to cfa9a3fCompare April 22, 2026 00:07

hjmjohnson changed the title ~~WIP: ENH: Add performance benchmark CI builds~~ENH: Add performance benchmark CI buildsApr 22, 2026

hjmjohnson force-pushed the ci-performance-builds branch from cfa9a3f to c4e39a6Compare April 23, 2026 00:51

github-actionsBot added the type:Enhancement Improvement of existing methods or implementation label Apr 23, 2026

hjmjohnson mentioned this pull request Apr 23, 2026
COMP: Use ITK default branch origin/main in ASV config and docs InsightSoftwareConsortium/ITKPerformanceBenchmarking#113
Merged

hjmjohnson force-pushed the ci-performance-builds branch from c4e39a6 to 2359970Compare April 23, 2026 02:28

hjmjohnson marked this pull request as ready for review April 23, 2026 02:40

greptile-appsBot reviewed Apr 23, 2026
View reviewed changes

Comment thread.github/workflows/perf-benchmark.yml Outdated
Comment thread.github/workflows/perf-benchmark.yml Outdated
Comment thread.github/workflows/perf-benchmark.yml

This was referenced Apr 23, 2026
ENH: Cache ExternalData object store across CI runs #6109
Merged
ENH: Ingest ITKMontage remote module into Modules/Registration/Montage #6103
Merged

hjmjohnson force-pushed the ci-performance-builds branch from 2359970 to a677cfcCompare April 24, 2026 18:18

jhlegarreta reviewed Apr 27, 2026
View reviewed changes

Comment thread.github/workflows/perf-benchmark.yml Outdated
Comment thread.github/workflows/perf-benchmark.yml Outdated
Comment thread.github/workflows/perf-benchmark.yml Outdated
Comment thread.github/workflows/perf-benchmark.yml Outdated
Comment thread.github/workflows/perf-benchmark.yml Outdated

hjmjohnson added 2 commits April 27, 2026 09:42

jhlegarreta approved these changes Apr 28, 2026
View reviewed changes

Uh oh!

Uh oh!

Conversation

thewtex commented Mar 21, 2019

Uh oh!

jhlegarreta commented Mar 25, 2019

Uh oh!

kwrobot-v1Bot commented Apr 23, 2019

Uh oh!

thewtex commented Apr 24, 2019

Uh oh!

mathstuf commented Apr 24, 2019

Uh oh!

hjmjohnson commented Apr 23, 2026

Uh oh!

hjmjohnson commented Apr 23, 2026

Uh oh!

greptile-appsBot commented Apr 23, 2026

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Sequence Diagram

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hjmjohnson commented Apr 25, 2026

Uh oh!

jhlegarreta left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hjmjohnson commented Apr 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants