Skip to content

[CI] Apply parity fallback to baseline distributed tests - #3651

Draft
ethanwee1 wants to merge 2 commits into
developfrom
ethanwee1-fix-baseline-distributed-fallback
Draft

ethanwee1 wants to merge 2 commits into
developfrom
ethanwee1-fix-baseline-distributed-fallback

Conversation

@ethanwee1

@ethanwee1 ethanwee1 commented Sep 16, 2026

Copy link
Copy Markdown

Problem

Commit-vs-commit parity only applies configured workflow fallbacks to the current commit. If the baseline lacks the primary distributed workflow, the exception is downgraded to a warning and the report incorrectly shows zero baseline distributed tests even when the fallback workflow contains them.

Run 35114338058 showed this asymmetry: b285e55a used the MI350 trunk fallback and reported 5,172 tests, while 58acd33b reported zero despite all three distributed shards existing in its trunk run.

Tracked in AIPYTORCH-1223. Stacked on #3650.

Changes

Apply the configured distributed fallback to baseline downloads and carry its workflow-specific job prefix into baseline log collection.

Validation

Both workflows passed. The personal-fork report contains 5,172 distributed tests and 369,710 total tests on each commit, with 100% overall agreement and zero failed tests. Source logs were disabled to isolate artifact collection from the historical job-name change handled by the next stacked PR.

Match the current CUDA 13.2 and Noble ROCm Python 3.11 trunk jobs so parity selects the full push run instead of a scheduled periodic run.
Use the configured distributed fallback for baseline commits just as the current-commit path does, preserving its workflow-specific log prefix.
Base automatically changed from ethanwee1-fix-parity-canonical-run to develop September 16, 2026 23:08
error_msg=f"Baseline distributed fallback not found for {baseline_sha}",
)

print(f"Baseline distributed workflow '{baseline_dist_workflow}' id: {baseline_dist_wf['id']}")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If baseline_dist_wf is None and arch is not in baseline_dist_fallbacks, will baseline_dist_wf not stay None and error here in baseline_dist_wf['id']?

@jithunnair-amd jithunnair-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are no fallbacks being used in the default and inductor cases for baseline commit? (although, see below for an alternative approach)

@jithunnair-amd

jithunnair-amd commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

Also, rocm-mi350 and periodic-rocm-mi350 should be removed from parity_job_config.json since they are effectively disabled now. And that might remove the need for the code to handle fallbacks specifically for distributed (keeping it consistent with the code for default and inductor, although arguably all of them should use fallbacks, if available). Looks like https://github.com/ROCm/pytorch/pull/3652/changes#diff-3a30832e69138957d1da0a40263cdfb3995c237e467f13fd84003c94b40a823c does exactly that.

And while you're cleaning up fallbacks, the fallbacks for mi200 should be removed too: trunk-rocm-sandbox doesn't run on mi200 anymore.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants