[release/2.13] Unskip ROCSHMEM triton UTs on rocm 7.14 - #3578
Merged
Merged
Conversation
The module-level sys.exit(0) from #3525 hid a device-library finder path bug rather than a Triton functional failure. Remove it so the tests collect. Keep the capability skip when the SHMEM backend is unavailable.
Honor ROCM_PATH when locating rocSHMEM Triton device bitcode. (cherry picked from commit 4a31e29) The test/inductor/test_ck_backend.py hunk from that commit is omitted.
chinmaydk99
marked this pull request as ready for review
August 19, 2026 00:35
2 tasks
jithunnair-amd
pushed a commit
that referenced
this pull request
Aug 25, 2026
…l) (#3594) ## Summary Carry-forward of ROCm-specific commits from `release/2.13` onto `release/2.14` ? the 2.14 analog of the 2.13 "IFU dedup" ([PR #3503](#3503)). Analyzes the full IFU delta from [PR #3591](#3591) at `release/2.13`'s tip (`6596a0d`): **81 commits** in `release/2.13` not in `release/2.14`, each either dropped (already in 2.14, or 2.13-only) or cherry-picked here. ## Method Same rules as the [release/2.13 cherry-pick process](https://amd.atlassian.net/wiki/spaces/MLSE/pages/1819593560): - **Drop ? already in 2.14**: patch-id equivalence (`git cherry`), upstream PR already present, or content already relocated/absorbed into 2.14 (verified during cherry-pick). - **Drop ? 2.13-only**: version bump, `related_commits`, dependency/requirements pins, release-line CD/build changes. - **Carry-forward**: ROCm-specific fixes not present in 2.14 in any form. ## Commits carried forward (13) | Source | Notes | |--------|-------| | Fix reentrant deadlock in `torch.cuda._lazy_call` (#3496) | clean | | Skip ROCm-failing tests in test_cuda/test_nn (#3504) | **conflict resolved** ? kept 2.14's `test_graph_capture_error...` cleanup + took `@skipIfRocm`; kept 2.14's `test_allocator_backend` (already fixes LD_LIBRARY_PATH via `subprocess_env()`) | | skip test_hip_device_count (rocprofiler-sdk) | clean | | skip flaky test_Embedding_discontiguous_cuda | clean | | Skip profiler check for foreach tests on ROCm (#3262) | clean | | Skip test_is_pinned_no_context on python 3.14+ | clean | | Skip flaky test_reentrant_parent_error_on_cpu | clean | | Filter out CK gemms for gfx1250-only arch (#3545) | clean | | Fix hipify skipping sources under Windows subst drives (ROCM-29365) (#3561) | clean | | Fixed gfx1250_preferred_backend test (#3546) | clean | | [ROCm] Optimize AMD normalization backward kernel (pytorch#189405) | clean | | [release/2.13] Build rocSHMEM for the requested gfx target (#3582) | clean | | fix PicklingError handling for Python 3.14 | clean | ## Removed from this PR (per team) - [RELEASE_ONLY] skip test_autoheuristic in-code (pytorch#180927) ? not needed on 2.14. Upstream [pytorch#180927](pytorch#180927) is `RELEASE_ONLY`, merged to upstream `release/2.12` only (never on `main` or upstream `release/2.14`). ## Reviewed and excluded (team decision: not needed for 2.14) | Commit | Reason | |--------|--------| | Update composable_kernel submodule with gfx1033 support (#3499) | 2.14's CK pin already includes gfx1033; carrying would move the submodule pointer **backward**. | | Revert "Skip cudagraphs for kernel-free inductor graphs" (pytorch#187347) | 2.14 already has the fixed upstream **re-land** on top of the revert; carrying = regressing 2.14. | | Revert "[Reland] Port index_add fast path" (pytorch#188196) | 2.14 already has the newer **re-land (pytorch#189506)** on top of the revert; carrying = regressing 2.14. | ## Dropped as already in 2.14 (verified during cherry-pick) - rocm-smi deprecation (#3434) ? 2.14 already has zero `rocm_smi` refs. - Enable linalg via hipSOLVER, remove Magma (#3455) ? all 6 upstream PRs already in 2.14. - Stabilize ROCm 7.14 testing (#3525) ? 2.14 already carries the guards + skip messages. - Add gfx1250 support in GroupedBlas (#3532) ? already in 2.14 via upstream pytorch#190703 (`ScaledBlasDeviceUtils`). - Fix uninitialized embedding indices `test_embedding_meta_indices` (pytorch#188002) ? upstream already in 2.14. - Unskip ROCSHMEM triton UTs on rocm 7.14 (#3578) ? targets a 2.13-only 7.14 module skip that 2.14 never had; tests already run on 2.14. Remaining commits drop as 2.13-only (version/pins/related_commits/CD) or as already-in-2.14 by patch-id/upstream. ## Test plan - [x] TheRock full-stack dev build (torch + torchaudio + torchvision + triton) on gfx94X-dcgpu for Python 3.10-3.14, all **Build** jobs green (ROCm `10.1.0a20260822`, on the `aws-linux-scale-rocm-prod` pool), after bringing the branch up to its base so `related_commits` resolves torchaudio/torchvision: - py3.10 https://github.com/ROCm/TheRock/actions/runs/32863299013 - py3.11 https://github.com/ROCm/TheRock/actions/runs/32863305363 - py3.12 https://github.com/ROCm/TheRock/actions/runs/32863311391 - py3.13 https://github.com/ROCm/TheRock/actions/runs/32863318109 - py3.14 https://github.com/ROCm/TheRock/actions/runs/32863325370 - [x] TheRock CI wheel build on gfx94X-dcgpu for Python 3.10-3.14 all green (ROCm `10.1.0a20260822`, torch-only, on the `aws-linux-scale-rocm-prod` pool, ~1h40m each): - py3.10 https://github.com/ROCm/TheRock/actions/runs/32818564344 - py3.11 https://github.com/ROCm/TheRock/actions/runs/32818568751 - py3.12 https://github.com/ROCm/TheRock/actions/runs/32818572249 - py3.13 https://github.com/ROCm/TheRock/actions/runs/32818576288 - py3.14 https://github.com/ROCm/TheRock/actions/runs/32818580229 ## Processed Commits List Spreasheet with each commit evaluated for cherry-pick & resolution (drop/include) with reasoning: [pr3594-commit-outcomes.xlsx](https://github.com/user-attachments/files/31390971/pr3594-commit-outcomes.xlsx) --------- Signed-off-by: pablo-garay <pablo-garay@users.noreply.github.com> Co-authored-by: Iurii Paikov <Iurii.Paikov@amd.com> Co-authored-by: Chinmay Dattanand Kuchinad <40351312+chinmaydk99@users.noreply.github.com> Co-authored-by: rocm-repo-management-api-6[bot] <212817015+rocm-repo-management-api-6[bot]@users.noreply.github.com> Co-authored-by: Dmitry Nikolaev <139769634+dnikolaev-amd@users.noreply.github.com> Co-authored-by: srinivamd <52507740+srinivamd@users.noreply.github.com> Co-authored-by: Subramanian <ssubrama1@amd.com> Co-authored-by: zjliu-amd <zhuojin.liu@amd.com> Co-authored-by: Andres Lugo <108368282+alugorey@users.noreply.github.com> Co-authored-by: pablo-garay <pablo-garay@users.noreply.github.com> Co-authored-by: Bo Li <110066325+BLOrange-AMD@users.noreply.github.com> Co-authored-by: Anatoliy Litvinenko <alitvine@amd.com> Co-authored-by: Jeff Daily <jeff.daily@amd.com> Co-authored-by: Luca Bruni <luca.bruni@amd.com> Co-authored-by: albmalamd <albert.malewski@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ROCm/pytorch#3525 (
[release/2.13] Stabilize ROCm 7.14 testing on release/2.13) added a module-levelsys.exit(0)intest/distributed/test_shmem_triton.pyforROCM_VERSION >= (7, 14).This PR does two things:
sys.exit(0). Keep the existing capability skip when the SHMEM backend is unavailable.4a31e29([ROCm][CI] Fix trunk rocSHMEM and CK failures, from pytorch/pytorch#188429): search$ROCM_PATH/libwhenROCM_PATHis set, else/opt/rocm/lib.We did not take the rest of
4a31e29. That commit also adds@skipIfRocmVersionAtLeast([7, 14])ontest/inductor/test_ck_backend.py. That CK skip is already onrelease/2.13and is unrelated to this UT. Fullcherry-pick -xof4a31e29would just restage it. We also did not backport the rest of [ROCm][CI] Switch rocm-preview to TheRock wheels and upgrade CI to 7.14 pytorch/pytorch#188429 (TheRock CI/docker switch).Test plan
test_pytorch_wheels_full.yml,test_configs=distributed,tests_to_include=distributed/test_shmem_tritonlinux-gfx942-1gpu-ccs-csp-ossci-rocm: https://github.com/ROCm/TheRock/actions/runs/32195967020/job/95904033020@skip_if_rocm_multiprocess.