[release/2.14] Carry-forward ROCm cherry-picks from release/2.13 - #3593
Closed
pablo-garay wants to merge 9 commits into
Closed
pablo-garay wants to merge 9 commits into
pablo-garay wants to merge 9 commits into
Conversation
This fixes deadlock with torch.compile while running bigger models in MAD engine. This is a cherry-pick of upstream PR: pytorch#182948 Made with [Cursor](https://cursor.com) (cherry picked from commit 24cf333)
Adds @skipIfRocm to 4 tests that consistently fail in the TheRock wheel test lane (test_pytorch_wheels.yml) on gfx94X/MI300 with torch==2.13.0a0+rocm7.14.0. - test_allocator_backend - test_graph_capture_error_releases_reserved_segments - test_lazy_call_reentrant_set_rng_state_does_not_deadlock - test_linear_cross_entropy_loss_default Also removes the now-dead @skipIfRocmVersionLessThan((7, 14)) from test_graph_capture_error_releases_reserved_segments — with @skipIfRocm on top it is unreachable on ROCm. Test plan - Validated via ROCm/TheRock test_pytorch_wheels.yml run #30218037322 on gfx94X-dcgpu with torch==2.13.0a0+rocm7.14.0 — all 4 tests now SKIPPED, run passes - Core UTs passing on 2.13: https://github.com/ROCm/TheRock/actions/runs/30218037322 (cherry picked from commit 523576b)
…d via issue) (pytorch#180927) [ROCm] skip test_autoheuristic in-code (already disabled via issue) (cherry picked from commit 366eac9) (cherry picked from commit 2a5642e)
…OCM-21749) (#3262) ## Summary Fixes ROCM-21749: `test_big_num_tensors__foreach_norm` fails on all ROCm GPUs. ## Root Cause `ForeachFuncWrapper.__call__` profiles a `_foreach_norm` call and checks for `multi_tensor_apply_kernel` in profiler output to verify the multi-tensor fast-path was taken. On CUDA, Kineto captures demangled kernel names like `multi_tensor_apply_kernel[...]`. On ROCm, ROCTracer/rocprofiler captures different symbols (`hipLaunchKernel`, `vectorized_elementwise_kernel`, etc.), so `mta_called` is always `False` and the assertion fails. ## Fix Add `or TEST_WITH_ROCM` to `skip_profiler_check` so the profiler-based assertion is bypassed on ROCm, consistent with the existing skip for CUDA 12.6/12.8 flakiness. ## Test Plan - `test_big_num_tensors__foreach_norm_*` should now pass on ROCm GPUs (MI210, MI300X, MI350X) - No change in behavior on CUDA Jira: https://amd-hub.atlassian.net/browse/ROCM-21749 Co-authored-by: Subramanian <ssubrama1@amd.com> (cherry picked from commit f8efdb3) (cherry picked from commit e143818)
2 tasks
Author
|
has a mistake; close it |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Carry-forward of ROCm-specific commits from
release/2.13ontorelease/2.14— the 2.14 analog of the 2.13 "IFU dedup" (PR #3503).Starting point was the IFU delta surfaced by PR #3591 (
release/2.13→release/2.14): 74 commits inrelease/2.13that are not inrelease/2.14. Each was analyzed and either dropped (already in 2.14, or 2.13-only plumbing) or cherry-picked here.Method
Applied the same rules as the release/2.13 cherry-pick process:
git cherry), by upstream PR number already present, or by content (e.g. an upstream PR backported to 2.13 that 2.14 already carries natively).related_commits, dependency/requirements pins, and release-line CD/build changes (CUDA 12.9 revival, docker image pins, binary-build timeout, RC/PTX strip, etc.).Of the 74: 42 already in 2.14, 20 dropped as 2.13-only (incl. 2 release-line reverts that 2.14 intentionally keeps active), and 12 carry-forward candidates. During cherry-pick, 4 candidates were found to be already in 2.14 (their content had been relocated/absorbed by upstream) and were dropped, leaving 9 commits here.
Commits carried forward (9)
torch.cuda._lazy_call(#3496)test_graph_capture_error...cleanup body + took the@skipIfRocm; kept 2.14'stest_allocator_backend(it already fixes the LD_LIBRARY_PATH root cause viasubprocess_env(), so the 2.13 blanket skip was dropped)Candidates dropped during cherry-pick (already in 2.14)
release/2.14already has zerorocm_smireferences (replaced by the hipfile logic).skip_if_rocm_ver_atleast_multiprocess/getRocmVersion() >= (7,14)guards and the same skip messages.ScaledBlasDeviceUtils, which already includesgfx1250underROCM_VERSION >= 71400in 2.14.Test plan