Uh oh!
There was an error while loading. Please reload this page.
Add ROCm 6.4 support - #812
Draft
Zheyu Shen (arsdragonfly) wants to merge 1 commit into
Draft
Conversation
Add dockerfile/rocm6.4.x.dockerfile and required cross-cutting changes: - gpu_stream: register on Platform.ROCM and port the microbenchmark to HIP. - third_party/rccl-tests: bump submodule for ROCm 6.4 compatibility. - dockerfile/rocm6.2.x.dockerfile: bump Intel MLC to v3.12 (3.10 mirror gone). - CI: add rocm6.4 entry to build-image workflow.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds ROCm 6.4 container support and extends the existing gpu-stream microbenchmark so it can be built/registered on ROCm (HIP) in addition to CUDA.
Changes:
- Added a new
rocm6.4.xDockerfile and wired it into the CI image build matrix. - Ported
gpu-streamC++ microbenchmark build logic to support a ROCm/HIP build path (hipify + rocm_smi) and updated the benchmark registration to includePlatform.ROCM. - Updated the ROCm 6.2 image to use a new Intel MLC v3.12 download URL.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| superbench/benchmarks/micro_benchmarks/gpu_stream/gpu_stream.cu | Adds ROCm SMI path to retrieve memory clock rate under HIP/ROCm. |
| superbench/benchmarks/micro_benchmarks/gpu_stream/gpu_stream_utils.hpp | Switches NVML include to ROCm SMI include when building for HIP/AMD. |
| superbench/benchmarks/micro_benchmarks/gpu_stream/CMakeLists.txt | Adds CUDA-vs-HIP build split and hipify-based ROCm build pipeline; links rocm_smi. |
| superbench/benchmarks/micro_benchmarks/gpu_stream.py | Registers gpu-stream for Platform.ROCM. |
| dockerfile/rocm6.4.x.dockerfile | Introduces new ROCm 6.4 build image with updated dependency/tooling steps. |
| dockerfile/rocm6.2.x.dockerfile | Updates Intel MLC download URL to v3.12. |
| .github/workflows/build-image.yml | Adds rocm6.4 to the Docker build matrix. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+537
to
+546
| rsmi_frequencies_t freq{}; | ||
| ret = rsmi_dev_gpu_clk_freq_get(static_cast<uint32_t>(gpu_id), RSMI_CLK_TYPE_MEM, &freq); | ||
| if (ret != RSMI_STATUS_SUCCESS) { | ||
| std::cerr << "Failed to get memory clock from ROCm SMI: status=" << ret << std::endl; | ||
| rsmi_shut_down(); | ||
| return -1.0f; | ||
| } | ||
| // freq.current is the index of the active frequency level; values are in Hz. | ||
| float clock_mhz = static_cast<float>(freq.frequency[freq.current]) / 1.0e6f; |
Comment on lines
+110
to
+116
| if(HIP_UNCACHED_MEMORY) | ||
| add_compile_definitions(HIP_UNCACHED_MEMORY) | ||
| endif() | ||
| set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -O2") | ||
| add_executable(gpu_stream ${HIP_SOURCES}) |
Comment on lines
+177
to
+188
| # Fix: copy the set before iterating. Patch all joblib instances system-wide. | ||
| RUN pip install "joblib>=1.4.2" && \ | ||
| find / -path '*/joblib/parallel.py' -not -path '*/.git/*' -exec sed -i \ | ||
| 's/timeout_control_job = next(iter(self\._jobs_set), None)/timeout_control_job = next(iter(set(self._jobs_set)), None)/' {} + | ||
| RUN cd third_party && \ | ||
| git clone -b release-staging/rocm-rel-6.4 https://github.com/ROCmSoftwarePlatform/hipBLASLt.git && \ | ||
| sed -i 's/host-x86_64-unknown-linux,/host-x86_64-unknown-linux-gnu,/' \ | ||
| hipBLASLt/tensilelite/Tensile/BuildCommands/SharedCommands.py && \ | ||
| cd hipBLASLt && ./install.sh -dc && \ | ||
| find /opt -path '*/joblib/parallel.py' -not -path '*/.git/*' -exec sed -i \ | ||
| 's/timeout_control_job = next(iter(self\._jobs_set), None)/timeout_control_job = next(iter(set(self._jobs_set)), None)/' {} + && \ | ||
| cp -v build/release/clients/staging/hipblaslt-bench /opt/superbench/bin/ |
Comment on lines
+177
to
+187
| # Fix: copy the set before iterating. Patch all joblib instances system-wide. | ||
| RUN pip install "joblib>=1.4.2" && \ | ||
| find / -path '*/joblib/parallel.py' -not -path '*/.git/*' -exec sed -i \ | ||
| 's/timeout_control_job = next(iter(self\._jobs_set), None)/timeout_control_job = next(iter(set(self._jobs_set)), None)/' {} + | ||
| RUN cd third_party && \ | ||
| git clone -b release-staging/rocm-rel-6.4 https://github.com/ROCmSoftwarePlatform/hipBLASLt.git && \ | ||
| sed -i 's/host-x86_64-unknown-linux,/host-x86_64-unknown-linux-gnu,/' \ | ||
| hipBLASLt/tensilelite/Tensile/BuildCommands/SharedCommands.py && \ | ||
| cd hipBLASLt && ./install.sh -dc && \ | ||
| find /opt -path '*/joblib/parallel.py' -not -path '*/.git/*' -exec sed -i \ | ||
| 's/timeout_control_job = next(iter(self\._jobs_set), None)/timeout_control_job = next(iter(set(self._jobs_set)), None)/' {} + && \ |
Comment on lines
119
to
+120
| BenchmarkRegistry.register_benchmark('gpu-stream', GpuStreamBenchmark, platform=Platform.CUDA) | ||
| BenchmarkRegistry.register_benchmark('gpu-stream', GpuStreamBenchmark, platform=Platform.ROCM) |
Hongtao Zhang (polarG) added a commit
that referenced
this pull request
Aug 24, 2026
**Description** Adds a ROCm 6.4.4 SuperBench Docker image targeting AMD MI300X (gfx942), based on #812, plus fixes discovered while building and verifying the image end-to-end on MI300X hardware. **Major Revision** - Add `dockerfile/rocm6.4.x.dockerfile` — new ROCm 6.4.4 image (Ubuntu 24.04, Python 3.12, PyTorch 2.7.1, gfx942), base `rocm/pytorch:rocm6.4.4_ubuntu24.04_py3.12_pytorch_release_2.7.1`. - OFED upgraded to DOCA-Host 25.10 user-space (`doca-ofed-userspace=3.2.3-019000`), matching the target host's OFED version; MLNX_OFED standalone tarballs are gone upstream. - Embedded Docker pinned to client-only `29.6.2` (extracts only the `docker` client binary, dropping dockerd/containerd/runc/shims to shrink the CVE surface — SuperBench only shells out to the Docker client, never runs its own daemon). - Add `flex` to the apt package list (fixes an OpenMPI "developer build"/`autogen.pl` failure that requires it). - Set `AMDGPU_TARGETS=gfx942` explicitly (previously unset, a regression vs `rocm6.3.x.dockerfile`) — without a GPU present at `docker build` time, hipcc silently defaulted to `gfx906`, which caused incorrect zero-copy reads / `gpu-copy-bw:correctness` failures on MI300X. - Bump `third_party/rccl-tests` submodule to `0039629` (adds a missing `<cstring>` include required for ROCm 6.4's HIP headers, ROCm/rccl-tests#132). - Fix `third_party/Makefile` rccl-tests build: repeat the HIP platform defines (`-x hip -D__HIP_PLATFORM_AMD__ -D__HIPCC__`) inside the `HIPCUFLAGS` override, since setting `AMDGPU_TARGETS` causes the override to replace (not append to) rccl-tests' own `HIPCUFLAGS`, which otherwise drops those defines and breaks the HIP headers. - Register `rocm6.4` in `.github/workflows/build-image.yml`. **Minor Revision** - CMake `if/else` shell fix (missing `;` before `else`). - `rm -rf .git` cleanup in the final `make postinstall` step (image size). - `apt-get clean` in the apt/OFED install blocks. **Verification** A full `sb run` was executed on an 8x MI300X host: 460/462 executor runs passed. The 3 failures (`resnet` rendezvous port collision, `gpu-copy-bw:perf` OOM, `gpu-copy-bw:correctness`) were root-caused; the correctness failure was the `AMDGPU_TARGETS` issue above, confirmed in-container with the full correctness suite passing 321/321 after rebuilding with `gfx942`. The other two failures are fixed outside this Dockerfile (runner/config changes) and are not blockers for this PR. --------- Co-authored-by: Ubuntu <perfgateuser@CVL08PrdGPC01000027.ff0hegz40zmexpxxrbwn0pra2d.cbnx.internal.cloudapp.net> Co-authored-by: Ubuntu <perfgateuser@CVL08PrdGPC0100002H.ff0hegz40zmexpxxrbwn0pra2d.cbnx.internal.cloudapp.net> Co-authored-by: Hongtao Zhang <hongtaozhang@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a ROCm 6.4 dockerfile and the cross-cutting changes that first become
relevant at this version.
This PR is the first of three stacked PRs that split #810 by ROCm version:
Add ROCm 6.4 supportAdd ROCm 7.0 support(stacked on this one)Add ROCm 7.2 support(stacked on the 7.0 PR)Changes
dockerfile/rocm6.4.x.dockerfile— new ROCm 6.4 image.superbench/benchmarks/micro_benchmarks/gpu_stream*— register thegpu-streammicrobenchmark onPlatform.ROCMand port it to HIP so itbuilds for any ROCm dockerfile.
third_party/rccl-tests— submodule bump for ROCm 6.4 compatibility.dockerfile/rocm6.2.x.dockerfile— bump Intel MLC to v3.12 (the v3.10download mirror is gone).
.github/workflows/build-image.yml— addrocm6.4matrix entry.Notes
Splits the original branch
arsdragonfly/rocm-refresh(existing PR #810) byROCm version so each image can be reviewed and built independently.