fix: migrate GPU topology from rocm_smi to amd_smi (ROCm 10.1+) by srinivamd · Pull Request #663 · ROCm/mori

srinivamd · 2026-09-11T20:09:16Z

TheRock#6852 (merged 2026-08-31) removed rocm_smi_lib from ROCm 10.1+. This breaks MORI's GPU topology build (rocm_smi/rocm_smi.h not found).

Keep both code paths, version-guarded via ROCM_VERSION from rocm_version.h:

MORI_USE_AMDSMI=1 (ROCm >= 10.1): amd_smi API, socket/processor handles
MORI_USE_AMDSMI=0 (ROCm <= 10.0): rocm_smi API, flat device indices

Key API migration:

rsmi_init(0) → amdsmi_init(AMDSMI_INIT_AMD_GPUS)
rsmi_num_monitor_devices → amdsmi_get_socket/processor_handles
rsmi_dev_pci_id_get → amdsmi_get_gpu_device_bdf (BDF struct)
rsmi_is_P2P_accessible → amdsmi_topo_get_p2p_status (4 args)
RSMI_IO_LINK_TYPE → amdsmi_link_type_t
rsmi_status_string → amdsmi_status_code_to_string

Uses __has_include for rocm_version.h path portability.

Fixes #638

Motivation

MORI build fails on ROCm 10.1+ with fatal error: 'rocm_smi/rocm_smi.h' file not found.

ROCm/TheRock#6852 (merged 2026-08-31) disabled
rocm_smi_lib from ROCm 10.1+ builds. MORI's GPU topology code (gpu.cpp, gpu.hpp, check.hpp)
includes rocm_smi/rocm_smi.h and uses the rsmi_* API, which no longer ships with ROCm 10.1+.

Fixes #638

Technical Details

Version-guarded dual code using ROCM_VERSION_MAJOR/ROCM_VERSION_MINOR from rocm_version.h. Both old
and new code paths live side-by-side:

MORI_USE_AMDSMI=1 (ROCm >= 10.1): amdsmi_* API with socket/processor handle model
MORI_USE_AMDSMI=0 (ROCm <= 10.0): rsmi_* API with flat device indices (unchanged)

Key API migration:

rocm_smi	amd_smi	Notes
`rsmi_init(0)`	`amdsmi_init(AMDSMI_INIT_AMD_GPUS)`
`rsmi_num_monitor_devices(&n)`	`amdsmi_get_socket_handles` + `amdsmi_get_processor_handles`
Socket/processor handle model
`rsmi_dev_pci_id_get(i, &id)`	`amdsmi_get_gpu_device_bdf(handle, &bdf)`	BDF struct instead of
packed uint64
`rsmi_is_P2P_accessible(i, j, &ok)`	`amdsmi_topo_get_p2p_status(h_i, h_j, &type, &cap)`	4 args;
returns link type + capability struct
`RSMI_IO_LINK_TYPE`	`amdsmi_link_type_t`	Enum type change
`rsmi_status_string`	`amdsmi_status_code_to_string`

Uses __has_include for rocm_version.h path portability (<rocm-core/rocm_version.h> on ROCm 7.2+,
<rocm_version.h> fallback for older).

Files changed:

include/mori/application/topology/gpu.hpp — version guard, include switch, amdsmi_link_type_t type
field
include/mori/application/utils/check.hpp — version guard, dual ROCM_SMI_CHECK macro
src/application/topology/gpu.cpp — dual Load() implementations with OpenAmdSmi/OpenRocmSmi

Test Plan

Build MORI from source on ROCm 10.1.0 nightly (TheRock build) — verifies MORI_USE_AMDSMI=1 path
Build MORI from source on ROCm 10.0.0 GA — verifies MORI_USE_AMDSMI=0 path (no regression)
Both tested as part of vllmsrc docker build (MORI is a build dependency of vLLM)

Test Result

ROCm 10.1.0 nightly (MORI_USE_AMDSMI=1): MORI build + import mori smoke test pass — CI run
#34608518940
ROCm 10.0.0 GA (MORI_USE_AMDSMI=0): MORI build passes (downstream torchaudio ABI issue unrelated
to this change)

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/GOVERNANCE.md#pull-requests.

TheRock#6852 (merged 2026-08-31) removed rocm_smi_lib from ROCm 10.1+. This breaks MORI's GPU topology build (rocm_smi/rocm_smi.h not found). Keep both code paths, version-guarded via ROCM_VERSION from rocm_version.h: - MORI_USE_AMDSMI=1 (ROCm >= 10.1): amd_smi API, socket/processor handles - MORI_USE_AMDSMI=0 (ROCm <= 10.0): rocm_smi API, flat device indices Key API migration: - rsmi_init(0) → amdsmi_init(AMDSMI_INIT_AMD_GPUS) - rsmi_num_monitor_devices → amdsmi_get_socket/processor_handles - rsmi_dev_pci_id_get → amdsmi_get_gpu_device_bdf (BDF struct) - rsmi_is_P2P_accessible → amdsmi_topo_get_p2p_status (4 args) - RSMI_IO_LINK_TYPE → amdsmi_link_type_t - rsmi_status_string → amdsmi_status_code_to_string Uses __has_include for rocm_version.h path portability. Fixes ROCm#638

Formatting only — no logic changes: - check.hpp: align ROCM_SMI_CHECK macro trailing backslashes - gpu.cpp: compress candidates[] array, break long if-statement

QizhouZhang97

Thanks for resolving this issue. Please address the following issue before merging it

QizhouZhang97 · 2026-09-16T02:37:57Z

+  std::vector<amdsmi_socket_handle> sockets(socketCount);
+  ROCM_SMI_CHECK(amdsmi_get_socket_handles(&socketCount, sockets.data()));
+
+  uint32_t numGpus = 0;


could you check socketCount before you accessing sockets array

QizhouZhang97 · 2026-09-16T02:39:17Z


-// Bootstrap-only helper: this macro terminates the process and should not be
-// used in per-connection or per-transfer runtime paths.
 #define SYSCALL_RETURN_ZERO(stmt)                                                               \


retain this comment

QizhouZhang97 · 2026-09-16T02:42:10Z

+  ROCM_SMI_CHECK(amdsmi_get_socket_handles(&socketCount, sockets.data()));
+
+  uint32_t numGpus = 0;
+  ROCM_SMI_CHECK(amdsmi_get_processor_handles(sockets[0], &numGpus, nullptr));


the enumeration only walks sockets[0]. In amd_smi a "socket" is one per GPU, not one per host socket. I verified this against the real libamd_smi.so on this 8-GPU box: socket_count=8, each with processor_count=1. So numGpus comes out as 1, seven GPUs disappear, p2ps is empty, and GetGpuByLogicalId(1..7) returns nullptr — which system.cpp:66 dereferences unconditionally. Segfault on every non-zero rank. Fix is to loop over all sockets (or skip sockets entirely and use amdsmi_get_processor_handles_by_type).

… guard 1. gpu.cpp: Loop over ALL sockets to collect processor handles. amd_smi models one socket per GPU (not per host socket), so sockets[0] only sees 1 GPU on an 8-GPU box. Collect from all. 2. gpu.cpp: Guard socketCount == 0 before accessing sockets array. 3. check.hpp: Restore "Bootstrap-only helper" comment on SYSCALL_RETURN_ZERO.

srinivamd · 2026-09-20T14:43:17Z

@QizhouZhang97 — All three review comments from your Sep 16 review have been addressed in the latest commits:

Loop all sockets (not just sockets[0]) — Fixed in 1a55e2d. amd_smi models one socket per GPU, so the code now iterates all sockets to collect processor handles from each.
Guard socketCount == 0 — Fixed in 1a55e2d. Added early-exit with error message before accessing the sockets array.
Restore "Bootstrap-only helper" comment on SYSCALL_RETURN_ZERO — Fixed in 1a55e2d.

CI status on latest commit (3017c3c):

Check	Result
pre-commit (clang-format)	passed
mori build (MI355X_AINIC)	passed
mori build (MI300X_MLX)	passed
mori intranode test (MI355X_AINIC)	passed
mori intranode test (MI300X_MLX)	passed
mori intranode test (ROCm 10, MI355X_AINIC)	passed
mori JAX intranode test (MI355X_AINIC)	passed
mori umbp unit test (MI355X_AINIC)	passed
mori internode test (MI300X_MLX)	passed
mori internode test (MI355X_AINIC)	failed — `HIP error no ROCm-capable device` (runner infra, not code)
CCO unit test (MI355X_AINIC)	failed — same MI355X_AINIC runner infra issue

The two failures are both on the MI355X_AINIC internode runner reporting "no ROCm-capable device is detected" — a GPU device-passthrough issue on that CI node, not related to this PR's code changes. All builds and all other test configurations passed.

Ready for re-review and approval when you get a chance. Thanks!

jhchouuu assigned QizhouZhang97 Sep 14, 2026

jhchouuu requested a review from QizhouZhang97 September 14, 2026 05:48

style: apply clang-format to pass pre-commit checks

0fa114c

Formatting only — no logic changes: - check.hpp: align ROCM_SMI_CHECK macro trailing backslashes - gpu.cpp: compress candidates[] array, break long if-statement

QizhouZhang97 requested changes Sep 16, 2026

View reviewed changes

srinivamd added 2 commits September 19, 2026 10:43

style: unwrap ROCM_SMI_CHECK line to pass clang-format

3017c3c

Navigation Menu

fix: migrate GPU topology from rocm_smi to amd_smi (ROCm 10.1+) - #663

fix: migrate GPU topology from rocm_smi to amd_smi (ROCm 10.1+)#663
srinivamd wants to merge 4 commits into
ROCm:mainfrom
srinivamd:fix/migrate-rocm-smi-to-amd-smi-v2

srinivamd commented Sep 11, 2026

Uh oh!

QizhouZhang97 left a comment

Uh oh!

QizhouZhang97 Sep 16, 2026

Uh oh!

QizhouZhang97 Sep 16, 2026

Uh oh!

QizhouZhang97 Sep 16, 2026

Uh oh!

srinivamd commented Sep 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

srinivamd commented Sep 11, 2026

Motivation

Technical Details

Test Plan

Test Result

Submission Checklist

Uh oh!

QizhouZhang97 left a comment

Choose a reason for hiding this comment

Uh oh!

QizhouZhang97 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

QizhouZhang97 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

QizhouZhang97 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

srinivamd commented Sep 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants