Skip to content

Add multi-NIC OFI/libfabric backend for Slingshot/CXI - #674

Open
vinaynkaranth wants to merge 1 commit into
ROCm:mainfrom
vinaynkaranth:hpe-mori-libfabric
Open

vinaynkaranth wants to merge 1 commit into
ROCm:mainfrom
vinaynkaranth:hpe-mori-libfabric

Conversation

@vinaynkaranth

Copy link
Copy Markdown

Add a libfabric (OFI) transport for MORI-IO targeting HPE Slingshot / CXI
fabrics, with per-NIC endpoint management, GPU→NIC affinity selection, and
inline descriptor-based address/MR exchange.

--- OFI backend (new) ---

  • src/io/ofi/backend_impl.{hpp,cpp}: FI_EP_RDM + FI_MR_ENDPOINT + FI_MR_PROV_KEY data plane (fi_read / fi_writemsg with offset-based RMA), HMEM (FI_HMEM_ROCR)
    registration for GPU buffers with host-only fallback, inline synchronous CQ
    drain, and a background multi-CQ poller.
  • Wire up the backend: src/io/engine.cpp (OFI create branch), src/io/CMakeLists.txt, include/mori/io/{backend.hpp (OfiBackendConfig), engine.hpp, enum.hpp, common.hpp},
    src/pybind/pybind_io.cpp, python/mori/io/{init.py, engine.py}.

--- Multi-NIC endpoints + affinity ---

  • OfiManager opens one fabric/domain/AV/CQ/endpoint per unique provider domain (per physical NIC) from fi_getinfo, and selects the local NIC per registration
    by GPU/NUMA affinity via TopoSystem (SelectLocalNic), binding each MR to its
    chosen endpoint.
  • Per-transfer destination fabric address is resolved from the remote MR's owning NIC: OfiMrBlob carries {key, nicId}; EngineDesc advertises a per-NIC
    address list (OfiEngineBlob); addresses are keyed (localNic, engineKey,
    remoteNic) since each AV is endpoint-scoped.

--- Topology: CXI NIC enumeration ---

  • src/application/topology/net.cpp: enumerate HPE Slingshot (CXI) NICs from /sys/class/cxi when ibverbs finds no NICs, parsing the PCI BDF and the link
    speed from device/port/0/link/speed (e.g. ck400G → 400 Gbps). ibverbs enumeration is now fault-tolerant. (include/.../topology/net.hpp)

--- Robustness ---

  • src/application/topology/system.cpp: guard null GPU / PCI-node lookups in CollectAndSortCandidates (previously segfaulted when the GPU BDF was missing
    from the PCI tree) and skip NICs absent from the tree before sorting.
  • src/application/topology/gpu.cpp: minor topology load hardening.
  • OfiManager::SelectLocalNic wraps affinity lookup in try/catch and falls back to a stable NIC on any topology error.

--- Tests / scripts ---

  • tests/cpp/io/ofi_slingshot_test.cpp: OFI backend test.
  • tests/scripts/{build_ofi_portage.sh, run_ofi_portage.sh, run_fi_write_basic.sh}: build/run harness for the OFI path.

Motivation

Technical Details

Test Plan

Test Result

Submission Checklist

Add a libfabric (OFI) transport for MORI-IO targeting HPE Slingshot /
CXI
fabrics, with per-NIC endpoint management, GPU→NIC affinity selection,
and
inline descriptor-based address/MR exchange.

--- OFI backend (new) ---
* src/io/ofi/backend_impl.{hpp,cpp}: FI_EP_RDM + FI_MR_ENDPOINT +
  FI_MR_PROV_KEY
  data plane (fi_read / fi_writemsg with offset-based RMA), HMEM
(FI_HMEM_ROCR)
  registration for GPU buffers with host-only fallback, inline
synchronous CQ
  drain, and a background multi-CQ poller.
* Wire up the backend: src/io/engine.cpp (OFI create branch),
  src/io/CMakeLists.txt,
  include/mori/io/{backend.hpp (OfiBackendConfig), engine.hpp, enum.hpp,
common.hpp},
  src/pybind/pybind_io.cpp, python/mori/io/{__init__.py, engine.py}.

--- Multi-NIC endpoints + affinity ---
* OfiManager opens one fabric/domain/AV/CQ/endpoint per unique provider
  domain
  (per physical NIC) from fi_getinfo, and selects the local NIC per
registration
  by GPU/NUMA affinity via TopoSystem (SelectLocalNic), binding each MR
to its
  chosen endpoint.
* Per-transfer destination fabric address is resolved from the remote
  MR's
  owning NIC: OfiMrBlob carries {key, nicId}; EngineDesc advertises a
per-NIC
  address list (OfiEngineBlob); addresses are keyed (localNic,
engineKey,
  remoteNic) since each AV is endpoint-scoped.

--- Topology: CXI NIC enumeration ---
* src/application/topology/net.cpp: enumerate HPE Slingshot (CXI) NICs
  from
  /sys/class/cxi when ibverbs finds no NICs, parsing the PCI BDF and the
link
  speed from device/port/0/link/speed (e.g. ck400G → 400 Gbps). ibverbs
  enumeration is now fault-tolerant. (include/.../topology/net.hpp)

--- Robustness ---
* src/application/topology/system.cpp: guard null GPU / PCI-node lookups
  in
  CollectAndSortCandidates (previously segfaulted when the GPU BDF was
missing
  from the PCI tree) and skip NICs absent from the tree before sorting.
* src/application/topology/gpu.cpp: minor topology load hardening.
* OfiManager::SelectLocalNic wraps affinity lookup in try/catch and
  falls back
  to a stable NIC on any topology error.

--- Tests / scripts ---
* tests/cpp/io/ofi_slingshot_test.cpp: OFI backend test.
* tests/scripts/{build_ofi_portage.sh, run_ofi_portage.sh,
  run_fi_write_basic.sh}:
  build/run harness for the OFI path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant