Add multi-NIC OFI/libfabric backend for Slingshot/CXI - #674
Open
vinaynkaranth wants to merge 1 commit into
Open
vinaynkaranth wants to merge 1 commit into
vinaynkaranth wants to merge 1 commit into
Conversation
Add a libfabric (OFI) transport for MORI-IO targeting HPE Slingshot /
CXI
fabrics, with per-NIC endpoint management, GPU→NIC affinity selection,
and
inline descriptor-based address/MR exchange.
--- OFI backend (new) ---
* src/io/ofi/backend_impl.{hpp,cpp}: FI_EP_RDM + FI_MR_ENDPOINT +
FI_MR_PROV_KEY
data plane (fi_read / fi_writemsg with offset-based RMA), HMEM
(FI_HMEM_ROCR)
registration for GPU buffers with host-only fallback, inline
synchronous CQ
drain, and a background multi-CQ poller.
* Wire up the backend: src/io/engine.cpp (OFI create branch),
src/io/CMakeLists.txt,
include/mori/io/{backend.hpp (OfiBackendConfig), engine.hpp, enum.hpp,
common.hpp},
src/pybind/pybind_io.cpp, python/mori/io/{__init__.py, engine.py}.
--- Multi-NIC endpoints + affinity ---
* OfiManager opens one fabric/domain/AV/CQ/endpoint per unique provider
domain
(per physical NIC) from fi_getinfo, and selects the local NIC per
registration
by GPU/NUMA affinity via TopoSystem (SelectLocalNic), binding each MR
to its
chosen endpoint.
* Per-transfer destination fabric address is resolved from the remote
MR's
owning NIC: OfiMrBlob carries {key, nicId}; EngineDesc advertises a
per-NIC
address list (OfiEngineBlob); addresses are keyed (localNic,
engineKey,
remoteNic) since each AV is endpoint-scoped.
--- Topology: CXI NIC enumeration ---
* src/application/topology/net.cpp: enumerate HPE Slingshot (CXI) NICs
from
/sys/class/cxi when ibverbs finds no NICs, parsing the PCI BDF and the
link
speed from device/port/0/link/speed (e.g. ck400G → 400 Gbps). ibverbs
enumeration is now fault-tolerant. (include/.../topology/net.hpp)
--- Robustness ---
* src/application/topology/system.cpp: guard null GPU / PCI-node lookups
in
CollectAndSortCandidates (previously segfaulted when the GPU BDF was
missing
from the PCI tree) and skip NICs absent from the tree before sorting.
* src/application/topology/gpu.cpp: minor topology load hardening.
* OfiManager::SelectLocalNic wraps affinity lookup in try/catch and
falls back
to a stable NIC on any topology error.
--- Tests / scripts ---
* tests/cpp/io/ofi_slingshot_test.cpp: OFI backend test.
* tests/scripts/{build_ofi_portage.sh, run_ofi_portage.sh,
run_fi_write_basic.sh}:
build/run harness for the OFI path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a libfabric (OFI) transport for MORI-IO targeting HPE Slingshot / CXI
fabrics, with per-NIC endpoint management, GPU→NIC affinity selection, and
inline descriptor-based address/MR exchange.
--- OFI backend (new) ---
registration for GPU buffers with host-only fallback, inline synchronous CQ
drain, and a background multi-CQ poller.
src/pybind/pybind_io.cpp, python/mori/io/{init.py, engine.py}.
--- Multi-NIC endpoints + affinity ---
by GPU/NUMA affinity via TopoSystem (SelectLocalNic), binding each MR to its
chosen endpoint.
address list (OfiEngineBlob); addresses are keyed (localNic, engineKey,
remoteNic) since each AV is endpoint-scoped.
--- Topology: CXI NIC enumeration ---
speed from device/port/0/link/speed (e.g. ck400G → 400 Gbps). ibverbs enumeration is now fault-tolerant. (include/.../topology/net.hpp)
--- Robustness ---
from the PCI tree) and skip NICs absent from the tree before sorting.
--- Tests / scripts ---
Motivation
Technical Details
Test Plan
Test Result
Submission Checklist