[Qinghua Zhou (seagater)]

Summary

• Add an optional GPUNetIO/GDAKI backend to the existing PortChannel API. CPU-proxy channels remain the default; GPU-initiated channels are constructed explicitly. Preserve main's proxy triggers and accumulate semantics.
• Fetch the unmodified official NVIDIA-DOCA/gpunetio dependency with CMake FetchContent, pinned to v4.0.1 commit bfe3e5484f16a01ac91a906b2f0046dbc8bd61a4. No Git submodule or copied NVIDIA sources are maintained in this tree.
• Expose owned connection, memory-registration and semaphore handles through GpuNetIoService. Bind channels with PortChannel(semaphore, remoteDestination, localSource); each channel uses its selected peer/QP, registration-relative offsets and private counters. Include per-QP atomic scratch, CUDA-device-aware teardown, path-MTU negotiation, installed dependency headers and license.
• Support sparse collective setup with setup(peerQpCounts, tag): 0-64 QPs per peer, zero for self, reciprocal counts on each edge, unequal counts across edges, and idle ranks. Allocate QPs/CQs and atomic scratch only for requested connections; exchange QP metadata only between requested peers. Legacy setup overloads retain uniform full-mesh behavior.
• Require HCA RDMA atomic support, direct GPU doorbells, valid DBRs and noncollapsed GPU-resident CQs; reject unsupported configurations collectively rather than selecting CPU-assisted fallback. Each local setup phase exchanges status before advancing, including QP transitions and final device-context publication.
• Add P2P, LL correctness/latency, multi-QP bandwidth, independent-channel and sparse-connection mp-unit coverage, with CPU regression tests for bindings, keys, queue selection, scratch isolation, policy and setup helpers.
• Target main independently of the EP integration (EP: integrate GPUNetIO and rank-major top-k-expanded layout #908). One explicit HCA per service; no EP kernels/layouts, Python EP interfaces, automatic HCA selection or multi-HCA policy.

Build and Use

With a supported CUDA toolchain, InfiniBand development dependencies, CMake and Git installed, configure from the repository root:

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
  -DMSCCLPP_USE_CUDA=ON -DMSCCLPP_USE_IB=ON \
  -DMSCCLPP_USE_GPUNETIO=ON -DMSCCLPP_BUILD_TESTS=ON
cmake --build build -j2

The first ON configure downloads the pinned dependency automatically and needs network access to GitHub. No git submodule update step is required. MSCCLPP_USE_GPUNETIO defaults to OFF; OFF builds do not declare, download or build GPUNetIO targets. The backend requires CUDA and IB and is not supported with ROCm. Select MSCCLPP_GPU_ARCHS as appropriate for the target GPUs when cross-compiling.

For an existing offline GPUNetIO checkout, add this option to the configure command:

-DFETCHCONTENT_SOURCE_DIR_GPUNETIO=/absolute/path/to/gpunetio

Verify that checkout is clean at bfe3e5484f16a01ac91a906b2f0046dbc8bd61a4: the FetchContent source override bypasses fetching and pin enforcement. Other project dependencies must also be available for a fully offline build. Reconfigure older build directories after the submodule-to-FetchContent migration.

Use the public <mscclpp/gpu_net_io_service.hpp> API in this order: setup(peerQpCounts, tag), connect(peer, qpIndex), registerMemory, exchangeMemory, buildSemaphore, then construct a PortChannel. Both endpoints request matching connection counts and matching exchange tags. Register source/destination payloads separately; offsets are registration-relative. See backend setup and lifetime contracts for the full example and installed-header usage. CMake consumers linking mscclpp or mscclpp_static inherit MSCCLPP_USE_GPUNETIO and the GPUNetIO include paths in ON builds. Installed consumers use find_package(mscclpp CONFIG REQUIRED) and link mscclpp::mscclpp or mscclpp::mscclpp_static, with CMAKE_PREFIX_PATH set to the install prefix. OFF targets export neither the feature macro nor DOCA header paths. Non-CMake/manual compilation still requires the documented flags and include paths.

Focused Checks

CPU regressions (no GPU/NIC execution):

ctest --test-dir build -R '^gpunetio_.*_test$' --output-on-failure

On configured GPU/RDMA ranks, with launcher-specific environment settings supplied as needed:

# Two-rank P2P and LL correctness
timeout 300s mpirun -np 2 build/bin/mp_unit_tests \
  --filter=PortChannelOneToOneTest.GpuNetIoP2P
timeout 300s mpirun -np 2 build/bin/mp_unit_tests \
  --filter=PortChannelOneToOneTest.GpuNetIoLLPingPong --exclude-perf-tests
# Four-rank sparse plan: unequal counts and one idle rank
timeout 300s mpirun -np 4 build/bin/mp_unit_tests \
  --filter=PortChannelSparseTest.GpuNetIoSparseConnections
# Two-rank, single-HCA multi-QP bandwidth benchmark
export MSCCLPP_GPUNETIO_QPS_PER_PEER=4
timeout 300s mpirun -np 2 -x MSCCLPP_GPUNETIO_QPS_PER_PEER \
  build/bin/mp_unit_tests --filter=PortChannelOneToOneTest.GpuNetIoMultiQpBandwidth

The bandwidth benchmark retains 256 B-8 MiB per QP, 10 warmup and 256 timed puts; it does not validate received payloads. The legacy MSCCLPP_EP_GPUNETIO_QPS_PER_PEER variable is a test-only fallback. Production configuration is explicit. Test filters are substring matches; performance tests are included unless excluded.

Validation

• Recorded validation for the exact tree published in 36f54ef: CUDA 13.0.88 Release ON/OFF shared/static core, unit_tests and mp_unit_tests builds for SM80/90/100/120. ON header probes inherit feature/include requirements from shared/static targets. Existing test warnings remain.
• Six ON and three OFF CPU regression tests pass. The actual-setup-body regression covers 68 four-rank fault cases, including missing atomic support and asymmetric QP-transition failures; threaded phase checks verify that peers do not advance after a reported local failure. CPU stubs do not prove GPU memory visibility, NIC ordering or real device cleanup.
• Independent shared/static CUDA consumers compile and link at SM90 against relocated ON and OFF installs using only the exported CMake package, without manual GPUNetIO flags or paths. Exports contain no build-tree paths; OFF installs contain no GPUNetIO headers. Backend/core/header install scripts were used; full installation of unrelated extensions was not a validation gate.
• Repository C++ lint and formatting checks for all 131 Python files pass.
• The author reported a passing two-node P2P run and verification of the subsequent sparse-connection, GID-index and Mahdieh-review patches. Exact runtime configurations/logs were not supplied here; this is not blanket qualification of all hardware, tests or performance.
• The above results apply to 36f54ef; later changes are not represented as newly tested here. Hosted CI status and ROCm compatibility are not claimed.

Limitations

• Sparse setup is still collective: every bootstrap rank participates, including idle ranks. Count-plan validation all-gathers O(worldSize^2) integers per rank. connect selects from the completed plan; fully on-demand/noncollective connection creation is not implemented.
• The unmodified upstream pin still contains known GPU_CPU allocation-fallback and CPU doorbell-service shutdown defects. This integration excludes those paths using GPU_SM_DB, valid DBRs, GPU-resident noncollapsed CQs, no CPU UMEM and no DOCA CPU progress service. It does not fix the upstream defects or support AUTO/CPU-assisted fallback.
• Local setup exceptions are reported collectively at phase boundaries before peers advance or publish success. Service construction (including status-buffer allocation) must succeed on all ranks first. Process loss, hung drivers and failures inside bootstrap communication still require launcher-level recovery; paired registration/semaphore exchanges after setup are outside this collective protocol. Keep an external launcher timeout and do not retry failed setup on the same service. A setup skip is not a hardware pass. Rebuild consumers after device-context layout changes.
• GPUNetIO now honors MSCCLPP_IB_GID_INDEX and validates the port/GID collectively before QP creation. Remaining transport review findings are tracked in the open review threads. Hardware correctness, performance and existing CPU-proxy runtime coverage still require configuration-specific verification.

Performance: GPUNetIO vs. IB CPU proxy

PortChannelOneToOneTest perf tests, GB300, 2 nodes in different racks, 1 GPU per node with its PCIe-local HCA, mean of 2 runs.

Metric	IB proxy (Host)	IB proxy (HostNoAtomic)	GPUNetIO
PingPong latency	27.0 µs	29.0 µs	32.5 µs
Bidirectional bandwidth, 128 MB	85.6 GB/s	85.5 GB/s	85.4 GB/s
1 QP bandwidth, 128 MB	87.7 GB/s	87.8 GB/s	87.6 GB/s
4 QPs bandwidth, 512 MB	97.5 GB/s	97.4 GB/s	97.3 GB/s

Why is GPUNetIO PingPong latency higher?

Each PingPong iteration sends a 4-byte putWithSignal, then waits for the peer's reply. At that size the time goes to per-operation overhead, not to moving data. The difference is in who pays the cost of posting the RDMA request:

• IB proxy: the GPU thread only pushes a 16-byte trigger into the host-pinned FIFO. A dedicated CPU proxy thread, which is always polling, does the rest: it builds the work request and rings the NIC doorbell from host memory.
• GPUNetIO: the GPU thread does all of that work itself.

Measured per iteration on rank 0, using in-kernel %globaltimer timers (temporary instrumentation, not committed):

Phase (µs)	IB proxy	GPUNetIO
putWithSignal returns on the GPU	1.15	4.8
wait for the peer's reply (includes the peer's own putWithSignal)	8.6	11.9
flush	15.6	14.9
Total per iteration	26.9	32.6

The GPUNetIO putWithSignal time breaks down as follows (µs):

Step	Time
__threadfence_system() before the post	0.7
Reserve two send-queue slots (a GPU atomic on the queue pair)	0.8
Write the two work requests (RDMA write plus the atomic that delivers the signal)	0.3
mark_wqes_ready (a release fence plus an atomic)	0.8
Submit: update the doorbell record, then MMIO-write the doorbell register on the NIC over PCIe	1.4
Memory-table lookup and bounds checks	~0.8

So each GPUNetIO post costs about 3.6 µs more on the issuing GPU thread. Both ranks pay it in every round trip: rank 0 directly in putWithSignal, and rank 1 inside rank 0's wait. That accounts for the ~5.5 µs gap. flush, which is bounded by the NIC completion, is the same or slightly faster.

Removing the __threadfence_system() only recovers about 0.6 µs (32.5 → 31.7 µs). The remainder is inherent to posting from a single GPU thread: device-scope atomics and fences, plus an uncached MMIO doorbell write across PCIe from the SM.

The bandwidth rows show this cost is amortized away for large messages. In real workloads, many GPU threads post concurrently on separate queue pairs, and posting overlaps with compute, without a CPU proxy thread or FIFO round trip. That is the case GPUNetIO targets, for example EP dispatch/combine.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Device completion and peer-validation defects remain alongside acknowledged upstream blockers and unverified GPU/NIC runtime behavior.

Review effort: Balanced
Findings: 1 · 3 · 3

Open (7)

• Reject self and out-of-range peer ranks during construction · New
• Validate peer before indexing peer arrays · New
• Release builds discard EBUSY instead of completing bounded flush · New
• Avoid enabling backend with broken pinned upstream implementation · New
• Add required MIT license header to public C++ header · New
• Add required MIT license header to C++ header · New
• Add required MIT license header to C++ test file · New

What changed in this PR

Adds an optional GPUNetIO/GDAKI backend to PortChannel while retaining CPU proxy behavior by default.

Changes:

• Adds collective GPUNetIO service setup and multi-QP device operations.
• Integrates the upstream GPUNetIO submodule and build/install configuration.
• Adds CPU-stub, P2P, and bandwidth tests plus usage documentation.

File	Description
.gitmodules	Registers the GPUNetIO submodule.
CMakeLists.txt	Adds and validates the build option.
docs/​quickstart.md	Documents optional backend setup.
include/​mscclpp/​gpu_net_io_service.hpp	Declares the host service API.
include/​mscclpp/​internal/​port_channel_gpunetio_device_impl.hpp	Implements device RDMA operations.
include/​mscclpp/​port_channel.hpp	Adds GPUNetIO channel construction.
include/​mscclpp/​port_channel_device.hpp	Routes device operations by backend.
include/​mscclpp/​port_channel_gpunetio_device.hpp	Defines the device context API.
src/​CMakeLists.txt	Includes the optional component.
src/​core/​CMakeLists.txt	Links GPUNetIO objects.
src/​core/​include/​ib.hpp	Exposes internal verbs handles.
src/​core/​port_channel.cc	Constructs GPUNetIO channels.
src/​gpunetio/​CMakeLists.txt	Builds and installs the dependency.
src/​gpunetio/​PROVENANCE.md	Documents provenance and limitations.
src/​gpunetio/​host/​gpu_net_io_qp_table.hpp	Builds sparse device QP tables.
src/​gpunetio/​host/​gpu_net_io_service.cpp	Sets up QPs, memory, and device state.
src/​gpunetio/​vendor	Pins the upstream GPUNetIO dependency.
test/​mp_unit/​CMakeLists.txt	Enables GPUNetIO MP tests.
test/​mp_unit/​port_channel_tests.cu	Adds P2P and bandwidth coverage.
test/​unit/​CMakeLists.txt	Registers focused unit tests.
test/​unit/​gpunetio_channel_test.cc	Tests routing, QPs, and host API behavior.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The GPU/NIC transport remains hardware-unverified, and the review found documentation, licensing, and build-diagnostic issues.

Review effort: Balanced
Findings: 2 · 3

Open (5)

• Update PR metadata and build instructions for FetchContent · New
• Restrict -w to vendored sources · New
• Add required Microsoft MIT license header · New
• Add required Microsoft MIT license header · New
• Update comment for compact qpHl indexing · New

Resolved since last review (7)

• Reject self and out-of-range peer ranks during construction
• Avoid enabling backend with broken pinned upstream implementation
• Release builds discard EBUSY instead of completing bounded flush
• Validate peer before indexing peer arrays
• Add required MIT license header to C++ test file
• Add required MIT license header to C++ header
• Add required MIT license header to public C++ header

[Copilot]

🔵 Needs a closer look

GPU/NIC ordering, concurrent queue progress, and resource teardown require final human review backed by configuration-specific hardware validation.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Update GpuNetIO instructions and test names for this revision

include/​mscclpp/​core.hpp:399

The PR description's GpuNetIoService example cannot compile against this revision, and its GpuNetIoP2P, GpuNetIoLLPingPong, and GpuNetIoSparseConnections filters match no tests. Update the usage instructions to use EndpointConfig::Ib::Mode::GpuNetIo with the ProxyService factories, as documented in docs/guide/gpunetio.md, and use current test names such as PortChannelOneToOneTest.PingPongIbGpuNetIoMode. Also distinguish validation of the earlier service-based implementation from validation of this revision.

2 resolved since last review

• Fix direct PortChannel construction memory bindings
• Reject unsupported non-1 RoCE ports or resolve MAC by port

_{Review effort: Balanced}

[Binyang Li (Binyang2014)]

Please add perf test results and compare it with IB perf test with CPU proxy

[Copilot]

🟡 Changes recommended

Unresolved memory-publication, connection-compatibility and resource-cleanup issues affect backend correctness.

1 open issue

• Add system-scope fence before standalone signal atomic · New

3 previously missed issues

In code that hasn't changed since last review

Reject incompatible HostNoAtomic and GpuNetIo endpoints symmetrically

src/​core/​context.cc:98

[P2] With a HostNoAtomic endpoint opposite a GpuNetIo endpoint, only the GPU side rejects the incompatible modes; the host side can return successfully and then wait indefinitely for its peer's semaphore exchange. Validate the incompatibility before selecting either connection implementation so both endpoints reject it. Use the resolved ibNoAtomic_ flag to cover automatic fallback as well as explicit configuration.

Release QP resources when QPN query fails during construction

src/​core/​gpunetio.cc:166

[P2] If the QPN query fails, construction throws after the QP has been created, but GpuNetIoQp::~GpuNetIoQp() is not called. Impl holds only a raw pointer, so the QP and its associated resources leak. Clean up around this query or give Impl ownership that also releases resources on constructor failure.

Update PR description for the endpoint-mode API and current tests

docs/​guide/​gpunetio.md:36

[P2] The PR description still instructs users to use GpuNetIoService and sparse setup(peerQpCounts, tag), but this tree implements the endpoint-mode API documented here. Its GpuNetIoP2P, GpuNetIoLLPingPong, sparse-connection and bandwidth filters also do not match the current tests, so those commands do not exercise this backend. Update the description to use EndpointConfig::Ib::Mode::GpuNetIo and ProxyService, list the actual test names, and distinguish earlier validation from this implementation.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

GPU/NIC ordering and shared-queue completion require human validation, and the identified constructor cleanup issue remains unresolved.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Clean up QP resources when constructor initialization fails

src/​core/​gpunetio.cc:166

If doca_verbs_qp_get_qpn returns an error, this constructor throws after allocating the QP and its NIC/GPU resources. The failed construction does not run GpuNetIoQp::~GpuNetIoQp(), and Impl does not release the QP, so those resources leak. Clean up the QP before rethrowing, or move ownership into a resource-owning member.

1 resolved since last review

• Add system-scope fence before standalone signal atomic

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Dependency cleanup needs correction, and GPU/NIC ordering and concurrent completion behavior still require hardware-backed human review.

1 open issue

• Prevent double-free when CUDA completion-queue initialization fails · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

A constructor failure path leaks networking resources, and the published usage and validation commands do not match the implementation.

No open issues 🎉

2 previously missed issues

In code that hasn't changed since last review

Clean up QP resources when QPN lookup fails

src/​core/​gpunetio.cc:166

If the SDK-backed doca_verbs_qp_get_qpn fails, this throws after QP creation succeeds, leaking the QP and its associated networking/GPU resources. Constructor failure skips GpuNetIoQp::~GpuNetIoQp(), and Impl holds only a raw pointer; the upstream export-failure backport does not cover this path. Destroy the QP before propagating the exception, or give it an owning cleanup wrapper.

Update PR description for endpoint workflow and current tests

docs/​guide/​gpunetio.md:36

The PR description still requires mscclpp/gpu_net_io_service.hpp and GpuNetIoService::setup(...), which are absent from this endpoint-based implementation, so its usage instructions cannot compile. Its four focused mp-unit filters also match no registered tests. Update the description to the EndpointConfig/ProxyService workflow shown here and the current ...IbGpuNetIoMode test names, and remove the obsolete sparse-service setup claims and test command.

1 resolved since last review

• Prevent double-free when CUDA completion-queue initialization fails

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The enabled build omits a required dependency source, and GPU/NIC ordering still requires hardware-specific validation and human review.

1 open issue

• Add missing CUDA wrapper source to GPUNetIO object library · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

An unresolved channel-binding crash needs correction, and GPU/NIC ordering, completion handling, and teardown require hardware validation and final human review.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Handle empty memory registrations during GPUNetIO binding

src/​core/​gpunetio.cc:331

[P2] Binding a GPUNetIO channel dereferences a null implementation if any ProxyService memory-table entry is a default-constructed RegisteredMemory, even when the channel's source and destination entries are valid. Unused placeholders are already added in src/core/executor/executor.cc:335-341. Add a nullable internal accessor, skip empty registrations while leaving their table entries zero-initialized, and cover an unused placeholder alongside valid bindings in a regression test.

1 resolved since last review

• Add missing CUDA wrapper source to GPUNetIO object library

_{Review effort: Balanced}