Skip to content

Fix MtLink P2P detection with partially visible GPUs - #14

Merged
yeahdongcn merged 1 commit into
MooreThreads:mainfrom
froststeam:fix/partial-visibility-mtlink
Aug 20, 2026
Merged

yeahdongcn merged 1 commit into
MooreThreads:mainfrom
froststeam:fix/partial-visibility-mtlink

Conversation

@froststeam

Copy link
Copy Markdown
Contributor

Summary

  • prefer the pairwise mtmlDeviceCountMtLinkLayouts() query for NVLink-compatible P2P detection
  • avoid materializing handles for host GPUs hidden by MTHREADS_VISIBLE_DEVICES
  • retain the per-link scan for older runtimes that do not support the layout API
  • add regression tests for the pairwise path and the legacy fallback

Root cause

In an 8-GPU host with only four GPUs exposed, mtmlDeviceGetMtLinkSpec() still reports all 14 physical links as UP. mtmlDeviceGetMtLinkRemoteDevice() correctly returns MTML_ERROR_NOT_FOUND for links whose remote endpoint is outside the visible device set. Pairwise P2P detection should not enumerate those hidden endpoints.

Validation

  • 15 passed (test_mtlink_visibility.py, test_abi_layout.py, test_library_loading.py)
  • S5000, driver 5.2.0-server, MTHREADS_VISIBLE_DEVICES=0,1,2,3: all six visible pairs returned NVML_P2P_STATUS_OK, each reported two MtLink layouts, and the patched pairwise path performed zero remote-device lookups
  • legacy fallback test skips MTMLError_NotFound links and still finds a visible peer

The repository baseline currently has pre-existing full-file Black/flake8 failures; the new test file passes Black and isort checks.

@froststeam
froststeam force-pushed the fix/partial-visibility-mtlink branch from dcee0fb to 5343a22 Compare August 20, 2026 06:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants