[Ziming Wang (ZenAlexa)]

implements the first opt-in slice proposed in #942 for review. CollectiveProgram.diagnose_synchronization() reports per-channel token mismatches and dependency cycles before a plan reaches serialization or execution. the API and its initial scope remain subject to maintainer review.

behavior

#806 checks totals per rank pair and channel type. balanced totals can include a wait-before-signal cycle or signals and waits on different semaphore instances. both cases currently serialize successfully.

the diagnostic resolves each memory channel using the executor's per-peer occurrence pairing, matches signal and wait ordinals, and combines those dependencies with each thread block's authored order. graphlib.TopologicalSorter identifies a concrete cycle in linear time and space. the report contains a status, a message, and zero-based rank/thread-block/operation locations.

• acyclic means the balanced token graph is acyclic.
• mismatch identifies the channel, counts, and first unmatched token.
• cycle includes a closed dependency witness with channel and token details.
• unsupported identifies the boundary that prevents analysis.

the call is read-only and optional. existing to_json() checks, instruction order, fusion, synchronization insertion, replication, and runtime execution remain unchanged. this keeps diagnostics separate from the intra-rank ordering work in #627.

scope

supports straight-line Simple plans with paired MemoryChannel endpoints, SyncType.both, and one ordered producer and one ordered consumer thread block per directed semaphore. normal and relaxed operations have the same token dependencies. positive instance replication creates independent copies of the graph.

the token model assumes fresh semaphore state or a previously balanced invocation, without counter overflow. device residency, transport progress, payload visibility, and data races require separate validation.

pipelines, packet operations, port/switch channels, cross-block barriers, intra-rank semaphores, thread-block groups, resource reuse, unpaired endpoints, shared producer/consumer blocks, nondefault signal/wait synchronization, and post-processed IR return unsupported. their existing compilation behavior is preserved.

validation

• CPU-only source-DSL scenarios passed on Python 3.12 and 3.15, loading the actual DSL independently of native GPU package initialization.
• 1,467 comparisons against an independent token-counter simulator covered balanced signal/wait permutations, repeated tokens, two-channel orderings, three ranks, different peer creation orders, different thread-block IDs, and normal/relaxed synchronization. cycle-witness edges were checked against the authored scenarios.
• lifecycle scenarios checked repeated diagnosis, plan completion after a mismatch, unsupported boundaries, fusion and replication, post-processing, and unchanged serialized output, including an allgather plan.
• 62 existing language scenarios were compared against the same mainline baseline. 59 produced identical serialized output. three failed identically on both revisions due to the existing MSCCLPPProgram reference in allgather_ring.py, the put_packet call in alltoall_pkt.py, and the signal/wait imbalance in put_with_signal_fuse_test.py.
• bash tools/lint.sh py dry passed for all 147 tracked Python files.
• built and installed the native wheel on Linux x86_64 with Python 3.12, GCC 13.3, and CUDA 12.9.86, targeting SM90 with MSCCLPP_USE_IB=OFF. the standard isolated build included the native bindings and the default collective, NCCL, and expert-parallel extensions. fixed FetchContent sources and offline wheels supplied the build dependencies.
• imported the installed package from site-packages and ran the scenario entry point through that package. all 1,467 token-oracle comparisons and lifecycle boundaries passed. all 62 existing language scenarios also ran through the installed package, with 59 successes and the same three pre-existing failures listed above.
• the IB-enabled configuration stopped at CMake dependency discovery because the image lacks the IBVerbs library. the Azure Pipelines trigger returned insufficient privileges. please run /azp run from an authorized account for the existing CUDA 12.9/13.0 workflow in the project's build image. GPU execution and collective throughput have not been measured.

added the scenario entry point to the existing CUDA 12.9/13.0 DSL workflow after package installation.

current CI integration

f6d2346 merges current main (f5643de9) and fixes the second import failure in the scenario step. at 6a35910 both CUDA 12.9 and 13.0 jobs built the wheel and got past libcuda.so.1 through the image's /usr/local/cuda/compat directory, then failed loading the native bindings with libnvidia-ml.so.1 missing, because the build links CUDA::nvml and the CPU runner has no driver NVML. the scenario step now links libnvidia-ml.so.1 to the toolkit stub /usr/local/cuda/lib64/stubs/libnvidia-ml.so in a temp directory for that one command, and fails early if the stub is absent. the same images expose LIBRARY_PATH=/usr/local/cuda/lib64/stubs and NV_NVML_DEV_VERSION, which is where the wheel build resolved NVML. actionlint passes. both lanes need to run on this revision to confirm the import and the scenarios.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Ziming Wang (ZenAlexa)]

/azp run

[azure-pipelines]

Commenter does not have sufficient privileges for PR 946 in repo microsoft/mscclpp

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

[Ziming Wang (ZenAlexa)]

added the image's existing driver library path so the CPU runner can import the installed package, and the scenario still goes through the native wheel. could you approve the current CUDA 12.9 and 13.0 workflow runs?

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.