[Bug] A5 URMA first GET stalls waiting for CQE on physical cards 5 to 6 #2969

Platform

a5 (Ascend 950 hardware)

Runtime Variant

tensormap_and_ringbuffer

Description

On the original st-onboard-a5 runner, a two-rank URMA GET from physical card 5 to physical card 6 consistently leaves the first CQE pending until the scheduler reports S3 and the AICPU stream returns 507018. The reverse transfer on the same pair completes. This reproduces before domain A is released and with domain B absent.

Steps to Reproduce

1. Build and install the A5 package at the commit below with PTO-ISA 90a3796597a5fabc2ce5066e73f785c2e85822f7 and CANN 9.3.0 on suzhou-blue128.

2. On the original physical cards, run the retained-domain test from the checkout using its installed environment.

   task-submit --device 5,6 --run '
     SIMPLER_SCHEDULER_TIMEOUT_MS=2000 \
     SIMPLER_OP_EXECUTE_TIMEOUT_US=3000000 \
     SIMPLER_STREAM_SYNC_TIMEOUT_MS=6000 \
     python -m pytest \
       tests/st/a5/tensormap_and_ringbuffer/urma_deferred_completion/test_urma_domain_lifecycle.py::test_retained_urma_domain_release_preserves_other_domain \
       --resource-child --platform a5 --runtime tensormap_and_ringbuffer \
       --device "$TASK_DEVICE" --exclude-level 4 --require-pto-isa \
       --pto-session-timeout 1200 --manual exclude -v
   '

Expected Behavior

Both first GETs complete, both dependent consumers run and match their golden outputs, and the test subsequently verifies that releasing domain A preserves domain B.

Actual Behavior

Card 5 emits FIN for the first GET. The host-readable failure snapshot shows both mailbox messages consumed, the normal completion sentinel received, and one software submission. The first CQ slot still has its pending owner phase when the scheduler exits.

AICPU stream 61, task 17: 507018
scheduler code=100 S3 completed=0/2 running=0 ready=1 waiting=1 orch_done=1
head=2 tail=2 normal_done=1 row=58 target_bb=1 target_cqe=1
sq_head=1 sq_tail=0 submitted=1 cq_tail=0 depth=32768 cqe_dw0=0x0

The snapshot confirms what the AICPU poller observed. Transport doorbell delivery and visibility of the CQE from another device-side observer remain to be checked.

Git Commit ID

Original CI checkout e465bd65ae032aac4b5fd2e90aaca54a0813f76e. The same first fault reproduced on #2966 head 8a5305056f244f15b5432c8f9ff258036efbebe8. The local diagnostic build 5e8c43449a37e7d22a925e9e68aee5d933627691 adds the read-only snapshot.

CANN Version

9.3.0

Driver Version

25.7.rc1.12, as reported by npu-smi

Host Platform

Linux (x86_64)

Additional Context

Original A5 CI job. #2966 covers the secondary force-reset crash after the failed communication run. The initial URMA stall remains reproducible with that protection.

Related to #2860, which records an intermittent chunked URMA GET failure on the same physical pair. The reproduction here uses the first small GET, with the completion and rank/peer controls above. A shared cause remains unverified.