[Bug] Dynamic domain handshake markers exhaust /tmp inodes before communicator teardown #2783

Platform

a2a3 (Ascend 910B/C hardware)

Runtime Variant

All / Unknown

Description

The a2a3 host HCCL backend leaves per-allocation handshake files in /tmp after a dynamic communication domain is released. For a two-rank Fabric allocation, each unique allocation ID produces two fabric announce files, two fabric-ready markers, and two release markers. comm_release_domain_windows frees the window and erases the allocation record, but does not remove these files. Marker count therefore grows with completed allocations while the base Worker remains alive.

During a Worker's lifetime, the marker sweep runs only during base comm_destroy after a successful final destroy handshake. comm_init also sweeps its own prefix at startup. If final teardown fails or the process exits first, the files remain. A new Worker uses a PID-scoped rootinfo prefix, so its startup sweep cannot reclaim the previous process's files. The sweep currently ignores individual unlink and directory-iteration errors, which can also leave files behind without reporting failure.

Steps to Reproduce

1. On two a2a3 devices, start from the existing hardware test tests/ut/py/test_worker/test_dynamic_alloc_hw.py::test_two_rank_allocate_release_round_trip. It creates one Worker and calls Worker.run with orch.allocate_domain repeatedly.
2. Increase the a2a3 repetition count. Before Worker.close, count /tmp/barrier_alloc*.ready after each completed Worker.run.
3. On the Fabric path, observe approximately six additional files per successful two-rank allocation/release; the count does not fall when the domain is released.
4. To exercise the stale-generation path, end the Worker without a successful final destroy handshake and start a new Worker. The old PID-scoped markers remain.

Steps 1-4 describe a targeted reproduction based on the source path. The server incident below is observed evidence; this targeted test has not been run for this report.

Expected Behavior

Completed domain allocations should not cause unbounded /tmp inode growth while a Worker remains active. Final cleanup should safely reclaim stale markers after teardown failure, and should report failures to remove them.

Actual Behavior

On 2026-10-30, a shared aarch64 host's /tmp tmpfs had 1,048,576/1,048,576 inodes used, despite only 23 GiB of approximately 1,006 GiB data capacity being used. It contained 773,027 barrier_pto_multi_comm_*.ready files accumulated from 2026-10-27 through 2026-10-30. One exited process group left 260,520 files, consistent with 43,420 two-rank allocations at six files each. Ordinary temporary-file creation failed with "No space left on device" (observed with sort).

Before cleanup, 772,883 of these files belonged to 31 process-ID groups; none of those PIDs was running, and all files were older than two hours. Removing only those stale files reduced /tmp inode use from 100% to 27%.

Git Commit ID

0497e9d

Host Platform

Linux (aarch64)

Additional Context

The commit above is the current main source inspected for this report. The exact deployed binary commit from the incident was not captured.

• Worker rootinfo path under /tmp, scoped by PID and object ID
• Per-allocation Fabric announce name, fabric-ready barrier, and release path without marker deletion
• Cleanup gated on successful destroy handshake; prefix-limited sweep and ignored per-entry errors

A fix should safely retire markers for completed allocations without racing participating ranks, bound stale-generation residue after failed teardown, and report sweep errors. A focused regression can extend the existing two-rank hardware test to assert bounded marker count across repeated Worker.run calls and verify the stale-generation recovery path.

Related: #824 discusses repeated dynamic domain allocation cost, but not marker lifetime or inode exhaustion.