IB: size the ownership hand-off arrays by the neighborhood radius

[sbryngelson]

Description

s_handoff_ib_ownership (src/simulation/m_ibm.fpp) overruns its neighbor arrays when ib_neighborhood_radius >= 2, which crashes multi-node runs with a moving IB in 3D.

Root cause. max_nbrs = 26 is a hard-coded parameter, and requests(2*max_nbrs), recv_neighbor_list(max_nbrs) and recv_bufs(:, max_nbrs) are sized from it. The receive loop, though, visits all (2R+1)^num_dims - 1 offsets (124 for R = 2, 342 for R = 3 in 3D) and writes recv_neighbor_list(nbr_idx) and requests(nreqs) for each one. The per-neighbor tags were already widened for R > 1, but the arrays were not. Separately, each receive buffer is sized for num_local_ibs_max patches (about 5 MB per neighbor), so every step posts megabytes of MPI_PACKED receives even when the sizes are legal.

Fix.

• Allocate requests, recv_neighbor_list and recv_bufs from max_nbrs = (2R+1)**num_dims - 1, and reuse max_nbrs in the unpack loop.
• Size each buffer for min(num_local_ibs_max, num_gbl_ibs) patches. A rank can hand off at most every global patch.

Behavior is unchanged for R = 1, apart from the smaller buffers. CFD results are unchanged.

Verification

This was run in production on OLCF Frontier (CCE 19, OpenACC GPU build, --case-optimization) with the same patch applied to 3dc5b2f, which is the current master:

• Before: 3D case, one moving IB (prescribed-kinematics plate, 912x400x544 ≈ 198M cells), 64 ranks on 8 nodes, ib_neighborhood_radius = 3. The run aborted in the first steps after about 65,000 cxil_map: write error lines, then failed with:
  MPI_Irecv(buf=..., count=5120004, MPI_PACKED, src=31, tag=216, ...) failed ... OFI tagged recv failed (Invalid argument).
• After: the same configuration ran clean, and so did every later 3D moving-IB run: 16-25 nodes, 128-200 ranks, automatic R = 2 and explicit R = 1 and R = 3. All ran to completion.

The out-of-bounds writes happen on any multi-rank run with R >= 2, because recv_neighbor_list(nbr_idx) is written for every offset, including MPI_PROC_NULL ones. With few ranks, though, they usually corrupt memory silently instead of crashing. The production crash needed many real neighbors (64 ranks across 8 nodes).

Regression test (added): 3D -> 2 MPI Ranks -> IBM Moving Sphere -> ib_neighborhood_radius=2 (2440B4AE), ppn = 2. It is the existing 2 MPI Ranks -> IBM Sphere setup (30x30x50 cells), with the sphere given a prescribed velocity (moving_ibm = 1, vel(1) = 0.01) so the hand-off runs every step, and ib_neighborhood_radius = 2. The sphere moves less than 1% of a cell in 50 steps, so no cell crosses its surface and the golden is not sensitive to roundoff. The golden was generated on a Frontier CPU compute node with GNU 12.3 (gcc-native/12.3) + Cray MPICH, MPI, Release, matching the existing 2 MPI Ranks goldens. Results:

code	build	result
this branch	Release	pass
this branch	Debug (-fcheck=all)	pass
master (m_ibm.fpp at 3dc5b2f)	Debug (-fcheck=all)	abort: At line 1782 of m_ibm.fpp: Index '27' of dimension 1 of array 'recv_neighbor_list' above upper bound of 26
master	Release	passes (the overrun is silent at 2 ranks)

So, to be explicit: this test guards the fix only in bounds-checked builds. The CI reldebug GNU lanes compile with -fcheck=bounds,pointer, which would catch a regression. In Release builds it is coverage of the R > 1 multi-rank hand-off path. The existing tests AFBACA70 (2-rank IB sphere), CE232828 (3D 2 ranks) and D7E7DE04 (moving IB, pitch ramp) still pass.

simulation also builds with CCE 19 on CPU. Precheck passes apart from two test_thermochem cases that fail on the Frontier login node because they compile with the system /usr/bin/gfortran (addressed by #1943).

Contribution Policy

We do not accept pull requests generated primarily by AI without genuine understanding or real-world usage context.

All contributions are expected to demonstrate:

• A clear understanding of the codebase
• Alignment with product direction
• Thoughtful reasoning behind changes
• Evidence of real-world usage or hands-on experience with the problem

If these expectations are not met, we would prefer to implement the changes ourselves rather than spend time reviewing low-effort submissions.

---

Acknowledgement

• I confirm this PR meets the above expectations and reflects my own understanding and real-world context.

---

This PR was prepared with the assistance of an AI tool (Claude Code). The bug was found, and the fix exercised, in multi-node production runs on Frontier.

PR template credit: junegunn

[Copilot]

Copilot review overview

🟡 Changes recommended

The corrected multi-rank radius-dependent path needs an automated regression test.

Review effort: Balanced
Findings: 1

Open (1)

• Add MPI moving-IB regression with ppn>1 and radius 2 · New

What changed in this PR

Fixes moving-IB ownership hand-off buffer overruns for neighborhood radii greater than one.

Changes:

• Dynamically sizes neighbor/request arrays from radius and dimensionality.
• Reduces per-neighbor buffer capacity using the global IB count.
• Reuses the computed neighbor count during unpacking.

File	Description
src/​simulation/​m_ibm.fpp	Corrects MPI hand-off array and buffer sizing.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[sbryngelson]

Added in 29de766: 3D -> 2 MPI Ranks -> IBM Moving Sphere -> ib_neighborhood_radius=2 (2440B4AE, ppn=2). It is the existing 2-rank IB sphere with moving_ibm = 1, vel(1) = 0.01 (under 1% of a cell over 50 steps, so no cell crosses the surface) and ib_neighborhood_radius = 2. The golden is GNU 12.3 + MPI, Release, generated on a Frontier CPU compute node.

Evidence:

• This branch passes in both Release and Debug (-fcheck=all).
• master in Debug aborts with m_ibm.fpp:1782: Index '27' of dimension 1 of array 'recv_neighbor_list' above upper bound of 26.
• master in Release passes: at 2 ranks the overrun is silent. So the test catches a regression in bounds-checked builds (the CI GNU reldebug lanes use -fcheck=bounds,pointer), and is coverage of the R>1 path otherwise.

[codecov]

Codecov Report

❌ Patch coverage is 60.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.98%. Comparing base (3dc5b2f) to head (29de766).

Files with missing lines	Patch %	Lines
src/simulation/m_ibm.fpp	60.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1944      +/-   ##
==========================================
+ Coverage   62.64%   62.98%   +0.34%     
==========================================
  Files          86       86              
  Lines       22425    22426       +1     
  Branches     3325     3325              
==========================================
+ Hits        14048    14126      +78     
+ Misses       6119     6018     -101     
- Partials     2258     2282      +24     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.