Probes: own a probe on a rank face by one rank only

[sbryngelson]

Description

A probe that lies exactly on a face shared by two MPI sub-domains gets written with every output value doubled.

Root cause. s_write_probe_files (src/simulation/m_data_output.fpp) tests ownership with a closed interval, x_cb(-1) <= probe%x <= x_cb(m), in the 1D, 2D and 3D branches. So both neighboring ranks claim a probe on their shared face, and the per-probe s_mpi_allreduce_sum returns twice the value. Probes on "nice" coordinates such as z = 0 in a symmetric domain with an even number of ranks in z hit this every time.

Fix. The new helper f_probe_owned applies a half-open test, lo < v <= hi, so the face goes to the lower rank. The first rank in each direction also owns its lower face, so a probe on the global lower boundary is still sampled. The probe sampler reads the cell to the left of the first face at or beyond the probe (j - 2), so the lower rank holds exactly the cell serial samples, and multi-rank probe output now equals serial output. In serial (num_procs == 1), the single rank owns the closed interval exactly as before.

Probes that are not on a rank face are unaffected, and so are serial runs. Probe output for probes on a face changes from the sum of two ranks to the serial value.

Verification

Production (Frontier, CCE 19, OpenACC GPU build, applied to 3dc5b2f). These runs used an earlier version of this fix in which the face went to the upper rank. In this uniform flow both versions give the same values. The case was a 3D 912x400x544 grid with a uniform free stream (rho = 1, u = 1, p = 17.857) and probes at z = 0, which lies on a rank face.

run	ranks	probe1_prim.dat at t = 0 (rho, u, v, w, p, gamma)
before	64	2.000, 2.000, 0, 0, 35.714, 5.0
after	200	1.000, 1.000, 0, 0, 17.857, 2.5

Before the fix, the other probes at z = 0 (probes 3-7) were also exactly doubled, and probes off the face were correct.

Regression test (added): 1D -> 2 MPI Ranks -> Probe on rank face (B227B20E), ppn = 2. It uses 32 cells on [0, 1], split 16/16, so the rank face is exactly x = 0.5 (dyadic, so no roundoff), and puts the probe there. D/probe1_prim.dat is part of the golden; the packer keeps every probe column. The golden was generated on a Frontier CPU compute node with GNU 12.3 (gcc-native/12.3) + Cray MPICH, MPI, Release, matching the existing 2 MPI Ranks goldens (GNU, MPI, Release). Results:

code	build	result
this branch	Release	pass
this branch	Debug (-fcheck=all)	pass
master (m_data_output.fpp at 3dc5b2f)	Release	fail: probe1_prim.dat variable 2 is 1.0 against a golden of 0.5 (rho summed by both ranks)
master	Debug	fail (same mismatch)

A serial run of the same case on this branch gives a probe1_prim.dat identical (max abs difference 0) to the 2-rank golden. The existing tests 0FCCE9F1 (1D 2 ranks), 8C7AA13B (2D 2 ranks) and 18DB27D5 (2D probe) still pass.

simulation also builds with CCE 19 on CPU. Precheck passes apart from two test_thermochem cases that fail on the Frontier login node because they compile with the system /usr/bin/gfortran (addressed by #1943).

Contribution Policy

We do not accept pull requests generated primarily by AI without genuine understanding or real-world usage context.

All contributions are expected to demonstrate:

• A clear understanding of the codebase
• Alignment with product direction
• Thoughtful reasoning behind changes
• Evidence of real-world usage or hands-on experience with the problem

If these expectations are not met, we would prefer to implement the changes ourselves rather than spend time reviewing low-effort submissions.

---

Acknowledgement

• I confirm this PR meets the above expectations and reflects my own understanding and real-world context.

---

This PR was prepared with the assistance of an AI tool (Claude Code). The bug was found, and the fix exercised, in multi-node production runs on Frontier.

PR template credit: junegunn

[Copilot]

Copilot review overview

🟡 Changes recommended

The MPI-only ownership behavior lacks a deterministic two-rank regression test.

Review effort: Balanced
Findings: 1

Open (1)

• Add two-rank regression for MPI probe ownership double-counting · New

What changed in this PR

Fixes duplicate MPI probe sampling on shared rank faces.

Changes:

• Adds half-open probe ownership checks.
• Preserves ownership of global upper boundaries.

File	Description
src/​simulation/​m_data_output.fpp	Assigns each probe location to one rank.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[sbryngelson]

Added in c5bfb0e: 1D -> 2 MPI Ranks -> Probe on rank face (B227B20E, ppn=2), with 32 cells on [0, 1] split 16/16 and the probe at the face x = 0.5. D/probe1_prim.dat is in the golden (GNU 12.3 + MPI, Release, generated on a Frontier CPU compute node).

Evidence:

• This branch passes in Release and Debug.
• With master's m_data_output.fpp it fails: variable 2 reads 1.0 against a golden of 0.5, i.e. both ranks summed.

While writing the test I also moved the face to the lower rank (lo < v <= hi, first rank keeps its lower face). The sampler reads the cell left of the face, so this matches serial: a serial run of the same case gives a probe file identical to the 2-rank golden (max abs difference 0).

[github-actions]

Lines of Code

File	Lines	Diff
src/simulation/m_data_output.fpp	1578	+8

Directory	Lines	Diff
simulation	28062	+8
total	47016	+8

[codecov]

Codecov Report

❌ Patch coverage is 27.27273% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.67%. Comparing base (3dc5b2f) to head (c5bfb0e).

Files with missing lines	Patch %	Lines
src/simulation/m_data_output.fpp	27.27%	4 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1945      +/-   ##
==========================================
+ Coverage   62.64%   62.67%   +0.03%     
==========================================
  Files          86       86              
  Lines       22425    22430       +5     
  Branches     3325     3326       +1     
==========================================
+ Hits        14048    14059      +11     
+ Misses       6119     6110       -9     
- Partials     2258     2261       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.