Skip to content

268 ensemble multi rank mpi tasks launch multiple processes but each reports a singleton mpi world #269

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
markcoletti wants to merge 3 commits into
base: 0.9.0-prerelease
Choose a base branch
from
Open

268 ensemble multi rank mpi tasks launch multiple processes but each reports a singleton mpi world #269

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c55963a
Returned --bind-to core for prun
markcoletti Nov 5, 2026
1a92fa4
Added a documentation note regarding possible issues with `mpi4py` be…
markcoletti Nov 5, 2026
ee81a65
fix: clarify mpi4py installation requirements for IPS tasks in docume…
markcoletti Nov 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions doc/user_guides/ensembles.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -366,6 +366,28 @@ components within each running ensemble instance:
(assuming one core per MPI process). For example, two concurrent four-process
tasks need a budget of at least eight cores per instance.

.. note::

IPS tasks that call python scripts that use ``mpi4py`` must ensure that ``mpi4py`` package in your
environment is linked to OpenMPI, not MPICH or another MPI
implementation. An ``mpi4py`` package
linked to MPICH can cause every process launched by an IPS component to
report rank 0 in an ``MPI.COMM_WORLD`` of size 1 even though the user
requested multiple ranks.

If you encounter this symptom, check the library loaded by the Python
environment used for the task:

.. code-block:: bash

python -c 'from mpi4py import MPI; print(MPI.Get_library_version())'

In the affected environment, rebuilding and reinstalling ``mpi4py`` from
source after loading the ``openmpi`` module and setting
``MPICC="$(which mpicc)"`` will resolve this issue. Verify that the rebuilt
package reports Open MPI, and rerun a multi-rank task to confirm that its
communicator has the requested size and distinct ranks.

Portal
------

Expand Down
3 changes: 2 additions & 1 deletion ipsframework/task_manager.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -453,7 +453,7 @@ def build_launch_cmd(
if not mpicmd:
raise Exception('Missing %s command in $PATH' % (mpi_binary))

nproc_flag = '-np'
nproc_flag = '-n'

@Lance-Drane Lance-Drane Nov 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this actually do? I checked the output of prun --help using version 3.0.8 and I noted that these flags are referenced differently, but have the same description:

+----------------------+-----------------------------------------------+
|                      | Launch Options                                |
+----------------------+-----------------------------------------------+
| Option               | Description                                   |
|======================|===============================================|
| "--runtime-options   | Comma-delimited list of runtime directives    |
| <arg0>"              | for the job (e.g., do not abort if a process  |
|                      | exits on non-zero status)                     |
+----------------------+-----------------------------------------------+
| "-c" | "--np         | Number of processes to run                    |
| <num_procs>"         |                                               |
+----------------------+-----------------------------------------------+
| "-n" | "--n          | Number of processes to run                    |
| <num_procs>"         |                                               |
+----------------------+-----------------------------------------------+

I couldn't find any documentation for mpirun 5.0.7 (perlmutter), but for version 4.1.6 I did find this, where the options are also listed separately but have the same description:

mpirun (Open MPI) 4.1.6

Usage: mpirun [OPTION]...  [PROGRAM]...
Start the given program using Open RTE

-c|-np|--np <arg0>       Number of processes to run
-h|--help <arg0>         This help message
   -n|--n <arg0>         Number of processes to run

In any case this seems to work fine on my end

ppn_flag = '-npernode'
host_select = '-H'
if mpi_binary == 'prun':
Expand Down Expand Up @@ -483,6 +483,7 @@ def build_launch_cmd(
)
if cpp:
cmd += f' --map-by slot:PE={cpp}'
cmd += ' --bind-to core'
elif smp_node:
cmd = ' '.join([mpicmd, nproc_flag, str(nproc)])
else:
Expand Down
Loading