Skip to content

Re-enable BDDC and FDM after pyop3 - #5541

Open
pbrubeck wants to merge 6 commits into
mainfrom
pbrubeck/bddc-pyop3
Open

pbrubeck wants to merge 6 commits into
mainfrom
pbrubeck/bddc-pyop3

Conversation

@pbrubeck

@pbrubeck pbrubeck commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Re-enables the BDDC and FDM tests that #3318 (pyop3) disabled. Fixes the bugs that they exposed in pyop3 and Firedrake.

 pyop3/buffer.py                       MATIS preallocation
-  global MATPREALLOCATOR                # records owned rows only, so ghost rows lost entries
+  MATIS with a local MATPREALLOCATOR; untouched local DoFs masked out of the l2g maps
 pyop3/insn/base.py                    C-string kernel argument shapes
-  arg.size                              # symbolic for unindexed Dats
+  get_local_max(arg.size)
 firedrake/bcs.py                      DirichletBC.set
+  accepts a Function value
 firedrake/preconditioners/bddc.py
-  op2.Kernel / op2.par_loop             # get_entity_coordinates
+  firedrake.par_loop with a reference-cell averaging matrix
-  BrokenDirichletBC.nodes
+  BrokenDirichletBC._nodes              # the attribute that DirichletBC.node_set uses
 firedrake/preconditioners/fdm.py
-  raise NotImplementedError             # condense, extrude_interior_facet_maps
+  ported to op3.loop / pack and the pyop3 facet maps

The block_diagonal branch of FDMPC.assemble_coefficients is unchanged. It depends on matrix-free context attributes from an unmerged PR.

Evidence

Test Before After
regression/test_assemble.py::test_assemble_matis (unskipped) allclose(A_is - A_aij, 0) fails 10 pass, n=3
regression/test_bddc.py (module skip removed) n=3 deadlock in create_matis; MatZeroRowsColumnsLocal "new nonzero" 53 pass (17 at n=1, 36 at n=3)
regression/test_fdm.py (dg/rt xfail removed, test_static_condensation_amat new) NotImplementedError 52 pass (18 at n=1, 32 at n=2, 2 at n=3)
regression/test_bcs.py::test_set_function_on_bc_nodes (new) assert np.isscalar pass (test_bcs.py: 102 at n=1, 1 at n=2)
pyop3/insn/test_loop.py::test_c_string_kernel_with_unindexed_global (new) loopy assertion pass (test_loop.py: 23 pass, 1 existing xfail)

Also run: test_facet_split.py 6 pass (n=1, 3), test_star_pc.py 22 pass (n=1, 2), make srclint clean.

Merge Danger

Door: two-way

Blast Radius: moderate

  • MATIS matrices assembled through pyop3 (mat_type="is") get a new local pattern and masked l2g maps. AIJ/BAIJ/nest preallocation is unchanged.

Contributions in this PR were assisted by Claude Code (Claude Opus 5.5) and Codex.

@pbrubeck pbrubeck added the LLM used An LLM was used in the production of this PR label Oct 21, 2026
@pbrubeck pbrubeck added the pyop3 label Oct 22, 2026
@connorjward

Copy link
Copy Markdown
Contributor

If this claims to fix about 10 bugs can they please be separate PRs.

@pbrubeck

Copy link
Copy Markdown
Contributor Author

If this claims to fix about 10 bugs can they please be separate PRs.

I don't think that's necessary. The commit history is clean enough

@connorjward

Copy link
Copy Markdown
Contributor

If this claims to fix about 10 bugs can they please be separate PRs.

I don't think that's necessary. The commit history is clean enough

It's a question of reviewability. It is vastly easier for me to review small changes. We have a policy of squashing commits anyway so it would be better for our commit history as well.

@pbrubeck

Copy link
Copy Markdown
Contributor Author

It's a question of reviewability. It is vastly easier for me to review small changes. We have a policy of squashing commits anyway so it would be better for our commit history as well.

It should be fine.

@pbrubeck
pbrubeck changed the base branch from main to pbrubeck/mesh-uid-comm-precursor October 22, 2026 10:49
Comment thread firedrake/preconditioners/fdm.py Outdated
@pbrubeck pbrubeck added the base:main Run this PR using a main (dev) build label Oct 23, 2026
pbrubeck and others added 4 commits October 26, 2026 09:53
A MATIS was preallocated with a global MATPREALLOCATOR, which records
only the rows that a rank owns. The local rows of ghost degrees of
freedom therefore had no nonzero pattern, so assembly in parallel lost
their contributions, and Dirichlet BCs failed on them.

The preallocator of a MATIS is now a MATIS whose local matrix is a
MATPREALLOCATOR, so the sparsity loop records the local pattern. Local
degrees of freedom that no local insertion touches are masked out of the
local-to-global maps, as PyOP2 did, so the local matrices have no empty
rows. This re-enables test_assemble_matis.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
get_entity_coordinates computes the entity coordinates with a
firedrake.par_loop instead of an op2.Kernel. BrokenDirichletBC overrides
_nodes, which DirichletBC uses for its node set, so that cellwise BCs
take effect. The par_loop in get_primal_indices indexes the marker with
the (node, component) shape of par_loop arguments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Dat that is passed to a C string kernel without a loop index (for
example the Dat of a Constant) has a symbolic size. Loopy needs an
integer shape, so take the local maximum of the size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pbrubeck
pbrubeck force-pushed the pbrubeck/bddc-pyop3 branch from d2792d1 to 3dbfc3d Compare October 26, 2026 09:53
@pbrubeck
pbrubeck changed the base branch from pbrubeck/mesh-uid-comm-precursor to main October 26, 2026 09:55
@pbrubeck
pbrubeck force-pushed the pbrubeck/bddc-pyop3 branch from 3dbfc3d to 4ec2436 Compare October 26, 2026 09:56
pbrubeck and others added 2 commits October 26, 2026 10:09
…ments

Port FDMPC.condense, which applies the exact Schur complement on the
facets with a local KSP that inverts the interior block on each cell.
The local KSP is attached to an empty Mat because a pyop3 local kernel
can only receive PETSc objects of type Mat. Add a test that uses
pc_use_amat with static condensation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port extrude_interior_facet_maps, which the SIPG terms of PoissonFDMPC
use. Extruded meshes are DMPlex meshes, so the maps come from the
interior vertical and horizontal facet sets instead of the layers.
Re-enable the DG and RT cases of test_ipdg_direct_solver.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pbrubeck
pbrubeck force-pushed the pbrubeck/bddc-pyop3 branch from 4ec2436 to 294b666 Compare October 26, 2026 10:10
Comment thread pyop3/buffer.py
) -> None:
if not isinstance(obj, PETSc.Object):
raise TypeError(f"'obj' must be a PETSc object, not {type(obj).__name__}.")
name = utils.maybe_generate_name(name, prefix, self.DEFAULT_PREFIX)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The maybe_ is weird. Why not name = name or utils.generate_name(prefix, self.DEFAULT_PREFIX)?
This is just a comment. I am not going to continue this discussion.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

base:main Run this PR using a main (dev) build LLM used An LLM was used in the production of this PR pyop3

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants