Skip to content

Support dS and interior_facets on extruded meshes - #5542

Open
pbrubeck wants to merge 11 commits into
pbrubeck/bddc-pyop3from
pbrubeck/extruded-dS
Open

pbrubeck wants to merge 11 commits into
pbrubeck/bddc-pyop3from
pbrubeck/extruded-dS

Conversation

@pbrubeck

Copy link
Copy Markdown
Contributor

Stacked on #5541; review only the last four commits.

Summary

dS and mesh.interior_facets now work on extruded meshes, so FDM no longer needs extrude_interior_facet_maps.

Form compilation splits dS before TSFC:

 compile_form(form)
   for each split form f
+    if a mesh of f is extruded
+      replace each dS integral by dS_v + dS_h
     TSFCKernel(f)

On an extruded mesh, the interior facets are one entity set with one component per stratum:

mesh.interior_facets                       # op3.Slice with two components
├── interior_facet_vert
└── interior_facet_horiz
V.interior_facet_node_list                 # op3.Dat.from_components(vert map, horiz map)
mesh.interior_facet_local_facet_indices    # horizontal facets numbered after the vertical ones

op3.Dat.from_components is new. It builds a Dat on a set with several components from one Dat per component, in the layout of the axes. Owned entries of all components come first, so [:nfacets] still selects the owned facets.

FDM reads the interior facet maps of every mesh in the same way:

-index_facet, local_facet_data, nfacets = extrude_interior_facet_maps(V)
-index_coef, _, _ = extrude_interior_facet_maps(Gq_facet or Gq)
+nfacets = mesh.interior_facets.owned.local_size
+index_facet = V.interior_facet_node_list
+index_coef = coefficient.function_space().interior_facet_node_list
+local_facet_data = mesh.interior_facet_local_facet_indices.data_ro.reshape((-1, 2))

Evidence

  • Before: assemble(Constant(1)*dS(domain=extruded_mesh)) raises ValueError: interior_facet integral cannot be used with a TensorProductCell. mesh.interior_facets raises TypeError on extruded meshes.
    After: test_interior_facet_dS_extr gets an interior facet area of 3.0 on the unit cube with 2x2x2 cells. test_interior_facet_maps_extr checks that, on every rank, the node list and the local facet numbers describe the same owned facets, and that the ranks together own the 10 interior facets once. Both pass at 1 and 3 ranks. The maps test fails at 3 ranks if the subset Dats are concatenated instead of laid out by pyop3.
  • tests/pyop3/expr/tensor/test_dat.py::test_from_components checks that owned entries of all components are stored before the ghost entries.
  • tests/firedrake/extrusion/ and the facet regression tests (test_facet_*.py, test_facets.py, test_interior_facets.py, test_mixed_interior_facets.py): 586 passed at 1 rank, 54 passed at 3 ranks, 0 failed.
  • test_fdm.py::test_ipdg_direct_solver passes for cg/dg/rt-Box and dg/rt-Rectangle.

Merge Danger

Door: two-way

Blast Radius: mesh

_entity_axes_by_name builds every named entity set, so a regression would show up in facet assembly on all meshes. On extruded meshes, V.interior_facet_node_list returns only owned rows, while on other meshes it also returns ghost rows.

This PR was prepared with Claude Code.

🤖 Generated with Claude Code

pbrubeck and others added 11 commits October 22, 2026 00:28
_new_uid kept one counter per communicator, so a mesh on COMM_SELF could
receive the same ufl_id as a mesh on COMM_WORLD. UFL compares meshes by
ufl_id, so a Submesh on COMM_SELF compared equal to its parent, and
Function.assign from the parent took the single-mesh path. This gave
wrong values and could deadlock a halo exchange.

Each communicator now receives, on the first request, an index on which
all its ranks agree and that no other communicator on those ranks uses.
The uid is the Cantor pairing of that index and the per-communicator
counter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A MATIS was preallocated with a global MATPREALLOCATOR, which records
only the rows that a rank owns. The local rows of ghost degrees of
freedom therefore had no nonzero pattern, so assembly in parallel lost
their contributions, and Dirichlet BCs failed on them.

The preallocator of a MATIS is now a MATIS whose local matrix is a
MATPREALLOCATOR, so the sparsity loop records the local pattern. Local
degrees of freedom that no local insertion touches are masked out of the
local-to-global maps, as PyOP2 did, so the local matrices have no empty
rows. This re-enables test_assemble_matis.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
get_entity_coordinates computes the entity coordinates with a
firedrake.par_loop instead of an op2.Kernel. BrokenDirichletBC overrides
_nodes, which DirichletBC uses for its node set, so that cellwise BCs
take effect. The par_loop in get_primal_indices indexes the marker with
the (node, component) shape of par_loop arguments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Dat that is passed to a C string kernel without a loop index (for
example the Dat of a Constant) has a symbolic size. Loopy needs an
integer shape, so take the local maximum of the size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port FDMPC.condense, which applies the exact Schur complement on the
facets with a local KSP that inverts the interior block on each cell.
The local KSP is attached to an empty Mat because a pyop3 local kernel
can only receive PETSc objects of type Mat. Add a test that uses
pc_use_amat with static condensation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port extrude_interior_facet_maps, which the SIPG terms of PoissonFDMPC
use. Extruded meshes are DMPlex meshes, so the maps come from the
interior vertical and horizontal facet sets instead of the layers.
Re-enable the DG and RT cases of test_ipdg_direct_solver.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Split each interior_facet integral over an extruded mesh into its
interior_facet_vert and interior_facet_horiz parts before calling TSFC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dat.from_components stores each Dat in its root component, in the layout
of the axes, so that the owned entries of all components come first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The interior facets of an extruded mesh are the union of its vertical and
horizontal interior facets, with one component for each. The DoF maps and
the local facet numbers on this set are built from those of the two
components. The local facet numbers of horizontal facets follow the
vertical facets, as in the facet numbering of a tensor-product cell.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Extruded meshes now support dS and interior_facets, so FDM reads the
interior facet maps of every mesh directly and no longer assembles
dS_h + dS_v.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pbrubeck pbrubeck added base:main Run this PR using a main (dev) build LLM used An LLM was used in the production of this PR pyop3 labels Oct 22, 2026
@pbrubeck
pbrubeck force-pushed the pbrubeck/bddc-pyop3 branch 3 times, most recently from 4ec2436 to 294b666 Compare October 26, 2026 10:10

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

base:main Run this PR using a main (dev) build LLM used An LLM was used in the production of this PR pyop3

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant