Skip to content

Validate face buffer of subdivision meshes - #631

Merged
stefanatwork merged 9 commits into
masterfrom
fix/subdiv-face-valence-validation
Oct 15, 2026
Merged

stefanatwork merged 9 commits into
masterfrom
fix/subdiv-face-valence-validation

Conversation

@stefanatwork

Copy link
Copy Markdown
Collaborator

The half edge of a face is looked up through the prefix sum of the face valences stored in the face buffer, but this buffer was never validated.

A face with zero vertices produced a start offset that pointed behind the half edge array, thus dereferencing the half edge of such a face during commit read out of bounds. A face buffer whose valences sum up to more than the size of an index buffer caused the half edge array, which is sized through the index buffer, to get written out of bounds. The sum was further accumulated in 32 bit, thus it could also wrap around.

The face buffer now gets validated before the half edge structures are built, and inconsistent face and index buffers raise an RTC_ERROR_INVALID_OPERATION error. The check is performed before any state is updated, thus a rejected geometry is left unmodified.

The half edge of a face is looked up through the prefix sum of the face
valences stored in the face buffer, but this buffer was never validated.

A face with zero vertices produced a start offset that pointed behind the
half edge array, thus dereferencing the half edge of such a face during
commit read out of bounds. A face buffer whose valences sum up to more
than the size of an index buffer caused the half edge array, which is
sized through the index buffer, to get written out of bounds. The sum was
further accumulated in 32 bit, thus it could also wrap around.

The face buffer now gets validated before the half edge structures are
built, and inconsistent face and index buffers raise an
RTC_ERROR_INVALID_OPERATION error. The check is performed before any
state is updated, thus a rejected geometry is left unmodified.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment thread doc/src/api/RTC_GEOMETRY_TYPE_SUBDIVISION.md Outdated
Comment thread kernels/common/scene_subdiv_mesh.cpp Outdated
Comment thread doc/src/api/RTC_GEOMETRY_TYPE_SUBDIVISION.md Outdated
@stefanatwork
stefanatwork requested a review from svenwoop October 12, 2026 11:27
@stefanatwork

Copy link
Copy Markdown
Collaborator Author

I ditched the excessive AI slop and went for a minimally invasive fix:
A face with zero edges at the end of a subd mesh would get starting index of the size of the halfEdge array, after running the parallel prefix sum. Later on, the code tried to check the validity of that half edge, but didn't check if it was out of bounds. My fix is now to skip looking at half edges when the face has zero edges, and to clamp the last index coming out of the prefix sum.

Comment thread kernels/common/scene_subdiv_mesh.cpp Outdated
@stefanatwork
stefanatwork merged commit 89109c8 into master Oct 15, 2026
14 of 16 checks passed
@stefanatwork stefanatwork self-assigned this Oct 15, 2026
@stefanatwork
stefanatwork deleted the fix/subdiv-face-valence-validation branch October 15, 2026 15:39
svenwoop pushed a commit that referenced this pull request Oct 15, 2026
In certain cases, subdivision faces with zero vertices/edges could lead to buffer overruns when building the half edge data structure. Added extra checks to skip these faces.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants