Skip to content

BUG: Correct filter results, pipeline copies, and output checks - #1974

Merged
imikejackson merged 20 commits into
BlueQuartzSoftware:developfrom
jmarquisbq:fix/small-data-corrections
Nov 19, 2026
Merged

imikejackson merged 20 commits into
BlueQuartzSoftware:developfrom
jmarquisbq:fix/small-data-corrections

Conversation

@jmarquisbq

@jmarquisbq jmarquisbq commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Keep feature attributes aligned with randomized IDs, preserve legacy numeric values, and measure edge samples from the starting endpoint. Preserve pipeline metadata, stop stalled threshold replacement loops, and return errors when output path inspection fails. 38 files changed, +2171 / −111 lines.

The companion SimplnxOoc #42 updates a test for an interface already on develop. The two PRs can merge in either order.

1. Filter results and termination

  • src/simplnx/Utilities/ClusteringUtilities.cpp moves each feature tuple to its mapped ID through complete permutation cycles. Feature zero stays fixed.
  • src/simplnx/Parameters/StringParameter.cpp preserves the value of floating-point legacy JSON parameters when it converts them to text. Integer inputs use the existing integer conversion.
  • src/simplnx/Utilities/StringInterpretationUtilities.hpp treats complete nonzero fractional text as true in the shared Boolean converter. ConditionalSetValue uses that converter for both execution paths.
  • src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/PointSampleEdgeGeometry.cpp includes the half-subdivision offset in each distance from the starting endpoint.
  • src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ReplaceElementAttributesWithNeighborValuesFilter.cpp requires a comparison array in the geometry's Cell Attribute Matrix in loop mode. A stalled pass returns -15002. The public filter reports cancellation as -1. Tuples copied before either result remain changed.

2. Pipeline copies and output checks

  • src/simplnx/Pipeline/Pipeline.cpp preserves disabled state through pipeline copies and value operations. Filter and placeholder copies retain metadata used by Write DREAM3D-NX File and the extracted pipeline.
  • src/simplnx/Utilities/FilterUtilities.cpp and src/simplnx/Utilities/FileUtilities.cpp return errors -4090 and -4092 when output path inspection fails. Directory creation error -4010 includes the operating system error code and message.

Test Plan

  • Linux Release passes all 4,056 in-core tests with both algorithm paths enabled.
  • Linux Release passes all 1,052 nonbenchmark out-of-core tests and 142 focused checks. The store-contract tests verify backed arrays.
  • Linux Debug passes 138 focused tests, including public filter cancellation and the feature permutation regression.
  • Windows MSVC Release passes 4,054 in-core tests. Two tests require POSIX directory search permissions and are skipped on Windows.
  • All 30 changed C++ files pass clang-format 19.1.1.
  • GitHub CI passes for the published revision.

The builds used simplnx 72beb73ab on develop 8cad1e818 with EbsdLib 3.1.4. The paired out-of-core build used SimplnxOoc 8bfd2ee7 and DREAM3D-NX 8e6d7403.

Scoped clang-tidy returned a nonzero result. Only assertion-macro diagnostics remained on lines changed by the review fixes.

Related Issue

Closes #1785

Closes #1944

Closes #1797

Closes #1806

Closes #1807

Closes #1856

Related to #1878 (PointSampleEdgeGeometryFilter item 4)

Related to #1879 (RandomizeFeatureIdsFilter item 1)

@jmarquisbq jmarquisbq self-assigned this Nov 13, 2026
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Comment thread test/FileUtilitiesTest.cpp Resolved
Comment thread docs/Coding_Knowledge_Base.md Resolved
@jmarquisbq

jmarquisbq commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Review feedback checklist

All nine items from Joey Kleingers’s review, with the original priorities.

High

  • Stop Loop Until Gone when the comparison array is outside Cell Data. A same-size array outside the geometry’s Cell Attribute Matrix can keep producing donor selections while its values never change. Validate array ownership in preflight, detect a pass that does not reduce the failing-cell count, or apply both checks. Make the preflight modification declaration match the arrays execution writes. Add a filter-level regression and update the filter docs and V&V path table. Review thread (1084981f3)

Medium

  • Give each FileUtilities test a unique scratch directory. Use a UUID in the fu1856- directory name and update the destructor’s filename safety check. This prevents parallel or interrupted runs from interfering with other cases. Review thread (617511735)
  • Add a core regression for the feature ID permutation. Call ClusterUtilities::RandomizeFeatureIds(store, totalFeatures, featureIArrays) from a [simplnx] test. Verify that feature zero stays fixed, every tuple reaches its mapped ID in a cycle longer than two, and rows at or beyond totalFeatures stay unchanged. Review thread (af5ed3ac7)
  • Run the cancellation regression through the filter. Use ReplaceElementAttributesWithNeighborValuesFilter::execute with the existing arguments, message handler, and cancel flag. Keep the assertions that cancellation takes priority over the stall error. Review thread (afadaa0c4)
  • Correct the threshold replacement V&V summary. Replace “No deviations” with a statement that stalled-loop termination differs from DREAM3D 6.5.171, and point to the termination contract for BUG: Stop threshold neighbor replacement when no donor passes #1806. Review thread (62fa6112d)
  • Decide where fractional Boolean text conversion belongs. Move the complete numeric-text check into StringInterpretationUtilities::Convert<bool>, add a core [simplnx] regression, and remove the local ConditionalSetValue helper. If the fix stays local to ConditionalSetValue, file a follow-up for the shared conversion and reference it from the helper. Review thread (6c1b6912a)

Nits

  • Make the V&V evidence traceable. In the threshold replacement and Write DREAM3D V&V reports, replace references to the task validation record and controller with the tested commit, platform, and result, or link the relevant PR evidence. Review thread (62fa6112d)
  • Read the placeholder disabled flag once. Use the Boolean value from the validated iterator directly, or share the isDisabled key constant so the lookups cannot drift. Review thread (762faf38c)
  • Clarify the Pipeline child-sharing contract in the knowledge base. State how Pipeline copy and move construction and assignment share child nodes with the source, and preserve that behavior when copying additional fields. Review thread (edd3334c3)

All fixes are included in af5ed3ac7. The full V&V refresh remains deferred to the planned team pass.

@joeykleingers joeykleingers left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The summary of my review of 2026-11-13 is covered by the review feedback checklist.

@jmarquisbq
jmarquisbq force-pushed the fix/small-data-corrections branch from af5ed3a to 72beb73 Compare November 16, 2026 19:57
@jmarquisbq
jmarquisbq requested review from joeykleingers and removed request for joeykleingers November 16, 2026 21:05

@joeykleingers joeykleingers left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. I fixed the last five review nits in a7f2cd2: the Loop Until Gone help text now says that a stalled loop returns an error, the -15001/-15003 preflight codes and the isDisabled key are named constants, NumberToStringFilterParameterConverter::convert has Doxygen, the Boolean conversion test uses SIMPLNX_RESULT_REQUIRE_VALID, and the V&V summary mentions error -15002.

@imikejackson

Copy link
Copy Markdown
Contributor

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Move each feature tuple to its remapped ID through complete cycles.
Test scalar and component associations with resident and disk-backed stores.
Preserve feature zero and unused trailing feature rows.

Fixes BlueQuartzSoftware#1785

Signed-off-by: Jessica Marquis <jessica.marquis@bluequartz.net>
Keep legacy numbers accurate in floating-point and integer arrays.
Apply numeric truth to Boolean replacement and matching values.
Test precision limits, both algorithms and disk-backed Boolean stores.

Fixes BlueQuartzSoftware#1944

Signed-off-by: Jessica Marquis <jessica.marquis@bluequartz.net>
Include the half-subdivision offset used by sample placement.
Preserve sample coordinates and check distances against geometric values.

Fixes BlueQuartzSoftware#1797

Signed-off-by: Jessica Marquis <jessica.marquis@bluequartz.net>
Return an error when Loop Until Gone cannot replace any remaining
failing cell. Preserve tuples copied before the stalled pass.
Cover no donors, repeated propagation, partial progress and
cancellation with resident and disk-backed arrays.

Closes BlueQuartzSoftware#1806

Signed-off-by: Jessica Marquis <jessica.marquis@bluequartz.net>
Keep disabled flags and filter comments through node copies and
pipeline value operations. Restore Boolean flags for placeholders
while retaining their raw JSON and prior malformed-flag fallback.
Cover copying, saving, extraction and replay with a disabled reader.

Closes BlueQuartzSoftware#1807

Signed-off-by: Jessica Marquis <jessica.marquis@bluequartz.net>
Replace the pending checkpoint with the verified seven-case results.
Keep the historical sign-offs and limit the storage claim to metadata
and replay behavior.

Signed-off-by: Jessica Marquis <jessica.marquis@bluequartz.net>
Return invalid Results when filesystem status queries fail. Preserve
missing-path and concurrent-directory-creation behavior. Include the full
operating-system message when directory creation fails.

Add POSIX permission regressions and explicit platform skips.

Refs BlueQuartzSoftware#1856

Signed-off-by: Jessica Marquis <jessica.marquis@bluequartz.net>
Require Loop Until Gone to use the selected geometry's Cell Data.
Declare only the numeric arrays that execution can modify.
Check both external input placements, single-pass behavior, and
geometries with no Cell Attribute Matrix.

Refs BlueQuartzSoftware#1806

Signed-off-by: Jessica Marquis <jessica.marquis@bluequartz.net>
Use a UUID for each fixture directory. Create the directory
before using it and remove only the exact path the fixture owns.

Refs BlueQuartzSoftware#1856

Signed-off-by: Jessica Marquis <jessica.marquis@bluequartz.net>
Record the stalled-loop deviation in the report summary.
Point readers to the PR test plan and retain the dated
Write DREAM3D regression results without workflow wording.

Signed-off-by: Jessica Marquis <jessica.marquis@bluequartz.net>
State that Pipeline copies share their child nodes and that
move operations transfer the existing collection.

Signed-off-by: Jessica Marquis <jessica.marquis@bluequartz.net>
Check the public cancellation result before execution and
after a pass. Retain the progress and tuple assertions.

Signed-off-by: Jessica Marquis <jessica.marquis@bluequartz.net>
Read the Boolean from the guarded lookup and keep the raw
fallback JSON.

Signed-off-by: Jessica Marquis <jessica.marquis@bluequartz.net>
Preserve nonzero fractions in the shared converter while
retaining the existing integer and text fallbacks. Use the
shared path from ConditionalSetValue and add a core test.

Signed-off-by: Jessica Marquis <jessica.marquis@bluequartz.net>
Exercise the core utility with scalar IDs and component tuples.
Verify feature zero, a long cycle, and unchanged trailing rows.

Signed-off-by: Jessica Marquis <jessica.marquis@bluequartz.net>
Document the public cancellation error and the algorithm checkpoint separately.

Signed-off-by: Jessica Marquis <jessica.marquis@bluequartz.net>
Write exact decimal values at Float32 rounding ties. Test legacy comparison and replacement at ties and adjacent values through the public filter.

Document Boolean replacements and use the shared disk-backing assertion.

Signed-off-by: Jessica Marquis <jessica.marquis@bluequartz.net>
jmarquisbq and others added 3 commits November 18, 2026 16:22
Keep path diagnostics in UTF-8 and report a follow-up inspection error
when directory creation reports no error. Let threshold loops use an
alternate path to the same comparison array in Cell Data.

Clarify the native pipeline replay guarantee and identify the tested
revision in the writer verification report.

Signed-off-by: Jessica Marquis <jessica.marquis@bluequartz.net>
Signed-off-by: Jessica Marquis <jessica.marquis@bluequartz.net>
- The Loop Until Gone help text says that a stalled loop returns an
  error, as the filter page and the V&V report do.
- Name the -15001 and -15003 preflight error codes.
- PlaceholderFilter reads the disabled flag through
  AbstractPipelineNode::k_IsDisabledKey instead of a second literal.
- Document NumberToStringFilterParameterConverter::convert.
- The Boolean conversion test asserts with SIMPLNX_RESULT_REQUIRE_VALID.
- The V&V summary says that a stalled loop returns error -15002.
@imikejackson
imikejackson force-pushed the fix/small-data-corrections branch from a7f2cd2 to 89de3e0 Compare November 18, 2026 21:22
@imikejackson

Copy link
Copy Markdown
Contributor

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@imikejackson
imikejackson merged commit 7b9983e into BlueQuartzSoftware:develop Nov 19, 2026
5 checks passed
@imikejackson
imikejackson deleted the fix/small-data-corrections branch November 19, 2026 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment