Repository navigation
BUG: Validate sampling, resampling and geometry count values - #1964
joeykleingers wants to merge 11 commits into
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
4b7d655 to
8e77db8
Compare
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
8e77db8 to
93e1900
Compare
93e1900 to
9b7907c
Compare
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
imikejackson
left a comment
There was a problem hiding this comment.
This whole PR is going to be redesigned as a NumberParameter and VectorParameter update instead.
9b7907c to
53871a0
Compare
Why this pull request has "Request changes"The defects that this pull request fixes are real, and the analysis behind them is careful. The changes are requested because of the design, not because of the individual fixes. 1. The same check is written again in each filterMost of the preflight changes check one value that the user typed: finite, greater than 0, at least 1, or in a range. This pull request adds about 20 of these checks. Each has its own error code (-86100 to -86174) and its own test. #1952 and #1973 do the same thing in other filters. Each new filter must remember to write the check again. simplnx already has the place for this check. #1991 adds optional bounds to params.insert(std::make_unique<Float32Parameter>(k_HatchSpacing_Key, "Hatch Spacing",
"The orthogonal distance between each generated vector.", 0.14f, Float32Parameter::Bounds::Positive()));The parameter layer gets its own error codes. A filter's preflight no longer checks a single value that the user typed, and the per-filter codes for these conditions (the new -861xx codes and the old -3012 to -3014, -7362 to -7364, -11000, -11001, -11500, -11502 and -23508 for these conditions) are removed. We will add the bounds on this branch and convert the filters that this pull request changes. The checks that must stay in a filter stay: relations between parameters, products and overflow, and values that depend on the final geometry (checked when the filter executes). 2. Findings in the parts that stayThese findings apply to the code that does not move to the parameter layer:
3. Defect found that this pull request did not causeResample Image Geom in Exact Dimensions mode can make one cell fewer than the user asks for (s = 1, d = 1, E = 7 gives 6 cells). This is now item L99 in #1908. Next stepsThis branch was rebased onto |
Sampling, resampling and scan-path filters accepted zero, negative, NaN or infinite spacing, resolutions, sigmas, origins, angles and coordinates, which led to division by zero, loops that never end, float-to-integer conversions of nonfinite values and NaN output. - Preflight rejects invalid active values in Create AM Scan Paths, Point Sample Edge Geometry, Resample Image Geom, Approximate Point Cloud Hull, Uncertain Regular Grid Sample Surface Mesh, Interpolate Point Cloud To Regular Grid (Gaussian sigmas), Compute Coordinate Threshold (sphere), Extract Feature Boundaries 2D (custom Z) and Regular Grid Sample Surface Mesh (origin). Inactive options are not checked. - Values derived from the final geometry (resampled spacing and dimensions, hull grid bounds, edge sample counts, resampling axis maps) are checked when the filter executes, before any conversion or allocation; preflight warns and uses safe placeholders when the current geometry gives an unrepresentable layout. - Add ValidateResamplingParameters() and ComputeResampledImageLayout() for the shared resampling code. Output for every previously valid value is unchanged. Part of BlueQuartzSoftware#1873. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Geometry-creation, partitioning, resampling, curvature, point-sampling and interpolation filters accepted zero or negative counts, signed values that wrapped to huge unsigned dimensions, nonfinite kernel sizes and dimension products that overflow, which led to empty or corrupt geometries, division by zero, huge allocations and float-to-integer conversions that are undefined behavior. - Preflight rejects negative Partition Geometry counts, partition products and ID ranges that cannot be represented (Basic, Advanced and Bounding Box modes), zero dimensions in Create Geometry (Image), Create Image Geometry, Resample Image Geom (Exact Dimensions) and Resample Rect Grid To Image Geom, Neighborhood Ring Count below 1 in Feature Face Curvature, negative Number of Sample Points in Point Sample Triangle Geometry and nonfinite Kernel Size in Interpolate Point Cloud To Regular Grid. Inactive options are not checked. - Values derived from final geometry are checked when the filter executes, before conversion or allocation: an existing partition grid's metadata, product and ID range, each coordinate-to-partition index conversion, and the interpolation kernel radius, size and storage. Preflight warns when the current geometry gives an unrepresentable kernel. - Add GeometryUtilities::ValidateImageDimensions() for the shared positive-dimension check. - A zero sample count now returns an empty vertex geometry without drawing samples; masked vertices are skipped before their coordinates are converted. Output for every previously valid value is unchanged. Part of BlueQuartzSoftware#1873. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Read Image Stack accepted zero Exact 2D Dimensions and nonfinite Scaling, and read the Exact 2D Dimensions vector even when another resampling mode was selected. Zero dimensions produced infinite spacing in the resampled slices. - Preflight checks only the active resampling option, before reading slice metadata or creating output actions: Exact 2D Dimensions must be at least 1 with a representable product, and Scaling must be finite and at least 1 percent. These use the shared resampling checks. - Inactive resampling options are no longer read or checked. Output for every previously valid value is unchanged. Part of BlueQuartzSoftware#1873. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Check numeric parameter values before filter preflight. Reject NaN and infinity by default. Check filter defaults in tests. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Use parameter bounds for sampling and resampling inputs. Prevent stalled hatch loops and check partition coordinates. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Use the names of the parameters in the application. Describe how Read Image Stack sets the X and Y dimensions. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Plan Read Image Stack X/Y resampling from one slice and check the stack product. Keep the worker error when resampling is canceled. Give the outside ID for an empty existing partition grid. Give each hull grid failure its own code and a recovery step. Merge split algorithm documentation blocks. Remove unused includes. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Give each interpolation preview warning and the RectGrid bound check their own codes. Check the Read Image Stack size one time. Name only the active controls in the resampling messages. Use CamelCase names for the NumberBounds members. Restore the knowledge base rules that bounds cannot express. Make the resampling cancellation test independent of thread count. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Declare the Gaussian Sigmas range on the parameter. Remove the filter check of the sigma squares, because the range keeps 2 * sigma * sigma a finite, nonzero value. Add an input data warning about NaN values to the Approximate Point Cloud Hull documentation. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
ecc5cf8 to
c06e312
Compare
|
@jmarquisbq - This PR needs to be reviewed by you since both @joeykleingers and I have commits. |
NaN and infinity in input data (array values, file values and the origin, spacing, bounds and coordinates of an input geometry) are no longer rejected. A check stays only where such a value would reach an integer conversion, an index, a loop bound, an ordering or a float32 narrowing. Typed number and vector parameters still reject NaN and infinity through their bounds. - Partition Geometry no longer rejects a partition grid with a nonfinite origin or spacing, or input coordinates and rectilinear bounds that are nonfinite. An exterior point gets the outside ID; a point that reaches the cell conversion with a nonfinite or out-of-range quotient fails with -86215. The Direct algorithm uses the real axis offsets and checks actual tuples when an axis map cannot be proven safe. Rectilinear bounds are read in blocks. - Resampling (Resample Data, Read Images [3D Stack]) no longer rejects a prepared geometry with a nonfinite origin, and keeps only the finite nonpositive spacing check. A destination cell exterior on any axis is zero-filled; -86137 is returned only when a cell that is inside on the other axes needs an unsafe conversion. - Interpolate Point Cloud to Regular Grid keeps only the finite nonpositive spacing check and checks each vertex's cell conversion after the mask and bounds checks (-86251), instead of calling the unchecked ImageGeom::getIndex(). - Approximate Point Cloud Hull checks each vertex's voxel before the int64 cast (-86248); a NaN vertex no longer reaches it. - Create AM Scan Paths checks the first and reconstructed hatch positions before narrowing to float32 (-86249, -86250) and sorts intersections with NaN after every number. - INodeGeometry0D::isPlane() compares every coordinate with the first instead of inserting them into a std::set<float32>, which NaN made undefined. - The shared resampling messages name the output spacing instead of New Spacing. - Removed the unused ResampleImageGeom::ComputeAxisSourceIndices(). Freed error codes: -86171, -86172, -86211, -86228, -86247. Part of BlueQuartzSoftware#1873. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Rewrite the knowledge base entry "Decide NaN and infinity handling by where the value comes from". Typed number and vector parameters still reject NaN and infinity by default. Array values, file values and input geometry data pass through without rejection, warning or a nonfinite-handling choice; a filter checks them only where they would reach undefined behavior (integer conversion, an index, size, count or loop bound, an ordering or equivalence, float32 narrowing or a standard library precondition). Separate the ordering rule (NaN after every number) from the equivalence rule (NaN equal only to NaN). Part of BlueQuartzSoftware#1873. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
|
@imikejackson @jmarquisbq I added two commits on top of Mike's rework:
They apply the NaN and infinity rule from our meeting. Typed number and vector parameters keep rejecting NaN and infinity through the new bounds. Array values, file values and input geometry metadata pass through, and a filter checks them only where they would reach undefined behavior. The knowledge base entry "Decide NaN and infinity handling by where the value comes from" now says this.
Validation on macOS, Release: affected tests 165/165 in-core and 169/169 out-of-core; full out-of-core suite 1053/1053. I also updated the description for the parameter bounds and this change. |
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
|
@joeykleingers I moved this PR back to draft and set it to In Progress. This is about your last two commits ( Why. Your commits remove the nonfinite checks on geometry data and guard only the integer conversions. That follows the "garbage in, garbage out" rule, and for values in data arrays I understand the reasoning. But an The plan. Issue #1994 makes geometry validity an invariant at the point where the values are set:
I am implementing this on the branch Merge order:
Knowledge base. The rewritten entry in Separately, I am no longer sure about the "garbage in, garbage out" rule for NaN and infinity in data arrays either. When a float array contains NaN or infinity, we cannot know whether the values are intentional, and the output is then undefined for the user. I would like to talk with you about that on its own, outside this PR. |
Summary
Sampling, resampling, scan-path and geometry-creation filters now reject spacing, resolution, sigma, origin, angle, count and dimension values outside their domain, instead of looping forever, dividing by zero, converting NaN or infinite values to integers, or wrapping signed counts to huge unsigned sizes. Values that a user types are checked by the parameter itself:
NumberParameterandVectorParametertake optional bounds, and every floating-point number and vector parameter rejects NaN and infinity. Values that depend on the final geometry (resampled layouts, hull grids, edge sample counts, partition indices, interpolation kernels) are checked when the filter executes. NaN and infinity in input data (array values and input geometry metadata) pass through; a filter checks them only where they would reach undefined behavior. 105 files changed, +6430 / −592 lines.1. Parameter bounds
src/simplnx/Parameters/NumberBounds.hpp—NumberBounds<T>: optional inclusive or exclusive lower and upper limits, presetsPositive(),NonNegative(),GreaterThan(),AtLeast(),LessThan(),AtMost(),Between()src/simplnx/Parameters/NumberParameter.{hpp,cpp},VectorParameter.{hpp,cpp}— store the bounds and check them invalidate(); a vector bound applies to every componenttest/DynamicFilterInstantiationTest.cpp— the bounds, and a check that every filter's default values satisfy its own parametersIFilter::preflight()already validates every active parameter beforepreflightImpl()and skips parameters that a linked option turns off, so the GUI,nxrunnerand Python all get these checks. The parameter layer has its own codes: -86200 (NaN or infinity), -86201 (below the lower limit) and -86202 (above the upper limit). Each message names the parameter by its on-screen name and gives the value.Every floating-point
NumberParameterandVectorParameterin every filter rejects NaN and infinity by default, also without bounds. A parameter can allow them withsetAllowNonFinite(true); no filter needs that today. Parameter UUIDs and the JSON format of the values do not change. The bounds are not yet shown in the DREAM3D-NX parameter widgets or the generated parameter tables.2. Spatial sampling and resampling values
src/simplnx/Utilities/ImageProcessing/ImageGeometryResample.{hpp,cpp}—ComputeResampledImageLayout()and the checked resampling axis mapssrc/Plugins/SimplnxCore/src/SimplnxCore/Filters/*Filter.cpp— parameter bounds and the remaining preflight checkstest/ImageGeometryResampleTest.cpp— tests for the shared resampling codeBounds on the parameters:
The finite requirement covers the angles, origins, uncertainty, sphere and Custom Z Value of these filters. Compute Coordinate Threshold keeps its preflight check that the sphere radius is greater than 0, because a vector bound applies to every component.
Resampled spacing, dimensions and tuple products, the hull's voxel bounds and grid product, and each edge's sample count depend on the final geometry. Preflight warns and uses unit placeholder dimensions and spacing when the current geometry gives a layout that cannot be represented; execution recomputes the layout from the final metadata and returns an error before converting or allocating. The hull check covers the grid with its one-voxel empty border, which is added in integer arithmetic so that it is kept for voxel indices beyond 2^53.
Create AM Scan Paths counts its hatch lines as integers instead of adding Hatch Spacing to a
float32position, so a spacing too small to move the position stops with an error instead of looping forever.3. Geometry and sampling counts
src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/PartitionGeometry*.cpp— final-grid checks and the checked coordinate-to-partition conversionsrc/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/InterpolatePointCloudToRegularGrid.{hpp,cpp}—ComputeKernelLayout()Bounds on the parameters:
usizebefore the<= 0check, so negative values passed); Cell Length > 0; Starting Feature ID ≥ 0. Preflight also rejects partition products that overflow and partition IDs that do not fitint32.Checked during execution: an existing partition grid's spacing, product and ID range; each coordinate-to-partition-index conversion; and the interpolation kernel's radius, size and storage, from the final spacing. Preflight warns when the current spacing gives a kernel that cannot be represented.
4. Read Image Stack resampling
src/Plugins/NXImageProcessing/src/NXImageProcessing/Filters/ReadImageStackFilter.cppScaling is at least 1 percent and Exact 2D Dimensions at least 1, through parameter bounds that apply only to the selected resampling option. In Exact 2D Dimensions mode, the output X and Y dimensions are the given values. The X/Y layout is planned from one slice, and the stack product is checked before reading the slices. Exact 2D Dimensions is no longer read when another mode is selected.
5. NaN and infinity in input data
docs/Coding_Knowledge_Base.md— "Decide NaN and infinity handling by where the value comes from"src/simplnx/DataStructure/Geometry/INodeGeometry0D.cpp—isPlane()Array values, file values and the origin, spacing, bounds and coordinates of an input geometry are not rejected for being NaN or infinite. A filter checks them only where they would reach undefined behavior: an integer conversion, an index, size, count or loop bound, an ordering, or a
float32narrowing.ImageGeom::getIndex(), whose comparisons NaN passes.int64conversion (-86248). A NaN vertex never wins the extent comparisons, so the extent check alone did not catch it.float32(-86249, -86250); intersections are sorted with NaN after every number.INodeGeometry0D::isPlane()compares every coordinate with the first instead of inserting the coordinates into astd::set<float32>, which is undefined with NaN.Finite spacing at or below 0 is still an error, with each filter's existing code (for example -86136 and -86212).
The knowledge base entry now says: typed number and vector parameters reject NaN and infinity; input data passes through; a filter guards only the undefined-behavior points above. For an ordering it uses NaN after every number, and for deduplication NaN is equal only to NaN.
Preflight is the one place for checks on the values a user types. The algorithms state the properties they rely on as
@preconditions and check only values that preflight cannot know.Changed behavior
preflightImpl()(-86200).Family entries fixed
This pull request fixes these entries of #1873; the issue stays open for its other parameter classes:
CreateAMScanPathsFilterhatch spacing (CreateAMScanPaths.cpp:190) and rotation angle (:421)PointSampleEdgeGeometryFiltersampling spacing (PointSampleEdgeGeometry.cpp:73)ResampleImageGeomFilterspacing and exact dimensions (ImageGeometryResample.cpp:260)ApproximatePointCloudHullFiltergrid resolution (ApproximatePointCloudHullFilter.cpp:86) and empty-neighbor maximum (ApproximatePointCloudHull.cpp:178)UncertainRegularGridSampleSurfaceMeshFilterspacing, origin and uncertainty (:138)InterpolatePointCloudToRegularGridFilterGaussian sigmas (:172) and kernel size (InterpolatePointCloudToRegularGrid.cpp:417)ComputeCoordinateThresholdFiltersphere (:165)ExtractFeatureBoundaries2DFiltercustom Z (ExtractFeatureBoundaries2D.cpp:265)RegularGridSampleSurfaceMeshFilterorigin (:200)PartitionGeometryFiltercell counts and Advanced origin (the count part of the:550entry;:206)ResampleRectGridToImageGeomFilterdimensions (:113)CreateGeometryFilter/CreateImageGeometryFilterdimensions (the dimension part of theCreateImageGeometryAction.cpp:58entry)FeatureFaceCurvatureFilterring count (:58)PointSampleTriangleGeometryFiltersample count (PointSampleTriangleGeometry.cpp:64)ReadImageStackFilterexact 2D dimensions (:302)Not part of this pull request: the Partition Geometry bounding-box and Compute Coordinate Threshold rectangle relations, and the Read Image Stack origin and spacing overrides (later pull requests for #1873); bounds on the other filters' parameters, and showing bounds in the DREAM3D-NX widgets (#1991); cancellation in the Create AM Scan Paths hatch loop (#1876); results that overflow
float32from finite values (#1874); and Partition Geometry reading coordinates during preflight (#1798).Test plan
developwhere that is safe (accepted invalid spacing, sigmas, origins, angles, counts and dimensions; zero exact dimensions that give infinite spacing) and pass with the fix. Cases wheredevelopreaches undefined behavior (nonfinite-to-integer conversions, non-terminating loops, oversized allocations) are tested against the fixed code only. Controls check inactive options, valid boundaries, zero sample points, provisional pipeline geometry that becomes valid during execution, and both Direct and Scanline partition paths.Number Parameter Value Bounds,Vector Parameter Value Bounds,Numeric Parameter Defaults).isPlane()with NaN; a 4096 × 2 × 1 rectilinear grid across the bound-reading block boundary on both partition paths.DREAM3DNX_ENABLE_OOC=ON,SIMPLNX_TEST_ALGORITHM_PATH=1) on macOS: 169/169.ctest -L "^ooc$" -LE benchmark: 1053/1053.Related Issue
Related to #1873
Related to #1991