Skip to content

COMP: Set the strict store-copy format with a configured header - #1982

Merged
imikejackson merged 3 commits into
BlueQuartzSoftware:developfrom
imikejackson:topic/store_copy_strict_format_header
Nov 19, 2026
Merged

imikejackson merged 3 commits into
BlueQuartzSoftware:developfrom
imikejackson:topic/store_copy_strict_format_header

Conversation

@imikejackson

@imikejackson imikejackson commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The strict store-copy format switch (SIMPLNX_STORE_COPY_STRICT_FORMAT) was a compile definition that the out-of-core build added to the simplnx target as PUBLIC, so it reached every file that uses simplnx. This pull request makes it a simplnx cache option that writes a small generated header, and fixes a CTest ordering bug found while testing it.

  • Add the cache option SIMPLNX_STORE_COPY_STRICT_FORMAT (default OFF). When ON, a store copy to a storage format with no registered I/O manager fails instead of falling back to in-memory storage. With the option OFF, behavior is unchanged.
  • Generate simplnx/Common/StoreCopyConfig.hpp with #cmakedefine01. Only four translation units depend on StoreCopyConfig.hpp: StoreCopyUtilities.cpp, DataArrayTest.cpp, MemorySafetyTest.cpp and StorageFormatPlanTest.cpp. It is a separate header, not part of SimplnxConfig.hpp, so that files that include SimplnxConfig.hpp do not rebuild when the option changes.
  • Move the test-only PlaceholderPlanManager into UnitTestCommon and register it for the HDF5-OOC format in the strict-mode test branches. The simplnx tests no longer include code from the private out-of-core storage plugin, and a simplnx-only build with the option ON now compiles and runs them. Registration keeps an I/O manager that is already registered for that format.
  • Run include(CTest) before add_subdirectory(src/apps). BUILD_TESTING comes from include(CTest), so in a new build directory the first configure registered none of the nxrunner tests guarded by if(BUILD_TESTING) (the 7 nxrunner_pipeline_flush_* tests, nxrunner OOC threshold preferences, nxrunner cache memory budget CLI). The redundant enable_testing() in src/apps is removed.

Merge order

DREAM3D-NX sets the new option from DREAM3DNX_ENABLE_OOC before it adds simplnx: BlueQuartzSoftware/DREAM3DNX#1256. Merge that pull request first. If this one merged first, the generated header would define the switch as 0 in out-of-core builds and override the definition they pass today, so those builds would silently fall back to memory. After both merge, the out-of-core plugin stops passing the definition and requires the option instead.

Test Plan

Release builds, macOS arm64, ci-macos-arm64 preset with the Python bindings off:

  • Option OFF, rebased on develop (8cad1e8): full ctest -j 1: 4029/4029 passed.
  • Option ON, simplnx only: full ctest -j 1: 3928/3928 passed, plus the 9 nxrunner tests 9/9.
  • Option OFF and ON after the review fixes: DataArray, MemorySafety, StorageFormatPlan and placeholder tests: 317/317 each.
  • Fresh build directory, one configure: the 9 nxrunner tests and 11 NXINFO tests are registered.
  • DREAM3D-NX out-of-core build with this branch and DREAM3DNX#1256: no macro-redefinition warning; nxrunner, NXINFO, DataArray, MemorySafety, StoreCopy and StorageFormatPlan tests 375/375; all out-of-core labeled tests 1051/1052 (one DREAM3D-NX histogram test failed once and passed 6 of 6 times alone; it looks intermittent and unrelated).
  • After the review fixes, simplnx only, option OFF and ON: StorageFormatPlan, C8, DataArray, MemorySafety and placeholder tests 93/93 each; the 3 C8 finalizer tests run in both builds.
  • DREAM3D-NX out-of-core CI configuration (ci-linux-x64, Ubuntu 24.04, gcc 13) with this branch rebased on develop (7b9983e) and DREAM3DNX#1256: header defines 1; full ctest -j 1 5563/5563 and focused OOC regressions 11/11.
  • clang-format 19.1.1 and git diff --check.
  • CI passes.

Related Issue

Related to #1777 (this changes how the switch is set, not the unknown-format policy that issue decides)

@imikejackson imikejackson self-assigned this Nov 14, 2026
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@imikejackson

Copy link
Copy Markdown
Contributor Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@jmarquisbq jmarquisbq left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all nine files in simplnx #1982 at 4e54317986da4d047ea2d73c5bc5431a93c8cb5a. Please restore the three finalizer tests in standalone strict builds and clarify how the option affects numeric DataArray creation.

1. Findings

  • 1.1 [Minor] Run the finalizer tests when no import finalizer is registered. The guard at line 1265 compiles out three cases whenever strict store-copy policy is enabled. Standalone simplnx can now enable that policy without registering an import finalizer. I enabled only those cases in a scratch copy and all three passed against the strict library. Gate the cases on finalizer availability so this configuration exercises import completion, warning order, and rollback. StorageFormatPlanTest.cpp
  • 1.2 [Minor] Describe the factory requirement and its effect on array creation. A registered manager can support numeric arrays and lack a string-store factory for the same format. The cache help and generated-header comment should say “Reject a requested storage format when no registered factory supports the requested store type.” Please also mention that this applies to numeric array creation and import planning as well as copies. CMake option · Header template
  • 1.3 [Nit] Narrow the rebuild claim in the PR description. Only four translation units depend on the new header. Toggling the option rebuilt 24 translation units in my Windows Ninja build because reconfiguration also regenerated simplnx_test_dirs.hpp. “Only four translation units depend on StoreCopyConfig.hpp” matches the dependency graph.

2. Verification and merge order

  • 2.1 Local evidence. Focused Windows Release tests passed 51/51 with strict mode on and 54/54 with it off. A fresh configure registered all nine nxrunner and eleven NXINFO tests. clang-format 19.1.1 and git diff --check passed. clang-tidy reported diagnostics on the four changed translation units, including diagnostics from Catch macros and the generated configuration macro.
  • 2.2 DataArray consequences and integration. With a compatible factory registered, both settings use it. If a filter or resolver selects HDF5-OOC and its numeric factory is missing, strict mode can reject the array during preflight. With strict mode off, preflight plans an in-memory array. Execution can then consume substantially more RAM or fail the memory check. Conversely, enabling strict mode in a standalone build can expose filters or imports that previously relied on fallback. These differences arise when a build changes policy and the selected format is unavailable. Array creation · Format selection

Please merge DREAM3D-NX #1256 first so OOC builds retain strict behavior. For paired validation, please identify a passing filter test that executes and verifies a disk-backed numeric output. Preflight creates a metadata placeholder and does not invoke the value-store factory. My local tests used standalone simplnx.

REQUEST CHANGES pending the test-coverage and configuration-description fixes.

imikejackson and others added 3 commits November 19, 2026 10:14
* Add the cache option SIMPLNX_STORE_COPY_STRICT_FORMAT (default OFF).
  When ON, a store copy to a storage format that has no registered I/O
  manager fails instead of falling back to in-memory storage.
* Generate simplnx/Common/StoreCopyConfig.hpp from the option. Only
  StoreCopyUtilities.cpp and the tests that check the behavior include
  it, so changing the option recompiles only those files. Before, an
  out-of-core build passed the macro as a PUBLIC compile definition to
  every file that uses simplnx.
* Print the option in the configuration summary.
* Move PlaceholderPlanManager into UnitTestCommon and register it for
  the HDF5-OOC format in the strict-mode tests, so the simplnx tests no
  longer include the out-of-core storage plugin and a simplnx-only
  build with the option ON compiles and runs them.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
* Run include(CTest) before add_subdirectory(src/apps). BUILD_TESTING
  is defined by include(CTest), so the nxrunner tests and the
  nxrunner_pipeline_flush tests were registered only after a second
  configure of a new build directory.
* Remove the enable_testing() call in src/apps, which the earlier
  include(CTest) makes redundant.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
… option

The three C8 finalizer tests now compile in strict builds. They return early
only when an import finalizer is already registered. The option help and the
generated-header comment now say that strict mode rejects a requested storage
format when no registered factory supports the requested store type, for
numeric array creation, import planning and store copies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@imikejackson
imikejackson force-pushed the topic/store_copy_strict_format_header branch from 4e54317 to a502e60 Compare November 19, 2026 15:50
@imikejackson

Copy link
Copy Markdown
Contributor Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@jmarquisbq jmarquisbq left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all nine files in simplnx #1982 at a502e600e405551442d74e6b35f5051d945a9c60. The previous review requests are addressed.

1. Finding

  • 1.1 [Minor, confirmed] Report the three C8 finalizer cases as skipped when a finalizer is already active. This branch replaces REQUIRE_FALSE(collection.anyManagerFinalizesImport()) with a warning and an early return in all three cases. When that guard fires, CTest records a pass even though the case never checks import completion, warning order, or rollback. A Catch2 2.13.6 probe reproduced the pass with zero assertions. First guard · Second guard · Third guard

    Use WARN("SIMPLNX_IMPORT_FINALIZER_SKIPPED: an import finalizer is already active."); in each guard and extend the existing CTest property to SKIP_REGULAR_EXPRESSION "SIMPLNX_POSIX_PATH_STATUS_SKIPPED|SIMPLNX_IMPORT_FINALIZER_SKIPPED". This preserves the existing POSIX skip and lets the finalizer cases run normally when no finalizer is active. I verified that matching the warning in the skip expression changes the probe’s CTest result to Skipped. CTest registration

2. Architecture and verification

  • 2.1 Architecture. Numeric array creation resolves the format before both preflight and execution. The test helper preserves an existing manager, and only four translation units include the generated policy header. Merge DREAM3D-NX #1256 first to retain strict behavior in out-of-core builds.
  • 2.2 Local checks. Fresh Windows Release configurations generated 0 by default and 1 with strict mode enabled. The first strict configure registered all nine nxrunner tests and all eleven NXINFO tests. clang-format 19.1.1, the comment audit and git diff --check passed. Changed-line clang-tidy reported diagnostics involving the generated macro, MSVC’s /MP flag, naming and test macros.
  • 2.3 Runtime evidence. The author reports 93/93 focused tests in each standalone mode and 5,563/5,563 in the paired Linux out-of-core run. My local verification covered configuration, test registration and the skip-reporting probe. The PR’s Windows, Linux and macOS CI checks passed. simplnx CI

APPROVE. Item 1.1 is nonblocking.

@imikejackson
imikejackson merged commit 6962879 into BlueQuartzSoftware:develop Nov 19, 2026
5 checks passed
@imikejackson
imikejackson deleted the topic/store_copy_strict_format_header branch November 19, 2026 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants