Repository navigation
COMP: Set the strict store-copy format with a configured header - #1982
imikejackson merged 3 commits into
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
Reviewed all nine files in simplnx #1982 at 4e54317986da4d047ea2d73c5bc5431a93c8cb5a. Please restore the three finalizer tests in standalone strict builds and clarify how the option affects numeric DataArray creation.
1. Findings
- 1.1 [Minor] Run the finalizer tests when no import finalizer is registered. The guard at line 1265 compiles out three cases whenever strict store-copy policy is enabled. Standalone simplnx can now enable that policy without registering an import finalizer. I enabled only those cases in a scratch copy and all three passed against the strict library. Gate the cases on finalizer availability so this configuration exercises import completion, warning order, and rollback. StorageFormatPlanTest.cpp
- 1.2 [Minor] Describe the factory requirement and its effect on array creation. A registered manager can support numeric arrays and lack a string-store factory for the same format. The cache help and generated-header comment should say “Reject a requested storage format when no registered factory supports the requested store type.” Please also mention that this applies to numeric array creation and import planning as well as copies. CMake option · Header template
- 1.3 [Nit] Narrow the rebuild claim in the PR description. Only four translation units depend on the new header. Toggling the option rebuilt 24 translation units in my Windows Ninja build because reconfiguration also regenerated
simplnx_test_dirs.hpp. “Only four translation units depend onStoreCopyConfig.hpp” matches the dependency graph.
2. Verification and merge order
- 2.1 Local evidence. Focused Windows Release tests passed 51/51 with strict mode on and 54/54 with it off. A fresh configure registered all nine nxrunner and eleven NXINFO tests.
clang-format19.1.1 andgit diff --checkpassed.clang-tidyreported diagnostics on the four changed translation units, including diagnostics from Catch macros and the generated configuration macro. - 2.2 DataArray consequences and integration. With a compatible factory registered, both settings use it. If a filter or resolver selects
HDF5-OOCand its numeric factory is missing, strict mode can reject the array during preflight. With strict mode off, preflight plans an in-memory array. Execution can then consume substantially more RAM or fail the memory check. Conversely, enabling strict mode in a standalone build can expose filters or imports that previously relied on fallback. These differences arise when a build changes policy and the selected format is unavailable. Array creation · Format selection
Please merge DREAM3D-NX #1256 first so OOC builds retain strict behavior. For paired validation, please identify a passing filter test that executes and verifies a disk-backed numeric output. Preflight creates a metadata placeholder and does not invoke the value-store factory. My local tests used standalone simplnx.
REQUEST CHANGES pending the test-coverage and configuration-description fixes.
* Add the cache option SIMPLNX_STORE_COPY_STRICT_FORMAT (default OFF). When ON, a store copy to a storage format that has no registered I/O manager fails instead of falling back to in-memory storage. * Generate simplnx/Common/StoreCopyConfig.hpp from the option. Only StoreCopyUtilities.cpp and the tests that check the behavior include it, so changing the option recompiles only those files. Before, an out-of-core build passed the macro as a PUBLIC compile definition to every file that uses simplnx. * Print the option in the configuration summary. * Move PlaceholderPlanManager into UnitTestCommon and register it for the HDF5-OOC format in the strict-mode tests, so the simplnx tests no longer include the out-of-core storage plugin and a simplnx-only build with the option ON compiles and runs them. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
* Run include(CTest) before add_subdirectory(src/apps). BUILD_TESTING is defined by include(CTest), so the nxrunner tests and the nxrunner_pipeline_flush tests were registered only after a second configure of a new build directory. * Remove the enable_testing() call in src/apps, which the earlier include(CTest) makes redundant. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
… option The three C8 finalizer tests now compile in strict builds. They return early only when an import finalizer is already registered. The option help and the generated-header comment now say that strict mode rejects a requested storage format when no registered factory supports the requested store type, for numeric array creation, import planning and store copies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4e54317 to
a502e60
Compare
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
jmarquisbq
left a comment
There was a problem hiding this comment.
Reviewed all nine files in simplnx #1982 at a502e600e405551442d74e6b35f5051d945a9c60. The previous review requests are addressed.
1. Finding
-
1.1 [Minor, confirmed] Report the three C8 finalizer cases as skipped when a finalizer is already active. This branch replaces
REQUIRE_FALSE(collection.anyManagerFinalizesImport())with a warning and an early return in all three cases. When that guard fires, CTest records a pass even though the case never checks import completion, warning order, or rollback. A Catch2 2.13.6 probe reproduced the pass with zero assertions. First guard · Second guard · Third guardUse
WARN("SIMPLNX_IMPORT_FINALIZER_SKIPPED: an import finalizer is already active.");in each guard and extend the existing CTest property toSKIP_REGULAR_EXPRESSION "SIMPLNX_POSIX_PATH_STATUS_SKIPPED|SIMPLNX_IMPORT_FINALIZER_SKIPPED". This preserves the existing POSIX skip and lets the finalizer cases run normally when no finalizer is active. I verified that matching the warning in the skip expression changes the probe’s CTest result toSkipped. CTest registration
2. Architecture and verification
- 2.1 Architecture. Numeric array creation resolves the format before both preflight and execution. The test helper preserves an existing manager, and only four translation units include the generated policy header. Merge DREAM3D-NX #1256 first to retain strict behavior in out-of-core builds.
- 2.2 Local checks. Fresh Windows Release configurations generated
0by default and1with strict mode enabled. The first strict configure registered all nine nxrunner tests and all eleven NXINFO tests. clang-format 19.1.1, the comment audit andgit diff --checkpassed. Changed-line clang-tidy reported diagnostics involving the generated macro, MSVC’s/MPflag, naming and test macros. - 2.3 Runtime evidence. The author reports 93/93 focused tests in each standalone mode and 5,563/5,563 in the paired Linux out-of-core run. My local verification covered configuration, test registration and the skip-reporting probe. The PR’s Windows, Linux and macOS CI checks passed. simplnx CI
APPROVE. Item 1.1 is nonblocking.
Summary
The strict store-copy format switch (
SIMPLNX_STORE_COPY_STRICT_FORMAT) was a compile definition that the out-of-core build added to thesimplnxtarget as PUBLIC, so it reached every file that uses simplnx. This pull request makes it a simplnx cache option that writes a small generated header, and fixes a CTest ordering bug found while testing it.SIMPLNX_STORE_COPY_STRICT_FORMAT(defaultOFF). WhenON, a store copy to a storage format with no registered I/O manager fails instead of falling back to in-memory storage. With the optionOFF, behavior is unchanged.simplnx/Common/StoreCopyConfig.hppwith#cmakedefine01. Only four translation units depend onStoreCopyConfig.hpp:StoreCopyUtilities.cpp,DataArrayTest.cpp,MemorySafetyTest.cppandStorageFormatPlanTest.cpp. It is a separate header, not part ofSimplnxConfig.hpp, so that files that includeSimplnxConfig.hppdo not rebuild when the option changes.PlaceholderPlanManagerintoUnitTestCommonand register it for theHDF5-OOCformat in the strict-mode test branches. The simplnx tests no longer include code from the private out-of-core storage plugin, and a simplnx-only build with the optionONnow compiles and runs them. Registration keeps an I/O manager that is already registered for that format.include(CTest)beforeadd_subdirectory(src/apps).BUILD_TESTINGcomes frominclude(CTest), so in a new build directory the first configure registered none of the nxrunner tests guarded byif(BUILD_TESTING)(the 7nxrunner_pipeline_flush_*tests,nxrunner OOC threshold preferences,nxrunner cache memory budget CLI). The redundantenable_testing()insrc/appsis removed.Merge order
DREAM3D-NX sets the new option from
DREAM3DNX_ENABLE_OOCbefore it adds simplnx: BlueQuartzSoftware/DREAM3DNX#1256. Merge that pull request first. If this one merged first, the generated header would define the switch as 0 in out-of-core builds and override the definition they pass today, so those builds would silently fall back to memory. After both merge, the out-of-core plugin stops passing the definition and requires the option instead.Test Plan
Release builds, macOS arm64,
ci-macos-arm64preset with the Python bindings off:OFF, rebased ondevelop(8cad1e8): fullctest -j 1: 4029/4029 passed.ON, simplnx only: fullctest -j 1: 3928/3928 passed, plus the 9 nxrunner tests 9/9.OFFandONafter the review fixes: DataArray, MemorySafety, StorageFormatPlan and placeholder tests: 317/317 each.OFFandON: StorageFormatPlan, C8, DataArray, MemorySafety and placeholder tests 93/93 each; the 3 C8 finalizer tests run in both builds.ci-linux-x64, Ubuntu 24.04, gcc 13) with this branch rebased ondevelop(7b9983e) and DREAM3DNX#1256: header defines 1; fullctest -j 15563/5563 and focused OOC regressions 11/11.git diff --check.Related Issue
Related to #1777 (this changes how the switch is set, not the unknown-format policy that issue decides)