Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -370,6 +370,8 @@ set(SIMPLNX_HDRS
${SIMPLNX_SOURCE_DIR}/Common/Any.hpp
${SIMPLNX_SOURCE_DIR}/Common/Array.hpp
${SIMPLNX_SOURCE_DIR}/Common/AtomicFile.hpp
${SIMPLNX_SOURCE_DIR}/Common/FileReplacement.hpp
${SIMPLNX_SOURCE_DIR}/Common/FileReplacementDetails.hpp
${SIMPLNX_SOURCE_DIR}/Common/Bit.hpp
${SIMPLNX_SOURCE_DIR}/Common/BoundingBox.hpp
${SIMPLNX_SOURCE_DIR}/Common/SimplnxConstants.hpp
Expand Down Expand Up @@ -673,6 +675,7 @@ set(SIMPLNX_GENERATED_HEADERS

set(SIMPLNX_SRCS
${SIMPLNX_SOURCE_DIR}/Common/AtomicFile.cpp
${SIMPLNX_SOURCE_DIR}/Common/FileReplacement.cpp
${SIMPLNX_SOURCE_DIR}/Common/RgbColor.cpp
${SIMPLNX_SOURCE_DIR}/Common/Range.cpp
${SIMPLNX_SOURCE_DIR}/Common/Range2D.cpp
Expand Down
37 changes: 37 additions & 0 deletions docs/Coding_Knowledge_Base.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ repository, but contains no code from one.
- [Build DataPaths from DataPaths](#build-datapaths-from-datapaths)
- [Never drop a Result](#never-drop-a-result)
- [Check the file close and the stream state before you report a successful write](#check-the-file-close-and-the-stream-state-before-you-report-a-successful-write)
- [Replace files through InstallReplacementFile](#replace-files-through-installreplacementfile)
- [Check for cancel in every long loop](#check-for-cancel-in-every-long-loop)
- [Write floating-point data as text that reads back to the same value](#write-floating-point-data-as-text-that-reads-back-to-the-same-value)
- [Write floating-point output through a saturating cast](#write-floating-point-output-through-a-saturating-cast)
Expand Down Expand Up @@ -663,6 +664,42 @@ written and closed without a state check; an atomic-file commit that follows eit
`DataStructureWriter` does with `closeChecked()` in PR #1859:
`return MergeResults(std::move(writeResult), fileWriter.closeChecked());`.

### Replace files through InstallReplacementFile

**Rule:** use `InstallReplacementFile()` to replace an existing file with a complete new file. Do not use a bare
replacing `std::filesystem::rename()` or `MoveFileExW()`. Close the replacement file's writer before the rename.

**Example of the broken rule:** `AtomicFile::commit()` used a direct `MoveFileExW()` call on Windows, and
`WriteDREAM3D` used a direct rename for the XDMF file. The out-of-core storage plugin also used a conventional
replace (`BlueQuartzSoftware/SimplnxOoc#39`). A kernel trace showed the
Windows Search indexer (`SearchProtocolHost.exe`) opening the destination six microseconds before a failed replace.
A workload that replaced the same file thousands of times failed in 11 of 17 runs on the CI machine where the
Windows Search service runs, and in 0 of 8 runs on the one where it is disabled.

The following results were measured on NTFS with a `GENERIC_READ` handle open on the destination. The trace recorded the
indexer's share mode, and its access mask is unknown.

| Destination handle share mode | `MoveFileExW` replace | POSIX-semantics replace on NTFS |
| --- | --- | --- |
| Read, write and delete (Windows Search indexer) | `ERROR_ACCESS_DENIED` (5) | Succeeds |
| Read and write without delete | `ERROR_ACCESS_DENIED` (5) | `ERROR_SHARING_VIOLATION` (32) |

**Why:** any open destination handle blocks a conventional Windows replace, even when the handle shares read,
write and delete access. The indexer can hold that handle for only a few microseconds. On NTFS, a POSIX-semantics rename
can replace a destination while a handle with delete sharing is open.

**What to look for:**

- A file writer that publishes a replacement with a direct `std::filesystem::rename()` call.
- A direct `MoveFileExW(MOVEFILE_REPLACE_EXISTING)` call.

**Pattern to use:** call `InstallReplacementFile(replacement, destination)` and report any failure with
`DescribeFileReplacementFailure()`. To request write-through on Windows, use `FileReplacementDurability::WriteThrough`
as `AtomicFile::commit()` does. The helper uses one rename on POSIX. On Windows it tries a POSIX-semantics rename first.
If that step fails, it tries a conventional rename. Other Windows file systems use the conventional rename with the same
bounded retry policy. If both steps fail, the helper returns the POSIX-semantics rename error if it permits a retry.
For other primary errors, it returns the conventional rename error.

### Check for cancel in every long loop

**Rule:** every loop that can run for a long time checks `shouldCancel` (or `getCancel()`) and has a progress
Expand Down
7 changes: 7 additions & 0 deletions src/Plugins/SimplnxCore/docs/WriteDREAM3DFilter.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,13 @@ The `.xdmf` file does not contain a copy of the data — it is a metadata wrappe

Enabling XDMF costs almost nothing (the file is small and fast to write) and is the recommended setting whenever visualization outside DREAM3D-NX is a possibility. It can safely be left off for intermediate pipeline outputs that will only be re-read by DREAM3D-NX itself.

If the filter reports error `-78500`, it saved the `.dream3d` file and could not save its `.xdmf` file.
An existing `.xdmf` file at that path may not match the new `.dream3d` file.

- If the message says the file may be open in another program, close programs that use the file and run the filter again.
This message can also appear if you do not have permission to replace the file.
- If the message says the file or output folder may be read-only, fix any read-only settings or permissions and run the filter again.

### Required Input Sources

None — this filter writes whatever currently exists in the **DataStructure** to the output file. If the file already exists it is overwritten. The one exception is geometry connectivity: the vertex-incidence and element-neighbor lists that some filters build for an Edge, Triangle, Quad, Tetrahedral or Hexahedral geometry. These lists are not saved. Filters that need them rebuild them after the file is read. The geometry's vertex, edge, face and cell lists are always saved.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,21 +1,31 @@
#include "WriteDREAM3D.hpp"

#include "simplnx/Common/AtomicFile.hpp"
#include "simplnx/Common/FileReplacement.hpp"
#include "simplnx/Common/Result.hpp"
#include "simplnx/Common/Types.hpp"
#include "simplnx/DataStructure/DataGroup.hpp"
#include "simplnx/DataStructure/IO/HDF5/DataStructureWriter.hpp"
#include "simplnx/Pipeline/Pipeline.hpp"
#include "simplnx/Pipeline/PipelineFilter.hpp"
#include "simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp"
#include "simplnx/Utilities/Parsing/HDF5/IO/FileIO.hpp"

#include <fmt/format.h>
#include <fmt/std.h> // IWYU pragma: keep (filesystem path formatter)

#include <filesystem>
namespace fs = std::filesystem;
#include <string_view>
#include <utility>

using namespace nx::core;

namespace fs = std::filesystem;

namespace
{
constexpr int32 k_FailedFindPipelineError = -15;
constexpr int32 k_XdmfReplacementError = -78500;
} // namespace

// -----------------------------------------------------------------------------
Expand All @@ -40,8 +50,8 @@ Result<> WriteDREAM3D::operator()()
}
AtomicFile atomicFile = std::move(atomicFileResult.value());

auto exportFilePath = atomicFile.tempFilePath();
auto writeXdmf = m_InputValues->WriteXdmfFile;
const auto exportFilePath = atomicFile.tempFilePath();
const bool writeXdmf = m_InputValues->WriteXdmfFile;

Pipeline pipeline;

Expand All @@ -59,27 +69,37 @@ Result<> WriteDREAM3D::operator()()
nx::core::HDF5::DataStructureWriter::WriteOptions writeOptions;
writeOptions.compressionLevel = m_InputValues->UseCompression ? m_InputValues->CompressionLevel : 0;
auto results = DREAM3D::WriteFile(exportFilePath, m_DataStructure, pipeline, writeXdmf, writeOptions);
if(results.valid())
if(results.invalid())
{
if(m_ShouldCancel)
{
return MakeErrorResult(-1, "Filter cancelled");
}
Result<> commitResult = atomicFile.commit();
if(commitResult.invalid())
{
return commitResult;
}
if(writeXdmf)
return results;
}
if(m_ShouldCancel)
{
return MakeErrorResult(-1, "Filter cancelled");
}
auto commitResult = atomicFile.commit();
if(commitResult.invalid())
{
return MergeResults(std::move(results), std::move(commitResult));
}

if(writeXdmf)
{
fs::path temporarySidecar = exportFilePath;
temporarySidecar.replace_extension(".xdmf");
fs::path destinationSidecar = m_InputValues->ExportFilePath;
destinationSidecar.replace_extension(".xdmf");
const auto outcome = InstallReplacementFile(temporarySidecar, destinationSidecar);
if(outcome.Error)
{
fs::path xdmfFilePath = exportFilePath.replace_extension(".xdmf");
std::error_code errorCode;
fs::rename(xdmfFilePath, m_InputValues->ExportFilePath.parent_path() / m_InputValues->ExportFilePath.stem().concat(".xdmf"), errorCode);
if(errorCode)
{
std::string ss = fmt::format("Failed to rename xdmf file with error: '{}'", errorCode.message());
return MakeErrorResult(errorCode.value(), ss);
}
const std::string_view recoveryAdvice = outcome.RetriesExhausted ? "The .xdmf file may be open in another program, or you may not have permission to replace it. "
"Close programs that use the .xdmf file and run the filter again." :
"The .xdmf file or the output folder may be read-only, or you may not have permission to replace the file. "
"Fix any read-only settings or permissions and run the filter again.";
const auto message = fmt::format("The .dream3d file '{}' was saved. Its .xdmf file '{}' could not be saved. "
"An existing .xdmf file at that path may not match the new .dream3d file. {} {}",
m_InputValues->ExportFilePath, destinationSidecar, recoveryAdvice, DescribeFileReplacementSystemError(outcome));
return MergeResults(std::move(results), MakeErrorResult(k_XdmfReplacementError, message));
}
}
return results;
Expand Down
35 changes: 35 additions & 0 deletions src/Plugins/SimplnxCore/test/DREAM3DFileTest.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,10 @@
#include "SimplnxCore/Filters/WriteDREAM3DFilter.hpp"

#include "SimplnxCore/SimplnxCore_test_dirs.hpp"
#include "simplnx/Common/Result.hpp"
#include "simplnx/Common/ScopeGuard.hpp"
#include "simplnx/Common/Types.hpp"
#include "simplnx/Common/Uuid.hpp"

#include "simplnx/Core/Application.hpp"
#include "simplnx/DataStructure/AttributeMatrix.hpp"
Expand Down Expand Up @@ -50,6 +53,8 @@
#include "simplnx/Utilities/Parsing/HDF5/IO/FileIO.hpp"

#include <catch2/catch.hpp>
#include <fmt/format.h>
#include <fmt/std.h> // IWYU pragma: keep (filesystem path formatter)

#include <algorithm>
#include <chrono>
Expand All @@ -59,6 +64,7 @@
#include <mutex>
#include <stdexcept>
#include <string>
#include <system_error>
#include <utility>
#include <vector>

Expand Down Expand Up @@ -1218,6 +1224,35 @@ void CheckXdmfFile(const fs::path& dream3dPath)
REQUIRE(referenceCount > 0);
}

TEST_CASE("SimplnxCore::WriteDREAM3DFilter: Reports a blocked XDMF replacement", "[SimplnxCore][WriteDREAM3DFilter]")
{
const auto directory = fs::path(unit_test::k_BinaryTestOutputDir.view()) / ("write_dream3d_" + Uuid::GenerateV4().str());
const auto destination = directory / "output.dream3d";
const auto blockedSidecar = GetXdmfPath(destination);
const auto cleanup = MakeScopeGuard([&directory]() noexcept {
std::error_code error;
fs::remove_all(directory, error);
});
REQUIRE(fs::create_directories(blockedSidecar));

DataStructure dataStructure = CreateTestDataStructure();
Arguments args;
args.insertOrAssign(WriteDREAM3DFilter::k_ExportFilePath, destination);
args.insertOrAssign(WriteDREAM3DFilter::k_WriteXdmf, true);
const WriteDREAM3DFilter filter;
const auto result = filter.execute(dataStructure, args).result;
SIMPLNX_RESULT_REQUIRE_INVALID(result); // NOLINT(cppcoreguidelines-avoid-do-while) Project Result assertion macro.
REQUIRE((result.errors().size() == 1));
CHECK((result.errors().front().code == -78500));
CHECK_THAT(result.errors().front().message, Catch::Matchers::StartsWith(fmt::format("The .dream3d file '{}' was saved. Its .xdmf file '{}' could not be saved. "
"An existing .xdmf file at that path may not match the new .dream3d file. "
"The .xdmf file or the output folder may be read-only, or you may not have permission to replace the file. "
"Fix any read-only settings or permissions and run the filter again. Operating system error: ",
destination, blockedSidecar)));
CHECK(fs::is_regular_file(destination));
CHECK(fs::is_directory(blockedSidecar));
}

TEST_CASE("WriteDREAM3D:Pipeline / WriteXdmf combinations", "[ReadDREAM3DFilter][WriteDREAM3DFilter]")
{
UnitTest::LoadPlugins();
Expand Down
20 changes: 4 additions & 16 deletions src/simplnx/Common/AtomicFile.cpp
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
#include "AtomicFile.hpp"

#include "simplnx/Common/FileReplacement.hpp"
#include "simplnx/Common/Result.hpp"
#include "simplnx/Common/Types.hpp"
#include "simplnx/Utilities/FileUtilities.hpp"
Expand All @@ -17,10 +18,6 @@

#if defined(_WIN32)
#include <fstream>
#ifndef NOMINMAX
#define NOMINMAX
#endif
#include <windows.h>
#endif

#if defined(__linux__) || defined(__APPLE__)
Expand Down Expand Up @@ -233,20 +230,11 @@ Result<> AtomicFile::commit()
return MakeErrorResult(-15781, fmt::format("Temporary output file '{}' does not exist. Write the output before calling commit().", m_TempFilePath.string()));
}

#if defined(_WIN32)
if(MoveFileExW(m_TempFilePath.c_str(), m_FilePath.c_str(), MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH) == 0)
{
const std::error_code renameError(static_cast<int>(GetLastError()), std::system_category());
return MakeErrorResult(-15782, fmt::format("Could not replace output file '{}' with temporary file '{}': {}.", m_FilePath.string(), m_TempFilePath.string(), renameError.message()));
}
#else
std::error_code renameError;
fs::rename(m_TempFilePath, m_FilePath, renameError);
if(renameError)
const auto outcome = InstallReplacementFile(m_TempFilePath, m_FilePath, FileReplacementDurability::WriteThrough);
if(outcome.Error)
{
return MakeErrorResult(-15782, fmt::format("Could not replace output file '{}' with temporary file '{}': {}.", m_FilePath.string(), m_TempFilePath.string(), renameError.message()));
return MakeErrorResult(-15782, DescribeFileReplacementFailure(outcome, m_FilePath));
}
#endif

return {};
}
Expand Down
5 changes: 4 additions & 1 deletion src/simplnx/Common/AtomicFile.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,10 @@ class SIMPLNX_EXPORT AtomicFile

/**
* @brief Atomically replaces the destination with the temporary file.
* @return Valid result on success or a detailed filesystem error.
*
* On Windows, this operation retries transient rename errors and requests write-through.
* @return Valid result on success or a detailed filesystem error. Error -15782 reports a failed replacement.
* @post Failure leaves the destination unchanged. Any temporary file remains available until removal or destruction.
*/
[[nodiscard]] Result<> commit();

Expand Down
Loading