Repository navigation
BUG: Replace files on Windows while another program has them open - #1990
Open
imikejackson wants to merge 4 commits into
Open
imikejackson wants to merge 4 commits into
imikejackson wants to merge 4 commits into
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
imikejackson
marked this pull request as ready for review
November 18, 2026 21:31
imikejackson
marked this pull request as draft
November 18, 2026 22:20
On Windows, a replacing rename fails with "Access is denied" while a handle to the destination file is open in a process, also when that handle shares read, write and delete access. The Windows Search indexer opens new files in that way for some microseconds. An operation that replaces a file many times then fails on a machine where the Windows Search service runs. * Add InstallReplacementFile(). On Windows it uses a rename with POSIX semantics, which an open handle with delete sharing does not block. It uses MoveFileExW() when the file system does not support that rename, and it retries a sharing error for a maximum of 2 seconds. On Linux and macOS it renames one time. * Use it in AtomicFile::commit() with the write-through option. Error code -15782 does not change. * Add tests for the retry limits, for real files, and for Windows handles that hold the destination. * Add the rule to the coding knowledge base. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
* Use MoveFileExW() also when the open of the replacement file reports an operation that the file system does not support. * Give the public members of the helper types CamelCase names, as the clang-tidy configuration requires. * Return the new error code -15785 from AtomicFile::commit() when the rename succeeded and only the flush to disk failed. WriteDREAM3D then installs the XDMF file before it returns the error. * Install the XDMF file of WriteDREAM3D with InstallReplacementFile() and report a failure with the new error code -78500. * Stop after one attempt when the destination is read-only or a directory. The other access-denied errors keep the limited retry. * Open the Windows test handles with read access, because Windows does not apply the share mode to a handle with attribute access only. * Make each failure description say whether the destination changed. * Add tests for a relative destination, an absent destination, a read-only destination and a held replacement file. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
A design review of the file replacement helper required these changes. * Try MoveFileExW() in the same attempt after each failure of the rename with POSIX semantics. The list of "not supported" error codes is removed, because it was measured on three file systems only. When the two renames fail, report the first error if it permits a retry, and the error of MoveFileExW() if it does not. * Flush the replacement file before the rename when the write-through option is set. A failed flush then leaves the destination unchanged. Remove error code -15785, AtomicFile::commitResult() and the special case in WriteDREAM3D, which were necessary only for a flush after the rename. * Move the retry steps, the retry limits and the fallback function to FileReplacementDetails.hpp in the namespace nx::core::detail. The public header keeps only the functions that callers use. * Say "on NTFS" where the documentation describes the rename with POSIX semantics. * Add tests for the fallback order and for a failed commit that keeps the destination and the temporary file. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
An independent review found no Critical or Important problem. These are its Minor and Nit findings. * Give the read-only and permission advice when the replacement stops after one attempt, and the "open in another program" advice when the retries are used up. Before, a read-only destination got the least specific text. * Make the -78500 message of WriteDREAM3D say that the .dream3d file was saved, that an existing .xdmf file can be out of date, and what to do. Use the same words in the filter documentation. * Keep the warnings of the writer when WriteDREAM3D returns the commit error or the -78500 error. * Change the retry detail to "(tried N times over M ms)". * Name the WriteDREAM3D test after its plugin and write its files in the unit test output folder. * Correct the clang-tidy findings in the new code. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
imikejackson
force-pushed
the
fix/windows-file-replace
branch
from
November 19, 2026 17:00
1ac8454 to
309f858
Compare
imikejackson
marked this pull request as ready for review
November 19, 2026 18:20
Contributor
Author
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On Windows, a replacing rename fails with "Access is denied" while a handle to the destination file is open in a process, also when that handle shares read, write and delete access. The Windows Search indexer opens new files in that way for some microseconds.
InstallReplacementFile()is a new helper that replaces a file with a rename that such a handle does not block.AtomicFile::commit()and the.xdmffile ofWriteDREAM3Dnow use it. On Linux and macOS the helper renames one time, as before. 13 files changed, +948 / −39 lines.Companion pull request: BlueQuartzSoftware/SimplnxOoc#48 uses this helper in the out-of-core storage plugin. That pull request cannot merge before this one.
1. Cause
The defect was found in the out-of-core storage plugin (BlueQuartzSoftware/SimplnxOoc#39), where one filter replaces a file several thousand times. The measurements were made on Windows 11, on NTFS.
SearchProtocolHost.exe(Windows Search) opening the destination file 6 microseconds before a failed replace.MoveFileExWreplace and one replace with POSIX semantics in the same folders in each iteration, for 30 minutes and 1.26 million iterations. TheMoveFileExWreplace failed 12 times. The replace with POSIX semantics failed 0 times.GENERIC_READhandle held open on the destination, the two types of replace gave these results:MoveFileExWreplaceERROR_ACCESS_DENIED(5)ERROR_ACCESS_DENIED(5)ERROR_SHARING_VIOLATION(32)AtomicFile::commit()used oneMoveFileExW()call, so each filter that writes over an existing output file can fail in the same way, with a lower probability.2. File replacement helper
src/simplnx/Common/FileReplacement.hpp,src/simplnx/Common/FileReplacement.cpp—InstallReplacementFile(),DescribeFileReplacementFailure()andDescribeFileReplacementSystemError().src/simplnx/Common/FileReplacementDetails.hpp— the retry steps, the retry limits andRenameWithFallback()in the namespacenx::core::detail. The tests use them to script each step.On Windows, one attempt does these steps:
SetFileInformationByHandle()andFileRenameInfoExwith POSIX semantics.MoveFileExW(MOVEFILE_REPLACE_EXISTING). On exFAT, FAT32 and an SMB share, step 1 fails withERROR_INVALID_PARAMETERand step 2 does the rename (measured). No list of "not supported" error codes is used, because other file systems were not measured.The helper retries
ERROR_ACCESS_DENIED,ERROR_SHARING_VIOLATIONandERROR_LOCK_VIOLATIONwith waits that double from 1 ms to 64 ms, and it stops when the waits reach 2,000 ms. It does not retryERROR_ACCESS_DENIEDwhen the destination is read-only or a directory. Each other error returns after one attempt.FileReplacementDurability::WriteThroughflushes the data of the replacement file to the device (FlushFileBuffers()) before the rename. If the flush fails, the destination is unchanged. The rename itself is not flushed. The default does not flush, because a caller can replace a file thousands of times. On Linux and macOS this option has no effect (#1992).DescribeFileReplacementFailure()gives the destination, says that it is unchanged, gives one action for the user, and gives the operating-system message last.3. AtomicFile
src/simplnx/Common/AtomicFile.cpp,src/simplnx/Common/AtomicFile.hpp—commit()calls the helper withWriteThrough.Error code -15782 does not change. The message text changes: it now comes from
DescribeFileReplacementFailure().Behavior change: before,
commit()usedMOVEFILE_WRITE_THROUGH. Now it flushes the complete temporary file before the rename. This is a stronger guarantee, and it can make the save of a large file slower. The time was not measured.4. WriteDREAM3D
src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteDREAM3D.cpp— the.xdmffile is installed with the helper. A failure returns the new error code -78500. The message says that the.dream3dfile was saved and that an existing.xdmffile can be out of date. The two error returns keep the warnings of the writer.src/Plugins/SimplnxCore/docs/WriteDREAM3DFilter.md— describes error -78500.5. Tests and knowledge base
test/FileReplacementTest.cpp— the retry limits with scripted steps, the order and the reported error of the two rename steps, the messages, real files, and Windows cases in which a second handle holds the destination or the replacement.test/AtomicFileTest.cpp— a failed commit keeps the destination and the temporary file (all platforms), and a Windows case in whichcommit()replaces a file while a handle with full sharing is open on it.src/Plugins/SimplnxCore/test/DREAM3DFileTest.cpp— error -78500 when the.xdmfpath cannot be replaced.docs/Coding_Knowledge_Base.md— a new entry: replace files throughInstallReplacementFile().Test plan
ctest -R "InstallReplacementFile|FileReplacement|AtomicFile|StoreFileReplacement|NumericPrepared|WriteDREAM3D|DREAM3DFile|MergeTwins|Resize|HDF5Chunked"passes, 199 of 199 tests.AtomicFile::CommitReplacesFileWithFullSharingHandleand the Windows cases of the companion pull request fail. With the flag, they pass.ctest -R "InstallReplacementFile|FileReplacement|AtomicFile|WriteDREAM3D|DREAM3DFileTest|MergeTwins"passes, 34 of 34 tests.ctest -L "^ooc$" -LE "benchmark"passes, 1061 of 1061 tests.MoveFileExW()step has a scripted test only. The helper was not run on exFAT, FAT32, an SMB share or ReFS. The behavior of the two Win32 calls on exFAT, FAT32 and SMB was measured with a separate script.FlushFileBuffers()fail, and that the data is on the disk after a power failure was not tested.WriteDREAM3Dkeeps on its error returns. No existing fixture makes the writer succeed with a warning.develop, with the companion branch and DREAM3DNXdevelop:VizTestrendering tests andPerf: Offscreen render (empty scene). The run was started through ssh without a desktop session, and the log shows that the render window got no OpenGL pixel format. They were not run ondevelopfor comparison.Related Issue
Related to BlueQuartzSoftware/SimplnxOoc#39