Repository navigation
Skip non-Zarr objects silently when listing group members - #4471
Open
mayuriphad wants to merge 4 commits into
Open
mayuriphad wants to merge 4 commits into
mayuriphad wants to merge 4 commits into
Conversation
LocalStore.list, list_prefix and list_dir returned the '<stem>.<uuid>.partial' files created by _atomic_write while a write was in progress. Readers then treated them as store keys and emitted ZarrUserWarning for each one. Filter files matching the temp-file pattern in _list_files and _list_dir and add a regression test. Fixes zarr-developers#4161.
Contributor
|
i think we want the store to show these keys. we should remove the warning from member listing instead. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4471 +/- ##
==========================================
- Coverage 94.69% 94.68% -0.01%
==========================================
Files 94 94
Lines 13597 13595 -2
==========================================
- Hits 12875 12873 -2
Misses 722 722
🚀 New features to boost your workflow:
|
…ering LocalStore Per review: keep LocalStore listings faithful to what is on disk (including in-progress .partial temp files) and drop the ZarrUserWarning emitted when group member iteration encounters an object that is not a Zarr node.
Author
|
Thanks @d-v-b, makes sense. I've reworked this: \LocalStore\ listings are back to reporting everything on disk (including in-progress .partial\ files), and group member iteration now skips objects that aren't Zarr nodes silently instead of emitting a \ZarrUserWarning. Updated \ est_group_members\ accordingly and the changelog entry. |
Contributor
|
Can you update the pr title and description |
Author
|
Thanks @d-v-b. I updated the title and the description to match the new approach. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4161.
Problem
While
LocalStorewrites a key, it first writes to a temporary file named<name>.<uuid>.partialand then renames it into place. A reader that lists the store while another process is writing sees these files, andGroup.members()emitted aZarrUserWarning("Object at ... is not recognized as a component of a Zarr hierarchy") for each one. Any other object in the store that is not a Zarr node triggered the same warning.Change
As discussed in review, the store listings are unchanged and keep showing everything on disk. Instead, listing the members of a group now skips objects that are not Zarr nodes without a warning.
_iter_membersno longer warns in theKeyErrorbranch. It skips the object.test_group_membersnow checks that listing members does not warn about extra objects. The expected warning about consolidated metadata with Zarr format 3 is kept.test_use_consolidated_for_children_membersno longer expects the warning.changes/4161.bugfix.md.Testing
pytest tests/test_group.py tests/test_metadata/test_consolidated.py: 1010 passed, 209 skipped. The CI checks on this PR pass.