Skip to content

Skip non-Zarr objects silently when listing group members - #4471

Open
mayuriphad wants to merge 4 commits into
zarr-developers:mainfrom
mayuriphad:fix/localstore-partial-listing
Open

mayuriphad wants to merge 4 commits into
zarr-developers:mainfrom
mayuriphad:fix/localstore-partial-listing

Conversation

@mayuriphad

@mayuriphad mayuriphad commented Oct 3, 2026 •

Copy link
Copy Markdown

Fixes #4161.

Problem

While LocalStore writes a key, it first writes to a temporary file named <name>.<uuid>.partial and then renames it into place. A reader that lists the store while another process is writing sees these files, and Group.members() emitted a ZarrUserWarning ("Object at ... is not recognized as a component of a Zarr hierarchy") for each one. Any other object in the store that is not a Zarr node triggered the same warning.

Change

As discussed in review, the store listings are unchanged and keep showing everything on disk. Instead, listing the members of a group now skips objects that are not Zarr nodes without a warning.

  • _iter_members no longer warns in the KeyError branch. It skips the object.
  • test_group_members now checks that listing members does not warn about extra objects. The expected warning about consolidated metadata with Zarr format 3 is kept.
  • test_use_consolidated_for_children_members no longer expects the warning.
  • Changelog entry: changes/4161.bugfix.md.

Testing

pytest tests/test_group.py tests/test_metadata/test_consolidated.py: 1010 passed, 209 skipped. The CI checks on this PR pass.

LocalStore.list, list_prefix and list_dir returned the '<stem>.<uuid>.partial'
files created by _atomic_write while a write was in progress. Readers then
treated them as store keys and emitted ZarrUserWarning for each one.

Filter files matching the temp-file pattern in _list_files and _list_dir and
add a regression test. Fixes zarr-developers#4161.
Copilot AI balanced review requested due to automatic review settings October 3, 2026 11:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@d-v-b

d-v-b commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

i think we want the store to show these keys. we should remove the warning from member listing instead.

@codecov

codecov Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.68%. Comparing base (df18415) to head (7039c56).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4471      +/-   ##
==========================================
- Coverage   94.69%   94.68%   -0.01%     
==========================================
  Files          94       94              
  Lines       13597    13595       -2     
==========================================
- Hits        12875    12873       -2     
  Misses        722      722              
Files with missing lines Coverage Δ
src/zarr/core/group.py 95.67% <100.00%> (-0.01%) ⬇️

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…ering LocalStore

Per review: keep LocalStore listings faithful to what is on disk (including
in-progress .partial temp files) and drop the ZarrUserWarning emitted when
group member iteration encounters an object that is not a Zarr node.
@mayuriphad

Copy link
Copy Markdown
Author

Thanks @d-v-b, makes sense. I've reworked this: \LocalStore\ listings are back to reporting everything on disk (including in-progress .partial\ files), and group member iteration now skips objects that aren't Zarr nodes silently instead of emitting a \ZarrUserWarning. Updated \ est_group_members\ accordingly and the changelog entry.

@d-v-b

d-v-b commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Can you update the pr title and description

@mayuriphad mayuriphad changed the title Exclude in-progress atomic-write temp files from LocalStore listings Don't warn about non-Zarr objects when listing group members Oct 4, 2026
@mayuriphad

Copy link
Copy Markdown
Author

Thanks @d-v-b. I updated the title and the description to match the new approach.

@mayuriphad mayuriphad changed the title Don't warn about non-Zarr objects when listing group members Skip non-Zarr objects silently when listing group members Oct 5, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LocalStore listing lists .partial temp files

4 participants