Repository navigation
Conversation
Implements URL pipeline support (https://github.com/jbms/url-pipeline): '|'-chained URLs resolve through pluggable adapters registered under the 'zarr.url_adapters' entry-point group (entry-point name = URL scheme). - zarr.abc.url_pipeline: PipelineSegment, AdapterResolution, PipelineContext, URLPipelineAdapter (single-classmethod contract) - zarr.storage._url_pipeline: parse_pipeline / resolve_pipeline; the root sub-URL delegates to make_store so existing file/memory/fsspec routing is unchanged - registry: register_url_adapter / get_url_adapter / list_url_adapter_schemes (name check only; no adapter imports) - make_store/make_store_path route strings containing '|' (or a registered root scheme) through the resolver; residual store paths combine with the user-supplied path - StorePath gains a zarr_format attribute (populated by format segments in a follow-up) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e-core # Conflicts: # src/zarr/errors.py # src/zarr/registry.py
…thoring - make_store validates the access mode before routing a URL pipeline string to adapters, matching every other StoreLike branch - make_store docstring lists URL pipeline strings; StorePath.open documents the invalid-mode ValueError - user guide: StoreLike bullet for pipeline strings and an adapter authoring section in extending.md (runnable example) - changelog: drop PR-relative wording, name the public parser entry points Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
get_url_adapter took a non-reentrant lock across entry_point.load(), so an adapter module that resolved another scheme at import time deadlocked the process. Matching entry points are now taken off the pending list under the lock and imported outside it; import failures are wrapped in URLPipelineError and leave the entry point discoverable for a retry. A same-named entry point is discarded with a ZarrUserWarning when the scheme is already registered (e.g. by a builtin), instead of staying pending forever; duplicate entry-point names warn and the first wins. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ent contracts - The resolver closes the store an adapter returned when it cannot be made read-only for mode 'r', instead of leaking it. - resolve_pipeline documents every error type it lets through; the resolve_preceding docstring states the local-file-root limitation. - AdapterResolution and PipelineContext compare and hash by identity, since a Store is not hashable and frozen=True implied otherwise. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… caller The zarr2:/zarr3: format carried on StorePath.zarr_format was merged only in zarr.api.asynchronous; create_array, from_array, Group.from_store, Group.open, Array.open and the deprecated AsyncArray._create dropped it and swallowed explicit conflicts. The merge now lives on StorePath.resolve_zarr_format and is applied at each site. To let the pipeline's format apply when the caller does not specify one, create_array (sync and async) and Group.from_store default zarr_format to None, resolving to the configured default (3) as before; Array.open gains a zarr_format parameter mirroring Group.open. Tests: registry deadlock/race/import-failure/shadowing, resolver leak, zarr_format merge at each core site, and pins for the documented memory:/ file: divergences and the local-file-root limitation (xfail, strict). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- The extending guide's wrapper example dropped the preceding residual path while its comment claimed the opposite; it now joins the two and a tested snippet shows root|a|b resolving to a/b. - Changelog and storage guide state the memory:// routing split with fsspec, the undecoded percent-escapes in file: roots, the local-file-root limitation, the entry-point collision warnings, and the zarr_format default changes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ature/url-pipeline-core # Conflicts: # src/zarr/api/asynchronous.py # src/zarr/core/group.py
…leSystemWrapper fsspec < 2024.12.0 (the min_deps job) cannot open a plain memory:// URL at all, so the routing comparison only applies with fsspec>=2024.12.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`Group.open`, `Array.open` and their async variants defaulted to `zarr_format=3`, so a pipeline with a `zarr2:` segment raised a conflict unless the caller passed `zarr_format=None`. The default is now a private "unspecified" sentinel. When the caller does not pass a format, a URL pipeline decides: its format segment if any, otherwise auto-detection. Every other store keeps the default of 3, so the metadata probing of non-pipeline callers is unchanged. An explicit format (including None) behaves as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The format adapters from the URL pipeline spec (schemes/zarr.md). The segment body is a path within the preceding resource and is joined onto the preceding residual path; `zarr2:` / `zarr3:` record the Zarr format and `zarr:` leaves it to auto-detection (keeping a format pinned by an earlier segment). Every other field of the preceding resolution is carried forward with `dataclasses.replace`, so a format segment also works as an intermediate segment. The builtins are registered as lazily loaded entry points ahead of the third-party ones in `_collect_entrypoints`, so importing zarr does not import them, and a third-party `zarr.url_adapters` entry point with the same name is ignored with a warning. They are adapter-only schemes: a string without `|` such as `zarr3:foo` is never routed to them as a pipeline root, so its meaning is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uide Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4 of 7 tasks
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4476 +/- ##
==========================================
+ Coverage 94.68% 94.78% +0.09%
==========================================
Files 94 97 +3
Lines 13596 13889 +293
==========================================
+ Hits 12874 13164 +290
- Misses 722 725 +3
🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(not ready for review yet)
Summary
Adds the builtin
zarr:,zarr2:andzarr3:URL pipeline adapters from the spec(
schemes/zarr.md), andmakes
Group.open/Array.openwork with them without passingzarr_format=None(F3).Adapters (
src/zarr/storage/_url_adapters/_format.py). The segment body is a pathwithin the preceding resource and is joined onto the preceding residual path.
zarr2:/zarr3:setAdapterResolution.zarr_format;zarr:leaves itNone(auto-detect). Everyother field is carried forward with
dataclasses.replace, so a format segment also works asan intermediate segment (
root|zarr2:a|other:b). A query, a./..path segment, or twoformat segments that disagree (
|zarr2:|zarr3:) raiseURLPipelineError.zarr:after azarr2:keeps the earlier format instead of erasing it.Registration (
registry.py). The builtins are lazily loaded entry points prepended tothe
zarr.url_adapterspending list in_collect_entrypoints. Importing zarr does not importthem (tested in a subprocess), and there are no new import cycles. A third-party entry point
with the same name is ignored with a
ZarrUserWarning("the builtin ... takes precedence").An explicit
register_url_adapter("zarr3", ...)call still overrides a builtin, silently._collect_entrypointscan be called repeatedly (the test fixtures do) without duplicatingbuiltins.
Adapter-only schemes.
zarr,zarr2andzarr3are never dispatched as a pipelineroot. So a string without
|, such aszarr3:foo, behaves exactly as it does onmain.F3.
Group.open,Array.open,AsyncGroup.openandAsyncArray.opennow default to aprivate "unspecified" sentinel (
zarr.core.common._UNSPECIFIED). When the caller does not passa format:
3, so metadata probing for non-pipelinecallers is unchanged (pinned by
test_open_classmethod_default_without_pipeline).An explicit
zarr_format(includingNone) behaves as before, and a conflict still raisesValueError. The pinned tests inTestZarrFormatMergeInCoreare updated.Depends on
feature/url-pipeline-core, so untilfeat: url-pipeline core — parser, adapter ABC, registry, store hooks #4192 merges the diff here includes its commits. Review only the commits after
03ab3b05.This is PR 1 of a stack. Later PRs: read-only
zip:, the file-resource primitive plus awritable
zip:, root semantics/Windows/API polish, and the user guide.Closes
Group.open/Array.opendefaultzarr_format=3).zarr2:/zarr3:half of follow-up F9 (builtin adapters, registered through_collect_entrypoints).Decision needed
F3: how
Group.open/Array.openchoose the default format. This is isolated in thecommit "feat: let URL pipelines pick the format in Group.open / Array.open".
is no format segment). Everything else keeps
3. This adds no probing cost for existingcallers. One subtlety: a pipeline without a format segment (for example
x.zip|zip:, in alater PR) auto-detects, while the same store passed as a
Storeobject defaults to 3.Noneeverywhere, likezarr.open_group. This issimpler, but every existing
Group.open(store)caller would then fetch both v2 and v3metadata keys.
3and documentzarr_format=Nonefor pipelines (asin feat: url-pipeline core — parser, adapter ABC, registry, store hooks #4192). This works against
zarr2:and the spec'szarr:auto-detection.zarr:after another format segment.root|zarr2:|zarr:keeps format 2. Thealternative is to have
zarr:reset the format to auto-detect, which the strict reading of"
zarr:maps toNone" implies. Conflicting explicit segments (|zarr2:|zarr3:) raise.For reviewers
register_url_adaptercalls at import,the right mechanism? It keeps
import zarrcheap and reuses the collision handling.get_url_adaptergained a small special case so that a pending builtin is dropped silentlywhen the scheme was registered explicitly.
|is meant to be unchanged;is_url_pipeline("zarr3:foo")isFalse.Tests
Full suite: 13009 passed, 1295 skipped, 9 xfailed (baseline on
feature/url-pipeline-core: 12930 passed, 1295 skipped, 9 xfailed). ruff, mypy and pre-commit are clean.New tests are in
tests/test_url_pipeline/test_format_adapters.py:zarr.open,open_group,open_array,create_array,Group.openandArray.openwith|zarr2:,|zarr3:and|zarr:over local andmemory:roots;zarr_formatand the segment;root|zarr3:a/b) and a format segment used as an intermediate segment;schemes/zarr.mdexample;Author attestation
TODO
docs/user-guide/*.mdchanges/🤖 Generated with Claude Code