Skip to content

feat: builtin zarr:, zarr2: and zarr3: URL pipeline adapters - #4476

Draft
jhamman wants to merge 13 commits into
zarr-developers:mainfrom
jhamman:feature/url-pipeline-format-adapters
Draft

jhamman wants to merge 13 commits into
zarr-developers:mainfrom
jhamman:feature/url-pipeline-format-adapters

Conversation

@jhamman

@jhamman jhamman commented Oct 3, 2026 •

Copy link
Copy Markdown
Member

(not ready for review yet)

Summary

Adds the builtin zarr:, zarr2: and zarr3: URL pipeline adapters from the spec
(schemes/zarr.md), and
makes Group.open / Array.open work with them without passing zarr_format=None (F3).

zarr.open_group("data/|zarr2:", mode="w")
zarr.open_array("file:///data/|zarr2:path/to/array")
zarr.Group.open("s3://bucket/x.zarr|zarr:")   # format auto-detected
  • Adapters (src/zarr/storage/_url_adapters/_format.py). The segment body is a path
    within the preceding resource and is joined onto the preceding residual path. zarr2: /
    zarr3: set AdapterResolution.zarr_format; zarr: leaves it None (auto-detect). Every
    other field is carried forward with dataclasses.replace, so a format segment also works as
    an intermediate segment (root|zarr2:a|other:b). A query, a ./.. path segment, or two
    format segments that disagree (|zarr2:|zarr3:) raise URLPipelineError. zarr: after a
    zarr2: keeps the earlier format instead of erasing it.

  • Registration (registry.py). The builtins are lazily loaded entry points prepended to
    the zarr.url_adapters pending list in _collect_entrypoints. Importing zarr does not import
    them (tested in a subprocess), and there are no new import cycles. A third-party entry point
    with the same name is ignored with a ZarrUserWarning ("the builtin ... takes precedence").
    An explicit register_url_adapter("zarr3", ...) call still overrides a builtin, silently.
    _collect_entrypoints can be called repeatedly (the test fixtures do) without duplicating
    builtins.

  • Adapter-only schemes. zarr, zarr2 and zarr3 are never dispatched as a pipeline
    root. So a string without |, such as zarr3:foo, behaves exactly as it does on main.

  • F3. Group.open, Array.open, AsyncGroup.open and AsyncArray.open now default to a
    private "unspecified" sentinel (zarr.core.common._UNSPECIFIED). When the caller does not pass
    a format:

    • a URL pipeline decides: its format segment if there is one, otherwise auto-detection;
    • every other store keeps today's default of 3, so metadata probing for non-pipeline
      callers is unchanged (pinned by test_open_classmethod_default_without_pipeline).

    An explicit zarr_format (including None) behaves as before, and a conflict still raises
    ValueError. The pinned tests in TestZarrFormatMergeInCore are updated.

Depends on

This is PR 1 of a stack. Later PRs: read-only zip:, the file-resource primitive plus a
writable zip:, root semantics/Windows/API polish, and the user guide.

Closes

  • Follow-up F3 (Group.open / Array.open default zarr_format=3).
  • The zarr2:/zarr3: half of follow-up F9 (builtin adapters, registered through
    _collect_entrypoints).
  • Roadmap row Z1.
  • Part of Support ZEP 8 URL Syntax #2943.

Decision needed

F3: how Group.open / Array.open choose the default format. This is isolated in the
commit "feat: let URL pipelines pick the format in Group.open / Array.open".

  • Implemented: a private sentinel default. Pipelines pick the format (auto-detect when there
    is no format segment). Everything else keeps 3. This adds no probing cost for existing
    callers. One subtlety: a pipeline without a format segment (for example x.zip|zip:, in a
    later PR) auto-detects, while the same store passed as a Store object defaults to 3.
  • Alternative A: change the default to None everywhere, like zarr.open_group. This is
    simpler, but every existing Group.open(store) caller would then fetch both v2 and v3
    metadata keys.
  • Alternative B: leave the default at 3 and document zarr_format=None for pipelines (as
    in feat: url-pipeline core — parser, adapter ABC, registry, store hooks #4192). This works against zarr2: and the spec's zarr: auto-detection.

zarr: after another format segment. root|zarr2:|zarr: keeps format 2. The
alternative is to have zarr: reset the format to auto-detect, which the strict reading of
"zarr: maps to None" implies. Conflicting explicit segments (|zarr2:|zarr3:) raise.

For reviewers

  • Is lazy-loaded builtin entry points, rather than eager register_url_adapter calls at import,
    the right mechanism? It keeps import zarr cheap and reuses the collision handling.
    get_url_adapter gained a small special case so that a pending builtin is dropped silently
    when the scheme was registered explicitly.
  • Behavior for strings without | is meant to be unchanged; is_url_pipeline("zarr3:foo") is
    False.

Tests

uv sync --frozen
uv run --frozen ruff check src tests
uv run --frozen mypy src
uv run --frozen --with pytest-examples pytest -n auto tests
pre-commit run --all-files

Full suite: 13009 passed, 1295 skipped, 9 xfailed (baseline on feature/url-pipeline-core: 12930 passed, 1295 skipped, 9 xfailed). ruff, mypy and pre-commit are clean.

New tests are in tests/test_url_pipeline/test_format_adapters.py:

  • end-to-end zarr.open, open_group, open_array, create_array, Group.open and
    Array.open with |zarr2:, |zarr3: and |zarr: over local and memory: roots;
  • auto-detection of existing v2 and v3 data;
  • conflicts between an explicit zarr_format and the segment;
  • stacked paths (root|zarr3:a/b) and a format segment used as an intermediate segment;
  • every schemes/zarr.md example;
  • registration and laziness.

Author attestation

  • I am a human, these are my changes, and I have reviewed and understood every change and can explain why each is correct.

TODO

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/user-guide/*.md
  • Changes documented as a new file in changes/
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

🤖 Generated with Claude Code

jhamman and others added 13 commits August 19, 2026 09:16
Implements URL pipeline support (https://github.com/jbms/url-pipeline):
'|'-chained URLs resolve through pluggable adapters registered under the
'zarr.url_adapters' entry-point group (entry-point name = URL scheme).

- zarr.abc.url_pipeline: PipelineSegment, AdapterResolution,
  PipelineContext, URLPipelineAdapter (single-classmethod contract)
- zarr.storage._url_pipeline: parse_pipeline / resolve_pipeline; the root
  sub-URL delegates to make_store so existing file/memory/fsspec routing
  is unchanged
- registry: register_url_adapter / get_url_adapter /
  list_url_adapter_schemes (name check only; no adapter imports)
- make_store/make_store_path route strings containing '|' (or a
  registered root scheme) through the resolver; residual store paths
  combine with the user-supplied path
- StorePath gains a zarr_format attribute (populated by format segments
  in a follow-up)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e-core

# Conflicts:
#	src/zarr/errors.py
#	src/zarr/registry.py
…thoring

- make_store validates the access mode before routing a URL pipeline
  string to adapters, matching every other StoreLike branch
- make_store docstring lists URL pipeline strings; StorePath.open
  documents the invalid-mode ValueError
- user guide: StoreLike bullet for pipeline strings and an adapter
  authoring section in extending.md (runnable example)
- changelog: drop PR-relative wording, name the public parser entry points

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
get_url_adapter took a non-reentrant lock across entry_point.load(), so an
adapter module that resolved another scheme at import time deadlocked the
process. Matching entry points are now taken off the pending list under the
lock and imported outside it; import failures are wrapped in URLPipelineError
and leave the entry point discoverable for a retry.

A same-named entry point is discarded with a ZarrUserWarning when the scheme
is already registered (e.g. by a builtin), instead of staying pending forever;
duplicate entry-point names warn and the first wins.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ent contracts

- The resolver closes the store an adapter returned when it cannot be made
  read-only for mode 'r', instead of leaking it.
- resolve_pipeline documents every error type it lets through; the
  resolve_preceding docstring states the local-file-root limitation.
- AdapterResolution and PipelineContext compare and hash by identity, since
  a Store is not hashable and frozen=True implied otherwise.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… caller

The zarr2:/zarr3: format carried on StorePath.zarr_format was merged only in
zarr.api.asynchronous; create_array, from_array, Group.from_store,
Group.open, Array.open and the deprecated AsyncArray._create dropped it and
swallowed explicit conflicts. The merge now lives on
StorePath.resolve_zarr_format and is applied at each site.

To let the pipeline's format apply when the caller does not specify one,
create_array (sync and async) and Group.from_store default zarr_format to
None, resolving to the configured default (3) as before; Array.open gains a
zarr_format parameter mirroring Group.open.

Tests: registry deadlock/race/import-failure/shadowing, resolver leak,
zarr_format merge at each core site, and pins for the documented memory:/
file: divergences and the local-file-root limitation (xfail, strict).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- The extending guide's wrapper example dropped the preceding residual path
  while its comment claimed the opposite; it now joins the two and a tested
  snippet shows root|a|b resolving to a/b.
- Changelog and storage guide state the memory:// routing split with fsspec,
  the undecoded percent-escapes in file: roots, the local-file-root
  limitation, the entry-point collision warnings, and the zarr_format
  default changes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ature/url-pipeline-core

# Conflicts:
#	src/zarr/api/asynchronous.py
#	src/zarr/core/group.py
…leSystemWrapper

fsspec < 2024.12.0 (the min_deps job) cannot open a plain memory:// URL at
all, so the routing comparison only applies with fsspec>=2024.12.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`Group.open`, `Array.open` and their async variants defaulted to
`zarr_format=3`, so a pipeline with a `zarr2:` segment raised a conflict
unless the caller passed `zarr_format=None`.

The default is now a private "unspecified" sentinel. When the caller does
not pass a format, a URL pipeline decides: its format segment if any,
otherwise auto-detection. Every other store keeps the default of 3, so the
metadata probing of non-pipeline callers is unchanged. An explicit format
(including None) behaves as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The format adapters from the URL pipeline spec (schemes/zarr.md). The
segment body is a path within the preceding resource and is joined onto
the preceding residual path; `zarr2:` / `zarr3:` record the Zarr format and
`zarr:` leaves it to auto-detection (keeping a format pinned by an earlier
segment). Every other field of the preceding resolution is carried forward
with `dataclasses.replace`, so a format segment also works as an
intermediate segment.

The builtins are registered as lazily loaded entry points ahead of the
third-party ones in `_collect_entrypoints`, so importing zarr does not
import them, and a third-party `zarr.url_adapters` entry point with the
same name is ignored with a warning. They are adapter-only schemes: a
string without `|` such as `zarr3:foo` is never routed to them as a
pipeline root, so its meaning is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uide

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.71795% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.78%. Comparing base (25d390b) to head (37b2f49).

Files with missing lines Patch % Lines
src/zarr/storage/_url_pipeline.py 97.36% 3 Missing ⚠️
src/zarr/core/common.py 80.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4476      +/-   ##
==========================================
+ Coverage   94.68%   94.78%   +0.09%     
==========================================
  Files          94       97       +3     
  Lines       13596    13889     +293     
==========================================
+ Hits        12874    13164     +290     
- Misses        722      725       +3     
Files with missing lines Coverage Δ
src/zarr/abc/url_pipeline.py 100.00% <100.00%> (ø)
src/zarr/api/asynchronous.py 96.55% <100.00%> (+0.08%) ⬆️
src/zarr/api/synchronous.py 95.77% <ø> (ø)
src/zarr/core/array.py 98.40% <100.00%> (+<0.01%) ⬆️
src/zarr/core/group.py 95.69% <100.00%> (+0.01%) ⬆️
src/zarr/errors.py 100.00% <100.00%> (ø)
src/zarr/registry.py 93.53% <100.00%> (+1.77%) ⬆️
src/zarr/storage/__init__.py 100.00% <100.00%> (+5.00%) ⬆️
src/zarr/storage/_common.py 93.96% <100.00%> (+0.93%) ⬆️
src/zarr/storage/_url_adapters/_format.py 100.00% <100.00%> (ø)
... and 2 more
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant