Repository navigation
Conversation
Implements URL pipeline support (https://github.com/jbms/url-pipeline): '|'-chained URLs resolve through pluggable adapters registered under the 'zarr.url_adapters' entry-point group (entry-point name = URL scheme). - zarr.abc.url_pipeline: PipelineSegment, AdapterResolution, PipelineContext, URLPipelineAdapter (single-classmethod contract) - zarr.storage._url_pipeline: parse_pipeline / resolve_pipeline; the root sub-URL delegates to make_store so existing file/memory/fsspec routing is unchanged - registry: register_url_adapter / get_url_adapter / list_url_adapter_schemes (name check only; no adapter imports) - make_store/make_store_path route strings containing '|' (or a registered root scheme) through the resolver; residual store paths combine with the user-supplied path - StorePath gains a zarr_format attribute (populated by format segments in a follow-up) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e-core # Conflicts: # src/zarr/errors.py # src/zarr/registry.py
…thoring - make_store validates the access mode before routing a URL pipeline string to adapters, matching every other StoreLike branch - make_store docstring lists URL pipeline strings; StorePath.open documents the invalid-mode ValueError - user guide: StoreLike bullet for pipeline strings and an adapter authoring section in extending.md (runnable example) - changelog: drop PR-relative wording, name the public parser entry points Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
get_url_adapter took a non-reentrant lock across entry_point.load(), so an adapter module that resolved another scheme at import time deadlocked the process. Matching entry points are now taken off the pending list under the lock and imported outside it; import failures are wrapped in URLPipelineError and leave the entry point discoverable for a retry. A same-named entry point is discarded with a ZarrUserWarning when the scheme is already registered (e.g. by a builtin), instead of staying pending forever; duplicate entry-point names warn and the first wins. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ent contracts - The resolver closes the store an adapter returned when it cannot be made read-only for mode 'r', instead of leaking it. - resolve_pipeline documents every error type it lets through; the resolve_preceding docstring states the local-file-root limitation. - AdapterResolution and PipelineContext compare and hash by identity, since a Store is not hashable and frozen=True implied otherwise. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… caller The zarr2:/zarr3: format carried on StorePath.zarr_format was merged only in zarr.api.asynchronous; create_array, from_array, Group.from_store, Group.open, Array.open and the deprecated AsyncArray._create dropped it and swallowed explicit conflicts. The merge now lives on StorePath.resolve_zarr_format and is applied at each site. To let the pipeline's format apply when the caller does not specify one, create_array (sync and async) and Group.from_store default zarr_format to None, resolving to the configured default (3) as before; Array.open gains a zarr_format parameter mirroring Group.open. Tests: registry deadlock/race/import-failure/shadowing, resolver leak, zarr_format merge at each core site, and pins for the documented memory:/ file: divergences and the local-file-root limitation (xfail, strict). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- The extending guide's wrapper example dropped the preceding residual path while its comment claimed the opposite; it now joins the two and a tested snippet shows root|a|b resolving to a/b. - Changelog and storage guide state the memory:// routing split with fsspec, the undecoded percent-escapes in file: roots, the local-file-root limitation, the entry-point collision warnings, and the zarr_format default changes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ature/url-pipeline-core # Conflicts: # src/zarr/api/asynchronous.py # src/zarr/core/group.py
…leSystemWrapper fsspec < 2024.12.0 (the min_deps job) cannot open a plain memory:// URL at all, so the routing comparison only applies with fsspec>=2024.12.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`Group.open`, `Array.open` and their async variants defaulted to `zarr_format=3`, so a pipeline with a `zarr2:` segment raised a conflict unless the caller passed `zarr_format=None`. The default is now a private "unspecified" sentinel. When the caller does not pass a format, a URL pipeline decides: its format segment if any, otherwise auto-detection. Every other store keeps the default of 3, so the metadata probing of non-pipeline callers is unchanged. An explicit format (including None) behaves as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The format adapters from the URL pipeline spec (schemes/zarr.md). The segment body is a path within the preceding resource and is joined onto the preceding residual path; `zarr2:` / `zarr3:` record the Zarr format and `zarr:` leaves it to auto-detection (keeping a format pinned by an earlier segment). Every other field of the preceding resolution is carried forward with `dataclasses.replace`, so a format segment also works as an intermediate segment. The builtins are registered as lazily loaded entry points ahead of the third-party ones in `_collect_entrypoints`, so importing zarr does not import them, and a third-party `zarr.url_adapters` entry point with the same name is ignored with a warning. They are adapter-only schemes: a string without `|` such as `zarr3:foo` is never routed to them as a pipeline root, so its meaning is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uide Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`zip:` (schemes/zip.md) addresses an entry or directory inside the ZIP
archive that the preceding pipeline resolves to. The preceding pipeline is
resolved with mode "r", and the archive is read through that store with
async byte-range requests only: the central directory is fetched from the
end of the archive (parsed in a worker thread), and each member is fetched
and decompressed on demand, with true range reads for stored members.
A thin read-only reader over the preceding store's bytes is used instead
of `ZipStore`, because `ZipStore` does blocking file I/O and needs a local
path or a sync file object. The reader works the same way for local files,
fsspec objects, `memory:` entries and nested `zip:` archives, and never
blocks the IO event loop.
Reading a root that *is* the archive relies on `get("")` addressing the
root object (follow-up F2); that is isolated in `_read_resource_range` /
`_resource_size` so the file-resource primitive can replace it.
Writable modes ("w", "w-", "r+") raise `URLPipelineError`; "a" and None
open read-only. One leading "/" in the body is ignored and more than one is
invalid. The preceding store is closed when the archive cannot be opened,
and the reader closes it on `close()`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(not ready for review yet)
Summary
Adds a builtin, read-only
zip:URL pipeline adapter(
schemes/zip.md). This makesthe headline example from #2943 work:
How it works (
src/zarr/storage/_url_adapters/_zip.py):context.resolve_preceding(mode="r").The archive is the value at
preceding.pathinpreceding.store, or the store's rootobject when the path is empty (a local file or an fsspec object).
ZipReaderStoreis a thin, read-only store over that resource. It uses async byte-rangerequests only:
central directory with the stdlib
zipfileinasyncio.to_thread, over a sparsein-memory view; any range the view lacks is fetched and the parse retried. This handles
central directories larger than the initial read.
get, it reads the member's local header once (the offset is cached) and then themember's data.
whole and decompressed, in a thread above 1 MiB. CRC-32 is verified. Other methods (for
example LZMA) and encrypted members raise
NotImplementedError.dir/) are not keys.close()closes the preceding store, which the reader owns./is ignored, and//...is invalid. A query,.or..segmentsraise
URLPipelineError. The body becomes the residual path, and every other precedingfield (for example
zarr_format) is carried forward withdataclasses.replace."w","w-"and"r+"raiseURLPipelineError("zip: is read-only until writable ZIP support lands; ...")."r",Noneand"a"open read-only;"a"(thezarr.opendefault) serves the "open" half, per the contract from feat: url-pipeline core — parser, adapter ABC, registry, store hooks #4192.
URLPipelineErrorchained from the original exception, and the preceding store is closed.zipis an adapter-only scheme, likezarr*in feat: builtin zarr:, zarr2: and zarr3: URL pipeline adapters #4476. A string without|, such asfsspec's
zip://...orzip::file://..., behaves exactly as before.Why not
ZipStore?The adapter must work for local and remote (fsspec) roots and must not block zarr's I/O
event loop.
ZipStoreneeds a local path or a sync file object, and it does blockingzipfileI/O inside its async methods. Over an fsspec root that means either downloading thearchive or making sync, loop-blocking reads through an fsspec file object. The reader here
uses only the preceding store's async
get(..., byte_range=...)andgetsize. So one codepath serves local files, fsspec objects,
memory:entries, and archives nested in otherarchives. For an uncompressed archive (
ZipStore's default), a chunk read is a single rangerequest.
The
get("")quirk (F2)Reading an archive that is the pipeline root relies on
LocalStore/FsspecStoretreatingthe empty key as the root object. That reliance is isolated in two helpers,
_read_resource_rangeand_resource_size. The file-resource primitive in the next PRreplaces both.
Depends on
zarr*:adapters). This branch is stacked onfeature/url-pipeline-format-adapters, so review only the commits after37b2f490.Closes
zip:half of follow-up F9 and roadmap row Z2.Decision needed
ZipStore. Implemented: the thin reader described above.Alternative: open
ZipStoreon a local path (in a thread) for local roots, and on an fsspecfile object for remote roots. That is less code, but it blocks the event loop and does not
compose with nested archives.
"a"onzip:. Implemented: open read-only, following the open-or-createcontract. Alternative: reject every mode except
"r"until writes are supported. That isstricter, but
zarr.open(url)(default"a") would then fail on a perfectly readablearchive.
For reviewers
_SparseFile/_parse_central_directory: is lazily fetching ranges into the stdlibzipfileparser robust enough? It raises a private exception on a missing range andretries after fetching it.
once per read of the inner archive. Stored members, the common case for Zarr ZIP stores,
are read with ranges.
Tests
Full suite: 13053 passed, 1295 skipped, 9 xfailed. ruff, mypy and pre-commit are clean.
New tests are in
tests/test_url_pipeline/test_zip_adapter.py:ZipStore, opened asfile:/…/a.zip|zip:,file://…,file://localhost…and schemeless paths;/…/a.zip|zip:inner|zarr3:suband the headline example;local://and an in-memory fsspec filesystem standing in for a remoteobject store;
memory:entry, and nestedzip:(stored and deflated outer archives);first read; byte ranges; a CRC mismatch;
close().Author attestation
TODO
docs/user-guide/*.mdchanges/🤖 Generated with Claude Code