Repository navigation
feat: add a shigola mode that purges or seeds each metatile (MAPCO-11992) - #42
Open
NivGreenstein wants to merge 5 commits into
Open
NivGreenstein wants to merge 5 commits into
NivGreenstein wants to merge 5 commits into
Conversation
…992)
With app.map.provider set to shigola, retiler keeps shigola's tile cache
fresh instead of rendering tiles: each metatile job becomes one request to
shigola's Cache Admin API, POST {app.map.url}/cache/{purge|seed}, with the
metatile's lng/lat bbox and its zoom as both ends of the zoom range. A seed
always sends overwrite. No map is fetched, split or stored, and detiler is
not used.
The metatile processor is now chosen by the map provider: TileProcessor for
wms and arcgis, ShigolaCacheProcessor for shigola.
A 4xx that repeating cannot fix (400, 422) ends the job without a retry. As
pg-boss can only fail a fetched job into a retry, such a job is cancelled,
through a NonRetryableError the queue provider recognises. 401, 403, 404,
429, 5xx, timeouts and connection errors fail the job for pg-boss to retry,
since a token or url mistake is fixed by a deploy and would otherwise cancel
every metatile meanwhile.
Shigola mode has its own request, tile and duration metrics, labelled by
operation and outcome, and a shigola.cache span per job. The helm chart takes
the admin token from its secret or an existing one, and defaults the tile
matrix sets to WorldCRS84Quad, the grid the metatile zoom is in.
Needs the shigola config of @map-colonies/schemas' retiler v2 schema.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The shigola provider ships in a new retiler v3 schema, as released schemas are not modified. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…er installs Debian 11 reached end of life, and apt in node:24-bullseye now fails with 404s from bullseye-security. The pg client repo follows the container's release instead of naming one. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Temporary: points @map-colonies/schemas at the package built from MapColonies/schemas#181, so CI can run against the v3 schema before it is released. Replace it with the released version before merging. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each test sends a pg-boss job through the shigola cache processor against a mocked Cache Admin API: a purge and a seed complete with one request for the metatile's bbox and zoom, a refused request cancels the job, and an unavailable or unreachable shigola fails it for a retry. The integration suite measures coverage over all of src, so without these the shigola processor failed its thresholds. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issues: MAPCO-11992. Depends on MapColonies/schemas#181.
Further information:
With
app.map.provider: shigola, retiler keeps Shigola's tile cache fresh instead of rendering tiles. Each metatile job becomes onePOST {app.map.url}/cache/{purge|seed}with the metatile's lng/lat bbox and its zoom as both ends of the zoom range. A seed always sendsoverwrite: true. No map is fetched, split or stored, and detiler is not used.How a job ends
fail()always retries while retries remain, and a fetched job has no public terminal failure. So aNonRetryableErrormakes the queue provider cancel the job instead.Other changes
TileProcessorfor wms and arcgis,ShigolaCacheProcessorfor shigola.retiler_shigola_requests_count{operation, outcome, code},retiler_shigola_tiles_count{operation, result}andretiler_shigola_request_duration_seconds{operation, z}.shigola.cachespan per job.env.map.shigola.existingSecret). Tile matrix sets default to["WorldCRS84Quad"], the grid the metatile zoom is in.Before merging
@map-colonies/schemascurrently points at the tarball built from feat: add a shigola map provider in a new retiler v3 schema (MAPCO-11992) schemas#181, so CI can run against the newvectorRetilerV3schema before the release. Set it back to the released version (and update the lockfile) before merging.Testing:
helm templaterenders with the token from the chart's Secret, from an existing Secret, and for the arcgis default.🤖 Generated with Claude Code