Skip to content

feat: add a shigola mode that purges or seeds each metatile (MAPCO-11992) - #42

Open
NivGreenstein wants to merge 5 commits into
masterfrom
feat/shigola-mode
Open

NivGreenstein wants to merge 5 commits into
masterfrom
feat/shigola-mode

Conversation

@NivGreenstein

@NivGreenstein NivGreenstein commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator
Question Answer
Bug fix ✖
New feature ✔
Breaking change ✖
Deprecations ✖
Documentation ✔
Tests added ✔
Chore ✖

Related issues: MAPCO-11992. Depends on MapColonies/schemas#181.

Further information:

With app.map.provider: shigola, retiler keeps Shigola's tile cache fresh instead of rendering tiles. Each metatile job becomes one POST {app.map.url}/cache/{purge|seed} with the metatile's lng/lat bbox and its zoom as both ends of the zoom range. A seed always sends overwrite: true. No map is fetched, split or stored, and detiler is not used.

How a job ends

Response Job
2xx completed
400, 422 (bad bbox, unknown map, too many tiles) cancelled, never retried
401, 403, 404, 429, 5xx, timeout, connection error failed, retried by pg-boss
  • Why cancel: pg-boss's fail() always retries while retries remain, and a fetched job has no public terminal failure. So a NonRetryableError makes the queue provider cancel the job instead.
  • Deviation from the ticket: 401, 403 and 404 are retried, although the ticket says to end every 4xx except 429 without retry. They mean a token or URL mistake, which a deploy fixes, and would otherwise cancel every metatile in the meantime.

Other changes

  • Processor choice: the map provider now picks the metatile processor: TileProcessor for wms and arcgis, ShigolaCacheProcessor for shigola.
  • Metrics: retiler_shigola_requests_count{operation, outcome, code}, retiler_shigola_tiles_count{operation, result} and retiler_shigola_request_duration_seconds{operation, z}.
  • Traces: a shigola.cache span per job.
  • Helm: the token comes from the chart's Secret or an existing one (env.map.shigola.existingSecret). Tile matrix sets default to ["WorldCRS84Quad"], the grid the metatile zoom is in.
  • README: a new "Shigola mode" section.

Before merging

  1. Merge and release feat: add a shigola map provider in a new retiler v3 schema (MAPCO-11992) schemas#181.
  2. ⚠️ Replace the temporary dependency. @map-colonies/schemas currently points at the tarball built from feat: add a shigola map provider in a new retiler v3 schema (MAPCO-11992) schemas#181, so CI can run against the new vectorRetilerV3 schema before the release. Set it back to the released version (and update the lockfile) before merging.

Testing:

  • Unit tests pass (79). New cases cover every outcome class, token handling, the request shape, the metrics and span, the queue's cancel path, and the container wiring (Shigola mode never resolves a map provider or tile storage).
  • Typecheck and lint pass against the new schema build.
  • helm template renders with the token from the chart's Secret, from an existing Secret, and for the arcgis default.
  • The integration suite was not run locally.

🤖 Generated with Claude Code

…992)

With app.map.provider set to shigola, retiler keeps shigola's tile cache
fresh instead of rendering tiles: each metatile job becomes one request to
shigola's Cache Admin API, POST {app.map.url}/cache/{purge|seed}, with the
metatile's lng/lat bbox and its zoom as both ends of the zoom range. A seed
always sends overwrite. No map is fetched, split or stored, and detiler is
not used.

The metatile processor is now chosen by the map provider: TileProcessor for
wms and arcgis, ShigolaCacheProcessor for shigola.

A 4xx that repeating cannot fix (400, 422) ends the job without a retry. As
pg-boss can only fail a fetched job into a retry, such a job is cancelled,
through a NonRetryableError the queue provider recognises. 401, 403, 404,
429, 5xx, timeouts and connection errors fail the job for pg-boss to retry,
since a token or url mistake is fixed by a deploy and would otherwise cancel
every metatile meanwhile.

Shigola mode has its own request, tile and duration metrics, labelled by
operation and outcome, and a shigola.cache span per job. The helm chart takes
the admin token from its secret or an existing one, and defaults the tile
matrix sets to WorldCRS84Quad, the grid the metatile zoom is in.

Needs the shigola config of @map-colonies/schemas' retiler v2 schema.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
NivGreenstein and others added 4 commits October 15, 2026 19:21
The shigola provider ships in a new retiler v3 schema, as released schemas
are not modified.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…er installs

Debian 11 reached end of life, and apt in node:24-bullseye now fails with
404s from bullseye-security. The pg client repo follows the container's
release instead of naming one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Temporary: points @map-colonies/schemas at the package built from
MapColonies/schemas#181, so CI can run against the v3 schema before it is
released. Replace it with the released version before merging.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each test sends a pg-boss job through the shigola cache processor against a
mocked Cache Admin API: a purge and a seed complete with one request for the
metatile's bbox and zoom, a refused request cancels the job, and an
unavailable or unreachable shigola fails it for a retry.

The integration suite measures coverage over all of src, so without these
the shigola processor failed its thresholds.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant