Repository navigation
fix: delete only cache tiles that can touch the previous footprint (MAPCO-12035) - #124
Open
CL-SHLOMIKONCHA wants to merge 6 commits into
Open
CL-SHLOMIKONCHA wants to merge 6 commits into
CL-SHLOMIKONCHA wants to merge 6 commits into
Conversation
…APCO-12035) The update cache deletion job walked the whole update area from zoom 0 to the configured max zoom, so an update that adds a new area next to an existing layer produced millions of tiles above zoom 16 for ground that was never served and never cached. The update job now records the catalog footprint as previousFootprint at init, before the polygon parts are merged. The cache deletion creator clips the update area per zoom to the previous footprint buffered by one tile diagonal, and stops the walk at the first zoom whose clip is empty, since a deeper tile cannot touch both areas when its parent does not. When the footprint cannot be recorded the whole update area is walked as before. The buffer is taken in grid degrees: turf measures along the globe, and a buffer of many degrees around a footprint away from the equator bends around the sphere and misses the footprint, which ended the walk after zoom 0 for any real layer.
… coarse zooms (MAPCO-12035) turf measures a buffer along the globe, which is a superset of the grid distance as long as the buffer is not wide enough to bend around the sphere. The coarse zooms whose reach exceeds ten degrees are therefore walked over the whole update area, a handful of tiles that are nearly always shared, and the deeper zooms buffer the footprint as it is. This removes the flattening of the footprint onto an equatorial patch and the coordinate mapping that came with it.
… clip at coarse zooms (MAPCO-12035)" This reverts commit fd9d42f.
… ring (MAPCO-12035)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issues: MAPCO-12035
Further information:
Problem. The update cache deletion job walks the whole update area from zoom 0 to the configured max zoom (21). An update that adds a new area next to an existing layer therefore produces millions of deletion tiles, almost all above zoom 16, for ground that was never served and so never cached in redis.
Change.
updateJobHandler: at init, before the polygon parts are merged into the catalog, the current catalog footprint is stored on the job asadditionalParams.previousFootprint. This is best effort; if the catalog lookup fails a warning is logged and the deletion falls back to the full walk.cacheDeletionJobCreator: with a previous footprint on the job, the walk runs per zoom and clips the update area's tile ranges to the tiles touching the previous footprint's bbox. The box uses closed bounds (ceil - 1on the min side,flooron the max side), so an edge lying exactly on a grid line claims the tile on either side, the same rulefootprintToTileRangesapplies to footprints. This is plain range arithmetic in grid units, exact at any latitude, with no buffer. Because the bbox of an irregular footprint also holds ground the layer never covered, each clipped range is then checked against the footprint itself withbooleanIntersectsand dropped when its ground does not touch it. The first zoom with no surviving range ends the walk, because a child tile cannot be shared by both areas when its parent is not.jobParameters.schema:previousFootprint(Polygon or MultiPolygon, optional) on the update and swap finalize params.What gets deleted. The result is exactly the update tiles whose squares touch the previous footprint, at every zoom, which is every tile MapProxy could have rendered for the old layer inside the update. Whether a given tile is actually in redis is not known here; the cleaner unlinks the key either way. Tiles MapProxy may have served beyond the catalog footprint are left to the cache TTL, see below.
Example. A layer near 34.3°E 31.35°N updated with a disjoint polygon near 35.5°E 33.9°N, 2.7° away, now produces four ranges, one tile per zoom from 0 to 3, instead of about 1.9 million tiles across zooms 0 to 21. From zoom 4 on no tile covers both footprints and the walk stops.
Trade-offs, kept on purpose.
Tests. Unit tests cover the fallback walk, a disjoint update (walk stops at the last zoom with a shared tile, exhaustive oracle for tiles touching both), an overlapping update (reaches max zoom but stays inside the tiles touching the previous bbox), the same at latitude 32, an update in the notch of an L-shaped layer (stops early, all tiles touching both arms covered), and the footprint recording and its fallback in the update handler.