Repository navigation
fix: delete only cache tiles touching both the update area and the previous footprint (MAPCO-12035) - #125
Closed
CL-SHLOMIKONCHA wants to merge 2 commits into
Closed
fix: delete only cache tiles touching both the update area and the previous footprint (MAPCO-12035)#125CL-SHLOMIKONCHA wants to merge 2 commits into
CL-SHLOMIKONCHA wants to merge 2 commits into
Conversation
…APCO-12035) The update cache deletion job walked the whole update area from zoom 0 to the configured max zoom, so an update that adds a new area next to an existing layer produced millions of tiles above zoom 16 for ground that was never served and never cached. The update job now records the catalog footprint as previousFootprint at init, before the polygon parts are merged. The cache deletion creator clips the update area per zoom to the previous footprint buffered by one tile diagonal, and stops the walk at the first zoom whose clip is empty, since a deeper tile cannot touch both areas when its parent does not. When the footprint cannot be recorded the whole update area is walked as before. The buffer is taken in grid degrees: turf measures along the globe, and a buffer of many degrees around a footprint away from the equator bends around the sphere and misses the footprint, which ended the walk after zoom 0 for any real layer.
…ring (MAPCO-12035) The cache deletion creator descends the WorldCRS84 tile tree from the two root tiles and keeps a tile only when it touches both the update area and the previous footprint. A tile that misses either area is dropped with all its descendants, and a tile wholly inside both emits its subtree by arithmetic. Only the footprint edges that cross a tile are handed to its children, so the work shrinks as the walk narrows onto the boundaries. This replaces the per-zoom buffer and intersection: no geodesic buffer to flatten, no safety rim, and the emitted set is exactly the tiles touching both areas. On a 2000-vertex overlap it runs about six times faster than the buffered walk.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issues: MAPCO-12035
Alternative to #124: same problem and same job-level change, different range walk. Only one of the two should be merged.
Further information:
Problem. The update cache deletion job walks the whole update area from zoom 0 to the configured max zoom (21). An update that adds a new area next to an existing layer therefore produces millions of deletion tiles, almost all above zoom 16, for ground that was never served and so never cached in redis.
Change.
updateJobHandler: at init, before the polygon parts are merged into the catalog, the current catalog footprint is stored on the job asadditionalParams.previousFootprint. This is best effort; if the catalog lookup fails a warning is logged and the deletion falls back to the full walk.cacheDeletionJobCreator: with a previous footprint on the job, the creator descends the WorldCRS84 tile tree from the two root tiles and keeps a tile only when it touches both the update area and the previous footprint. A tile that misses either area is dropped with all its descendants. A tile that no footprint edge crosses is wholly inside or wholly outside that footprint, decided by its centre; a tile wholly inside both emits its subtree by arithmetic without being visited. Only the edges that cross a tile are handed to its children, so the work shrinks as the walk narrows onto the boundaries. The emitted set is exactly the tiles touching both areas. The grid helpers mirror the ones mc-utils uses forfootprintToTileRanges, applied to two footprints at once.jobParameters.schema:previousFootprint(Polygon or MultiPolygon, optional) on the update and swap finalize params.Compared with #124. That PR clips the update area per zoom to the previous footprint buffered by one tile diagonal, which needs a planar buffer and leaves a safety rim of up to two tiles. This walk needs no buffer or polygon intersection and has no rim. On an overlap benchmark (2000-vertex previous footprint, update half inside, to zoom 21) it emits 7.13M tiles in 0.27 s against 7.14M tiles in 1.8 s for the buffered walk, and 14.5M tiles in 0.08 s for today's full walk.
Example. A layer near 34.3°E 31.35°N updated with a disjoint polygon near 35.5°E 33.9°N, 2.7° away, produces four ranges, one tile per zoom from 0 to 3, instead of about 1.9 million tiles across zooms 0 to 21. From zoom 4 on the grid row boundary at 33.75° separates the two footprints, so no tile covers both.
Trade-offs, kept on purpose.
Tests. The footprint tests assert exact set equality with an exhaustive oracle of tiles touching both areas: a disjoint update (walk stops at the last zoom whose tile spans the gap), an overlapping update (reaches max zoom, nothing over the new-only ground), an update wholly inside the previous footprint (same tiles as mc-utils' plain walk), and a layer at latitude 32. The update handler tests cover the footprint recording and its fallback.