Skip to content

Rewrite VII interpolator to use dask map_blocks #135

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
djhoese wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Rewrite VII interpolator to use dask map_blocks #135

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
476bbcb
Fix VII interpolator producing Fortran-ordered arrays
djhoese Sep 30, 2026
f6f2382
Convert VII interpolator tests to pytest
djhoese Sep 30, 2026
5c2346a
Add dask arrays to tests and revert interp dim swap
djhoese Sep 30, 2026
25ac2b7
Rewrite VII interpolation to use map_blocks
djhoese Sep 30, 2026
523e6ed
Move to_cart condition into dask
djhoese Sep 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ These share almost no code and are **not** interchangeable.
|---|---|---|
| Generic spline/grid | `interpolator.py`, `geointerpolator.py` | Two generations in one file (below) |
| MODIS Cython | `modisinterpolator.py` + `_modis_interpolator.pyx`; `simple_modis_interpolator.py` + `_simple_modis_interpolator.pyx`; `_modis_utils.pyx` | Modern MODIS path |
| EPS-SG VII | `viiinterpolator.py` | **Not VIIRS.** Pure xarray, no Cython |
| EPS-SG VII | `viiinterpolator.py` | **Not VIIRS.** Pure numpy per scan (dask `map_blocks`), no Cython |
| Multilinear | `multilinear.py` + `multilinear_cython.pyx` | Regular cartesian grid; unrelated to geolocation |

Non-obvious details:
Expand Down Expand Up @@ -92,8 +92,16 @@ be exact. That is why the test fixture is an ocean scene and why tests assert *g
2. **`AbstractSingleInterpolator.interpolate_dask`** (`interpolator.py`) hand-builds a dask graph
(`normalize_chunks` + `tokenize` + a task dict), deliberately *not* `map_blocks`. Enabled by passing
`chunks=` to `interpolate()` / `interpolate_to_shape()`; `chunks="auto"` works.
3. **`viiinterpolator`** relies entirely on `xarray.DataArray.interp()` and *requires* xarray input. It
interpolates across-track then along-track as two 1-D passes because that is much faster than 2-D.
3. **`viiinterpolator`** *requires* `DataArray` input (numpy- or dask-backed) and returns `DataArray`s
that keep the name, attrs, and non-dimension coords. It is its own `map_blocks` wrapper, not
`scanline_mapblocks`: it rechunks to whole scans (`scan_alt_tie_points` tie point rows) and the
full width, and stacks `(lon, lat)` along a new first axis the same way. Each scan is interpolated
on its own with numpy (along then across track, tie-aligned pixels copied rather than weighted), so
results are identical for any chunking; tests enforce that, plus C-contiguous blocks and one
pixel-row chunk per tie-point-row chunk (Satpy's METimage reader relies on both).
`tie_points_geo_interpolation` never computes: the reductions that choose geodetic vs cartesian
(max |lat|, longitude range) are passed to `map_blocks` as 0-d dask arrays, so the choice is made
once per compute, over all the tie points, and is the same for every chunk. Never decide per chunk.

Everything else (the `__init__.py` legacy helpers, Gen-1 `Interpolator`/`GeoInterpolator`,
`multilinear`) is numpy-only; `__init__.py` parallelizes with `multiprocessing.Pool` instead.
Expand Down
Loading
Loading