Skip to content

Latest commit

 

History

History
197 lines (163 loc) · 25 KB

NEWS.md

File metadata and controls

197 lines (163 loc) · 25 KB

Version 2.6.0 (development version)

Added

  • Added run logs. When filename is given, connectedness() and beri() also write a small TOML file of the run's indicator, settings, inputs and output beside it, result.tif getting result.toml, in the format the R package and Eco-Connectivity Studio write. Dropping a log on Studio's window restores its settings. Logs are on by default: save_log=False turns them off for one call, and the environment variable CONNECTIVITY_SAVE_LOG set to 0, false, no or off for a session or job, in the R package as well. An existing run log is replaced, but another file of that name is kept, with a warning. Not writing a log is a warning, never an error. Results are unchanged.

Changed

  • Builds against connectivity-core 0.2.1, whose VERSION the run logs record. Results are unchanged.

Version 2.5.0

Added

  • Added a band argument to connectedness(), beri() and read_raster() that selects one band of a multi-band raster, such as one year of a condition time series. Bands are numbered from 1, as in GDAL, rasterio and terra. In connectedness() and beri(), band selects a band of condition_file. A pa_file or resistance_file with more than one band uses the same band, and a single-band one is used as it is. The beri() climate rasters keep all their bands. Without band, a multi-band condition_file now raises ValueError("condition_file must have exactly one layer; choose one with band") instead of RuntimeError("Reading condition failed: ..."). A band outside a raster's bands raises ValueError("band must be between 1 and <n>, the number of layers in <name>"). Only the selected band is read, and the default, band=None, gives the same results as before.

Changed

  • Changed the default window_size of connectedness() and beri() from 3 to 5, matching the R package and the examples in the README. Results with default settings change, and runs take longer because each focal cell's graph has more nodes. Pass window_size=3 to reproduce results from earlier versions.
  • Updated to connectivity-core 0.2.0, which checks its inputs before any work. Invalid levels, transforms and window sizes, and a path-cost overflow, now raise RuntimeError("Connectivity failed: <message>") from the low-level rust_conn.connectivity() instead of pyo3_runtime.PanicException, with the same messages except Level <n> is not a power of 2, which no longer starts with Failed average resampling.:. A mask or transgrid whose shape differs from the condition array, and transgrids with different band counts, now raise errors too. A transgrid that is not a float32 array now raises RuntimeError("Reading transgrids failed: ...") instead of PanicException. connectedness() and beri() already reject most of these inputs, and results are unchanged.
  • Open-border (closed_border=False) polygon_mask runs of connectedness() and beri() now return a float32 numpy.ndarray with NaN outside the polygon, like every other run, instead of a numpy.ma.MaskedArray. Code that reads the output's .mask should use np.isnan() instead.
  • connectedness() and beri() now check pa_file, resistance_file, current_file and each of future_files against condition_file before reading them, with the R package's messages. Each must have the CRS and resolution of condition_file (ValueError("<name> must have the same CRS, resolution and grid alignment as condition_file")) and lie on its grid over the analysis window (ValueError("<name> must cover the analysis window on the same grid as condition_file")). pa_file and resistance_file must have one band unless band is given ("<name> must have exactly one layer; choose one with band"), and every scenario must have as many bands as current_file ("Every climate scenario must have the same band count as current_file"). Previously only array shapes were compared, and not in every case: an input on a grid shifted by whole pixels, with the same shape, ran silently on misaligned cells, and a future scenario with fewer bands than current_file ran with a different result. Every input is now read on the analysis window of the condition grid, so a raster on that grid with a larger extent is accepted, as in R, instead of raising a shape mismatch. Results for aligned inputs are unchanged.
  • rust_conn.connectivity() and rust_conn.pixel_coverage() now release the GIL while they compute, so other Python threads, such as Jupyter widgets, GUIs or a thread reading the next tile, keep running during connectedness(), beri() and pixel_coverage(). Results are unchanged.
  • Declared affine and pyproj (2.5 or later) as dependencies, since the package imports them directly. Both were already installed with rasterio and geopandas.

Deprecated

  • Deprecated the margin_px argument of connectedness() and beri(), which the R package does not have. It is now ignored and raises a DeprecationWarning, and it will be removed in 3.0. It computed a ring of cells outside polygon_mask in open-border runs, left over from the grid-effect filtering removed in 2.0.0, which the crop then discarded. Open-border runs now compute only the cells that polygon_mask touches, as in R, so polygon runs are faster. The margin cells did not change the values of other cells, so results are unchanged.
  • Deprecated the matching valid_margin_px argument of read_raster(); it will be removed in 3.0, when the mask will mark only cells the polygon touches as valid. Its default and behaviour are unchanged in 2.5, but a margin that changes the mask (open-border mode with a polygon) now raises a DeprecationWarning. Pass valid_margin_px=0 for the 3.0 behaviour.

Fixed

  • Fixed open-border polygon_mask runs of connectedness() and beri() dropping the outermost rows and columns that the polygon touches: the crop window rounded the polygon bounds to the nearest pixel edge rather than outwards. The output now covers the polygon bounds rounded outwards to the pixel grid and contains every cell the polygon touches, by the same rule used to choose the computed cells, as in the R package. These edge cells are added with the values a whole-raster run gives them; the other cells and closed-border runs are unchanged.
  • Fixed open-border polygon_mask runs with a polygon in a different CRS from the rasters, which cropped the output with the polygon's unprojected bounds and usually failed with "Crop window is empty". connectedness() and beri() now reproject the polygon once, before reading, and use it for both the read and the crop.
  • Fixed distances for projected CRSs whose units are not metres. connectedness() and beri() now convert the transform to metres, as the R package does. Previously it was passed in the CRS's own units, so distances were about 3.28 times too large in US survey feet and 1000 times too small in kilometres. For example, the bundled condition values on a 1 km Albers grid gave a mean Connected Habitat of 0.473 in US survey feet, 0.507 in metres and 0.567 in kilometres; all three now give 0.507. Results for metre-based and geographic CRSs are unchanged. A projected CRS whose units cannot be converted to metres raises ValueError("Cannot determine the projected CRS units in metres").
  • connectedness() and beri() now warn when the condition raster has no CRS, with the R package's message: "Raster CRS is missing; assuming longitude/latitude degrees" or "Raster CRS is missing; assuming projected metres". The guess itself is unchanged.
  • A polygon and a raster must now both have a CRS, or both omit one, as in the R package. connectedness() and beri() raise ValueError("polygon_mask and the raster must both have a CRS, or both omit one"), and read_raster() and pixel_coverage() raise the same message for polygon. Previously connectedness(), beri() and read_raster() failed with an unclear geopandas error, and pixel_coverage() silently used the polygon's coordinates as they were.
  • Fixed connectedness() and beri() raising "valid_margin_px cannot exceed the non-closed read-buffer limit" for small settings, such as levels=[2, 4] with outer_window=3, even without polygon_mask. The default 32-pixel margin exceeded the read buffer; the margin is no longer used.
  • Fixed beri() with single-band current_file and future_files, which raised pyo3_runtime.PanicException: one-band climate rasters were read as 2D arrays. They now keep their band axis and work as in the R package.
  • write_raster(), which writes the filename output of connectedness(), beri() and pixel_coverage(), now always writes a GeoTIFF, as its docstring says. It used to copy the template raster's format, so a .vrt or GDAL vrt:// condition_file failed with "Writing through VRTSourcedRasterBand is not supported", and other formats were written in the template's format.
  • Fixed the beri() docstring, which said it returns None; it returns the BERI array.

Version 2.4.1

Changed

  • Updated PyO3 and rust-numpy to 0.30, clearing the PyO3 security advisories for versions below 0.29 (an out-of-bounds read in PyList/PyTuple iterators and a missing Sync bound on PyCFunction::new_closure); the extension uses neither API. PyO3 0.30 has no Python 3.8 stable ABI, so wheels are now tagged cp310-abi3, matching the existing requires-python >= 3.10. Results are unchanged.
  • Moved the Python package into the repository's python/ directory, beside the R package in r/connectivity. Installing from PyPI is unchanged; to build from a clone, run pip install ./python instead of pip install ., or install directly from GitHub with pip install "git+https://github.com/csiro/connectivity.git#subdirectory=python".

Version 2.4.0

Changed

  • Rebuilt per-cell graphs with flat node arrays and directly constructed compressed sparse row (CSR) edges, retaining integer Dijkstra. Per-thread buffers eliminate per-cell allocations. Together with the caching and BERI improvements below, these changes are about 4x faster for connectedness and PARC and 3x faster for BERI than v2.3 on the bundled example (8 threads); speedups against v2.2 are about twice as large. CPU/Rayon remains the only execution path.
  • Cached window area fractions and outer-edge flags in per-call templates; cells clipped by tile boundaries are still computed directly. Edge distances use per-call tables wherever values provably match the per-edge calculation bit for bit, with direct calculation retained for other edges, including skewed or inconsistent level transforms and distances near an f32 rounding boundary. Distances and integer costs are unchanged.
  • Batched BERI similarity calculations across all scenarios of each cell, preserving similarity values.
  • Fixed each node's outgoing edge order: same-level queen neighbours in the existing order, then fringe links in get_edge_neighbours order. Shortest-path costs are unchanged, but integer-cost ties can select different parents and adjusted path sums, and indicator accumulation follows a different destination order. On the bundled example 2-16% of cells change, by up to about 1e-4 (2.4e-4 relative); grids where most edges round to zero integer cost can change more. Repeated runs and different thread counts remain bit-identical on the same build and platform.
  • Improved Rust connectivity() validation: missing transforms are checked before processing, and skipped powers of two produce a clear panic message instead of an index error. Both previously surfaced when a focal cell first reached the affected level. Python connectedness() and beri() already validate both.
  • Moved the Rust numerical engine (overviews, window templates, graph building, routing, indicator metrics and pixel coverage) into a separate crate, connectivity-core 0.1.0 in crates/connectivity-core, with a documented array-based Rust API and no Python, GDAL or file I/O dependencies. The rust_conn extension is now a thin PyO3 wrapper around it: Python function signatures, outputs, exceptions and threading are unchanged. The core is versioned and tagged separately (connectivity-core-vX.Y.Z); only vX.Y.Z tags publish the Python package to PyPI.

Version 2.3.0

Changed

  • Reduced routing overhead by building one adjacency list for both adjusted and intact searches, caching raw adjusted-distance sums along the selected paths, and avoiding repeated path back-tracing and attribute copying. Integer cost scaling and shortest-path tie handling are unchanged.
  • Replaced the remaining standard (SipHash) hash maps in per-cell graph building with FxHashMap, and replaced pathfinding::dijkstra_all with an in-crate Dijkstra that follows the same steps, preserving parents, ties, summation order and outputs. pathfinding is now only a test dependency. Together with the routing optimizations above, these changes made v2.3 about 2x faster than v2.2 in testing.

Fixed

  • Made indicator summation deterministic by retaining the adjusted search visitation order instead of iterating over a randomized hash map. Repeated runs on the same build and platform now use a fixed summation order without sorting targets for every focal cell. Final scores may differ in the last few bits from earlier versions because floating-point addition depends on order.
  • Integer path costs are now checked for overflow. Inputs that push a path cost past the u32 range (for example a very large max_cost or unscaled resistance values) now raise an error instead of silently wrapping and corrupting the result.

Version 2.2.0

Added

  • Added an optional resistance_file argument (and resistance_scale) to connectedness() and beri(), backed by a new traversal argument on the Rust connectivity() binding. When supplied, the resistance raster is decoupled from condition and used only for path traversal — the edge weight becomes w = (1.0 - max_cost) * (1 - resistance) + max_cost — while condition still drives every indicator value (habitat area and the value multiplier). When omitted, condition is used for traversal as before, so output is unchanged. Resistance values are expected in [0, 1] (scaled via resistance_scale); cells valid in condition but missing resistance fall back to condition for traversal and emit a warning, so the analysis domain is never changed silently.
  • Added an optional pa_to_pa argument to connectedness() (PARC edition, requires pa_file). When enabled, only graph nodes that fall on a protected area contribute to the connectedness numerator: paths still route through the intervening non-PA landscape (adjusted/intact distances unchanged) and the denominator keeps the full reachable area exactly as when disabled, so only the numerator changes — non-PA destinations contribute nothing and coarse cells straddling a PA boundary contribute only their protected fraction. A protected area that can reach no other scores 0. Defaults to False (original PARC behaviour).
  • Added an optional filename argument to pixel_coverage(); when set, the per-pixel coverage array is also written to a GeoTIFF that inherits raster_file's grid and CRS, producing a raster directly usable as the pa_file input to connectedness().
  • Added bundled example datasets and an example_data_path() helper (exported from the top-level package) to resolve them by name — site_condition, pa_proportion, and the transgrids/* climate scenarios — so the example notebooks and quick tests run without external downloads.
  • Added an example notebook, examples/01_tasmania_indicators.ipynb, demonstrating the connectivity and BERI indicators on the bundled Tasmania data (shipped in the source distribution).

Changed

  • Raised the minimum supported Python to 3.10 (requires-python = ">=3.10").
  • Updated packaging ([tool.maturin]) to ship the example rasters in both wheels and source distributions, and the example notebooks in the source distribution.
  • Published the package on PyPI as eco-connectivity (pip install eco-connectivity), since the name connectivity is taken there; it is still imported as connectivity. Prebuilt wheels for Linux (x86_64, aarch64), macOS (Intel and Apple Silicon) and Windows (x64) mean installing no longer requires Rust. If you installed an earlier version from a locally built wheel, run pip uninstall connectivity first.

Version 2.1.0

Added

  • Added make_tile() and make_tiles() helpers for creating non-overlapping raster-pixel core tiles whose internal row/column breaks align to a requested aggregation level, e.g. align_to=max(levels), for tiled connectedness() and beri() runs. The helpers also support optional deterministic cost-balanced planning with balanced=True.

Fixed

  • Fixed adjusted path distances to use the same max_cost-scaled edge distances used by Dijkstra routing, avoiding a second hard-coded inverse-condition transform equivalent to max_cost = 2.
  • Fixed geographic area weights in tiled runs to use tile-local rows with the already shifted read-window transform, avoiding a double row-offset that could leave residual horizontal tile seams.

Version 2.0.0

Added

  • Added a window_mode option to connectedness(), beri(), and the Rust connectivity() binding.
  • Added window_mode="circular" for source-centered circular annuli with fractional area/count support at annulus boundaries.
  • Added window_mode="square" for source-centered square annuli.
  • Added resolution_info() for previewing internally generated level dimensions, resolutions, cell counts, approximate native/km search reach, and small-grid warnings before choosing levels.

Changed

  • Changed the default window mode to window_mode="circular".
  • connectedness() and beri() now validate that levels form a continuous power-of-two sequence before entering Rust graph construction.
  • Updated Rust Python bindings to pyo3 0.28 and numpy 0.28, removing the deprecated gil-refs and py-clone feature usage and migrating to the current PyO3/numpy APIs.
  • Updated the Python build configuration for current PyO3/maturin extension-module handling.
  • overview_info() is now a backward-compatible wrapper for resolution_info().
  • Reduced routing/window-construction overhead by avoiding per-node adjacency vector clones during Dijkstra, hoisting max-level lookup out of per-cell window construction, using faster hash maps for internal graph adjacency data, and writing parallel row results directly into the output array.
  • Reduced Python-side overhead by reading BERI scenario rasters concurrently, avoiding an extra temporary array when converting masked raster data to NaN-filled float32 arrays, and applying the option=3 square-root transform in place.

Removed

  • Removed window_mode="block" and the original snapped multi-resolution window construction from the public API and Rust implementation.
  • Removed automatic grid-effect filtering and the filter_kwargs arguments from connectedness() and beri().
  • Removed remove_grid_bias() from the package exports;
  • Removed the obsolete Rust inpaint_nans_diffusion Python binding and implementation, which were only used by the previous FFT notch-filter path.

Version 1.1.1

Fixed

  • Fixed a PARC-connectedness crash when the protected-area mask leaves no valid cells in the analysis window. Empty windows now return all-NaN output instead of failing during filtering/inpainting.
  • Filtered bounded outputs from connectedness() and beri() are now clipped back to the theoretical [0, 1] range after FFT notch filtering. This applies to connectedness (option = 1), connected habitat (option = 2), geometric-mean connected habitat (option = 3), PARC-connectedness, and BERI whenever filtering is enabled.
  • Float32 outputs written by write_raster() now set nodata = NaN in the GeoTIFF metadata, and masked float arrays are written with NaN cells so the raster profile matches the stored data.

Version 1.1.0

Added

  • Added remove_grid_effect() as a standalone FFT notch-filter post-processing function (also exported from the top-level package import).
  • Added a Rust inpaint_nans_diffusion implementation (Rayon-parallel), with optional thread control via n_threads.

Fixed

  • Fixed seam artifacts caused by swapped row/column arguments in graph distance coordinate conversion (Affine::xy now consistently receives (row, col)).
  • Fixed overview alignment across independently processed tiles by anchoring aggregation windows to global dataset pixel coordinates via tile offsets.
  • Fixed cross-level edge-neighbour mapping to be offset-aware (global-anchor consistent) when linking a level to its higher level.
  • Fixed occasional missing edge rows/cols in buffered tile reads by adding a half-pixel epsilon to the buffer extent.

Changed

  • Graph construction now passes tile offsets through the full path (core -> builder -> graph::fringe) to keep multi-resolution linking globally aligned.
  • Node IDs are now tile-invariant (NodeId = u64 from level/global-row/global-col), removing tile-local ID drift between adjacent runs.
  • Core traversal now processes overview levels in sorted order.
  • Non-closed tile reads are snapped to the coarsest overview grid, improving consistency of aggregated counts/weights at tile boundaries.
  • Added Area resampling for 2D overviews (latitude-weighted by cos(phi)), alongside Average, Sum, and Count.
  • 3D overview generation now explicitly supports only Average and Sum resampling.
  • overview_info() now reports estimated overview dimensions and does not require pre-existing file overviews.
  • connectedness() and beri() now apply FFT notch filtering (remove_grid_effect) instead of the previous Gaussian smoother.
  • Filtering in connectedness() and beri() is now controlled via filter_kwargs: pass None to disable filtering, or {} to apply default filtering (notch_width=3).
  • Inpainting used by filtering now relies on the Rust implementation and uses n_threads when provided.

Removed

  • Removed legacy smoothing_filter() backward-compatibility wrapper.
  • Removed SciPy-based inpainting fallback and dropped the scipy Python dependency.

Version 1.0.0

Added

  • Incorporated habitat area into connectivity calculations using per-node cell counts (aggregate-based weighting).
  • Integrated overview generation directly in Rust to eliminate preprocessing steps and reduce raster I/O bottlenecks.
  • Added the anyhow crate for improved, centralized error handling.
  • Refactored core functionality into a dedicated module for improved structure and maintainability.

Removed

  • Removed create_overviews() and has_overview(); overview handling is now fully internal.

Version 0.6.1

  • Fixed a bug for reading files with different overview levels.
  • Added additional early checks for grid shapes and values.

Version 0.6.0

  • Added closed_border argument to control whether cells outside the mask are excluded from the analysis.
  • Fixed the polygon-masked output to not return the buffered array.
  • Added compilation CPU optimisation flags (slower complie but faster run).
  • Changed the overview_levels argument of the create_overviews() function to levels for consistency. Also, file_path in read_raster() function.

Version 0.5.0

  • Added has_overview() function for checking overviews.

Version 0.4.2

  • A few internal structural changes; e.g., changes to the NumPy array memory layout and added structure in the Rust code for performance improvement.

Version 0.4.1

  • Fixed a bug involving a mismatch between conditions and transgrids for BERI by handling NANs.
  • Replaced Arc<T> with Rc<T> reference counting, since atomic operations and thread sharing are not needed.
  • Improved code readability by adding more structs.

Version 0.4.0

  • Added PARC-connectedness calculation to the connectedness() function.
  • Fixed a bug in reading transgrids with a polygon mask.

Version 0.3.0

  • Fixed the code to accept raster integer data type.
  • Added a scaling factor (scale arg) to convert condition data to the 0–1 range.
  • The output array/map is now cropped to the supplied polygon mask.
  • Added type hints to the main functions.
  • Made several improvements on the Rust side, such as using Arc<T> for cheap cloning and introducing the Graph struct for more organised code.

Version 0.2.3

  • Fixed isolated pixel calculation using their own values for a synthetic neighbour.
  • Increased the capacity of the graph to take more neighbours for high-resolution rasters.
  • Changed the default value of the sigma parameter to 1.0.

Version 0.2.2

  • Removed the dependency on GDAL, replacing it with rasterio.
  • Updated logic to ignore overview level 1 (both when reading and generating) since it corresponds to the original resolution and requires no additional computation.
  • Fixed the read_raster() and create_overviews() functions to ensure that the correct overview layers are created and read properly.

Version 0.2.1

  • Added Euclidean and Haversine distances for projected and unprojected grids, respectively, instead of using cell units.
  • Some changes in the arguments names including max_cost, window_size and outer_window.
  • Some small fixes for graph calculation.

Version 0.2.0

Added geographic distance (in km) instead of cell units Replaced the scale argument with max_cost Various minor improvements

Version 0.1.0

  • Initial version.