Skip to content

fix(geoconvert): GeoJSON to CSV handles features with different properties - #4837

Open
ybai08 wants to merge 1 commit into
dathere:masterfrom
ybai08:fix/geoconvert-geojson-csv-mixed-properties
Open

ybai08 wants to merge 1 commit into
dathere:masterfrom
ybai08:fix/geoconvert-geojson-csv-mixed-properties

Conversation

@ybai08

@ybai08 ybai08 commented Oct 18, 2026

Copy link
Copy Markdown

Fixes #2749

Summary

geozero's CsvWriter takes its columns from the first feature and expects every later feature to have the same properties in the same order (the TODO in georust/geozero#258). With real-world GeoJSON, qsv geoconvert x.geojson geojson csv:

  • panics with CSV features must all have the same column names when a later feature lists its properties in another order;
  • panics with index out of bounds when a later feature has a property the first one lacks;
  • fails with found record with 2 fields, but the previous record has 3 fields when a feature leaves a property out or has it null (geozero's reader skips nulls);
  • panics with index out of bounds from the second feature on when a null property comes before others, even if every feature has the same properties. The reader skips the null value but still counts it in the property index.

This takes the "workaround" route from the issue rather than waiting for upstream. It reuses the field-order wrapper that SHP to CSV got in #4800, renamed from DbfFieldOrder to FieldOrder because it isn't SHP-only any more:

  • geojson_property_names scans the document once with serde_json and collects every property name that has a non-null value in at least one feature, in the order first seen.
  • The features go through FieldOrder with fill_missing, so each row gets a cell for every column, left empty where the feature has no value. A feature with "properties": null gets an empty cell for each column too. Before, FieldOrder only filled cells in properties_end, and geozero never calls that for such a feature.

For files master already converted, the output is byte-identical: the same columns in the same order, and a property that is null in every feature still gets no column. The base and patched binaries were compared on a few such files, with and without --max-length. Only GeoJSON to CSV changes; SVG and GeoJSONL output still come straight from GeoJsonReader. The cost is a second JSON parse plus one extra copy of the input string. That happens only for CSV output, and geozero already reads the whole document into memory.

Tests

Three new tests in tests/test_geoconvert.rs:

  • geoconvert_geojson_to_csv_properties_in_different_order
  • geoconvert_geojson_to_csv_null_before_other_properties
  • geoconvert_geojson_to_csv_missing_and_null_properties: a property that is null, missing, or first appears in a later feature, plus a feature with "properties": null. It also runs the same input from stdin through --max-length.

Commands run (stable rustc 1.99.1, macOS arm64):

  • cargo test --locked -F feature_capable,geocode --test tests test_geoconvert gives 19 passed, 2 ignored (the existing TX_Cities fixture tests). With src/cmd/geoconvert.rs reverted to master, the three new tests fail. One panics with CSV features must all have the same column names, the other two with index out of bounds.
  • cargo test --locked -F feature_capable,geocode --test tests (the whole integration suite for that feature set) gives 2788 passed, 0 failed, 42 ignored. This was run before the third test was added; the source was the same.
  • cargo test --locked -F feature_capable,geocode --bin qsv geoconvert gives 4 passed.
  • cargo clippy --locked -F feature_capable,geocode --tests -- -W clippy::perf gives no warnings in the changed files.

Not run: cargo +nightly fmt (no nightly toolchain available; stable rustfmt shows no changes in the new code beyond the nightly-only struct field alignment, which follows the existing style), and -F all_features clippy/tests (polars build not attempted).

CHANGELOG: an entry is added under Unreleased > Fixed, linked to the issue for now. It can point to this PR's number instead.

AI use: the fix, the tests and this description were written with Claude Code (Claude Opus 5.5).

…rties

geozero's CsvWriter takes its columns from the first feature, and panics
unless every later feature has the same properties in the same order. So
GeoJSON to CSV panicked when a later feature listed its properties in
another order or had one the first feature lacked, and failed with "found
record with N fields, but the previous record has M fields" when a feature
left a property out or had it null. geozero's reader skips null values but
still counts them in the property index, so even when every feature had
the same properties, a null one that was not the last panicked (index out
of bounds) from the second feature on.

Scan the document for every property name that has a value in at least
one feature, in the order first seen, and run the features through the
field-order wrapper SHP to CSV already uses (renamed from DbfFieldOrder to
FieldOrder), so each row has a cell for every column, empty where the
feature has no value. A feature whose properties are null still gets a
blank cell per column.

Fixes dathere#2749
@ybai08
ybai08 force-pushed the fix/geoconvert-geojson-csv-mixed-properties branch from 80d05c8 to 14a92d6 Compare October 18, 2026 06:15
@codacy-production

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 20 complexity

Metric Results
Complexity 20

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

qsv geoconvert crashes when not all properties are available (or not in same order) when converting to CSV

1 participant