Repository navigation
Conversation
…rties geozero's CsvWriter takes its columns from the first feature, and panics unless every later feature has the same properties in the same order. So GeoJSON to CSV panicked when a later feature listed its properties in another order or had one the first feature lacked, and failed with "found record with N fields, but the previous record has M fields" when a feature left a property out or had it null. geozero's reader skips null values but still counts them in the property index, so even when every feature had the same properties, a null one that was not the last panicked (index out of bounds) from the second feature on. Scan the document for every property name that has a value in at least one feature, in the order first seen, and run the features through the field-order wrapper SHP to CSV already uses (renamed from DbfFieldOrder to FieldOrder), so each row has a cell for every column, empty where the feature has no value. A feature whose properties are null still gets a blank cell per column. Fixes dathere#2749
ybai08
force-pushed
the
fix/geoconvert-geojson-csv-mixed-properties
branch
from
October 18, 2026 06:15
80d05c8 to
14a92d6
Compare
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 20 |
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2749
Summary
geozero's
CsvWritertakes its columns from the first feature and expects every later feature to have the same properties in the same order (the TODO in georust/geozero#258). With real-world GeoJSON,qsv geoconvert x.geojson geojson csv:CSV features must all have the same column nameswhen a later feature lists its properties in another order;index out of boundswhen a later feature has a property the first one lacks;found record with 2 fields, but the previous record has 3 fieldswhen a feature leaves a property out or has itnull(geozero's reader skips nulls);index out of boundsfrom the second feature on when anullproperty comes before others, even if every feature has the same properties. The reader skips the null value but still counts it in the property index.This takes the "workaround" route from the issue rather than waiting for upstream. It reuses the field-order wrapper that SHP to CSV got in #4800, renamed from
DbfFieldOrdertoFieldOrderbecause it isn't SHP-only any more:geojson_property_namesscans the document once withserde_jsonand collects every property name that has a non-null value in at least one feature, in the order first seen.FieldOrderwithfill_missing, so each row gets a cell for every column, left empty where the feature has no value. A feature with"properties": nullgets an empty cell for each column too. Before,FieldOrderonly filled cells inproperties_end, and geozero never calls that for such a feature.For files master already converted, the output is byte-identical: the same columns in the same order, and a property that is null in every feature still gets no column. The base and patched binaries were compared on a few such files, with and without
--max-length. Only GeoJSON to CSV changes; SVG and GeoJSONL output still come straight fromGeoJsonReader. The cost is a second JSON parse plus one extra copy of the input string. That happens only for CSV output, and geozero already reads the whole document into memory.Tests
Three new tests in
tests/test_geoconvert.rs:geoconvert_geojson_to_csv_properties_in_different_ordergeoconvert_geojson_to_csv_null_before_other_propertiesgeoconvert_geojson_to_csv_missing_and_null_properties: a property that is null, missing, or first appears in a later feature, plus a feature with"properties": null. It also runs the same input from stdin through--max-length.Commands run (stable
rustc 1.99.1, macOS arm64):cargo test --locked -F feature_capable,geocode --test tests test_geoconvertgives 19 passed, 2 ignored (the existing TX_Cities fixture tests). Withsrc/cmd/geoconvert.rsreverted to master, the three new tests fail. One panics withCSV features must all have the same column names, the other two withindex out of bounds.cargo test --locked -F feature_capable,geocode --test tests(the whole integration suite for that feature set) gives 2788 passed, 0 failed, 42 ignored. This was run before the third test was added; the source was the same.cargo test --locked -F feature_capable,geocode --bin qsv geoconvertgives 4 passed.cargo clippy --locked -F feature_capable,geocode --tests -- -W clippy::perfgives no warnings in the changed files.Not run:
cargo +nightly fmt(no nightly toolchain available; stable rustfmt shows no changes in the new code beyond the nightly-only struct field alignment, which follows the existing style), and-F all_featuresclippy/tests (polars build not attempted).CHANGELOG: an entry is added under Unreleased > Fixed, linked to the issue for now. It can point to this PR's number instead.
AI use: the fix, the tests and this description were written with Claude Code (Claude Opus 5.5).