Skip to content

fix(io): keep null lists of structs when projecting Arrow data - #4065

Open
DG47 wants to merge 1 commit into
apache:mainfrom
DG47:fix/null-list-of-structs
Open

DG47 wants to merge 1 commit into
apache:mainfrom
DG47:fix/null-list-of-structs

Conversation

@DG47

@DG47 DG47 commented Oct 2, 2026

Copy link
Copy Markdown

Closes #3833

Rationale for this change

ArrowProjectionVisitor.list rebuilds a list whose element is a struct with pa.LargeListArray.from_arrays(list_array.offsets, value_array). The offsets alone cannot express a null list, so every null list<struct<...>> came out as an empty list. The visitor runs on both the read and the write path; on write the empty list is persisted into the Parquet file, so the null is gone for every reader, not only for pyiceberg.

Reproduction from the issue (pyarrow 25.0.1):

l_struct  in=[[{'x': 1}], [], None, [{'x': 3}]]  out=[[{'x': 1}], [], [], [{'x': 3}]]
l_int     in=[[1], [], None, [3]]                out=[[1], [], None, [3]]

This PR rebuilds the list with pa.Array.from_buffers, reusing the validity and offsets buffers (and the array offset) of the original list and swapping in the projected element array. Passing mask=list_array.is_null() to from_arrays would fix the unsliced case, but Arrow rejects a mask together with sliced offsets (Null bitmap with offsets slice not supported) for LargeListArray, so sliced batches would start raising. Reusing the buffers keeps the array offset and avoids that.

The integration test test_null_list_and_map asserted the corrupted value with the correct assertion commented out; the correct assertion is restored.

Are these changes tested?

  • New unit test test__to_requested_schema_null_list_of_structs in tests/io/test_pyarrow.py, parametrized over pa.list_ and pa.large_list, covering a full batch and a sliced one. It fails on main (At index 2 diff: [] != None) and passes with this change.
  • Restored the real assertion in tests/integration/test_reads.py::test_null_list_and_map.
  • make test and make lint pass locally.

Are there any user-facing changes?

Null lists of structs are now preserved on read and write instead of being turned into empty lists.

The projection visitor rebuilds a list of structs from its offsets alone,
which cannot represent a null list, so every null list<struct<...>> was
turned into an empty list. On the write path this is persisted into the
Parquet file and the null is lost for every reader.

Rebuild the list from the original validity and offsets buffers instead,
which also keeps the original array offset so sliced batches stay correct.

Closes apache#3833

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Null list<struct<...>> is written and read as an empty list

1 participant