Repository navigation
Scanning ArrowArrayStream twice raises InternalException: INTERNAL Error: ArrowArrayStream was released by another thread/library #105
Description
What happens?
When scanning an ArrowArrayStream twice, an internal error is raised.
Per https://duckdb.org/docs/stable/dev/internal_errors this should be raised as an issue.
Potentially related to #70
To Reproduce
import duckdb # 1.34.0
import polars as pl # 1.4.0
df = pl.DataFrame({"a": 1})
cap = df.__arrow_c_stream__()
rel = duckdb.query("SELECT * from cap")
# Printing this relation causes an error in .pl() below
print(rel)
# _duckdb.InternalException: INTERNAL Error: ArrowArrayStream was released by another thread/library
print(rel.pl())Output
┌───────┐
│ a │
│ int64 │
├───────┤
│ 1 │
└───────┘
Traceback (most recent call last):
File "/home/henry/development/adbc-polars-duck-bq-pg/duckdb_polars_no_pyarrow/duckdb_polars_error.py", line 23, in <module>
print(rel.pl())
~~~~~~^^
_duckdb.InternalException: INTERNAL Error: ArrowArrayStream was released by another thread/library
Stack Trace:
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb9Exception6ToJSONENS_13ExceptionTypeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x56) [0x79b7e0978f86]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb9ExceptionC2ENS_13ExceptionTypeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x14) [0x79b7e0979034]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb17InternalExceptionC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x11) [0x79b7e097c131]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(+0x7c49f1) [0x79b7dffc49f1]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb34PythonTableArrowArrayStreamFactory9GetSchemaEmRNS_18ArrowSchemaWrapperE+0x25) [0x79b7e04754f5]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb18ArrowTableFunction13ArrowScanBindERNS_13ClientContextERNS_22TableFunctionBindInputERNS_6vectorINS_11LogicalTypeELb1EEERNS5_INSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEELb1EEE+0x2a0) [0x79b7e072c740]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb18ArrowTableFunction17ArrowScanBindDumbERNS_13ClientContextERNS_22TableFunctionBindInputERNS_6vectorINS_11LogicalTypeELb1EEERNS5_INSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEELb1EEE+0x12) [0x79b7e072c8c2]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder25BindTableFunctionInternalERNS_13TableFunctionERKNS_16TableFunctionRefENS_6vectorINS_5ValueELb1EEESt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_NS_33CaseInsensitiveStringHashFunctionENS_29CaseInsensitiveStringEqualityESaISt4pairIKSF_S7_EEENS6_INS_11LogicalTypeELb1EEENS6_ISF_Lb1EEE+0x567) [0x79b7e0fd55f7]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder4BindERNS_16TableFunctionRefE+0xd68) [0x79b7e0fd8668]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder4BindERNS_8TableRefE+0x128) [0x79b7e0526a38]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder8BindNodeERNS_10SelectNodeE+0x3b) [0x79b7e0f7c1db]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder8BindNodeERNS_9QueryNodeE+0x48) [0x79b7e0527988]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder7BindCTEERNS_7CTENodeE+0x1ae) [0x79b7e0f7173e]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder8BindNodeERNS_7CTENodeE+0x12) [0x79b7e0f72c02]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder8BindNodeERNS_9QueryNodeE+0x96) [0x79b7e05279d6]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder4BindERNS_9QueryNodeE+0x2f) [0x79b7e0527a4f]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder4BindERNS_15SelectStatementE+0x3a) [0x79b7e0f85b1a]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder4BindERNS_12SQLStatementE+0x22d) [0x79b7e052792d]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb8Relation4BindERNS_6BinderE+0xea) [0x79b7e0cb79aa]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb13QueryRelation4BindERNS_6BinderE+0x5e) [0x79b7e0c8be8e]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder4BindERNS_17RelationStatementE+0x24) [0x79b7e0f85ad4]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder4BindERNS_12SQLStatementE+0xc5) [0x79b7e05277c5]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb7Planner10CreatePlanERNS_12SQLStatementE+0xa5) [0x79b7e05345a5]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb13ClientContext31CreatePreparedStatementInternalERNS_17ClientContextLockERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_10unique_ptrINS_12SQLStatementESt14default_deleteISC_ELb1EEENS_12optional_ptrISt13unordered_mapIS8_NS_18BoundParameterDataENS_33CaseInsensitiveStringHashFunctionENS_29CaseInsensitiveStringEqualityESaISt4pairIS9_SI_EEELb1EEE+0x28b) [0x79b7e0cdca6b]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb13ClientContext23CreatePreparedStatementERNS_17ClientContextLockERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_10unique_ptrINS_12SQLStatementESt14default_deleteISC_ELb1EEENS_12optional_ptrISt13unordered_mapIS8_NS_18BoundParameterDataENS_33CaseInsensitiveStringHashFunctionENS_29CaseInsensitiveStringEqualityESaISt4pairIS9_SI_EEELb1EEENS_21PreparedStatementModeE+0x347) [0x79b7e0cdd807]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb13ClientContext24PendingStatementInternalERNS_17ClientContextLockERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_10unique_ptrINS_12SQLStatementESt14default_deleteISC_ELb1EEERKNS_22PendingQueryParametersE+0xa4) [0x79b7e0ceaf84]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb13ClientContext35PendingStatementOrPreparedStatementERNS_17ClientContextLockERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_10unique_ptrINS_12SQLStatementESt14default_deleteISC_ELb1EEERNS_10shared_ptrINS_21PreparedStatementDataELb1EEERKNS_22PendingQueryParametersE+0xa0) [0x79b7e0cf2fa0]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb13ClientContext43PendingStatementOrPreparedStatementInternalERNS_17ClientContextLockERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_10unique_ptrINS_12SQLStatementESt14default_deleteISC_ELb1EEERNS_10shared_ptrINS_21PreparedStatementDataELb1EEERKNS_22PendingQueryParametersE+0x85) [0x79b7e0cf31c5]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb13ClientContext20PendingQueryInternalERNS_17ClientContextLockENS_10unique_ptrINS_12SQLStatementESt14default_deleteIS4_ELb1EEERKNS_22PendingQueryParametersEb+0xa9) [0x79b7e0cf47c9]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb13ClientContext20PendingQueryInternalERNS_17ClientContextLockERKNS_10shared_ptrINS_8RelationELb1EEEb+0x184) [0x79b7e0cfa764]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb13ClientContext12PendingQueryERKNS_10shared_ptrINS_8RelationELb1EEEb+0x44) [0x79b7e0cfa954]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(+0xc3ca22) [0x79b7e043ca22]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb16DuckDBPyRelation15ExecuteInternalEb+0x14) [0x79b7e043cac4]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb16DuckDBPyRelation14ExecuteOrThrowEb+0x43) [0x79b7e043ccd3]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb16DuckDBPyRelation20ToArrowTableInternalEmb+0x1a8) [0x79b7e043d878]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb16DuckDBPyRelation8ToPolarsEmb+0x31) [0x79b7e044e981]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(+0xcfbc8d) [0x79b7e04fbc8d]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(+0xb951bf) [0x79b7e03951bf]
/home/henry/.pyenv/versions/3.13.7/lib/libpython3.13.so.1.0(+0x15aade) [0x79b7e3d5aade]
/home/henry/.pyenv/versions/3.13.7/lib/libpython3.13.so.1.0(_PyObject_MakeTpCall+0x87) [0x79b7e3cf93d7]
/home/henry/.pyenv/versions/3.13.7/lib/libpython3.13.so.1.0(_PyEval_EvalFrameDefault+0x135d) [0x79b7e3c9217d]
/home/henry/.pyenv/versions/3.13.7/lib/libpython3.13.so.1.0(PyEval_EvalCode+0x165) [0x79b7e3e52335]
/home/henry/.pyenv/versions/3.13.7/lib/libpython3.13.so.1.0(+0x2bc1f1) [0x79b7e3ebc1f1]
/home/henry/.pyenv/versions/3.13.7/lib/libpython3.13.so.1.0(+0x2bc54f) [0x79b7e3ebc54f]
/home/henry/.pyenv/versions/3.13.7/lib/libpython3.13.so.1.0(+0x2be8ff) [0x79b7e3ebe8ff]
/home/henry/.pyenv/versions/3.13.7/lib/libpython3.13.so.1.0(+0x2bee4f) [0x79b7e3ebee4f]
/home/henry/.pyenv/versions/3.13.7/lib/libpython3.13.so.1.0(Py_RunMain+0x9a9) [0x79b7e3ee9c49]
/home/henry/.pyenv/versions/3.13.7/lib/libpython3.13.so.1.0(Py_BytesMain+0x5a) [0x79b7e3eea3ba]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x79b7e3a32083]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/bin/python3(_start+0x2e) [0x5a33c194e09e]
This error signals an assertion failure within DuckDB. This usually occurs due to unexpected conditions or errors in the program's logic.
For more information, see https://duckdb.org/docs/stable/dev/internal_errors
OS:
Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.31
DuckDB Package Version:
1.34.0
Python Version:
3.13.7
Full Name:
Henry Harbeck
Affiliation:
None
What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.
I have tested with a stable release
Did you include all relevant data sets for reproducing the issue?
Yes
Did you include all code required to reproduce the issue?
- Yes, I have
Did you include all relevant configuration to reproduce the issue?
- Yes, I have
Activity
paultiq commentedon Oct 21, 2025
Linking to my discussion comment here (since github doesn't link issues back to discussions): #132 (reply in thread)
If this issue is resolved, then duckdb/duckdb#13798 could be reverted which has a significant performance benefit for Polars scans.
kylebarron commentedon Oct 21, 2025
This is expected behavior with the Arrow PyCapsule Interface; you can't use a capsule multiple times because that capsule holds a C pointer that can only be used once.
The correct behavior should be to use
SELECT * from df, because that will call__arrow_c_stream__()on the Polars DataFrame producing a new capsule.In particular, I'd say it's an anti-pattern to allow querying from the capsule itself. Users should never be able to see the capsule and should only see other Arrow objects that can produce capsules.
paultiq commentedon Oct 21, 2025
I'm working up a proposed fix for the underlying issue in #70,
which I think will solve this too, which solves the double consumption problem but not if you explicitly call arrow_c_streamhenryharbeck commentedon Dec 23, 2025
For what its worth, the only reason I am doing this is as a workaround to not require PyArrow
ref: #132 / duckdb/duckdb#13827
(which I'm sure you're aware is a pain-point give you created the inital issue 😅)
Throw a reasonable error when an array stream was already consumed (#306
evertlammerts commentedon Feb 10, 2026
We're now throwing an InvalidInputException instead of an InternalException.