[el-marvin]

The main line's counterpart of #721, which moved the pin of v2.0-cyanoptera.

The nightly Release workflow on main builds the client from its sdist against the engine's main tip, and since duckdb/duckdb#28417 merged on 2026-11-06 the client no longer compiles there. Runs 1090 and 1093 stop in the sdist build:

src/pyfilesystem.cpp:88:42: error: 'TryGetClientContext' is not a member of 'duckdb::FileOpener'
src/pyconnection.cpp:1560:17: error: 'class duckdb::ClientContext' has no member named 'config'

The first error repeats at the seven other calls in src/pyfilesystem.cpp, and a build that keeps going after the first failure also stops at src/engine/connection.cpp:20, where ClientContext::registered_state is gone. Main's own CI stays green because it builds the pinned engine, 71f47d410f6 of 2026-10-30, which predates the rename. No 2.1.0 development wheel has been published since, and no automatic pin bump opens, because that waits for a green sdist.

external/duckdb moves from 71f47d410f6 to 23221edfd577, the engine's main tip on 2026-11-10. duckdb/duckdb#28314 added Context, which now owns the client configuration, the registered states and the file opener, and #28417 replaced FileOpener::TryGetClientContext with FileOpener::TryGetContext and removed the members of ClientContext that moved. The Python file system now hands PyUtil::QueryGil, the gate that decides whether engine work may enter Python, Context::TryGetClientContext(FileOpener::TryGetContext(opener)), which is the client it got before in every case: every client file opener is built over a client's Context, whose back reference Context::InitializeClient sets first, and a database opener's Context has none. The gate reads that client only after the module's atexit hook ran, on a thread other than the one that ran it, to admit the work of a query that thread started. ConnectionState::Get reads context.GetContext().GetRegisteredState(), as the engine's own code does. The Jupyter progress display goes into the ClientConfig that SetDefaultConfigArguments already holds, which ClientConfig::GetConfig now takes from the Context, where the engine reads it at query start.

The new pin also changes when the engine may run a table function fewer times than the plan names it. Since duckdb/duckdb#28324 (90fd938fd55) and d80575f2645, a table function that declares no stability counts as stable within the query; before, it counted as unknown and was treated as volatile. For python_arrow_scan, python_map_function and pandas_scan, which declared nothing, that reaches one place: the recursive CTE runtime may keep the build side of a join that does not depend on the recursion and reuse it on every iteration. Common subplan sharing and the self-join aggregate rewrite also need the two scans' bind data to compare equal, and theirs never do. So on the new pin a recursive CTE step that joins a registered single use reader reads it once and keeps its rows:

import duckdb
import pyarrow as pa

con = duckdb.connect()
con.register("src", pa.table({"id": [1, 2, 3]}).to_reader())
con.execute("""
    WITH RECURSIVE r(i) AS (SELECT 0 UNION ALL SELECT i + 1 FROM r JOIN src ON src.id = 1 WHERE i < 3)
    SELECT count(*) FROM r
""").fetchall()

On the old pin this raises "This RecordBatchReader has already been consumed", as the same query does through a replacement scan; without a declaration it returns [(4,)]. In the same position a reusable Arrow source is exported once, a map function runs once, and a pandas object column, whose conversion runs Python, is converted once, instead of once per iteration. All three functions now declare FunctionStability::VOLATILE, as the engine does for its own arrow_scan and for every table function registered through the C API, so they behave as before. The Arrow scan reads streams that may be single use or count their exports, map runs Python with side effects, and the pandas scan runs Python when it converts object columns; letting the pandas scan be shared is left for a separate change.

The engine's own file scans do change. read_csv, read_parquet and the read_json family fall under the new default too (duckdb/duckdb#28324), so a recursive step can open a file of a registered fsspec file system once per query instead of once per iteration, depending on how the join is planned. A join on a constant key, in a step that runs four times, opens a three row CSV 4 times on both pins, but the same join through a view opens it 4 times on the old pin and once on the new; an equi join on the recursive column, in a step that runs three times, opens it 3 times on the old pin and once on the new. A Python file system whose content changes between opens can therefore be read once per query; the client adopts that.

Two engine changes reach the predicates the Python Arrow scan offers its source. A folded range with an exclusive bound now prints as two comparisons, ((n > 1) AND (n < 8)), instead of the inclusive (n BETWEEN 1 AND 8) it never was (c66c283da9a, duckdb/duckdb#28448); only the text changes, and four cases follow. 5f52e4ac2b0 (duckdb/duckdb#28576) drops a cast that is invertible for equality from an equality, an inequality, an IN or a NOT IN, and that does change what the source receives. Predicates the walker refused on the old pin, so that DuckDB filtered above the scan, now reach a dataset, a LazyFrame or any other Arrow source as comparisons on the column (columns n INTEGER, ti TINYINT, r REAL):

n::VARCHAR = '5'                ("n" = 5)
n::VARCHAR <> '5'               ("n" != 5)
n::VARCHAR IN ('5', '7')        ("n" IN (5, 7))
n::VARCHAR NOT IN ('5', '7')    (NOT ("n" IN (5, 7)))
n::BIGINT IN (1, 2)             ("n" IN (1, 2))
ti IN (1, 300)                  ("ti" IN (1))
ti NOT IN (1, 300)              (NOT ("ti" IN (1)))
r::DOUBLE = 1.5                 ("r" = 1.5::DOUBLE)
r::DOUBLE <> 1.5                ("r" != 1.5::DOUBLE)

A dataset and a LazyFrame return the same rows as a native table for each. Both apply every form themselves except r::DOUBLE <> 1.5: NaN satisfies it, so they leave it to DuckDB, as they do every float predicate NaN can satisfy. When no value of the column can match, the engine folds the predicate instead: ti = 300 reaches the scan as constant_or_null(false, ti) and n::VARCHAR = '05' as constant_or_null(false, n), and both are refused. The case that pinned a refused cast now uses ti > 300, still offered as (CAST(ti AS INTEGER) > 300).

Two engine performance changes reach the client; both are to be reported to the engine. An aggregate over a Python source that is joined to a larger input runs slower, with identical plans and with or without the declarations. Engine commit 149e6f0de60 (duckdb/duckdb#28389) makes RadixPartitionedHashTable::MaxThreads give each source thread of a hash aggregate at least 256 KiB of data, so a 1,000 group aggregate scans its result on one thread. Python scans carry no distinct count statistics, so the optimizer puts the small aggregate on the probe side of the join, and the probe then runs on one thread too. Over 4 million rows with 8 threads, Release builds on a shared 192 core arm64 Linux machine under load, two alternating rounds:

query	old pin	new pin	new pin, that line reverted
pandas aggregate joined to its own frame, median of 9	41 to 55 ms	67 to 69 ms
Arrow aggregate joined to its own table, median of 9	41 to 55 ms	67 to 68 ms
pandas aggregate joined to a native table, best of 7	47 to 54 ms	60 to 73 ms	45 to 49 ms
native aggregate joined to a native table, best of 7	5.6 to 6.2 ms	5.9 to 6.8 ms	6.2 to 6.4 ms

The build with that line reverted also lacks the three declarations, which these plans do not depend on.

The second is the statement path. con.execute("SELECT 42").fetchall() costs 578,192 to 580,529 user space instructions of the calling thread on the old pin against 629,827 to 640,284 on the new, 8.5 to 10.7 percent or 49,000 to 62,000 more, over two measuring sessions on the same old pin build and two builds of the new pin. A prepared SELECT 42 executes at about the same cost on both (210,679 to 210,770 against 210,133 to 212,482) and parsing adds 1,800 to 2,400, so the growth is in binding, planning and optimizing. executemany pays it once per parameter row: executemany("INSERT INTO t_em VALUES (?, ?, ?)", rows) over 5,000 rows of three columns costs 1,318,128 to 1,318,532 instructions per row on the old pin against 1,376,342 to 1,376,378 on the new, about 58,000 or 4.4 percent more per row, counted the same way, and 2.37 to 2.45 s against 2.48 to 2.61 s of wall time per call. The counts come from an in-process perf_event_open counter of hardware instructions in user space around 20,000 statements with one engine thread, best of three, Release builds; wall time per statement went from 109 to 136 µs to 124 to 139 µs at a machine load of 100 to 155. SELECT 42 is the cheapest statement, so its 49,000 to 62,000 extra instructions are a floor that every execute, sql and relation submission pays. The benchmark nightly shows neither: benchmarks/baseline.json is at engine 71f47d410f, so until it is regenerated the comparison marks every exceeded gate as an engine difference and enforces nothing, and its Callgrind instruction counts cannot see the lost parallelism even after that.

Other engine changes in the range reach users of the client as they are. TIMESTAMP_NS + INTERVAL now returns TIMESTAMP_NS instead of TIMESTAMP, so df() gives datetime64[ns], and TIMESTAMP_NS '2000-01-01' + INTERVAL 300 YEAR raises "Cannot add interval to timestamp_ns" where it returned 2300-01-01. TIMESTAMPTZ_NS + INTERVAL now binds (both 23e628d9cfd, duckdb/duckdb#28279); df() returns such values as naive UTC wall times while fetchall() returns them zoned, a difference that predates this change and is now reachable through more SQL. A UDF registered with side_effects=True and used twice in a range filter is no longer folded into one BETWEEN, so WHERE f(i) > 2 AND f(i) < 8 over 10 rows calls it 17 times instead of 10, with the same result (c7de317279d, duckdb/duckdb#28458).

The tests cover each declaration with a case that fails without it and passes on the old pin: a single use reader registered and joined in a recursive step, a reusable Arrow source that stamps each export and must be seen three times there, a map over a VALUES relation whose function stamps each call, and a pandas object column whose __str__ stamps each conversion, through both registration and a replacement scan. Each form listed above has a case in the predicate oracle, and a dataset and a LazyFrame over rows with NULLs, NaN and the TINYINT limits, which ti IN (127, 128) and ti IN (-128, -129) select, return what a native table returns while the source applies the filter, or, for the float inequality, while DuckDB keeps it; the pushed ones fail on the old pin, where the filter stayed with DuckDB. A shutdown case runs a multi-file read through a registered fsspec file system with eight threads from an atexit handler that runs after the module's own, and asserts that some of the files were opened by threads other than the exit thread. It passes on the old pin and with this change, and fails with "the Python interpreter is shutting down" when the file system hands the gate no client, a wrong way to follow the rename that also compiles. In a child process with stand in IPython and ipywidgets modules, a query with the progress bar's wait time at zero must show one widget. The stand in display keeps what it shows, as the ipywidgets registry keeps every widget in a real kernel; one that drops it hits a separate crash, on released 2.0.0 too, because the widget is released without the GIL. The fast suite passes on the new pin with 16796 passed, 83 skipped and 223 xfailed, against 16752, 83 and 223 on the old pin; the slow exit tests pass too, and the sdist builds and installs against the new pin.

Follow-ups: regenerate benchmarks/baseline.json at the new engine, and report the aggregate thread cap and the planning cost to the engine. The nightly will still end red after this merges, at a later step: runs 1083 and 1086 on main (2026-10-30 and 2026-11-01, the two latest in which the sdist compiled and the test jobs passed) failed at "Upload Artifacts to S3" after the PyPI publish succeeded, because the staging upload fix of #750 is on v2.0-cyanoptera only.

[el-marvin]

Changes requested.

I reviewed d3a03d3 (merge base 5b8e7e4, the first round, so there is no earlier delta). The premise holds: main does not compile against the engine's main branch. I built base 5b8e7e4 (the merge base) and compiled its sources against the new engine headers. It stops at exactly the errors the description quotes: eight times TryGetClientContext is not a member of duckdb::FileOpener in src/pyfilesystem.cpp, ClientContext has no member named config at src/pyconnection.cpp:1560, and registered_state at src/engine/connection.cpp:20. The nightly Release runs 1090 and 1093 fail at the same lines in their sdist build. The approach sits at the right layer. The three source changes follow the engine's own accessors, and the three FunctionStability::VOLATILE declarations restore exactly what the old pin did for undeclared functions.

The code is right. What keeps this from merging is the record it leaves:

• One engine change now pushes predicates into Python Arrow sources that the client used to keep, and nothing in the predicate oracle pins them.
• Two engine performance regressions are described as smaller than they are: the statement path, and an aggregate over a Python source.

I built release builds of the head, of the merge base, and of a variant with every changed area reverted. I probed each claim against them and against the released 2.0.0, and ran the full fast suite on head and on base.

Findings

1. Engine commit 5f52e4ac2b0 makes the client push predicates into Python Arrow sources that it refused on the old pin, and no case in the predicate oracle pins them (tests/fast/arrow/scan/test_predicate_translation.py:224).
Severity: should fix. Category: missing coverage. Evidence: ran. Fix: root-cause. Origin: duckdb-python. Base: introduced.
Add SHAPES cases for the newly offered forms, and replace the description's sentence that what the source receives is unchanged. The evidence is in the inline comment.

2. The statement path costs 8 to 9 percent more instructions on the new pin, all of it in planning, but the description calls SELECT 42 unchanged within noise.
Severity: should fix. Category: performance. Evidence: ran. Fix: containment. Origin: engine. Base: introduced.
Replace the sentence with the figures below, and report the planning cost upstream with this reproducer.

The figures are user space instructions of the calling thread, one thread, Release builds made the same way, best of three runs, base against head:

statement	base	head
con.execute("SELECT 42").fetchall()	585,037	635,466
the same statement prepared once, then EXECUTE p	211,347	210,567
duckdb.extract_statements("SELECT 42")	66,928	69,231

• Over 100,000 statements the count rose from 56.35 G to 60.67 G in one pair of runs and from 56.57 G to 60.32 G in another. Run to run spread is 0.01 percent.
• Wall time went from 103 to 118 µs and from 111 to 131 µs per statement in two alternating rounds.
• Executing a prepared statement costs the same on both pins, and parsing adds 2,300 instructions. So the growth is in binding, planning and optimizing. The engine's profiler lists 47 optimizer passes on head against 45 on base.
• SELECT 42 is the cheapest statement, so 50,000 instructions is a floor that every execute, sql and relation submission pays.

3. The aggregate slowdown the description reports comes from engine commit 149e6f0de60 (#28389), and it reaches more queries than "an aggregate joined back to its own Python source".
Severity: should fix. Category: performance. Evidence: ran. Fix: containment. Origin: engine. Base: introduced.
Name the cause and the wider shape in the description, and report it upstream against #28389.

RadixPartitionedHashTable::MaxThreads (external/duckdb/src/execution/radix_partitioned_hashtable.cpp:939) now gives each hash aggregate source thread at least 256 KiB of data:

const auto threads_for_size = MaxValue<idx_t>(data_size / RadixHTConfig::MINIMUM_SOURCE_BYTES_PER_THREAD, 1);

A 1,000 group aggregate therefore runs its source on one thread. Python scans carry no distinct count statistics, so the optimizer puts the small aggregate on the probe side of the join, and the probe then runs on one thread. My measurements, at 8 threads with 4 million rows and best of seven:

• The description's shape (an aggregate joined back to its own source): Arrow 49 and 43 ms on base against 63 and 65 ms on head.
• The wider shape (a pandas aggregate joined to a native table), SELECT count(*) FROM (SELECT k, sum(v) s FROM df GROUP BY k) a JOIN t1 b USING (k): 53 and 48 ms on base, 68 and 65 ms on head.
• The same query on head with that one line changed back to the scheduler's thread count: 50 and 48 ms.
• With one thread, the user instruction counts of the self join shapes match to 0.03 percent: Arrow 1,157,974,952 against 1,158,304,420. The work is the same; only its parallelism is lost.
• The plans are identical on base, head and the variant without the declarations.

The benchmark nightly cannot catch this even after benchmarks/baseline.json is regenerated, because callgrind instruction counts do not see lost parallelism.

4. The description says a recursive step that joins read_csv over a registered fsspec file system now opens the file once per query, but that holds only when the scan lands on the build side of the join.
Severity: minor. Category: docs. Evidence: ran. Fix: root-cause. Origin: duckdb-python. Base: introduced.
Name the shape, or say "can open".

I counted fs._open calls through the registered memory file system. Base and head agree for the constant key join, the same shape as the description's own Arrow example:

shape	base	head
JOIN read_csv('memory://p/a.csv') s ON s.id = 1	4	4
the same with a glob, or as a cross join	4	4
equi join on the recursive column, ON s.id = r.i	3	1
through a view	4	1

5. The new test comment says the stand-in display keeps its widgets "like IPython's", but IPython's display keeps no reference (tests/fast/api/test_query_progress.py:82).
Severity: minor. Category: docs. Evidence: traced. Fix: root-cause. Origin: duckdb-python. Base: introduced.
Name the ipywidgets registry as the reason a real kernel never drops the last reference.

Smaller things

• The Jupyter display crash the description mentions predates this change. JupyterProgressBarDisplay releases its widget in its destructor, which runs at query end on the fetching thread without the GIL. With a display that does not keep the widget, the process aborts with "Fatal Python error: PyThreadState_Get: the function must be called with the GIL held". That happens on base, on head, and on released 2.0.0 under Python 3.13; my 2.0.0 probe under Python 3.12 did not abort. In a real kernel the ipywidgets registry keeps the widget, so the destructor's reference drop is a decrement without the GIL rather than a deallocation. Two more pre-existing items in the same display:
  • the FloatProgress call wraps its keyword arguments in an extra pair of parentheses (src/jupyter/jupyter_progress_bar_display.cpp:18), a C++ comma expression that passes only style;
  • cursor() (src/pyconnection.cpp:1374) never calls SetDefaultConfigArguments, so a cursor in a kernel gets no widget.
    These belong in their own issue, not here.
• The Arrow scan comment gives two reasons, but a test pins only one. The comment says a source "can hand out a one-shot stream or count its exports". The recursive case pins the first. A reusable exporter that stamps each export, joined in a recursive step and seen three times, would pin the second, and would protect the declaration against a later bind time relaxation for reusable sources.
• Other user visible engine changes in this range that the description does not name. A line each would help the release notes:
  • TIMESTAMP_NS + INTERVAL now returns TIMESTAMP_NS instead of TIMESTAMP, so df() gives datetime64[ns]. TIMESTAMP_NS '2000-01-01' + INTERVAL 300 YEAR now raises "Cannot add interval to timestamp_ns".
  • A UDF registered with side_effects=True and used twice in a range filter is no longer folded into one BETWEEN: WHERE f(i) > 2 AND f(i) < 8 over 10 rows calls it 17 times instead of 10 (engine c7de317279d). The results are the same.
  • TIMESTAMPTZ_NS + INTERVAL now binds. df() already returned naive UTC wall times for TIMESTAMPTZ_NS columns, while fetchall() returns zoned values (ConvertDateTimeTypes in src/pyresult.cpp localizes only TIMESTAMP_TZ). That pre-existing gap is now reachable through more SQL.
• The description is broader than what happens to the Python scans. It lists common subplan sharing and the self join aggregate rewrite among the optimizations an undeclared function would now get. For the three Python scans, only the recursive invariant reuse changes. On the variant without the declarations, the registered single use reader still raises in the union, self join and scalar subquery shapes, because their bind data never compare equal (TableFunctionData::Equals returns false).
• The new pandas recursive test duplicates a helper. It repeats the register or variable branching that _execute at the top of tests/fast/pandas/test_pandas_scan_bind_reads.py already does.
• The Submodule sanity check is red by design, not through this pull request's code. The log says the pin moved without the submodule-bump label, which the job requires: "If that is intended, add the 'submodule-bump' label and re-run". Its reachability check passed, because 23221edfd577 is on duckdb/duckdb main. The workflow reruns on labeled, so a maintainer adds the label.

Outside findings not taken

• Codex, should fix: missing coverage of reusable Arrow exporters in a recursive step. Taken as a smaller thing instead. Reverting the declaration already fails four register cases, and a conditional declaration would be a later design change with its own tests.
• The code review skill: blanket VOLATILE loses recursive reuse for a registered pyarrow Table. Not a regression against the old pin. It is consistent with the engine's own arrow_scan, and the description defers bind time stability on purpose.
• A finder: duckdb.decimal_type(260, 2) now raises an InternalException where it silently built DECIMAL(4,2). Invalid input that was already wrong.
• A finder: HUGEINT to float64 now rounds once. An engine fix.
• A finder: the new sibling scalar aggregate merger ignores stability. Not reachable for the Python scans: their bind data never compare equal, and two scalar subqueries over a registered or replaced single use reader still raise on head.
• A finder: query_progress() now reaches 100. Harmless; the tests assert ranges.
• A finder: rebinding at EXECUTE might change bind counts. It does not: an executemany over a map relation binds 9 times on 2.0.0, base and head.
• The code review skill: a gate on nb::gil_scoped_acquire in JupyterProgressBarDisplay::Update. The skill itself refuted it.

Prerequisites in the engine

RadixPartitionedHashTable::MaxThreads (finding 3).

• The cap comes from 149e6f0de60 (#28389) and is unchanged on engine main.
• #28389 precedes the #28417 rename, so every pin this client can compile against carries it.

The statement path planning cost (finding 2).

• I could not tie it to one commit. The extra instructions sit between parse and execution.

My recommendation is to ship the pin with the corrected record and report both upstream, rather than wait. Main's nightly cannot build its sdist until the client follows #28417, and no 2.1.0 development wheel has been published since 2.1.0.dev2610310630 on 2026-11-01.

What I verified

• Head and range. Head d3a03d3, merge base 5b8e7e4, one commit. The local diff matches gh-read pr diff.
• Builds. All are Release editable builds, Python 3.13, on one arm64 Linux machine:
  • the head;
  • the merge base, at engine 71f47d410f6;
  • a variant of the head with the three declarations removed, ClientOf returning null, and the display_create_func line removed;
  • that variant again, with the engine's thread cap line changed for finding 3.
• Released 2.0.0 from PyPI, under Python 3.12 and 3.13.
• Limited API compile. The six touched C++ files compile with Py_LIMITED_API=0x030C0000 on head.
• Touched tests on head. The five touched fast files gave 1014 passed and 2 xfailed. tests/slow/test_interpreter_shutdown.py gave 37 passed, with the new fsspec case among them.
• Head's test files on the base build. Only the five predicate text cases failed (the expected old texts) and the pre-existing flaky test_query_progress. The new map, pandas, Arrow register, Jupyter widget and fsspec shutdown cases pass on the old pin.
• The variant. Exactly the 9 cases that pin the reverted areas failed, each for its own reason:
  • four test_recursive_cte_raises[*-register]: DID NOT RAISE;
  • two pandas cases and the map case: [(1,)] == [(3,)];
  • the Jupyter widget case: a terminal bar printed;
  • the fsspec shutdown case: "the Python interpreter is shutting down", opens only on the exit thread.
• The premise probes. The recursive single use reader raises on 2.0.0, base and head, and returns [(4,)] on the variant. The map and pandas recursive probes give 3 distinct values on 2.0.0, base and head, and 1 on the variant.
• Full fast suite on head, 8 workers, 600 second timeout: 16761 passed, 83 skipped, 223 xfailed in 689.17s. This matches the description.
• Full fast suite on base: 1 failed, 16751 passed, 83 skipped, 223 xfailed in 830.92s. The failure is TestQueryProgress::test_query_progress ("query finished too quick"), a timing test that passed three times out of three when run alone on base and on head. That makes 16752 on the old pin, as the description says.
• The nine extra tests. Collected ids differ by exactly 9: 13 new ids, 4 old ones replaced. One Jupyter case, one predicate case, four extra recursive reader parameters, two pandas cases and one map case.
• Hot paths. At one thread over a million rows, the pandas scan, the Arrow scan, df(), Arrow output and fetchall() are equal or within noise. User instruction counts: pandas scan 264.05 M both, df() 977 M both. The df() wall time differed by 6 percent in one round and reversed in another.
• Outside reviews. Codex in read only mode at reasoning effort xhigh. The code review skill, which ran at effort max because it did not read the level from its arguments. Two read only finders over the engine range, one for value conversion and one for client services and planning. I verified every finding I kept with a probe or a quoted line.
• CI. Submodule sanity fails for the missing label. Code quality, the sdist build and the seed wheels pass. Several abi3 test jobs were still pending when I wrote this.

What held

• The file system gate. ClientOf gives QueryGil the same client as before in every path:
  • ClientContextFileOpener is built only in Context::InitializeClient (external/duckdb/src/main/context.cpp:249), after the back reference is set, and in http_util.cpp:599 from a client's own context;
  • DatabaseFileOpener holds db.Services(), whose client is null.
    Thirty repeated runs of the fsspec shutdown case under a machine load above 200, and fifteen each pinned to one and to two CPUs, all opened files off the exit thread and returned 32000 rows. The case is not flaky.
• The registered state and progress display moves. ConnectionState::Get and the progress display reach the same objects the engine reads (ClientConfig::GetConfig and client_context_execution.cpp:271).
• The declarations. VOLATILE reproduces the old pin's classification for every consumer. The optimizers and the recursive CTE runtime all test for REPEATABLE, and the old undeclared default was unknown.
• No other Python backed function needs a declaration. Scalar UDFs already map side_effects to VOLATILE or CONSISTENT, there are no Python aggregates, and scan_arrow_ipc reads buffers serialized before binding.
• The two predicate text changes match their engine commits. c66c283da9a prints a folded BETWEEN with an exclusive bound as two comparisons. 5f52e4ac2b0 folds ti = 300 to constant_or_null(false, ti).
• The description's other figures and facts check out:
  • the self join plans are identical across base, head and the variant;
  • benchmarks/baseline.json is at 71f47d410f6, and compare_baseline.py does not enforce when the submodule differs;
  • runs 1083 and 1086 failed only at "Upload Artifacts to S3" after "Publish Artifacts to PyPI" succeeded;
  • #750 merged into v2.0-cyanoptera.

[el-marvin]

Engine commit 5f52e4ac2b0 makes the client push predicates into Python Arrow sources that it refused on the old pin, and no case here pins them.
Severity: should fix. Category: missing coverage. Evidence: ran. Fix: root-cause. Origin: duckdb-python. Base: introduced.

The commit drops a cast that is invertible for equality, so a predicate the walker used to refuse now reaches it as a plain comparison on the column. I ran _duckdb._translate_predicate on both builds over columns n INTEGER, ti TINYINT, r REAL:

predicate	base (engine text, fragment)	head (engine text, fragment)
n::VARCHAR = '5'	(CAST(n AS VARCHAR) = '5'), refused	(n = 5), ("n" = 5)
n::VARCHAR = '05'	(CAST(n AS VARCHAR) = '05'), refused	constant_or_null(false, n), refused
ti IN (1, 300)	(CAST(ti AS INTEGER) IN (1, 300)), refused	(ti IN (1)), ("ti" IN (1))
r::DOUBLE = 1.5	(CAST(r AS DOUBLE) = 1.5), refused	(r = 1.5), ("r" = 1.5::DOUBLE)
n::BIGINT IN (1, 2)	(CAST(n AS BIGINT) IN (1, 2)), refused	(n IN (1, 2)), ("n" IN (1, 2))

The translations are right today: the engine proves each rewrite equivalent, and Constant still requires the constant's type to equal the column's. But a dataset, a LazyFrame or any other Arrow source now evaluates these where DuckDB did before. The description says the opposite: "Two engine changes alter only the engine's text of predicates offered to the Python Arrow scan; what the source receives is unchanged." That holds for c66c283da9a, not for 5f52e4ac2b0. This file is the oracle for what the walker hands a source. It changed two expectations for this commit and adds ti = 300, so the shapes this commit newly offers belong beside them.

Ask: add SHAPES cases for at least n::VARCHAR = '5', ti IN (1, 300) and r::DOUBLE = 1.5, and a REFUSALS case for n::VARCHAR = '05'. Replace the description sentence with what changed.

[el-marvin]

The comment says the stand-in display keeps its widgets "like IPython's", but IPython's display keeps no reference.
Severity: minor. Category: docs. Evidence: traced. Fix: root-cause. Origin: duckdb-python. Base: introduced.

In the tree's venv:

• IPython/core/display_functions.py: display() publishes a mimebundle per object (lines 273 to 284) and returns at most a DisplayHandle(display_id) (line 286); it keeps no reference to the object.
• ipywidgets/widgets/widget.py:46: what keeps a real widget alive is the module level _instances registry, which holds every widget until close().

The distinction matters because the real reason a kernel does not abort is the registry. A display path without ipywidgets would hit the pre-existing abort in JupyterProgressBarDisplay's destructor, which drops the widget without the GIL; I reproduced that on base, head and 2.0.0 under Python 3.13.

Ask: say that ipywidgets keeps every widget it creates, so the display never drops the last reference.

[el-marvin]

This is ready to merge.

I reviewed 8347c0c against merge base 5b8e7e4. It is one commit on top of d3a03d3, which the first round reviewed, and it changes only tests (five files) and the description. The C++ and the engine pin, 23221edfd577, did not move. I rebuilt the head as a Release build and reused the merge base build and the variant from the first round, whose sources are unchanged. Then I:

• reran every earlier probe on the new head;
• ran the new tests on the merge base build and on the variant without the three declarations;
• checked each sentence of the description against base and head;
• ran the full fast suite.

The premise was established in the first round, and the delta does not touch it: main does not compile against the engine's main tip. Nothing here blocks merging. The items below are nits the author may take or leave. Before merging, a maintainer still has to:

• add the submodule-bump label and re-run Submodule sanity, which fails only for the missing label;
• let "Tests and builds on PR" run on 8347c0c, which had not started when I wrote this (it passed on d3a03d3, apart from one cancelled job);
• report the two engine performance changes upstream, as the description says.

Since the last review

1. Pushed predicates (should fix): addressed. The commit adds nine SHAPES cases and one REFUSALS case. Over a dataset and a LazyFrame it adds five PUSHED rows predicates and twelve TINYINT and REAL rows cases. On the merge base build all 32 fail, and no other new case does:
   • the 22 rows cases fail because the filter stays above the scan (where False = all([[]]), or [[]] == [['("ti" IN (1))']]);
   • the 10 text cases fail on the old texts.
     The "unchanged" sentence is replaced by the list of forms. Adding <> and NOT IN, which the engine commit also uncasts, is better than what I asked for. Three kinds of predicate are still unpinned; that is finding 1 below.
2. Statement path (should fix): addressed. "Within noise" is replaced by instruction counts and the method, and the upstream report is named as a follow-up. My counts on the rebuilt head fall outside the stated range; that is finding 2.
3. Aggregate slowdown (should fix): addressed. The description names 149e6f0de60 (#28389), RadixPartitionedHashTable::MaxThreads, the 256 KiB minimum, the probe side placement and the wider shape. I checked the mechanism two more ways:
   • the plan puts the pandas aggregate, estimated at 4,000,000 rows, on the probe side;
   • at 200,000 groups the slowdown disappears.
     The figures are in "What I verified".
4. read_csv wording (minor): addressed. The sentence now says "can" and gives the shapes, and the open counts reproduce exactly. It pairs the view case with the wrong join; that is finding 3.
5. Widget comment (minor): addressed. It now names the ipywidgets registry. That matches ipywidgets/widgets/widget.py in the tree's environment: _instances at line 46, filled in open() at line 545 and emptied in close() at line 565.

The smaller things from the first round:

• Reusable exporter: addressed. test_a_recursive_step_exports_a_repeatable_object_on_every_iteration (tests/fast/arrow/scan/test_arrow_scan_streams.py:977) passes on base. It fails on the variant with [(1,)] == [(3,)].

• The pandas helper: addressed. The test now goes through _execute. Both paths pass on base and fail on the variant with [(1,)] == [(3,)].

• User visible engine changes: addressed. The new paragraph matches what base and head do, Asia/Tokyo session:

  statement	base	head
  TIMESTAMP_NS '2024-01-01 00:00:00.123456789' + INTERVAL 1 HOUR	TIMESTAMP, datetime64[us]	TIMESTAMP_NS, datetime64[ns]
  TIMESTAMP_NS '2000-01-01' + INTERVAL 300 YEAR	2300-01-01	"Out of Range Error: Cannot add interval to timestamp_ns"
  TIMESTAMPTZ_NS value + INTERVAL 1 HOUR	Binder Error	df() 01:00 naive, fetchall() 10:00+09:00
  side_effects=True UDF in f(i) > 2 AND f(i) < 8 over 10 rows	10 calls, [(5,)]	17 calls, [(5,)]

• Narrower optimization paragraph: addressed. I also checked the disjunctive join rewriter. It reads the new default without needing equal bind data (disjunctive_join_rewriter.cpp:100). On the variant without the declarations, t JOIN src ON t.a = src.id OR t.b = src.id behaves as on base:

  • a registered single use reader returns [(6,)];
  • map runs once;
  • the pandas object column converts once per row.
    So "that reaches one place" holds.

• The pre-existing Jupyter display defects: addressed by decision. They predate the pin and go to their own issue.

• Submodule sanity: addressed by decision. A maintainer adds the label.

Findings

All three are minor.

1. Three kinds of predicate that the engine's new cast removal hands a source have no case in the predicate oracle: a FLOAT column cast to DOUBLE under IN and NOT IN, an integer cast to a float type under IN, and a cast over arithmetic on the column (tests/fast/arrow/scan/test_predicate_translation.py:116).
Severity: minor. Category: missing coverage. Evidence: ran. Fix: root-cause. Origin: duckdb-python. Base: widened.
Add the four SHAPES cases and the one REFUSALS case named in the inline comment. Then either list them in the description or say that the list gives examples.

2. The statement path figures in the description are narrower than the counts: on the same two builds I measured 10.5 to 10.7 percent more for SELECT 42, and parsing adds about 2,300 instructions, not 1,900.
Severity: minor. Category: docs. Evidence: ran. Fix: root-cause. Origin: duckdb-python. Base: introduced.
Widen the ranges to what all runs show, and add that executemany pays the cost once per parameter row.

These are user space instructions of the calling thread, from the first round's stmt_split.py: one thread, 20,000 statements, best of three, two alternating rounds.

statement	base	head
con.execute("SELECT 42").fetchall()	578,214 and 578,192	640,284 and 639,094
EXECUTE p of a prepared SELECT 42	210,739 and 210,679	212,333 and 212,379
duckdb.extract_statements("SELECT 42")	66,960 and 66,890	69,254 and 69,207

• The spread is between sessions, not between builds. The base binary is unchanged since the first round, which measured it at 585,037; the description gives 580,473 to 580,529. One binary therefore moves by about 1 percent between sessions. Over the three sessions the new pin costs 8.5 to 10.7 percent more, 49,000 to 62,000 instructions.
• executemany pays the cost per row. Its cost per row rises by about the same 58,000 instructions as one SELECT 42. executemany("INSERT INTO t_em VALUES (?, ?, ?)", rows) over 5,000 rows of three columns costs:
  • 1,318,128 to 1,318,532 instructions per row on base;
  • 1,376,342 to 1,376,378 on head, 4.4 percent and about 58,000 more per row.
    Wall time per 5,000 rows went from 2.37 to 2.45 s to 2.48 to 2.61 s.

3. The read_csv sentence pairs the view case with the equi join and gives that join four iterations, but the view case is the constant key join, and the equi join runs three iterations.
Severity: minor. Category: docs. Evidence: ran. Fix: root-cause. Origin: duckdb-python. Base: introduced.
Pair each count with its join. For example: "A join on a constant key opens a three row CSV once per iteration on both pins, 4 times over four iterations, but through a view the same join opens it once on the new pin; an equi join on the recursive column opens it 3 times over three iterations on the old pin and once on the new."

In the first round's csv_opens.py:

• the view case is JOIN v s ON s.id = 1, with v a view over read_csv('memory://p/a.csv');
• the equi join starts at SELECT 1 with WHERE i < 3, so its step runs for i = 1, 2 and 3.

The fs._open counts, base then head:

shape	base	head
constant key	4	4
constant key, with a glob	4	4
constant key, as a cross join	4	4
equi join on the recursive column	3	1
constant key through a view	4	1

As written, the sentence suggests that a constant key join is never read once per query, and the view case contradicts that.

Smaller things

• Three functions, not two scans. "Common subplan sharing and the self-join aggregate rewrite also need the two scans' bind data to compare equal" holds for all three functions. MapFunctionData (src/map.cpp:25) does not override Equals either, and TableFunctionData::Equals returns false (external/duckdb/src/function/function.cpp:140).
• The native control row has a different plan, not just a different source. The native aggregate in the aggregate table is a Perfect Hash Group By, estimated at about 1,000 rows from the table's statistics and planned on the build side. The cap in RadixPartitionedHashTable therefore never applies to it. The row shows what table statistics buy; it does not show that a native aggregate is immune.
• The inequality counterpart of the new folded case has no case. n::VARCHAR <> '05' and ti NOT IN (300, 400) fold to constant_or_null(true, n) and constant_or_null(true, ti), both refused. A REFUSALS case would pin the <> counterpart of text-cast-equality-no-integer-matches-folded.
• The comment on the float inequality test names the wrong rule. "NaN satisfies it, so it falls under the float rule of NAN_MAY_SATISFY" (tests/fast/arrow/scan/test_arrow_scan_sources.py:672) names the test tuple. The predicate is not in that tuple: the rule that refuses it is nan_may_satisfy in duckdb/_sources/to_arrow.py:93.
• The executemany leak test runs close to its timeout on a loaded machine. tests/fast/test_binding_pressure_leak.py::test_executemany_leak runs 101 cycles of a 5,000 row executemany, so the extra 58,000 instructions per row add up 505,000 times. In my full run, at a machine load of about 350, it hit the 600 second timeout. Run alone, it passed in 556 s. CI passed it on d3a03d3. This is not a defect of this pull request.

Outside findings not taken

• Codex: no case for the NULL a cast IN keeps, or for an IN whose members all drop. The walker refuses every IN with a NULL member (value.IsNull(), src/arrow/predicate_translation.cpp:333), so a source never receives such a list. The engine's own test pins the NULL (test/optimizer/comparison_simplification_equality_casts.test:53). An IN whose members all drop folds to constant_or_null(false, ti), the shape of the existing ti = 300 case.
• Codex: no case for a widening integer cast under = or <>. The old rule already removed those casts: n::BIGINT <> 3 gives ("n" != 3) on both builds, so nothing new reaches a source.
• Codex: the recursive Arrow test does not prove that the declaration is what makes it pass. It does: on the variant without the declarations it fails with [(1,)] == [(3,)].
• Codex: integer to float casts and FLOAT to DOUBLE membership have no case. Taken as finding 1.
• Code review skill: integer to float under IN, the REAL round trip, and a cast over arithmetic have no case. Taken into finding 1. The fold of <> and of an IN whose members all drop is taken as a smaller thing, as is the comment wording.
• Code review skill: IS [NOT] DISTINCT FROM under a removed cast has no case. Refused by ComparisonOperator either way, and the existing distinct-from and not-distinct-from refusals pin that. The folded forms offer the source nothing.
• Code review skill: the TINYINT rows cases compare with a native table that goes through the same rewrite. source.applied == [[applied]] pins exactly which members survive, and the engine's own test pins the rewrite's semantics.
• Code review skill: the recursive tests should assert the export and conversion counts. That would pin engine scheduling: the Arrow source is exported once more, for the last, empty iteration. Three distinct stamps is what the declaration controls.
• Code review skill: no case pins the collation guard. n::VARCHAR COLLATE NOCASE = '5' binds as lower(CAST(n AS VARCHAR)), so it never reaches the guard.
• Code review skill: the narrow pair tests repeat the placement checks, and the Arrow recursive test sits in TestSchemaProbing and runs only the registered path. Style. The declaration belongs to the function, not to the entry path.

Prerequisites in the engine

No finding of this round has its origin in the engine. The two engine performance changes from the first round are now recorded in the description:

• RadixPartitionedHashTable::MaxThreads (external/duckdb/src/execution/radix_partitioned_hashtable.cpp:939) is unchanged on engine main, which is 136 commits past the pin.
• I still could not tie the planning cost to one commit.

My recommendation stands: ship the pin with this record and report both upstream.

What I verified

• Head and range. Head 8347c0c, merge base 5b8e7e4, two commits. The local diff matches gh-read pr diff, apart from hunk headers. The new commit changes only the five test files that the author's reply lists, and nothing beyond them.

• Builds. All are Release editable builds, Python 3.13, on one arm64 Linux machine:

  • the head, rebuilt for this round;
  • the merge base;
  • the first round's variant: the three declarations removed, ClientOf returning null, the display_create_func line removed, and the engine's thread cap line replaced.

• The touched tests on head. The seven touched fast files gave 1376 passed, 6 xfailed.

• The head's four changed test files on the merge base build. 37 failed, 1064 passed, 4 xfailed: the 32 new pushed and text cases, and the 5 text cases the first round already expected to fail there. The two kept float cases, the reusable exporter case and both pandas cases pass.

• The variant with the two new recursive test files. 3 failed, 654 passed, 4 xfailed: the reusable exporter case and the two pandas cases.

• Full fast suite on head, 8 workers, 600 second timeout: 1 failed, 16795 passed, 83 skipped, 223 xfailed in 1195.00s. The failure is test_executemany_leak, "Query interrupted" at the timeout. Run alone, 5 passed in 560.18s. That makes 16796 on the new pin, as the description says. The 35 new test ids account for the difference from the first round's 16761.

• Predicate probes, _duckdb._translate_predicate over 47 cast forms on base and on the new engine, and a rows probe over the new test file's own helpers. Each of the 14 forms I put through the rows probe, the unpinned ones included, returns the same rows as the native table over a dataset and a LazyFrame.

• The aggregate, 4 million rows, 8 threads, best of 7, two alternating rounds. Pandas aggregate joined to a native table:

  groups	base	head	variant without the cap
  1,000	50.1 and 46.6 ms	63.1 and 75.4 ms	47.4 and 43.8 ms
  200,000	55.3 and 57.7 ms	55.1 and 59.9 ms	54.5 and 64.1 ms

  The native aggregate joined to a native table took 5.3 to 6.4 ms on all three builds.

• The statement path and executemany, in-process perf_event_open counters as in finding 2.

• Outside reviews. Codex in read only mode at reasoning effort xhigh, on the delta and the engine commit. The code review skill at effort high, on the range. It reported reviewing this tree and ran the four changed test files once: 1101 passed, 4 xfailed.

• CI. For 8347c0c, Submodule sanity failed only at the label step: its reachability check, which runs first, passed, and the log says "If that is intended, add the 'submodule-bump' label and re-run". The same happened on d3a03d3. "Tests and builds on PR" for 8347c0c was still pending with no jobs when I wrote this. For d3a03d3 every job of it passed, except Test abi3: py3.15-macosx_x86_64, which was cancelled.

What held

• The newly pushed forms are correct, for both sources. NULL rows under NOT IN stay out, because the pyarrow translation wraps membership in pc.if_else(field.is_valid(), ...) (duckdb/_sources/to_arrow.py:89), and the polars one returns NULL for a NULL row too. Both pruning sources refuse the float <> and NOT IN through nan_may_satisfy. The TINYINT limits select what the native table selects.
• No new node kind reaches a source. Every node the cast removal now offers is one that the cast free spelling already produced on base: n = '5' gives ("n" = 5) on both builds, and r IN (1.5, 2.5) gives the same fragment as r::DOUBLE IN (1.5, 2.5) does on head.
• The folded forms never scan the source. For ti::VARCHAR = '05' and ti = 300 the source's stream was never called, as the author's reply says.
• Refusals that stay. A collated VARCHAR cast keeps its cast (n::VARCHAR COLLATE nocase = '5' stays refused), and the IS [NOT] DISTINCT FROM forms stay refused by ComparisonOperator.
• The aggregate mechanism. The cap is the line the description names. The pandas aggregate is planned on the probe side at an estimate of 4,000,000 rows, and the slowdown follows the data size.
• The engine commits named in the description. All seven (23e628d9cfd, c7de317279d, c66c283da9a, 5f52e4ac2b0, 90fd938fd55, d80575f2645 and 149e6f0de60) are in the new pin and not in the old one, with the titles and pull request numbers the description gives.

[el-marvin]

Three kinds of predicate that the engine's new cast removal hands a source have no case in this oracle: a FLOAT column cast to DOUBLE under IN and NOT IN, an integer cast to a float type under IN, and a cast over arithmetic on the column.
Severity: minor. Category: missing coverage. Evidence: ran. Fix: root-cause. Origin: duckdb-python. Base: widened.

CastIsInvertibleForEquality in 5f52e4ac2b0 (external/duckdb/src/planner/expression/bound_cast_expression.cpp:312) accepts four casts: an integer to VARCHAR, an integer to a float type that holds every value of it, an integer to a wider integer, and FLOAT to DOUBLE. The rule applies under = and <> (comparison_simplification.cpp) and under IN (in_clause_simplification_rule.cpp). Under IN it drops each member that does not survive the round trip, and the cast's child need not be a column. The new cases cover:

• the VARCHAR cast under all four operators;
• the wider integer under IN and NOT IN;
• FLOAT to DOUBLE under = and <>.

I ran _duckdb._translate_predicate on the base build and on the new engine (columns n INTEGER, ti TINYINT, r REAL). Each of these is refused on base and reaches the source on the new pin:

predicate	new pin, fragment
r::DOUBLE IN (1.5, 2.5)	("r" IN (1.5::DOUBLE, 2.5::DOUBLE))
r::DOUBLE NOT IN (1.5, 2.5)	(NOT ("r" IN (1.5::DOUBLE, 2.5::DOUBLE)))
r::DOUBLE IN (1.5, 0.1)	("r" IN (1.5::DOUBLE)), the member 0.1 dropped
n IN (1, 2.5::DOUBLE)	("n" IN (1)), the member 2.5 dropped
ti::FLOAT IN (1, 1.5)	("ti" IN (1)), the member 1.5 dropped
(n + 1)::VARCHAR = '6'	("n" = 5)

n::FLOAT = 5 stays (CAST(n AS FLOAT) = 5.0) and is refused on both builds, because FLOAT does not hold every INTEGER.

Nothing here is wrong:

• Over a dataset and a LazyFrame of the new register_narrow_pair rows, each form returns what the native table returns. For the integer forms I used ti in place of n.
• Both pruning sources refuse the float NOT IN, through the negation branch of nan_may_satisfy (duckdb/_sources/to_arrow.py:93).
• The uncast spellings r IN (1.5, 2.5) and r NOT IN (1.5, 2.5) already gave these fragments on base.

Still, the description presents its list as what the engine change hands a source, and this file is the oracle for that.

Ask: add SHAPES cases for:

• r::DOUBLE IN (1.5, 0.1), which also pins the round trip that drops a DOUBLE without an exact REAL value;
• r::DOUBLE NOT IN (1.5, 2.5);
• n IN (1, 2.5::DOUBLE);
• (n + 1)::VARCHAR = '6'.

Add a REFUSALS case for n::FLOAT = 5, which tells the lossless cast from the lossy one. Then either list the new forms in the description or say that the list gives examples.