Repository navigation
Retain compact arrays across numerical SQL programs - #303
Draft
cromwellian wants to merge 5 commits into
Draft
cromwellian wants to merge 5 commits into
cromwellian wants to merge 5 commits into
Conversation
Support prepared SQL gradients, arithmetic chunk-grid counts and direct Arrow registration. Preserve Context metadata through extension installation, expose target partitions and guard physical reuse against value-dependent filters. Co-Authored-By: GPT-6 <noreply@openai.com>
Materialize shared reachable CTEs once, reuse stage programs with fresh physical state, and retain ddx differentiation of original views. Add optional session-local float64 CPU and float32 MLX kernel execution with SQL validation and fallback, using the separately maintained EinFold feature build. Co-Authored-By: GPT-6 <noreply@openai.com>
Build the upstream-compatible feature commit from the contribution fork. Co-Authored-By: GPT-6 <noreply@openai.com>
Lower supported DataFusion and ddx stages through EinFold compact programs, importing live inputs on each ordinary query and expanding coordinates only at Arrow snapshot boundaries. Bound computed intermediates, preserve row/NULL masks and retain conservative SQL fallback. Co-Authored-By: GPT-6 <noreply@openai.com>
Keep the compact executor build reproducible without copying native or array-program sources into this repository. Co-Authored-By: GPT-6 <noreply@openai.com>
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
This was referenced Nov 2, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🛠️ Codex
Add
DenseExecutorto retain compact numerical arrays across shared DataFusion/ddx SQL stages through EinFold's optional array-program module. Axis domains, row-presence and NULL masks remain explicit; joins become gathers/contractions and UNION reductions accumulate gradients. Ordinary calls import current inputs afresh and produce independent Arrow snapshots. Bounded intermediates and strict refusal preserve original SQL fallback before caller-visible mutation.Depends on shared-kernel execution #302 and EinFold compact programs. Incremental review: cromwellian/xarray-sql@codex/shared-kernel-execution...codex/compact-sql-execution . The upstream fork PR's default diff includes predecessors until they merge.
Measured five-step 123.6M/context-16 training improved shared CPU 164.034→4.037 s (40.6×) and shared GPU 163.646→2.434 s (67.2×) per warm loss/update pair. GPU sampled RSS fell 40.0→17.91 GiB on that case. These use batch 1, tanh GELU, SGD and a padded 50,257-class vocabulary; they are separate from the real-character AdamW recipe. All losses/weight updates were checked against JAX outside the API timer. The complete performance/evidence report records first-use and full-context results and failed trials separately.
Full-stack validation: 714 non-integration tests passed on rebuilt ddx/EinFold wheels. Ordinary state still crosses the Arrow boundary; owned GPU state and whole-step compilation follow in the resident-training PR. No model/framework backward pass replaces ddx.