[cromwellian]

🛠️ Codex

Repeated training calls can prepare SQL grad/VJP/JVP statements once and write new Dataset values into existing input buffers. Fine-chunk registration counts partitions arithmetically and registers Arrow batches directly; extension handles retain Dataset metadata and invalidation state. Context(threads=...) exposes target parallel partitions, and physical reuse declines when dynamic filters retain value-dependent bounds.

This independently useful change starts the NanoGPT optimization stack; shared/compact execution, resident training and the evidence report follow separately.

Apple M5 Ultra measurements: one-cell-chunk registration 31.372→0.405 ms (77.44× for that specific case), update-and-sum 2.769→1.027 ms (2.70×), repeated-CTE preparation 1.709→1.307 ms (1.31×). Tiny-model warm time improved 1.11×, but its five-step total regressed due to setup; this does not claim universally faster startup.

Validation: 47 focused AD/preparation tests passed, covering current inputs, changing join-key ranges, old-result snapshots, coordinate rejection and shared Context invalidation. The complete downstream stack passed 714 non-integration tests. Changed Python files pass Ruff.

Review order: this PR → #302 (shared kernels) → #303 (compact arrays) → #305 (resident training). #304 contains the separate consolidated report and evidence.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn