Skip to content

fix(explore): pin the dominant-file aggregation to an edges-first plan (#2571) - #2575

Open
inth3shadows wants to merge 4 commits into
colbymchenry:mainfrom
inth3shadows:fix/2571-dominant-file-plan
Open

inth3shadows wants to merge 4 commits into
colbymchenry:mainfrom
inth3shadows:fix/2571-dominant-file-plan

Conversation

@inth3shadows

@inth3shadows inth3shadows commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #2571.

The dominant-file aggregation joins edges to nodes twice with plain JOINs, so SQLite picks the loop order from sqlite_stat1. runMaintenance samples those statistics (analysis_limit=1000 + PRAGMA optimize), and a sample can under-count nodes per file. The planner then starts from nodes, walks every node in the same file, and probes edges by (source, target). That is quadratic in nodes per file, and on the reporter's repo, with a generated R.java of 7,728 nodes, it cost ~100 s per explore.

Fix:

  • Pinned order. CROSS JOIN, which SQLite documents as fixing the loop order, is the reporter's suggested fix 2. The query now scans edges once, with a lookup by id for each end, whatever the statistics say.
  • Ties by path. ORDER BY edge_count DESC, n.file_path breaks ties by path. The old nodes-first plan happened to emit equal counts in path order and the edges-first plan does not. With a tie at the edge of the 20-row window, that would otherwise change the dominant file itself, so no plan now decides a tie.
  • Exported statement. The SQL is exported as DOMINANT_FILE_SQL so its test can read the plan directly.

Verified on this repo's own index (103,526 edges, 30,684 nodes, Node 22 / SQLite 3.51), with the final SQL:

statistics old query this PR top result
full ANALYZE (idx_nodes_file_path = 30684 26) 145 ms, SCAN n → SEARCH e (source=?) 187 ms, SCAN e → lookups by id name-matcher.ts, 3447 (both)
sampled (30684 6, planted) 3750 ms, SCAN n → SEARCH m USING INDEX idx_nodes_file_path → SEARCH e (source=? AND target=?) 176 ms, same plan as above same

On good statistics the edges-first plan costs about 1.3x the best plan: it does two lookups per edge and needs a temp B-tree for the GROUP BY. That is the price of never taking the quadratic one.

Tests (__tests__/dominant-file-plan.test.ts), on Node 22.22 and 22.23:

  • Plants the sampled sqlite_stat1 rows from the index above. It asserts that a plain JOIN takes the quadratic plan under them, as a control, and that the shipped statement starts with SCAN e. Without the fix: expected 'SCAN n USING INDEX idx_nodes_file_path' to match /^SCAN e\b/.
  • Two files with equal counts resolve to the first by path. With CROSS JOIN and no tiebreaker: expected 'src/zeta.ts' to be 'src/alpha.ts'.

The dominant-file and explore tests pass (62). The full suite on Linux / Node 22 (before the tiebreaker) gave Tests 1 failed | 6784 passed | 360 skipped (7145). The one failure, ui-server-api.test.ts > the engine's own busiest symbol, also fails on upstream/main in this checkout, because it reads a stale .codegraph here. This repo has no CI workflow that runs on pull requests.

Scope: this is the only query in src/ that uses two node aliases' file_path equality as a join key. The other file_path comparisons between node aliases are <> filters over bounded sets. Left as they were:

  • the sampled PRAGMA optimize itself (the reporter's fix 1);
  • the LIMIT 20 taken before generated and test files are filtered out in code. When generated files fill the window, that can understate nextEdgeCount or return null.

Related: this complements #2349 and #2115, which cache the result. Caching helps repeat calls, but the first compute after an index, and every fresh CLI or prompt-hook process before #2349, still pays for the plan. A trial merge is clean with #2115. With #2349 it conflicts only in the queries.ts import block.

Not verified: Node 24's bundled SQLite, Windows, and a cold-cache 1 GB database.

colbymchenry#2571)

The aggregation joined edges to nodes twice with plain JOINs, so the planner
chose the loop order from sqlite_stat1. runMaintenance samples those stats
(analysis_limit=1000 + PRAGMA optimize), and a sample can under-count nodes
per file (idx_nodes_file_path "N 6" where a full ANALYZE says "N 26"). The
planner then started from nodes, walked every node in the same file and
probed edges by (source, target) — quadratic in nodes per file, ~100 s per
explore on a repo with a generated R.java.

CROSS JOIN fixes the order: scan edges once, primary-key lookups for both
ends. Same answer, linear whatever the stats say.
…s test (colbymchenry#2571)

Review round 1 on colbymchenry#2575:
- The edges-first plan emits files with equal counts in another order than
  the old nodes-first plan, which happened to sort them by path. With a tie
  at the edge of the 20-row window that changed the dominant file itself.
  ORDER BY now breaks ties by path, so no plan decides it.
- The test reads the exported statement instead of patching prepare, and
  checks that a plain JOIN takes the quadratic plan under the planted
  statistics, so it keeps telling the two apart.
- Comment: a node lookup goes through the TEXT primary key's index, not the
  rowid.
- CHANGELOG wording.
…Builder's doc comment (colbymchenry#2571)

Review round 2 on colbymchenry#2575: the plan test read the exported constant, so an
inline query in computeDominantFile would have passed; it now also checks
that the constant is what getDominantFile prepares. The plain-JOIN control
warns instead of failing, so another SQLite's planner can't fail the build.
The constant no longer sits between QueryBuilder and its doc comment.
inth3shadows added a commit to inth3shadows/codegraph that referenced this pull request Oct 15, 2026
inth3shadows added a commit to inth3shadows/codegraph that referenced this pull request Oct 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sampled PRAGMA optimize (analysis_limit=1000) leaves stale stats → getDominantFile picks a quadratic plan, explore ~100s on large repos

1 participant