Repository navigation
fix(explore): pin the dominant-file aggregation to an edges-first plan (#2571) - #2575
Open
inth3shadows wants to merge 4 commits into
Open
inth3shadows wants to merge 4 commits into
inth3shadows wants to merge 4 commits into
Conversation
colbymchenry#2571) The aggregation joined edges to nodes twice with plain JOINs, so the planner chose the loop order from sqlite_stat1. runMaintenance samples those stats (analysis_limit=1000 + PRAGMA optimize), and a sample can under-count nodes per file (idx_nodes_file_path "N 6" where a full ANALYZE says "N 26"). The planner then started from nodes, walked every node in the same file and probed edges by (source, target) — quadratic in nodes per file, ~100 s per explore on a repo with a generated R.java. CROSS JOIN fixes the order: scan edges once, primary-key lookups for both ends. Same answer, linear whatever the stats say.
…s test (colbymchenry#2571) Review round 1 on colbymchenry#2575: - The edges-first plan emits files with equal counts in another order than the old nodes-first plan, which happened to sort them by path. With a tie at the edge of the 20-row window that changed the dominant file itself. ORDER BY now breaks ties by path, so no plan decides it. - The test reads the exported statement instead of patching prepare, and checks that a plain JOIN takes the quadratic plan under the planted statistics, so it keeps telling the two apart. - Comment: a node lookup goes through the TEXT primary key's index, not the rowid. - CHANGELOG wording.
…Builder's doc comment (colbymchenry#2571) Review round 2 on colbymchenry#2575: the plan test read the exported constant, so an inline query in computeDominantFile would have passed; it now also checks that the constant is what getDominantFile prepares. The plain-JOIN control warns instead of failing, so another SQLite's planner can't fail the build. The constant no longer sits between QueryBuilder and its doc comment.
inth3shadows
added a commit
to inth3shadows/codegraph
that referenced
this pull request
Oct 15, 2026
# Conflicts: # CHANGELOG.md
inth3shadows
added a commit
to inth3shadows/codegraph
that referenced
this pull request
Oct 15, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2571.
The dominant-file aggregation joins
edgestonodestwice with plainJOINs, so SQLite picks the loop order fromsqlite_stat1.runMaintenancesamples those statistics (analysis_limit=1000+PRAGMA optimize), and a sample can under-count nodes per file. The planner then starts fromnodes, walks every node in the same file, and probesedgesby(source, target). That is quadratic in nodes per file, and on the reporter's repo, with a generatedR.javaof 7,728 nodes, it cost ~100 s per explore.Fix:
CROSS JOIN, which SQLite documents as fixing the loop order, is the reporter's suggested fix 2. The query now scansedgesonce, with a lookup by id for each end, whatever the statistics say.ORDER BY edge_count DESC, n.file_pathbreaks ties by path. The old nodes-first plan happened to emit equal counts in path order and the edges-first plan does not. With a tie at the edge of the 20-row window, that would otherwise change the dominant file itself, so no plan now decides a tie.DOMINANT_FILE_SQLso its test can read the plan directly.Verified on this repo's own index (103,526 edges, 30,684 nodes, Node 22 / SQLite 3.51), with the final SQL:
ANALYZE(idx_nodes_file_path=30684 26)SCAN n→SEARCH e (source=?)SCAN e→ lookups by idname-matcher.ts, 3447 (both)30684 6, planted)SCAN n→SEARCH m USING INDEX idx_nodes_file_path→SEARCH e (source=? AND target=?)On good statistics the edges-first plan costs about 1.3x the best plan: it does two lookups per edge and needs a temp B-tree for the
GROUP BY. That is the price of never taking the quadratic one.Tests (
__tests__/dominant-file-plan.test.ts), on Node 22.22 and 22.23:sqlite_stat1rows from the index above. It asserts that a plainJOINtakes the quadratic plan under them, as a control, and that the shipped statement starts withSCAN e. Without the fix:expected 'SCAN n USING INDEX idx_nodes_file_path' to match /^SCAN e\b/.CROSS JOINand no tiebreaker:expected 'src/zeta.ts' to be 'src/alpha.ts'.The dominant-file and explore tests pass (62). The full suite on Linux / Node 22 (before the tiebreaker) gave
Tests 1 failed | 6784 passed | 360 skipped (7145). The one failure,ui-server-api.test.ts > the engine's own busiest symbol, also fails onupstream/mainin this checkout, because it reads a stale.codegraphhere. This repo has no CI workflow that runs on pull requests.Scope: this is the only query in
src/that uses two node aliases'file_pathequality as a join key. The otherfile_pathcomparisons between node aliases are<>filters over bounded sets. Left as they were:PRAGMA optimizeitself (the reporter's fix 1);LIMIT 20taken before generated and test files are filtered out in code. When generated files fill the window, that can understatenextEdgeCountor returnnull.Related: this complements #2349 and #2115, which cache the result. Caching helps repeat calls, but the first compute after an index, and every fresh CLI or prompt-hook process before #2349, still pays for the plan. A trial merge is clean with #2115. With #2349 it conflicts only in the
queries.tsimport block.Not verified: Node 24's bundled SQLite, Windows, and a cold-cache 1 GB database.