Skip to content

fix(index): a full index ends with complete planner statistics (#2571) - #2590

Open
danusha2345 wants to merge 1 commit into
colbymchenry:mainfrom
danusha2345:fix/2571-full-analyze-after-index
Open

danusha2345 wants to merge 1 commit into
colbymchenry:mainfrom
danusha2345:fix/2571-full-analyze-after-index

Conversation

@danusha2345

Copy link
Copy Markdown
Contributor

Fixes #2571.

What happens

After a full codegraph index, the maintenance step runs PRAGMA optimize under PRAGMA analysis_limit=1000. The statistics that leaves are a sample of the first rows of each index. On a repository with one generated file of thousands of symbols — the reporter's Android R.java with 7,728 nodes, a bundled .d.ts with 3,573 — that sample misses the big file, and the planner takes every file for a few symbols: idx_nodes_file_path comes out as 280356 6 where a full ANALYZE gives 280356 32.

With the underestimate, SQLite plans the dominant-file aggregation behind every codegraph_explore as "for every node, scan all nodes in the same file, then probe edges by (source, target)" — quadratic in the big files. The reporter measured ~100 s per explore on a 280k-node index, 1.5 s with exact statistics. The per-process memo from #2086 does not help a CLI explore or the prompt hook, which start a process per call, and in MCP mode the first explore holds the queue for the rest.

The change

A full index now ends with a full ANALYZE (analysis_limit=0) instead of the sampled optimize. It runs where the maintenance already ran — off-thread, after the WAL valve is quiesced — and the reporter's 1 GB index analyzes in 0.77 s, once, at the end of a run that took minutes. A sync keeps the incremental optimize: it re-analyzes only a table that grew by a lot, so the exact statistics a full index leaves stay in place.

runMaintenance takes { fullAnalyze }; indexAll passes it, sync does not.

Test

__tests__/index-full-analyze-stats.test.ts builds a database of one file with 3,000 symbols beside 3,000 files of one, where the big file sorts after the small ones, and reads sqlite_stat1 for idx_nodes_file_path:

  • after the sync maintenance: 6000 1 — the sample never reaches the big file;
  • after the full-index maintenance: 6000 2 — the exact average.

The second assertion fails on main (runMaintenance takes no options, and the stat stays 6000 1).

🤖 Generated with Claude Code

…mchenry#2571)

The maintenance after a bulk write ran `PRAGMA optimize` under
`analysis_limit=1000`: statistics from the first rows of each index. On a
repository with one generated file of thousands of symbols (an Android
`R.java`, a bundled `.d.ts`) that sample misses the big file, and the planner
takes every file for a few symbols. The dominant-file query behind every
`codegraph_explore` is then planned as a scan of each file's symbols per
symbol — ~100 s per explore on a 280k-node index, where exact statistics give
1.5 s; every CLI `explore` and every prompt hook paid it again, and the first
MCP explore held the queue for the rest.

A full index now ends with a full `ANALYZE` (under a second on a 1 GB index,
once, after a run that took minutes). A sync keeps the incremental
`optimize`, which re-analyzes only a table that grew by a lot.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sampled PRAGMA optimize (analysis_limit=1000) leaves stale stats → getDominantFile picks a quadratic plan, explore ~100s on large repos

1 participant