Repository navigation
fix(index): a full index ends with complete planner statistics (#2571) - #2590
Open
danusha2345 wants to merge 1 commit into
Open
danusha2345 wants to merge 1 commit into
danusha2345 wants to merge 1 commit into
Conversation
…mchenry#2571) The maintenance after a bulk write ran `PRAGMA optimize` under `analysis_limit=1000`: statistics from the first rows of each index. On a repository with one generated file of thousands of symbols (an Android `R.java`, a bundled `.d.ts`) that sample misses the big file, and the planner takes every file for a few symbols. The dominant-file query behind every `codegraph_explore` is then planned as a scan of each file's symbols per symbol — ~100 s per explore on a 280k-node index, where exact statistics give 1.5 s; every CLI `explore` and every prompt hook paid it again, and the first MCP explore held the queue for the rest. A full index now ends with a full `ANALYZE` (under a second on a 1 GB index, once, after a run that took minutes). A sync keeps the incremental `optimize`, which re-analyzes only a table that grew by a lot. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2571.
What happens
After a full
codegraph index, the maintenance step runsPRAGMA optimizeunderPRAGMA analysis_limit=1000. The statistics that leaves are a sample of the first rows of each index. On a repository with one generated file of thousands of symbols — the reporter's AndroidR.javawith 7,728 nodes, a bundled.d.tswith 3,573 — that sample misses the big file, and the planner takes every file for a few symbols:idx_nodes_file_pathcomes out as280356 6where a fullANALYZEgives280356 32.With the underestimate, SQLite plans the dominant-file aggregation behind every
codegraph_exploreas "for every node, scan all nodes in the same file, then probe edges by (source, target)" — quadratic in the big files. The reporter measured ~100 s per explore on a 280k-node index, 1.5 s with exact statistics. The per-process memo from #2086 does not help a CLIexploreor the prompt hook, which start a process per call, and in MCP mode the first explore holds the queue for the rest.The change
A full index now ends with a full
ANALYZE(analysis_limit=0) instead of the sampledoptimize. It runs where the maintenance already ran — off-thread, after the WAL valve is quiesced — and the reporter's 1 GB index analyzes in 0.77 s, once, at the end of a run that took minutes. A sync keeps the incrementaloptimize: it re-analyzes only a table that grew by a lot, so the exact statistics a full index leaves stay in place.runMaintenancetakes{ fullAnalyze };indexAllpasses it,syncdoes not.Test
__tests__/index-full-analyze-stats.test.tsbuilds a database of one file with 3,000 symbols beside 3,000 files of one, where the big file sorts after the small ones, and readssqlite_stat1foridx_nodes_file_path:6000 1— the sample never reaches the big file;6000 2— the exact average.The second assertion fails on
main(runMaintenancetakes no options, and the stat stays6000 1).🤖 Generated with Claude Code