Skip to content

docs: record the cross-language edge check and the test-selection measurement #456

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bompus merged 2 commits into from
Oct 11, 2026
Merged

docs: record the cross-language edge check and the test-selection measurement #456

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7da0726
docs: record the cross-language edge check and the test-selection mea…
bompus Oct 11, 2026
757da5c
docs: state the test-selection unit and drop the stale gap 7 next step
bompus Oct 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 19 additions & 5 deletions docs/design/competitive-landscape-adoption.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ Each row: does it help the agent stop reading, cost, and whether the downstream
|---|---|---|---|---|
| **A. Git-diff-seeded impact** (changed lines to affected symbols, flows, tests) | GitNexus, code-review-graph, codebase-memory-mcp, gortex, trace-mcp | Yes for review and "what does this branch break" prompts: one call instead of `git diff` plus N callers calls. No effect on discovery flows | Low: diff parsing plus line-to-symbol mapping over the existing impact machinery, rendered through explore | trace-mcp's 60-PR run: correctness flat, tokens down 73%. Not in the downstream project discovery bank |
| **B. Edge confidence on every edge** | graphify, code-review-graph, gortex, trace-mcp | Indirectly: an ambiguous label tells the agent which hop to verify instead of re-reading the neighborhood. A confident wrong caller is exactly what sends it to Grep | Low: the resolver knows how each edge was matched; synthesized edges already carry provenance | No |
| **C. Cross-language mis-wire guard** | tree-sitter-analyzer | Yes in polyglot repos: spurious callers inflate blast radius and force verification reads | Very low: callee language must match caller language unless an FFI or route edge explains it | No. Checkable on the golden corpus |
| **C. Cross-language mis-wire guard** | tree-sitter-analyzer | Yes in polyglot repos: spurious callers inflate blast radius and force verification reads | Very low: callee language must match caller language unless an FFI or route edge explains it | Checked 2026-10-11 (§6.3): the kernel already drops a same-name hit across language groups unless there is bridge evidence; a fresh index shows no such call edges. Nothing to build |
| **D. Index artifact export and import with incremental catch-up** | codebase-memory-mcp, Cursor, graphify | Adoption, not sufficiency: 9 of 11 worktrees had no index and an un-indexed root routes the agent to Grep | Moderate: copy the DB, then re-index files whose hash differs. Distinct from the borrowed-index bug the worktree guard exists for | Gap measured in the downstream project; fix not |
| **E. Git-aware staleness** (HEAD-keyed validity, branch-switch bulk resync, catch-up cap, post-commit nudge) | agentmap, tokensave, GitNexus | Yes: every avoided pending-sync banner is an avoided read burst | Low | Not measured; banner exists because the problem was observed |
| **F. Symbol-linked rationale memory** | trace-mcp, gortex, codebase-memory-mcp, graphify, Serena | Only for "why is it like this" questions | Moderate: link sessions hits to symbol ids; design in `sessions-memory-role.md` | the downstream project gates it on a recurring miss; none has fired |
Expand All @@ -57,7 +57,7 @@ Each row: does it help the agent stop reading, cost, and whether the downstream
## 3. Ranked shortlist

1. **Git-diff-seeded impact in explore (A).** Every peer with a review story ships it. CodeGraph has the impact machinery and none of the diff seeding. Add a "what does this diff break" task class to the agent-eval harness before shipping; the downstream project bank is discovery-only and cannot see it.
2. **Edge confidence in explore output plus the cross-language guard (B and C).** Cheapest item on the list and the one aimed most directly at the stated failure trigger. TSA's 1,259 spurious cross-language edges on HuggingFace `tokenizers` is a concrete regression probe. The golden dumps will show how many edges change tier. This fits the binding-model plan, which makes "how was this edge matched" a persisted fact.
2. **Edge confidence in explore output plus the cross-language guard (B and C).** Cheapest item on the list and the one aimed most directly at the stated failure trigger. TSA's 1,259 spurious cross-language edges on HuggingFace `tokenizers` is a concrete regression probe. The golden dumps will show how many edges change tier. This fits the binding-model plan, which makes "how was this edge matched" a persisted fact. Status 2026-10-11: B is done (gap 4 in §6) and C needs no work (§6.3).
3. **Index artifact export and import with catch-up (D).** The largest measured adoption hole. Keep the worktree mismatch guard: this copies then re-indexes, never borrows.
4. **Git-aware staleness (E).** HEAD-keyed validity, branch-switch bulk resync with a catch-up cap, an optional post-commit nudge.
5. **Long-tail language tier (H).** A generic walker turns "not in the index" into a signature-level answer. Interacts with the kernel-only plan's tail-language decisions: a generic kernel walker is a better answer for ArkTS, Terraform, VB.NET and COBOL than dropping them.
Expand Down Expand Up @@ -107,14 +107,14 @@ Where other tools do something this fork does not, or do it better by a publishe
| 1 | Call-edge precision and recall against a compiler, TypeScript first | repowise G4 | On 1.5.0, zod 0.73 precision / 0.37 recall and hono 0.81 precision, against repowise's August 2026 figures of 0.992 / 0.703 (zod) and 0.977 / 0.731 (hono). Repowise's main branch on 2026-10-03 reports 0.995 / 0.796 and 0.976 / 0.806 (its BENCHMARKS.md reran only itself, not the other tools). [precision-replay-2026-09.md](../benchmarks/precision-replay-2026-09.md) shows the fork keeps fewer TypeScript edges and names recall as the larger problem | Measured in section 6.1. The fork reaches 0.981 precision / 0.694 recall on zod and 0.987 / 0.664 on hono. Recall is the gap: zod 0.694 against repowise's 0.703 (August) and 0.796 (2026-10-03), hono 0.664 against 0.731 and 0.806, and 0.684 for 1.5.0 on hono. On hono, 125 of the 237 misses are scoring conventions and 112 are real gaps (section 6.1) |
| 2 | Natural-language file retrieval | repowise ContextBench; section 5 | On Python and Go questions repowise's local search found 0.74 of the needed files (0.88 with a hosted model) against 0.61 for CodeGraph 1.5.0. In the section 5 trial Octocode returned vite's `resolve.ts` at rank 1 for a natural-language query where the fork did not return it. One query, one repository | Measured in section 6.2. The fork finds 0.629 of the needed files on the same 42 questions, about the same as upstream 1.6.2 (0.666) and below repowise's 0.74. On vue core and svelte all builds find under a third of the needed files |
| 3 | Go precision on a large repository | repowise G4 | syft: CodeGraph 1.5.0 0.87 precision against 0.94 to 0.97 for the best arms in August (repowise main reports 0.979 and 0.982 on 2026-10-03); gitleaks within half a point of repowise and cobra about four points behind it (0.93 against 0.97). On interface-dispatch edges CodeGraph found 35 of 3,303 against repowise's 12 | Measured in section 6.1. The fork reaches 0.943 and 0.948 on the two syft cells, inside the 0.94 to 0.97 range of the best arms, and 0.995 on cobra. Syft recall stays low (0.56 and 0.28) |
| 4 | Edge origin and confidence on every edge | repowise, gortex | Strategy step 2. No shipped entry for a persisted origin or confidence shown in explore in `CHANGELOG.md` | Verify what has landed before scheduling |
| 4 | Edge origin and confidence on every edge | repowise, gortex | Strategy step 2. The resolver already stores `resolvedBy` and `confidence` in each edge's metadata, and the trail and Flow sections of explore label name-only hops `[name match, unverified]` | Done: the Relationships list was the one place still unlabelled (PR #455) |
| 5 | Memory floor | repowise | repowise builds its graph in about 75 MB. The fork's index run peaked at 208 MB (flask), 310 MB (gin) and 455 MB (vite) in section 5 | Strategy step 3. Set a target before optimizing |
| 6 | Structural pattern search; index-free live graph | Octocode | Section 5 and candidates S and T | Needs an agent-eval task class first |
| 7 | Test selection precision | repowise: 97.5% on 47 targets | The fork has `codegraph affected` and named-test output in explore; neither has been measured for precision | Unmeasured |
| 7 | Test selection precision | repowise: 97.5% on 47 targets | The fork has `codegraph affected` and named-test output in explore; neither had been measured for precision before §6.3 | Measured on Flask in §6.3. `affected` at its default depth selects almost every test; explore names correct tests but often lists none |
| 8 | Commit-history search, branch-delta indexes, one server for several repositories | Octocode | Commit search has no equivalent here. Sibling-worktree index seeding covers part of branch deltas. Several repositories per server is candidate J | Not scored; no measured miss |
| 9 | Dead code, health score, decision records, generated wiki | repowise | Deferred in the strategy doc; its health score was measured against defect data, the others are capabilities without a measurement | Reopen on a measured miss in our own use |

Items 1 and 3 shared one compiler-oracle experiment (section 6.1) and item 2 used a separate retrieval comparison (section 6.2). Item 7 still needs a corpus with known answers, so it is the next measurement to run. Results go in the survey doc with the date and the build.
Items 1 and 3 shared one compiler-oracle experiment (section 6.1) and item 2 used a separate retrieval comparison (section 6.2). Item 7 was measured on one repository in section 6.3. Results go in the survey doc with the date and the build.

### 6.1 Compiler-oracle replay, 2026-10-10

Expand Down Expand Up @@ -188,3 +188,17 @@ A last probe asked whether Octocode's embeddings recover instances the CodeGraph
| vue core `04c51be7` | 3 | 0 found, 7 files returned | 0 found, 11 returned | 0 found, 13 returned |

Octocode found one of six needed files where the two CodeGraph builds found none. That is two instances chosen because CodeGraph missed them, so it shows embeddings can help on a miss but says nothing about how often. It also cost far more: indexing took 8.1 and 9.0 minutes at about 5.8 GB peak, against 3 to 5 seconds for the CodeGraph builds. A svelte instance with 7,348 files took 28 minutes and 6.6 GB to index, and its search failed on the 500-character limit, so no svelte comparison was made.

### 6.3 Cross-language edges and test selection, 2026-10-11

**Cross-language call edges (candidate C).** On `huggingface/tokenizers` (261 files, Rust core with Python and Node bindings) a fresh index has no call edge between two different languages. The same holds for a fresh full index of this repository (1,685 files, many languages) and after a sync of the files involved. The kernel already refuses a same-name hit across language groups unless there is bridge evidence (`crosses_code_boundary` and `has_bridge_evidence`), so there is nothing to add. A sweep of 52 local indexes found 15 such edges, all in one long-lived index of this repository (name-only matches at confidence 0.3 to 0.5); they do not reproduce on a fresh build and look like leftovers of an older one.

**Test selection (gap 7).** Flask at `d086db8`, Python 3.14, 494 tests. The oracle is line coverage with one context per test; a test counts as exercising a file or function when it ran a line in it. Scored over the 20 source files that any test runs, against the 22 test modules (`tests/test_*.py`; the 494 tests live in them). `affected` returns test-file paths, so the unit is a test module paired with a source file: precision is the share of reported pairs the oracle confirms, recall the share of oracle pairs reported. Nothing is expanded to individual test cases:

| `codegraph affected` depth | Test modules reported, summed over the 20 files | Precision | Recall |
|---|---|---|---|
| 1 | 100 | 0.93 | 0.34 |
| 2 | 345 | 0.71 | 0.90 |
| 5 (default) | 421 | 0.65 | 1.00 |

Running every test has precision 0.62, so the default depth narrows almost nothing on a tightly coupled package. Explore's per-symbol test list was scored on every sixth executed function (57). Twenty-seven were not in its short blast-radius list, so they could not be scored. Of the other 30, twelve list direct tests (17 names, all correct, recall 0.31 where the list was not truncated), six list tests reached through callers (10 names, 7 correct) and twelve print "no tests found" although about nine test modules run each of them (for example `handle_http_exception`, `finalize_request`, `preprocess_request` and `from_mapping`). This is one repository and one language, and line coverage says a test ran the code, not that it would fail if the code broke. The definition behind repowise's 97.5% was not found, so the figures are not comparable with it.