From 761611c18e9b4bec6d3840a202267fa6b13e0b68 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 02:43:58 +0000 Subject: [PATCH 001/259] feat(angular): routes, navigation and the component tree for Angular apps (#2113) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An Angular app had no route nodes, no navigates edges and no link from a template to what it renders — the viewer's Steps tab said "No screens or endpoints" and there was no Screens tab. Routes (frameworks/angular-router.ts): every `Routes` array — typed `Routes` / `Route[]`, `RouterModule.forRoot/forChild`, `provideRouter`, a routes file's `export default [...]` / `as|satisfies Routes` — walked as objects. `component` binds by `references` (route-roots takes a class as a named handler); a lazy `loadComponent` binds to the file's `@Component` class. `children` join their parent's path; a lazily loaded file's routes are put under the path that loads them in postExtract (following an NgModule `loadChildren` to the routing module it imports), idempotently from the in-file path kept in the qualified name. A route with children is a layout: a screen only where no child claims its address, and linked from each screen inside it (`references`, `layout: true`). Paths written as route constants (`internalRoutes.account.path`, through destructured locals and `const x + '/:id'`) and `$localize` strings are read from the constant object. `redirectTo` aliases an address to the screen it sends the user to; a root `**` redirect answers `/` only. A `matcher` route and a `**` catch-all are not screens. Navigation: `router.navigate([...])`, `navigateByUrl`, a guard's `createUrlTree` / `parseUrl` — a command array (non-literal elements are holes), a static string, a route constant, a component property holding one, a function constant called with arguments, `.concat(id)`. A `relativeTo` navigation, a query-only `navigate([])` and a destination made only of holes (it would match any route of its length) are left unresolved. Templates (angular-template-synthesizer.ts): a component's `templateUrl` file or inline `template:` is read at synthesis time. `` by element selector (same app first; two candidates = none) is a `calls` edge from parent to child (`angular-template`), the edge a navigation in a child component rides to its screen. `routerLink` / `[routerLink]` and a `routerLink:` field written in the class (a tab bar's config) are navigates edges. Gated on an `@angular/core` dependency. Measured: - angular-realworld: 10 routes all bound; 11/11 navigation sites accounted for (10 resolved, 1 query-only); 31/31 routerLink sites resolved; 18 renders. - Ghostfolio: 84 routes all bound, 0 unresolved constant paths; 27 navigate resolved, 23 correctly not (8 relative, 9 query-only, 2 computed, 4 aimed at a matcher route); 162 routerLink edges, every one checked against its site; 170 renders. - ngx-admin (NgModule): 55 routes, 47 bound (8 are @nebular/auth's); no navigation calls in the source. - Controls byte-identical vs main (nodes and edges incl. metadata): koel, IceCubesApp, spring-petclinic, proshop. Ghostfolio index time at parity. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + README.md | 1 + __tests__/angular-router.test.ts | 291 ++++++ docs/design/framework-coverage.md | 20 +- docs/viewer-launch-changelog.md | 4 +- .../angular-template-synthesizer.ts | 294 ++++++ src/resolution/callback-synthesizer.ts | 3 + src/resolution/frameworks/angular-router.ts | 940 ++++++++++++++++++ src/resolution/frameworks/index.ts | 3 + src/ui-server/api/route-roots.ts | 22 +- src/ui-server/api/screens.ts | 14 +- 11 files changed, 1586 insertions(+), 7 deletions(-) create mode 100644 __tests__/angular-router.test.ts create mode 100644 src/resolution/angular-template-synthesizer.ts create mode 100644 src/resolution/frameworks/angular-router.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index e02a77eeb1..3858183a9b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - Laravel routes are now named by the path a request takes: a leading `/` is added where the routes file leaves it out, `Route::prefix()` and `Route::group(['prefix' => …])` groups are applied, and routes in `routes/api.php` carry the `/api` prefix Laravel serves them under, read from your `RouteServiceProvider`, `bootstrap/app.php` or any file that mounts a routes file. A front-end `fetch('/api/…')` now connects to the Laravel route that serves it. Re-index Laravel projects after upgrading. - Controllers that Spring or Laravel tests exercise by URL now count as tested. That covers MockMvc's `perform(post("/owners/new"))`, WebTestClient, TestRestTemplate, RestAssured, Laravel's `$this->postJson('api/me')`, Pest's `get('/about')`, and a project's own request helpers built on them. `codegraph_explore` now names the test suite that reaches such an endpoint, where before every one of them looked untested. Re-index Spring and Laravel projects after upgrading. +- Angular apps now have routes and navigation in the graph. Each screen in a `Routes` array is a route named by its full path, including children, lazily loaded `loadChildren` files and NgModules, paths written as route constants or `$localize` strings, and redirects. Each route is linked to its component. `router.navigate([...])`, `navigateByUrl`, a guard's `createUrlTree` and every `routerLink` in a component's template link to the screen they open, and a template's child components (``) are linked to the component that renders them. Questions like "where does this button go" and "what renders this component" now have answers in `codegraph_explore`. Re-index Angular projects after upgrading. - A SwiftUI view, a `@main` app, a view controller and a UIView subclass are now indexed once. Each used to get a second, one-line copy with no callers or calls, which showed up next to the real type in search and `codegraph_explore` answers as a dead end. Re-index Swift projects after upgrading. - A Swift class or struct that declares a protocol composition across lines, like `typealias Client = AutocompleteService.Client` followed by `& PostingService.Client` on the next line, is now indexed whole. That line used to break the parse of the entire enclosing type, so its methods, initializer and properties went missing, and calls into them landed on same-named methods elsewhere. Re-index Swift projects after upgrading. - A Swift reference to a type now links to the type's own declaration, not to a file that extends it. An `extension View { … }` or `extension Text { … }` used to stand in for SwiftUI's type, so every view, every `Text("…")` and every `Color.red` in an app linked to whichever file happened to extend it. Those files topped the most-depended-on lists and their impact reached the whole app. A bare name like `@State`, `@Test` or `Result<…>` no longer links to some other type's nested `State` or `Result`, and a qualified name like `Build.Id` links to the `Id` it names. Methods declared in an extension of an SDK type still resolve on the types that conform to it, and a protocol's methods declared in any of its extensions now connect to each conforming type's own implementation. Re-index Swift projects after upgrading. diff --git a/README.md b/README.md index 17efcad3d5..fc5d341bcc 100644 --- a/README.md +++ b/README.md @@ -346,6 +346,7 @@ These frameworks additionally emit **`navigates`** edges: the function that send | **TanStack Router** | `createFileRoute('/posts/$postId')` (file-based) and `createRoute({ path, getParentRoute })` composed up its parent chain (code-based); `_pathless` segments, `(group)` folders, `__root` and `` layouts are not addresses | `navigate({ to })`, a thrown `redirect({ to })`, `` / `` — where `to` is the route PATTERN and the values ride beside it in `params` | | **Vue Router** / **Nuxt** | `createRouter({ routes: [...] })` with the view each entry names, plus Nuxt `pages/` file-based routes, `server/api/` endpoints and route middleware | `router.push` / `replace`, `$router.push`, Nuxt's `navigateTo`, `` / `` / `` — **by route name** (`push({ name: 'profile' })`) as well as by path | | **SvelteKit** | `src/routes/**/+page.svelte` (`[slug]` → `:slug`, `[[opt]]` → `:opt?`), joined to the `+page.server.js` beside it so a loader's guard belongs to its page | `goto('/x')`, `redirect(status, '/x')` from a load or form action, and the plain `` that is a link in a SvelteKit app | +| **Angular** | `Routes` arrays (`RouterModule.forRoot` / `forChild`, `provideRouter`, a routes file's default export) with `component` or a lazy `loadComponent`; `children` and lazy `loadChildren` (an NgModule's through its routing module) joined into full paths; paths written as route constants or `$localize` strings; a route with children is a layout around the screens inside it | `router.navigate([...])`, `navigateByUrl`, a guard's `createUrlTree` / `parseUrl` — a command array, a route constant, or a component property holding one — `routerLink` / `[routerLink]` in the component's template, and `redirectTo`. Each template's child components (``) are linked to the component that renders them | In a repository holding several apps, each app's routes are matched only against navigation written inside that app. diff --git a/__tests__/angular-router.test.ts b/__tests__/angular-router.test.ts new file mode 100644 index 0000000000..f914dce9d0 --- /dev/null +++ b/__tests__/angular-router.test.ts @@ -0,0 +1,291 @@ +/** + * Angular Router: screens from `Routes` arrays, navigation from command + * arrays and `routerLink`, the component tree from templates. + * + * Validated on angular-realworld-example-app (standalone components, lazy + * `loadComponent`, a `loadChildren` routes file), ngx-admin (NgModule + * `loadChildren` through a routing module, layouts, redirects) and Ghostfolio + * (route constants with `$localize` paths, destructured locals, function + * constants, a `**` redirect home). + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import { commandsHref, localizeDefault, parseAngularRoutes, staticString } from '../src/resolution/frameworks/angular-router'; +import { buildScreens } from '../src/ui-server/api/screens'; + +describe('parseAngularRoutes', () => { + const file = (body: string) => `import { Routes } from '@angular/router';\n${body}`; + + it('reads screens, joining children to their parents, and skips what is not a screen', () => { + const { routes, mounts, redirects } = parseAngularRoutes( + file(`export const routes: Routes = [ + { path: '', loadComponent: () => import('./home/home.component') }, + { path: 'login', component: AuthComponent, canActivate: [guard] }, + { path: 'editor', children: [ + { path: '', loadComponent: () => import('./editor.component').then((m) => m.EditorComponent) }, + { path: ':slug', loadComponent: () => import('./editor.component').then((m) => m.EditorComponent) }, + ] }, + { path: 'profile', loadChildren: () => import('./profile/profile.routes') }, + { path: 'users', matcher: usersMatcher, component: UsersComponent }, + { path: 'old', redirectTo: 'login' }, + { path: '**', component: NotFoundComponent }, +];`) + ); + expect(routes.map((r) => [r.path, r.component])).toEqual([ + ['/', { name: null, spec: './home/home.component' }], + ['/login', { name: 'AuthComponent', spec: null }], + ['/editor', { name: 'EditorComponent', spec: './editor.component' }], + ['/editor/:slug', { name: 'EditorComponent', spec: './editor.component' }], + ]); + expect(mounts.map((m) => [m.prefix, m.spec])).toEqual([['/profile', './profile/profile.routes']]); + expect(redirects).toEqual([{ from: '/old', to: '/login', absolute: false }]); + }); + + it('treats a route with children as a layout: a screen only where no child claims its address', () => { + const { routes } = parseAngularRoutes( + file(`const routes: Routes = [ + { path: ':username', component: ProfileComponent, children: [ + { path: '', loadComponent: () => import('./articles.component') }, + { path: 'favorites', loadComponent: () => import('./favorites.component') }, + ] }, + { path: 'pages', component: PagesComponent, children: [{ path: 'dashboard', component: DashboardComponent }] }, +]; +export default routes;`) + ); + expect(routes.map((r) => [r.path, r.component?.spec ?? r.component?.name, r.layouts.map((l) => l.name)])).toEqual([ + ['/:username', './articles.component', ['ProfileComponent']], + ['/:username/favorites', './favorites.component', ['ProfileComponent']], + ['/pages/dashboard', 'DashboardComponent', ['PagesComponent']], + ['/pages', 'PagesComponent', []], + ]); + }); + + it('finds the arrays forRoot, forChild, provideRouter and a typed default export hold', () => { + for (const body of [ + `RouterModule.forRoot([{ path: 'a', component: A }])`, + `RouterModule.forChild([{ path: 'a', component: A }])`, + `bootstrapApplication(App, { providers: [provideRouter([{ path: 'a', component: A }])] })`, + `export default [{ path: 'a', component: A }] satisfies Routes;`, + `const table = [{ path: 'a', component: A }] as Routes;`, + ]) { + expect(parseAngularRoutes(file(body)).routes.map((r) => r.path)).toEqual(['/a']); + } + // An array that is not a routes array, in a file that imports the router. + expect(parseAngularRoutes(file(`const tabs = [{ path: 'a', component: A }];`)).routes).toEqual([]); + // No router import, no routes. + expect(parseAngularRoutes(`export const routes = [{ path: 'a', component: A }];`).routes).toEqual([]); + }); + + it('keeps a constant path as a placeholder for the cross-file pass, and reads $localize defaults', () => { + const { routes, redirects } = parseAngularRoutes( + file(`export const routes: Routes = [ + { path: internalRoutes.account.path, component: AccountComponent }, + { path: $localize\`:kebab-case@@routes.about:about\`, component: AboutComponent }, + { path: '**', redirectTo: 'home', pathMatch: 'full' }, +];`) + ); + expect(routes.map((r) => r.path)).toEqual(['/{internalRoutes.account.path}', '/about']); + expect(redirects).toEqual([{ from: '/**', to: '/home', absolute: false }]); + }); +}); + +describe('Angular destinations', () => { + it('reads a static string: literals, $localize defaults and + between them', () => { + expect(staticString(`'/' + $localize\`:kebab-case@@routes.about:about\``)).toBe('/about'); + expect(staticString(`'/editor/' + slug`)).toBeNull(); + expect(localizeDefault('$localize`Access`')).toBe('Access'); + }); + + it('reads an absolute command array, holes for what is computed, and nothing for a relative or all-hole one', () => { + expect(commandsHref(`['/article', article.slug]`)?.display).toBe('/article/${…}'); + expect(commandsHref(`['/profile', p.username, 'favorites']`)?.display).toBe('/profile/${…}/favorites'); + expect(commandsHref(`['/']`)?.display).toBe('/'); + expect(commandsHref(`['../', id]`)).toBeNull(); + expect(commandsHref(`['edit']`)).toBeNull(); + // Nothing static: it would match any route of its length. + expect(commandsHref('[`/${a}`, b, c]')).toBeNull(); + }); +}); + +// --------------------------------------------------------------------------- +// End to end +// --------------------------------------------------------------------------- + +const projects: string[] = []; +afterAll(() => { + for (const p of projects.splice(0)) fs.rmSync(p, { recursive: true, force: true }); +}); + +const component = (name: string, selector: string, template: string, body = '') => `import { Component } from '@angular/core'; +import { Router } from '@angular/router'; +@Component({ + selector: '${selector}', + ${template.startsWith('./') ? `templateUrl: '${template}'` : `template: \`${template}\``}, +}) +export class ${name} { + constructor(private readonly router: Router) {} +${body} +} +`; + +describe('an Angular app, indexed', () => { + let cg: CodeGraph; + let root: string; + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-angular-')); + projects.push(root); + const files: Record = { + 'package.json': JSON.stringify({ dependencies: { '@angular/core': '^19.0.0', '@angular/router': '^19.0.0' } }), + 'src/app/routes.constants.ts': `export const internalRoutes = { + account: { path: 'account', routerLink: ['/account'], subRoutes: { access: { path: 'access', routerLink: (id: string) => ['/account', 'access', id] } } }, + about: { path: $localize\`:kebab-case@@routes.about:about\`, routerLink: ['/' + $localize\`:kebab-case@@routes.about:about\`] }, +}; +`, + 'src/app/app.routes.ts': `import { Routes } from '@angular/router'; +import { AuthComponent } from './auth/auth.component'; +import { internalRoutes } from './routes.constants'; +const { about } = internalRoutes; +export const routes: Routes = [ + { path: 'home', loadComponent: () => import('./home/home.component') }, + { path: 'login', component: AuthComponent }, + { path: about.path, loadComponent: () => import('./about/about.component').then((m) => m.AboutComponent) }, + { path: internalRoutes.account.path, loadChildren: () => import('./account/account.routes') }, + { path: 'admin', loadChildren: () => import('./admin/admin.module').then((m) => m.AdminModule) }, + { path: '**', redirectTo: 'home' }, +]; +`, + 'src/app/account/account.routes.ts': `import { Routes } from '@angular/router'; +import { AccountComponent } from './account.component'; +import { AccessComponent } from './access.component'; +import { internalRoutes } from '../routes.constants'; +export default [ + { path: '', component: AccountComponent, children: [ + { path: '', loadComponent: () => import('./overview.component') }, + { path: internalRoutes.account.subRoutes.access.path + '/:id', component: AccessComponent }, + ] }, +] satisfies Routes; +`, + // An NgModule's routes live in the routing module it imports. + 'src/app/admin/admin.module.ts': `import { NgModule } from '@angular/core'; +import { AdminRoutingModule } from './admin-routing.module'; +@NgModule({ imports: [AdminRoutingModule] }) +export class AdminModule {} +`, + 'src/app/admin/admin-routing.module.ts': `import { NgModule } from '@angular/core'; +import { RouterModule, Routes } from '@angular/router'; +import { AdminUsersComponent } from './admin-users.component'; +const routes: Routes = [{ path: '', redirectTo: 'users', pathMatch: 'full' }, { path: 'users', component: AdminUsersComponent }]; +@NgModule({ imports: [RouterModule.forChild(routes)], exports: [RouterModule] }) +export class AdminRoutingModule {} +`, + 'src/app/home/home.component.ts': component('HomeComponent', 'app-home', './home.component.html', ` open(slug: string) { this.router.navigate(['/login']); } + more() { this.router.navigate([], { queryParams: { page: 2 } }); }`).replace('export class', 'export default class'), + 'src/app/home/home.component.html': ` +Sign in +About +Access +Relative +`, + 'src/app/shared/card.component.ts': component('CardComponent', 'app-card', ``, ` save() { this.router.navigate(internalRoutes.account.subRoutes.access.routerLink('me')); }`).replace( + "import { Router } from '@angular/router';", + "import { Router } from '@angular/router';\nimport { internalRoutes } from '../routes.constants';" + ), + 'src/app/auth/auth.component.ts': component('AuthComponent', 'app-auth', `Home`, ` done() { this.router.navigate(['/']); }`), + 'src/app/about/about.component.ts': component('AboutComponent', 'app-about', `

About

`), + // A tab bar built in the class, handed to a tabs component that binds each `tab.routerLink`. + 'src/app/account/account.component.ts': component( + 'AccountComponent', + 'app-account', + `Log out`, + ` tabs = [{ label: 'Access', routerLink: internalRoutes.account.subRoutes.access.routerLink('me') }];` + ).replace("import { Router } from '@angular/router';", "import { Router } from '@angular/router';\nimport { internalRoutes } from '../routes.constants';"), + 'src/app/account/overview.component.ts': component('OverviewComponent', 'app-overview', `

Overview

`).replace('export class', 'export default class'), + 'src/app/account/access.component.ts': component('AccessComponent', 'app-access', `

Access

`), + 'src/app/admin/admin-users.component.ts': component('AdminUsersComponent', 'app-admin-users', `

Users

`), + }; + files['src/app/home/home.component.ts'] = files['src/app/home/home.component.ts']!.replace( + "import { Router } from '@angular/router';", + "import { Router } from '@angular/router';\nimport { internalRoutes } from '../routes.constants';" + ).replace(' constructor(', ' protected readonly routerLinkAbout = internalRoutes.about.routerLink;\n constructor('); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); + }); + afterAll(() => cg.close()); + + const routeOf = (id: string) => cg.getNode(id)?.name; + + it('names each screen by the path a user takes to it, bound to its component', () => { + const routes = cg + .getNodesByKind('route') + .map((r) => { + const bound = cg.getOutgoingEdgesFrom([r.id], ['references']).map((e) => `${(e.metadata as Record)?.layout ? 'layout ' : ''}${cg.getNode(e.target)?.name}`); + return `${r.name} -> ${bound.sort().join(', ')}`; + }) + .sort(); + expect(routes).toEqual([ + '/about -> AboutComponent', + '/account -> OverviewComponent, layout AccountComponent', + '/account/access/:id -> AccessComponent, layout AccountComponent', + '/admin/users -> AdminUsersComponent', + '/home -> HomeComponent', + '/login -> AuthComponent', + ]); + }); + + it('draws navigation from calls and routerLink, through constants, properties and redirects', () => { + const navigations = cg + .getNodesByKind('route') + .flatMap((r) => cg.getIncomingEdgesTo([r.id], ['navigates']).map((e) => `${cg.getNode(e.source)?.name} -> ${r.name} (${(e.metadata as Record).navMethod})`)) + .sort(); + expect(navigations).toEqual([ + // `navigate(['/'])`: `/` matches no route, and the root `**` redirect sends it home. + 'AccountComponent -> /account/access/:id (routerLink)', + 'AccountComponent -> /login (routerLink)', + 'AuthComponent -> /home (routerLink)', + 'HomeComponent -> /about (routerLink)', + 'HomeComponent -> /account/access/:id (routerLink)', + 'HomeComponent -> /login (routerLink)', + 'done -> /home (navigate)', + 'open -> /login (navigate)', + // A route constant that builds its commands: `access.routerLink('me')`. + 'save -> /account/access/:id (navigate)', + ]); + }); + + it('links a template to the components it renders', () => { + const renders = cg + .getNodesByKind('class') + .flatMap((c) => + cg + .getOutgoingEdgesFrom([c.id], ['calls']) + .filter((e) => (e.metadata as Record | undefined)?.synthesizedBy === 'angular-template') + .map((e) => `${c.name} -> ${cg.getNode(e.target)?.name} <${(e.metadata as Record).via}>`) + ); + expect(renders).toEqual(['HomeComponent -> CardComponent ']); + }); + + it("draws the Screens picture: a child component's navigation on its screen, a layout's on each screen inside it", async () => { + const payload = await buildScreens(cg, root); + expect(payload.routed).toBe(true); + const links = payload.links.map((l) => `${routeOf(l.from) ?? cg.getNode(l.from)?.name} -> ${routeOf(l.to)}${l.via.length ? ` via ${l.via.map((v) => v.name).join('>')}` : ''}`).sort(); + expect(links).toEqual([ + '/account -> /account/access/:id', + '/account -> /login', + '/account/access/:id -> /account/access/:id', + '/account/access/:id -> /login', + '/home -> /about', + '/home -> /account/access/:id', + '/home -> /account/access/:id via CardComponent>save', + '/home -> /login', + '/home -> /login via open', + '/login -> /home', + '/login -> /home via done', + ]); + }); +}); diff --git a/docs/design/framework-coverage.md b/docs/design/framework-coverage.md index abf6a2346b..8e71012c9d 100644 --- a/docs/design/framework-coverage.md +++ b/docs/design/framework-coverage.md @@ -1,6 +1,6 @@ # Framework & language coverage — what is done, what is left -**Last verified: 2026-08-29** against the build at that date. Re-verify with the +**Last verified: 2026-08-29** (Angular row: 2026-09-29) against the build at that date. Re-verify with the queries in [Checking this file is still true](#checking-this-file-is-still-true) before trusting a row; this is a snapshot, not a live view. @@ -30,7 +30,7 @@ to write. That is why "add a router" is a small, self-contained job. ## Routers — routes AND navigation (done) -Six. Each reads a literal destination and leaves a computed one, a path no +Seven. Each reads a literal destination and leaves a computed one, a path no route serves, and a conditional whose arms disagree unresolved rather than guessed. @@ -42,14 +42,28 @@ guessed. | TanStack Router | `frameworks/tanstack-router.ts` | `tanstack-router-synthesizer.ts` | `tanstack-router.test.ts` | TanStack examples, fastapi-template frontend | | Vue Router / Nuxt | `frameworks/vue-router.ts` | `vue-router-synthesizer.ts` | `vue-router.test.ts` | vue-realworld (23 edges) | | SvelteKit | `frameworks/sveltekit-router.ts` | `sveltekit-synthesizer.ts` | `sveltekit-router.test.ts` | sveltekit-realworld (31 edges) | +| Angular | `frameworks/angular-router.ts` | `angular-template-synthesizer.ts` | `angular-router.test.ts` | angular-realworld (31 edges, 18 renders), Ghostfolio (189 edges, 170 renders), ngx-admin (routes and renders; its menus are config) | -Shared machinery all six use, in `frameworks/expo-router.ts`: `RouteTable` / +Shared machinery all seven use, in `frameworks/expo-router.ts`: `RouteTable` / `RootedRouteTable`, `routesForFile`, `addRouteTo`, `matchRoute`, `appRootFor`, `parseHrefExpression`, `readHrefViaLocal`, `nthArgumentText`, `readStringAt`, `toHref`. Plus `pageForHref` in `frameworks/nextjs.ts` (framework-agnostic despite where it lives) and the object-literal walker in `frameworks/object-literal.ts`. +Angular is the one whose markup is not indexed: a component's template is a +`templateUrl` file (or an inline `template:` string) read at synthesis time, +which also yields the component tree (`` by element selector) — the +edge a navigation in a child component rides to its screen. A `routerLink:` +field written in a component's class (a tab bar's or a menu's config, bound +in a loop elsewhere) counts as a link from that component. A route with +`children` is a layout: its component carries a `references` edge marked +`layout: true` from each screen nested in it, and `routeLayouts` in +`route-roots.ts` gives Screens every screen a layout serves. Known limits: a +route with a custom `matcher` has no static address, a relative navigation +(`relativeTo`) is left unresolved, and an edit to a template file alone is +picked up at the next sync of any source file (templates are not watched). + --- ## What is left diff --git a/docs/viewer-launch-changelog.md b/docs/viewer-launch-changelog.md index 74687d5929..d4f6de4624 100644 --- a/docs/viewer-launch-changelog.md +++ b/docs/viewer-launch-changelog.md @@ -5,7 +5,7 @@ These describe `codegraph ui` and its screens. They were taken out of `## [Unrel ## Highlights, as written for the viewer launch - **`codegraph ui` — your graph in a browser.** A local, read-only viewer for the project you already indexed: your code with its callers and callees in the margin, a map of the whole repository, and a strip that shows how one symbol reaches another. -- **See your app the way its users meet it.** A Screens tab draws every screen and the navigation between them, for Expo Router, React Router, Next.js, TanStack Router, Vue Router / Nuxt and SvelteKit apps. +- **See your app the way its users meet it.** A Screens tab draws every screen and the navigation between them, for Expo Router, React Router, Next.js, TanStack Router, Vue Router / Nuxt, SvelteKit and Angular apps. - **See what happens from a screen or an endpoint.** A Steps tab draws what one action sets in motion — the handlers it fires, the state it writes, the calls that leave your code, and every way it can answer — with the condition on each arrow. - **APIs too, and across tiers.** Endpoints in Express, NestJS, Fastify, Koa, Hono, FastAPI, Flask, Django, Spring, ASP.NET, Vapor and Gin, with a page's `fetch` following through to the route that serves it, a queued job to its consumer, an event to its handler. - **Read a handler in the order its code runs.** The same picture laid out by when things happen rather than by distance, so a reply sits below the token it carries. Where the code chooses, the condition is said once and each arrow answers it. @@ -15,6 +15,8 @@ These describe `codegraph ui` and its screens. They were taken out of `## [Unrel ## New Features +- **The Screens tab draws Angular apps.** Every screen in an Angular app's routes, with the navigation between them: `router.navigate(…)`, a guard's redirect and each template's `routerLink`, under the condition the code checks first. A button in a child component, like an article's favorite button, is drawn from the screen that renders it. A layout's tabs and buttons are drawn from every screen inside that layout. Re-index Angular projects after upgrading. + - **A big screen's picture stops wrapping into a column.** How wide a screen's lines run before they wrap was worked out with a formula, and the formula was wrong for the way these pictures are actually drawn: a part of a screen spends lines on its own structure — a step that fires things gets a line to itself, and what it fires starts another — so estimating the lines from the boxes alone badly undercounted them, and one screen's 98 boxes wrapped into a 4,356px column. Laying a picture out is cheap and exact, so the widths are now simply tried and the one that comes out closest to the shape of a window is kept. Across one app's 51 screens the tallest picture went from 4,356px to 3,796px, total height fell 8%, and — because a shorter picture is also a picture whose lines have less far to go — lines running over other boxes fell by a third and lines crossing each other went from 13 to 5. - **A screen's picture stops being mostly empty.** The parts of a screen were tiled a row at a time, each row as tall as its tallest member — so one short region beside a tall one left the whole rest of that row blank. Measured across one app: the median screen's canvas was only 55% picture and 45% nothing, and its busiest screen was 44% — 4,860px tall to hold about 2,160px of content, all of which a reader has to scroll through. Each part now goes as high as it can and then as far left as it can, so a short one tucks under another short one instead of waiting for the tall one beside it. Reading order is unchanged: parts are still placed in the screen's own source order, and an earlier one is never pushed below a later one. That screen is now 3,584px instead of 4,860px, and close to square rather than a ribbon. diff --git a/src/resolution/angular-template-synthesizer.ts b/src/resolution/angular-template-synthesizer.ts new file mode 100644 index 0000000000..936f340986 --- /dev/null +++ b/src/resolution/angular-template-synthesizer.ts @@ -0,0 +1,294 @@ +/** + * Angular templates: what a component renders and where it links. + * + * An Angular component's markup is a template — a `templateUrl` file beside + * it, or an inline `template:` string — that the index never parses, so two + * things a reader relies on were missing from the graph: + * + * - **The component tree.** `` in the + * home page's template renders `ArticleListComponent`, and the + * `` inside that renders the button whose click + * navigates. Without the edge, a navigation in a child component reached no + * screen. A `calls` edge from the parent class to the child class + * (`synthesizedBy: 'angular-template'`) stands for the render, the way + * `jsx-render` does for a React child. + * - **`routerLink`.** `routerLink="/login"`, `routerLink="/profile/{{ name }}"`, + * `[routerLink]="['/article', article.slug]"`, and a bound property that + * holds a route constant (`[routerLink]="routerLinkAbout"` with + * `routerLinkAbout = publicRoutes.about.routerLink`) each become a + * `navigates` edge from the component to the route it names. + * + * A child is matched by its element selector (`selector: 'app-article-list'`), + * in the same app first; a selector two components share there is left + * unmatched. A relative link and a destination no route serves draw nothing. + */ + +import type { Edge, Node } from '../types'; +import type { ResolutionContext } from './types'; +import type { MaybeYield } from './cooperative-yield'; +import { isTestPath } from '../search/query-utils'; +import { stripCommentsForRegex } from './strip-comments'; +import { matchBracket, readFields, skipString } from './frameworks/object-literal'; +import { dependsOn } from './frameworks/package-deps'; +import { appRootFor, HOLE, toHref, type HrefLiteral } from './frameworks/expo-router'; +import { destinationsForHref } from './frameworks/nextjs'; +import { angularDestination, angularRouteTable, angularRoutesFor, namesSomewhere, staticString, templatePathFor } from './frameworks/angular-router'; + +interface AngularComponent { + node: Node; + file: string; + selectors: string[]; + template: { file: string; text: string; firstLine: number; inline: boolean } | null; +} + +/** Links a single component may carry before it is a navigation menu rather than a decision. */ +const MAX_LINKS_PER_COMPONENT = 24; +/** Children a single template may render before the rest are left out. */ +const MAX_CHILDREN_PER_COMPONENT = 40; + +/** An element selector: `app-article-list`, not `[appDirective]` or `button[app-x]`. */ +const ELEMENT_SELECTOR = /^[a-zA-Z][\w-]*$/; + +const lineOfOffset = (text: string, at: number): number => { + let line = 1; + for (let i = 0; i < at && i < text.length; i++) if (text.charCodeAt(i) === 10) line++; + return line; +}; + +/** Every `@Component` class a file declares, with its selectors and template. */ +function componentsIn(file: string, content: string, nodes: readonly Node[], ctx: ResolutionContext): AngularComponent[] { + const out: AngularComponent[] = []; + const safe = stripCommentsForRegex(content, 'typescript'); + const decorator = /@Component\s*\(/g; + let m: RegExpExecArray | null; + while ((m = decorator.exec(safe)) !== null) { + const paren = m.index + m[0].length - 1; + const close = matchBracket(safe, paren); + if (close < 0) continue; + decorator.lastIndex = close; + const open = safe.indexOf('{', paren); + if (open < 0 || open > close) continue; + const objEnd = matchBracket(safe, open); + if (objEnd < 0) continue; + const cls = /^\s*(?:export\s+)?(?:default\s+)?(?:abstract\s+)?class\s+([A-Za-z_$][\w$]*)/.exec(safe.slice(close + 1, close + 400)); + if (!cls) continue; + const node = nodes.find((n) => n.kind === 'class' && n.name === cls[1]); + if (!node) continue; + const fields = readFields(safe, open, objEnd); + const selectorText = fields.get('selector')?.text; + const selector = selectorText ? staticString(selectorText) : null; + const selectors = (selector ?? '') + .split(',') + .map((s) => s.trim()) + .filter((s) => ELEMENT_SELECTOR.test(s)); + + let template: AngularComponent['template'] = null; + const url = fields.get('templateUrl')?.text; + const templateUrl = url ? staticString(url) : null; + if (templateUrl) { + const templateFile = templatePathFor(file, templateUrl); + const text = ctx.readFile(templateFile); + if (text) template = { file: templateFile, text, firstLine: 1, inline: false }; + } else { + const inline = fields.get('template'); + if (inline) { + // The template string's body, from the original source (stripping + // comments must not touch a template's text), at its own offset. + const quoteAt = content.indexOf(inline.text.trim()[0] ?? '`', inline.at + 'template'.length); + const end = quoteAt < 0 ? -1 : skipString(content, quoteAt); + if (end > quoteAt) { + template = { file, text: content.slice(quoteAt + 1, end), firstLine: lineOfOffset(content, quoteAt + 1), inline: true }; + } + } + } + out.push({ node, file, selectors, template }); + } + return out; +} + +/** Per context, renewed with the route list (whose identity changes when the resolver's caches do). */ +const componentIndexes = new WeakMap(); + +/** Every component in the project, read once per resolution state. */ +export function angularComponents(ctx: ResolutionContext): AngularComponent[] { + const source = ctx.getNodesByKind('route'); + const hit = componentIndexes.get(ctx); + if (hit && hit.source === source) return hit.components; + const components: AngularComponent[] = []; + for (const file of ctx.getAllFiles()) { + if (!/\.[cm]?ts$/.test(file) || file.endsWith('.d.ts') || isTestPath(file)) continue; + if (!(ctx.fileContains?.(file, '@Component') ?? ctx.readFile(file)?.includes('@Component'))) continue; + const content = ctx.readFile(file); + if (!content) continue; + components.push(...componentsIn(file, content, ctx.getNodesInFile(file), ctx)); + } + componentIndexes.set(ctx, { source, components }); + return components; +} + +// ============================================================================= +// Reading a template +// ============================================================================= + +/** `routerLink="…"` (plain) and `[routerLink]="…"` (bound), either quote. `routerLinkActive` is neither. */ +const ROUTER_LINK = /(\[routerLink\]|\brouterLink)\s*=\s*(?:"([^"]*)"|'([^']*)')/g; +/** A `routerLink:` field of an object literal written in a class. */ +const ROUTER_LINK_FIELD = /(?])/g; + +/** The destination a plain `routerLink` names: `/profile/{{ user.username }}` is `/profile/${…}`. */ +function plainHref(value: string): HrefLiteral | null { + const withHoles = value.trim().replace(/\{\{[\s\S]*?\}\}/g, HOLE); + return withHoles.startsWith('/') ? namesSomewhere(toHref(withHoles)) : null; +} + +// ============================================================================= +// The pass +// ============================================================================= + +export async function angularTemplateEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise { + if (!dependsOn(ctx, '@angular/core')) return []; + const components = angularComponents(ctx); + if (components.length === 0) return []; + + // Element selector → the components that answer to it, per app. + const bySelector = new Map(); + for (const c of components) { + for (const sel of c.selectors) { + const list = bySelector.get(sel); + if (list) list.push(c); + else bySelector.set(sel, [c]); + } + } + const childFor = (tag: string, parent: AngularComponent): AngularComponent | null => { + const all = bySelector.get(tag); + if (!all || all.length === 0) return null; + if (all.length === 1) return all[0]!; + const root = appRootFor(parent.file); + const near = all.filter((c) => c.file.startsWith(root)); + return near.length === 1 ? near[0]! : null; + }; + + const table = angularRouteTable(ctx); + const edges: Edge[] = []; + const seen = new Set(); + let scanned = 0; + for (const component of components) { + if ((++scanned & 31) === 0) await onYield(); + const template = component.template; + if (!template) continue; + const text = template.text; + // Where an edge is written: the tag's own line for an inline template; + // for a template file, the class — the file the edge's source is in. + const siteLine = (at: number) => (template.inline ? template.firstLine + lineOfOffset(text, at) - 1 : component.node.startLine); + const registeredAt = (at: number) => `${template.file}:${template.firstLine + lineOfOffset(text, at) - 1}`; + + let children = 0; + TAG.lastIndex = 0; + let t: RegExpExecArray | null; + while ((t = TAG.exec(text)) !== null && children < MAX_CHILDREN_PER_COMPONENT) { + const child = childFor(t[1]!, component); + if (!child || child.node.id === component.node.id) continue; + const key = `${component.node.id}>${child.node.id}`; + if (seen.has(key)) continue; + seen.add(key); + children++; + edges.push({ + source: component.node.id, + target: child.node.id, + kind: 'calls', + line: siteLine(t.index), + provenance: 'heuristic', + metadata: { synthesizedBy: 'angular-template', via: t[1]!, registeredAt: registeredAt(t.index) }, + }); + } + + const routes = angularRoutesFor(table, component.file); + if (!routes || routes.exact.size === 0) continue; + let links = 0; + // A tab bar or menu built in the class — `this.tabs = [{ label, routerLink: + // internalRoutes.home.subRoutes.summary.routerLink }]`, handed to a + // `` whose template binds `[routerLink]="tab.routerLink"` + // in a loop. The destination is the field this class wrote. + const source = ctx.readFile(component.file); + if (source) { + const lines = source.split('\n'); + const body = lines.slice(component.node.startLine - 1, component.node.endLine).join('\n'); + ROUTER_LINK_FIELD.lastIndex = 0; + let f: RegExpExecArray | null; + while ((f = ROUTER_LINK_FIELD.exec(body)) !== null && links < MAX_LINKS_PER_COMPONENT) { + const value = fieldValue(body, f.index + f[0].length); + const href = value ? angularDestination(value, component.file, component.node, ctx) : null; + if (!href || !href.path.startsWith('/')) continue; + const line = component.node.startLine + lineOfOffset(body, f.index) - 1; + for (const dest of destinationsForHref(href, routes)) { + const key = `${component.node.id}>${dest.node.id}`; + if (seen.has(key)) continue; + seen.add(key); + links++; + edges.push({ + source: component.node.id, + target: dest.node.id, + kind: 'navigates', + line, + provenance: 'heuristic', + metadata: { synthesizedBy: 'angular-router-link', href: dest.href.display, navMethod: 'routerLink', registeredAt: `${component.file}:${line}` }, + }); + } + } + } + ROUTER_LINK.lastIndex = 0; + let r: RegExpExecArray | null; + while ((r = ROUTER_LINK.exec(text)) !== null && links < MAX_LINKS_PER_COMPONENT) { + const value = r[2] ?? r[3] ?? ''; + const href = r[1] === '[routerLink]' ? angularDestination(value, component.file, component.node, ctx) : plainHref(value); + if (!href || !href.path.startsWith('/')) continue; + for (const dest of destinationsForHref(href, routes)) { + const key = `${component.node.id}>${dest.node.id}`; + if (seen.has(key)) continue; + seen.add(key); + links++; + edges.push({ + source: component.node.id, + target: dest.node.id, + kind: 'navigates', + line: siteLine(r.index), + provenance: 'heuristic', + metadata: { + synthesizedBy: 'angular-router-link', + href: dest.href.display, + navMethod: 'routerLink', + registeredAt: registeredAt(r.index), + template: true, + }, + }); + } + } + } + return edges; +} diff --git a/src/resolution/callback-synthesizer.ts b/src/resolution/callback-synthesizer.ts index ae619d01f1..fbd34b3303 100644 --- a/src/resolution/callback-synthesizer.ts +++ b/src/resolution/callback-synthesizer.ts @@ -33,6 +33,7 @@ import { nextLinkEdges } from './next-router-synthesizer'; import { reactRouterLinkEdges } from './react-router-synthesizer'; import { tanstackLinkEdges } from './tanstack-router-synthesizer'; import { vueRouterLinkEdges } from './vue-router-synthesizer'; +import { angularTemplateEdges } from './angular-template-synthesizer'; import { svelteKitLinkEdges, svelteKitPageComponentEdges } from './sveltekit-synthesizer'; import { createYielder, type MaybeYield } from './cooperative-yield'; import { crossTierEdges, hasCrossTierPattern, hasTestRequestPattern, testRequestEdges } from './tier-synthesizer'; @@ -3757,6 +3758,8 @@ export const SYNTH_PASSES: SynthPassDef[] = [ { name: 'reactRouterLinkEdges', gate: (has) => has(...JS_FAMILY), run: (_q, c, y) => reactRouterLinkEdges(c, y) }, { name: 'tanstackLinkEdges', gate: (has) => has(...JS_FAMILY), run: (_q, c, y) => tanstackLinkEdges(c, y) }, { name: 'vueRouterLinkEdges', gate: (has) => has('vue', ...JS_FAMILY), run: (_q, c, y) => vueRouterLinkEdges(c, y) }, + // An Angular template: the child components it renders and its `routerLink`s. + { name: 'angularTemplateEdges', gate: (has) => has('typescript'), run: (_q, c, y) => angularTemplateEdges(c, y) }, { name: 'svelteKitPageEdges', gate: (has) => has('svelte'), run: (_q, c, y) => svelteKitPageComponentEdges(c, y) }, { name: 'svelteKitLinkEdges', gate: (has) => has('svelte'), run: (_q, c, y) => svelteKitLinkEdges(c, y) }, { name: 'nixOptionEdges', gate: (has) => has('nix'), run: (q, _c, y) => nixOptionPathEdges(q, y) }, diff --git a/src/resolution/frameworks/angular-router.ts b/src/resolution/frameworks/angular-router.ts new file mode 100644 index 0000000000..02f7dc9d81 --- /dev/null +++ b/src/resolution/frameworks/angular-router.ts @@ -0,0 +1,940 @@ +/** + * Angular Router — routes declared in `Routes` arrays, navigation written as + * command arrays. + * + * export const routes: Routes = [ + * { path: 'login', component: AuthComponent }, + * { path: 'article/:slug', loadComponent: () => import('./article.component') }, + * { path: 'editor', children: [{ path: ':slug', loadComponent: … }] }, + * { path: 'profile', loadChildren: () => import('./profile/profile.routes') }, + * ]; + * + * `extract()` reads every such array — typed `Routes` / `Route[]`, passed to + * `RouterModule.forRoot/forChild(…)` or `provideRouter(…)`, or a routes + * file's `export default [...]` — into one `route` node per screen, named by + * its path the way every other framework's routes are. Angular paths are + * relative: a child's path joins its parent's, and a `loadChildren` file's + * routes sit under the path that lazy-loads them, which only a cross-file + * pass can know — so `extract()` names a routes file's routes as if it were + * mounted at `/`, and `postExtract()` puts each under its mount (following an + * NgModule `loadChildren` through to the routing module it imports). + * + * A route with `children` is a layout, not a screen: its component renders + * an outlet the children fill, and the `''` child is the screen at its + * address. It is a screen only when no child claims that address. A + * `redirectTo` entry and a `**` catch-all are not screens either. + * + * A path written as a constant (`path: internalRoutes.account.path`) is read + * from the object the constant names, `$localize` defaults included, in the + * same cross-file pass. + * + * A route binds to its component with a `references` edge — `route-roots.ts` + * takes a class as the handler a resolver named. A lazy `loadComponent` + * names a file, and its component is the `@Component` class that file + * exports. + * + * **Navigation is a command array.** `router.navigate(['/article', slug])` + * is `/article/${…}`; `navigateByUrl('/login')`, a guard's + * `router.createUrlTree(['/login'])` and `parseUrl('/login')` name a path the + * way every other router's calls do. A relative navigation (`relativeTo`) is + * left unresolved rather than guessed. `routerLink` is markup, read by + * `angular-template-synthesizer.ts`. + */ + +import * as path from 'path'; +import type { Node } from '../../types'; +import type { FrameworkExtractionResult, FrameworkResolver, ResolutionContext, ResolvedRef, UnresolvedRef } from '../types'; +import { resolveImportPath } from '../import-resolver'; +import { stripCommentsForRegex } from '../strip-comments'; +import { matchBracket, readFields, skipString, topLevelObjects } from './object-literal'; +import { dependsOn } from './package-deps'; +import { + addRouteTo, + appRootFor, + firstArgumentText, + HOLE, + toHref, + nthArgumentText, + parseHrefExpression, + readStringAt, + routesForFile, + type HrefLiteral, + type RootedRouteTable, + type RouteTable, +} from './expo-router'; +import { destinationsForHref } from './nextjs'; + +// ============================================================================= +// Reading a routes array +// ============================================================================= + +/** The component a route names: a class in scope, or a lazily imported file's (`name` null = its default export). */ +export type AngularComponentRef = { name: string; spec: null } | { name: string | null; spec: string }; + +export interface AngularRoute { + /** `/editor/:slug` within this file; a constant path segment is kept as `{expr}` until the cross-file pass reads it. */ + path: string; + line: number; + component: AngularComponentRef | null; + /** The layouts the screen renders inside, outermost first: each ancestor route's component, around its ``. */ + layouts: AngularComponentRef[]; +} + +export interface AngularRedirect { + /** `/` within this file. */ + from: string; + /** Where it sends the user, within this file — or, when `absolute`, in the app: `/home`. */ + to: string; + absolute: boolean; +} + +export interface AngularMount { + /** Where the lazily loaded routes sit within this file: `/profile`. */ + prefix: string; + /** The `loadChildren` import: `./profile/profile.routes`. */ + spec: string; + line: number; +} + +/** A file that could hold a routes array — the cheap gate before scanning. */ +const ROUTER_IMPORT = /['"]@angular\/router['"]/; + +/** + * Where a routes array opens: typed `Routes` / `Route[]`, handed to + * `RouterModule.forRoot/forChild` or `provideRouter`, or a routes file's + * default export. `children: [` is reached by walking an entry, never here. + */ +const ARRAY_OPENERS = + /(?::\s*(?:Routes|Route\s*\[\s*\])\s*=\s*|\bRouterModule\s*\.\s*for(?:Root|Child)\s*\(\s*|\bprovideRouter\s*\(\s*|\bexport\s+default\s+(?:<\s*Routes\s*>\s*)?|\b(?:const|let)\s+[A-Za-z_$][\w$]*\s*=\s*)\[/g; + +/** The fields that make an object a route rather than any object in an array. */ +const ROUTE_KEYS = ['component', 'loadComponent', 'children', 'loadChildren', 'redirectTo'] as const; + +/** A path value the pass can name later: `internalRoutes.account.path`. */ +const CONSTANT_PATH = /^[A-Za-z_$][\w$]*(?:\s*\.\s*[A-Za-z_$][\w$]*)+$/; + +/** A `$localize` tagged template's default text: `` $localize`:kebab-case@@routes.about:about` `` is `about`. */ +export function localizeDefault(text: string): string | null { + const m = /^\$localize\s*`((?:[^`\\$]|\\.)*)`$/.exec(text.trim()); + if (!m) return null; + const body = m[1]!; + // `:meaning|description@@id:text` — the metadata block is the leading `:…:`. + const meta = /^:[^:]*:/.exec(body); + return meta ? body.slice(meta[0].length) : body; +} + +/** + * A string written statically: a literal, a `$localize` default, or `+` + * between those — `'/' + $localize`:…:about``, as Ghostfolio builds its + * links. Null when any part is computed. + */ +export function staticString(text: string): string | null { + const parts: string[] = []; + let rest = text.trim(); + while (rest.length > 0) { + let part: string | null = null; + let used = 0; + const q = rest[0]; + if (q === '"' || q === "'") { + part = readStringAt(rest, 0); + used = part === null ? 0 : skipString(rest, 0) + 1; + } else if (rest.startsWith('$localize')) { + const tick = rest.indexOf('`'); + const end = tick < 0 ? -1 : skipString(rest, tick); + if (end > 0) { + part = localizeDefault(rest.slice(0, end + 1)); + used = end + 1; + } + } else if (q === '`') { + const end = skipString(rest, 0); + const body = end > 0 ? rest.slice(1, end) : ''; + if (end > 0 && !body.includes('${')) { + part = body; + used = end + 1; + } + } + if (part === null || used === 0) return null; + parts.push(part); + rest = rest.slice(used).trim(); + if (rest.length === 0) break; + if (rest[0] !== '+') return null; + rest = rest.slice(1).trim(); + } + return parts.length > 0 ? parts.join('') : null; +} + +/** A route's own path segment(s): a string, a `$localize` default, or a constant kept as `{expr}`; null when it cannot be named. */ +function pathSegments(text: string): string[] | null { + const trimmed = text.trim(); + const value = staticString(trimmed); + if (value !== null) { + if (value.includes('**')) return null; + return value.split('/').filter((s) => s.length > 0); + } + if (CONSTANT_PATH.test(trimmed)) return [`{${trimmed.replace(/\s+/g, '')}}`]; + // `internalRoutes.account.subRoutes.access.path + '/:id'`: each part a + // string or a constant, the constant a whole segment of its own. + if (trimmed.includes('+')) { + const joined: string[] = []; + for (const part of splitPlus(trimmed)) { + const literal = staticString(part); + if (literal !== null) joined.push(literal); + else if (CONSTANT_PATH.test(part.trim())) joined.push(`{${part.trim().replace(/\s+/g, '')}}`); + else return null; + } + const value = joined.join(''); + return value.includes('**') ? null : value.split('/').filter((seg) => seg.length > 0); + } + return null; +} + +/** An expression's `+` operands at depth 0, strings and brackets stepped over. */ +function splitPlus(text: string): string[] { + const parts: string[] = []; + let start = 0; + for (let i = 0; i < text.length; i++) { + const ch = text[i]!; + if (ch === '"' || ch === "'" || ch === '`') { + const end = skipString(text, i); + if (end < 0) return [text]; + i = end; + } else if (ch === '(' || ch === '[' || ch === '{') { + const end = matchBracket(text, i); + if (end < 0) return [text]; + i = end; + } else if (ch === '+') { + parts.push(text.slice(start, i)); + start = i + 1; + } + } + parts.push(text.slice(start)); + return parts; +} + +/** The component a `component:` / `loadComponent:` value names. */ +function componentRef(field: 'component' | 'loadComponent', text: string): AngularComponentRef | null { + if (field === 'component') { + const ident = /^\s*([A-Z][\w$]*)\s*$/.exec(text); + return ident ? { name: ident[1]!, spec: null } : null; + } + const lazy = lazyImport(text); + return lazy ? { name: lazy.member, spec: lazy.spec } : null; +} + +/** `() => import('./x')` → `./x`; `.then(m => m.X)` / `.then((c) => c.X)` names the member. */ +function lazyImport(text: string): { spec: string; member: string | null } | null { + const imp = /\bimport\s*\(\s*(['"`])([^'"`]+)\1\s*\)/.exec(text); + if (!imp) return null; + const then = /\.then\s*\(\s*\(?\s*([A-Za-z_$][\w$]*)\s*\)?\s*=>\s*\(?\s*\1\s*\.\s*([A-Za-z_$][\w$]*)/.exec(text.slice(imp.index + imp[0].length)); + return { spec: imp[2]!, member: then ? then[2]! : null }; +} + +function joinPath(segs: readonly string[]): string { + return '/' + segs.join('/'); +} + +/** + * Every screen and lazy mount a file's routes arrays declare. Entries are + * walked as objects (`object-literal.ts`), never read out of a window of + * text: `path` may be written after `component`. + */ +export function parseAngularRoutes(content: string): { routes: AngularRoute[]; mounts: AngularMount[]; redirects: AngularRedirect[] } { + const routes: AngularRoute[] = []; + const mounts: AngularMount[] = []; + const redirects: AngularRedirect[] = []; + if (!ROUTER_IMPORT.test(content)) return { routes, mounts, redirects }; + const safe = stripCommentsForRegex(content, 'typescript'); + const lineOf = (at: number) => safe.slice(0, at).split('\n').length; + const seenPaths = new Set(); + const walked = new Set(); + + const walk = (open: number, close: number, prefix: readonly string[], depth: number, layouts: readonly AngularComponentRef[]): void => { + if (walked.has(open) || depth > 12) return; + walked.add(open); + for (const obj of topLevelObjects(safe, open + 1, close)) { + const fields = readFields(safe, obj.start, obj.end); + if (!ROUTE_KEYS.some((k) => fields.has(k))) continue; + // A custom `matcher` decides the address at run time — no path to name. + if (fields.has('matcher')) continue; + const pathField = fields.get('path'); + // `{ path: '**', redirectTo: 'home' }`: where an address nothing else matches lands. + if (pathField && fields.has('redirectTo') && staticString(pathField.text) === '**') { + const target = staticString(fields.get('redirectTo')!.text); + if (target !== null && !target.includes('**')) { + const absolute = target.startsWith('/'); + redirects.push({ from: joinPath([...prefix, '**']), to: joinPath([...(absolute ? [] : prefix), ...target.split('/').filter((t) => t.length > 0)]), absolute }); + } + continue; + } + const own = pathField ? pathSegments(pathField.text) : []; + if (own === null) continue; // a computed path, or a `**` catch-all + const segs = [...prefix, ...own]; + const line = lineOf(pathField ? pathField.at : obj.start); + + const redirectTo = fields.get('redirectTo'); + if (redirectTo) { + // `{ path: '', redirectTo: 'home' }`: arriving at `/` is arriving at + // `/home`. A relative target is a sibling, under the same parent. + const target = staticString(redirectTo.text); + if (target !== null && !target.includes('**')) { + const absolute = target.startsWith('/'); + const to = joinPath([...(absolute ? [] : prefix), ...target.split('/').filter((t) => t.length > 0)]); + redirects.push({ from: joinPath(segs), to, absolute }); + } + continue; + } + + const loadChildren = fields.get('loadChildren'); + if (loadChildren) { + const lazy = lazyImport(loadChildren.text); + if (lazy) mounts.push({ prefix: joinPath(segs), spec: lazy.spec, line }); + continue; + } + const componentField = fields.get('component') ? 'component' : fields.get('loadComponent') ? 'loadComponent' : null; + const component = componentField ? componentRef(componentField, fields.get(componentField)!.text) : null; + + const children = fields.get('children'); + let childClaimsAddress = false; + if (children) { + const childOpen = safe.indexOf('[', children.at); + const childClose = childOpen < 0 ? -1 : matchBracket(safe, childOpen); + if (childOpen >= 0 && childClose > childOpen) { + for (const child of topLevelObjects(safe, childOpen + 1, childClose)) { + const childPath = readFields(safe, child.start, child.end).get('path'); + if (!childPath || pathSegments(childPath.text)?.length === 0) childClaimsAddress = true; + } + walk(childOpen, childClose, segs, depth + 1, component ? [...layouts, component] : layouts); + } + } + // A layout is a screen only where no child claims its address; a + // redirect and a componentless grouping are no screen at all. + if (!componentField || (children && childClaimsAddress)) continue; + const route = joinPath(segs); + if (seenPaths.has(route)) continue; + seenPaths.add(route); + routes.push({ path: route, line, component, layouts: [...layouts] }); + } + }; + + ARRAY_OPENERS.lastIndex = 0; + let m: RegExpExecArray | null; + while ((m = ARRAY_OPENERS.exec(safe)) !== null) { + const open = m.index + m[0].length - 1; + const close = matchBracket(safe, open); + if (close < 0) continue; + // `const x = [` is a routes array only when it says so: `] as Routes` / `] satisfies Routes`. + if (/^\b(?:const|let)\b/.test(m[0]) && !/^\]\s*(?:as|satisfies)\s+Routes\b/.test(safe.slice(close, close + 32))) continue; + walk(open, close, [], 0, []); + } + return { routes, mounts, redirects }; +} + +/** The id a config-declared route carries — reconstructed exactly, so the resolver recognises its own. */ +function routeId(filePath: string, line: number, inFilePath: string): string { + return `route:${filePath}:${line}:${inFilePath}:angular`; +} + +/** True for a route node this resolver emitted. */ +export function isAngularRoute(node: Node): boolean { + return node.kind === 'route' && node.id.endsWith(':angular') && node.id === routeId(node.filePath, node.startLine, inFilePath(node)); +} + +/** The path a route has within its own file, before any mount — kept in the qualified name so the cross-file pass is idempotent. */ +function inFilePath(node: Node): string { + const marker = '::route:'; + const at = node.qualifiedName.indexOf(marker); + return at < 0 ? node.name : node.qualifiedName.slice(at + marker.length); +} + +// ============================================================================= +// The component a file lazily exports +// ============================================================================= + +/** `@Component({…}) export default class X` / `export class X` — the classes a file declares as components. */ +const COMPONENT_CLASS = /@Component\s*\(/g; + +/** The `@Component` classes a file declares, in order, with whether each is the default export. */ +export function componentClassesIn(content: string): Array<{ name: string; isDefault: boolean; decoratorAt: number }> { + const out: Array<{ name: string; isDefault: boolean; decoratorAt: number }> = []; + const safe = stripCommentsForRegex(content, 'typescript'); + COMPONENT_CLASS.lastIndex = 0; + let m: RegExpExecArray | null; + while ((m = COMPONENT_CLASS.exec(safe)) !== null) { + const open = m.index + m[0].length - 1; + const close = matchBracket(safe, open); + if (close < 0) continue; + const after = /^\s*(export\s+)?(default\s+)?(?:abstract\s+)?class\s+([A-Za-z_$][\w$]*)/.exec(safe.slice(close + 1, close + 400)); + if (after) out.push({ name: after[3]!, isDefault: !!after[2], decoratorAt: m.index }); + COMPONENT_CLASS.lastIndex = close; + } + return out; +} + +/** The component class a lazily loaded file provides: the member named, else its default export, else its only component. */ +function lazyComponent(spec: string, member: string | null, fromFile: string, context: ResolutionContext): Node | null { + const file = resolveImportPath(spec, fromFile, 'typescript', context); + if (!file) return null; + let name = member; + if (!name) { + const content = context.readFile(file); + if (!content) return null; + const classes = componentClassesIn(content); + name = (classes.find((c) => c.isDefault) ?? (classes.length === 1 ? classes[0] : undefined))?.name ?? null; + if (!name) { + const plain = /\bexport\s+default\s+class\s+([A-Za-z_$][\w$]*)/.exec(content); + name = plain ? plain[1]! : null; + } + } + if (!name) return null; + return context.getNodesInFile(file).find((n) => n.kind === 'class' && n.name === name) ?? null; +} + +/** A path's segments with each `{expr}` read from the constant it names, as seen from `file` — where the route (or the mount) is written. */ +function resolvedSegments(pathText: string, file: string, context: ResolutionContext): string[] { + return pathText + .split('/') + .filter((seg) => seg.length > 0) + .flatMap((seg) => { + if (!seg.startsWith('{') || !seg.endsWith('}')) return [seg]; + const value = constantValue(seg.slice(1, -1), file, context); + return value === null ? [seg] : value.split('/').filter((v) => v.length > 0); + }); +} + +/** The class a route's component names, by its import (or this file) — or a lazy import's. */ +function routeComponent(encoded: string, fromFile: string, context: ResolutionContext): Node | null { + const lazy = LAZY_COMPONENT_REF.exec(encoded); + if (lazy) return lazyComponent(lazy[1]!, lazy[2] === 'default' ? null : lazy[2]!, fromFile, context); + const file = declaringFile(encoded, fromFile, context); + return file ? (context.getNodesInFile(file).find((n) => n.kind === 'class' && n.name === encoded) ?? null) : null; +} + +// ============================================================================= +// The cross-file pass: mounts and constant paths +// ============================================================================= + +/** The file a routes import names, and — for an NgModule — the routing modules it imports. */ +function routeFilesLoadedBy(spec: string, fromFile: string, context: ResolutionContext, routeFiles: ReadonlySet, mountFiles: ReadonlySet): string[] { + const target = resolveImportPath(spec, fromFile, 'typescript', context); + if (!target) return []; + if (routeFiles.has(target) || mountFiles.has(target)) return [target]; + // `loadChildren: () => import('./layout/layout.module').then(m => m.LayoutModule)`: + // the module holds no routes; the routing module it imports does. + const content = context.readFile(target); + if (!content) return []; + const out: string[] = []; + const imports = /\bimport\s+[^'"]*?from\s+(['"])([^'"]+)\1/g; + let m: RegExpExecArray | null; + while ((m = imports.exec(content)) !== null) { + const file = resolveImportPath(m[2]!, target, 'typescript', context); + if (file && (routeFiles.has(file) || mountFiles.has(file))) out.push(file); + } + return out; +} + +/** Per context, renewed with the route list (whose identity changes when the resolver's caches do). */ +const constantTexts = new WeakMap }>(); + +/** + * The source text a constant member chain names — `internalRoutes.account.path` + * is the `path:` field of `account` in `export const internalRoutes = {…}` — + * read from the file the chain's root is imported from (or declared in). + */ +export function constantText(expr: string, fromFile: string, context: ResolutionContext): string | null { + const source = context.getNodesByKind('route'); + let entry = constantTexts.get(context); + if (!entry || entry.source !== source) constantTexts.set(context, (entry = { source, memo: new Map() })); + const memo = entry.memo; + const key = `${fromFile}\0${expr}`; + if (memo.has(key)) return memo.get(key)!; + // `const { create } = internalRoutes.accounts.subRoutes;` / `const accounts = internalRoutes.accounts;` + // make `create.path` a chain rooted at the import. + const aliased = localAlias(expr, fromFile, context); + if (aliased !== null && aliased !== expr) { + const value = constantText(aliased, fromFile, context); + memo.set(key, value); + return value; + } + const [root, ...chain] = expr.split('.').map((part) => part.trim()); + let value: string | null = null; + const file = root ? declaringFile(root, fromFile, context) : null; + const content = file ? context.readFile(file) : null; + if (content && chain.length > 0) { + const safe = stripCommentsForRegex(content, 'typescript'); + const decl = new RegExp(String.raw`\b(?:const|let)\s+${root}\s*(?::[^=]+)?=\s*\{`).exec(safe); + if (decl) { + let start = decl.index + decl[0].length - 1; + let end = matchBracket(safe, start); + for (let i = 0; i < chain.length && end > start; i++) { + const field = readFields(safe, start, end).get(chain[i]!); + if (!field) break; + if (i === chain.length - 1) { + value = field.text.trim(); + break; + } + start = safe.indexOf('{', field.at); + end = start < 0 ? -1 : matchBracket(safe, start); + } + } + } + memo.set(key, value); + return value; +} + +/** `create.path` as the chain its root was destructured from in `fromFile`, or null when the root is no local alias. */ +function localAlias(expr: string, fromFile: string, context: ResolutionContext): string | null { + const content = context.readFile(fromFile); + if (!content) return null; + const [root, ...rest] = expr.split('.'); + if (!root) return null; + const destructure = /\b(?:const|let)\s*\{([^}]*)\}\s*=\s*([A-Za-z_$][\w$]*(?:\s*\.\s*[A-Za-z_$][\w$]*)*)\s*;?/g; + let m: RegExpExecArray | null; + while ((m = destructure.exec(content)) !== null) { + for (const binding of m[1]!.split(',')) { + const [prop, local] = binding.split(':').map((part) => part.trim()); + if ((local ?? prop) === root && prop) return [`${m[2]!.replace(/\s+/g, '')}.${prop}`, ...rest].join('.'); + } + } + const simple = new RegExp(String.raw`\b(?:const|let)\s+${root}\s*=\s*([A-Za-z_$][\w$]*(?:\s*\.\s*[A-Za-z_$][\w$]*)+)\s*;`).exec(content); + return simple ? [simple[1]!.replace(/\s+/g, ''), ...rest].join('.') : null; +} + +/** A constant path's value: `internalRoutes.account.path` → `account`. */ +function constantValue(expr: string, fromFile: string, context: ResolutionContext): string | null { + const text = constantText(expr, fromFile, context); + return text === null ? null : staticString(text); +} + +/** The file declaring `name` for code in `fromFile`: its import, else the file itself. */ +function declaringFile(name: string, fromFile: string, context: ResolutionContext): string | null { + const content = context.readFile(fromFile); + if (!content) return null; + const imports = /\bimport\s*\{([^}]*)\}\s*from\s*(['"])([^'"]+)\2/g; + let m: RegExpExecArray | null; + while ((m = imports.exec(content)) !== null) { + const names = m[1]!.split(',').map((s) => s.trim().split(/\s+as\s+/).pop()!.trim()); + if (names.includes(name)) return resolveImportPath(m[3]!, fromFile, 'typescript', context); + } + return fromFile; +} + +// ============================================================================= +// Route table +// ============================================================================= + +export type AngularRouteTable = RootedRouteTable; + +const tables = new WeakMap(); + +export function angularRouteTable(context: ResolutionContext): AngularRouteTable { + const all = context.getNodesByKind('route'); + const cached = tables.get(context); + if (cached && cached.source === all) return cached; + const byRoot = new Map(); + const byFile = new Map(); + for (const node of all) { + if (!isAngularRoute(node) || !node.name.startsWith('/')) continue; + const root = appRootFor(node.filePath); + let t = byRoot.get(root); + if (!t) byRoot.set(root, (t = { source: all, exact: new Map(), dynamic: [] })); + addRouteTo(t, node.name, node); + if (!byFile.has(node.filePath)) byFile.set(node.filePath, node); + } + // Redirects: arriving at `/` is arriving at wherever `redirectTo` sends + // the user. A file's mount prefix is what its routes' names add to their + // in-file paths. A chain (`/` → `/pages` → `/pages/dashboard`) settles in + // a few passes. + const aliases: Array<{ table: RouteTable; from: string; to: string }> = []; + for (const [file, sample] of byFile) { + const content = context.readFile(file); + if (!content || !content.includes('redirectTo')) continue; + const own = resolvedSegments(inFilePath(sample), file, context); + const full = sample.name.split('/').filter((seg) => seg.length > 0); + const prefix = full.slice(0, Math.max(0, full.length - own.length)); + const table = byRoot.get(appRootFor(file)); + if (!table) continue; + for (const r of parseAngularRoutes(content).redirects) { + const from = joinPath([...prefix, ...resolvedSegments(r.from, file, context)]); + const to = joinPath([...(r.absolute ? [] : prefix), ...resolvedSegments(r.to, file, context)]); + aliases.push({ table, from, to }); + } + } + for (let pass = 0; pass < 4; pass++) { + let added = false; + for (const a of aliases) { + const target = a.table.exact.get(a.to); + if (target && !a.table.exact.has(a.from)) { + a.table.exact.set(a.from, target); + added = true; + } + } + if (!added) break; + } + // `/` that no route or redirect serves falls through to the app's root + // `**` redirect. Only `/`: letting the wildcard answer any path would draw + // every destination this reading could not name as a trip home. + for (const a of aliases) { + if (a.from !== '/**') continue; + const target = a.table.exact.get(a.to); + if (target && !a.table.exact.has('/')) a.table.exact.set('/', target); + } + const table: AngularRouteTable = { source: all, byRoot }; + tables.set(context, table); + return table; +} + +/** + * The routes a file navigates among: its app's, or — for a shared library + * outside any app (an Nx `libs/ui`) — the one app's, when there is only one. + */ +export function angularRoutesFor(table: AngularRouteTable, filePath: string): RouteTable | null { + const own = routesForFile(table, filePath); + if (own && own.exact.size > 0) return own; + return table.byRoot.size === 1 ? [...table.byRoot.values()][0]! : null; +} + +// ============================================================================= +// Navigation +// ============================================================================= + +/** `this.router.navigate`, `router.navigateByUrl`, `this._router.createUrlTree`, `router.parseUrl`. */ +const NAV_CALL = /^(?:this\.)?_?[rR]outer\.(navigate|navigateByUrl|createUrlTree|parseUrl)$/; + +/** The destination a command array names: `['/article', slug]` is `/article/${…}`; null when it is relative or not a literal array. */ +export function commandsHref(text: string): HrefLiteral | null { + const arr = text.trim(); + if (arr[0] !== '[') return null; + const close = matchBracket(arr, 0); + if (close < 0) return null; + const parts: string[] = []; + let i = 1; + let element = ''; + const flush = (): boolean => { + const e = element.trim(); + element = ''; + if (e.length === 0) return true; + const literal = staticString(e); + if (literal !== null) { + parts.push(...literal.split('/').filter((seg) => seg.length > 0)); + return true; + } + // A matrix-params object says nothing about the path. + if (e.startsWith('{')) return true; + parts.push(HOLE); + return true; + }; + while (i < close) { + const ch = arr[i]!; + if (ch === '"' || ch === "'" || ch === '`') { + const end = skipString(arr, i); + if (end < 0) return null; + element += arr.slice(i, end + 1); + i = end + 1; + continue; + } + if (ch === '[' || ch === '{' || ch === '(') { + const end = matchBracket(arr, i); + if (end < 0) return null; + element += arr.slice(i, end + 1); + i = end + 1; + continue; + } + if (ch === ',') flush(); + else element += ch; + i++; + } + flush(); + // Only an absolute destination: `['/login']`. `['../', id]` and a bare + // `['edit']` are relative to wherever the component is. + const firstElement = arr.slice(1, close).split(',')[0] ?? ''; + const first = staticString(firstElement); + if (first === null || !first.startsWith('/')) return null; + const pathText = '/' + parts.filter((p) => p.length > 0).join('/'); + return namesSomewhere({ path: pathText, display: pathText.split(HOLE).join('${…}') }); +} + +/** + * The href, when it names at least one segment of its own. A path made only + * of holes (`[\`/${a}\`, b, c]`) would match any route of its length — a + * hole scores against a literal segment — so it names nothing. + */ +export function namesSomewhere(href: HrefLiteral | null): HrefLiteral | null { + if (!href) return null; + const segs = href.path.split('/').filter((seg) => seg.length > 0); + return segs.length === 0 || segs.some((seg) => !seg.includes(HOLE)) ? href : null; +} + +/** A class property's initializer, read from its body: `routerLinkAbout = publicRoutes.about.routerLink;`. */ +export function propertyInitializer(owner: Node, name: string, content: string): string | null { + const body = content.split('\n').slice(owner.startLine - 1, owner.endLine).join('\n'); + const decl = new RegExp(String.raw`(?:^|[\s;{])(?:(?:public|private|protected|readonly|static|override)\s+)*${name}\s*(?::[^=;\n]+)?=\s*`).exec(body); + if (!decl) return null; + let i = decl.index + decl[0].length; + const start = i; + while (i < body.length) { + const ch = body[i]!; + if (ch === '"' || ch === "'" || ch === '`') { + const end = skipString(body, i); + if (end < 0) return null; + i = end + 1; + continue; + } + if (ch === '[' || ch === '{' || ch === '(') { + const end = matchBracket(body, i); + if (end < 0) return null; + i = end + 1; + continue; + } + if (ch === ';' || ch === '\n') break; + i++; + } + return body.slice(start, i).trim() || null; +} + +/** The array an arrow function returns: `(id) => ['/x', id]` or `(id) => { return ['/x', id]; }`. */ +function arrowReturn(fn: string): string | null { + const arrow = fn.indexOf('=>'); + if (arrow < 0) return null; + let i = arrow + 2; + while (i < fn.length && /\s/.test(fn[i]!)) i++; + if (fn[i] === '(') i++; + if (fn[i] === '[') { + const end = matchBracket(fn, i); + return end > i ? fn.slice(i, end + 1) : null; + } + if (fn[i] === '{') { + const ret = /\breturn\s*\[/.exec(fn.slice(i)); + if (!ret) return null; + const open = i + ret.index + ret[0].length - 1; + const end = matchBracket(fn, open); + return end > open ? fn.slice(open, end + 1) : null; + } + return null; +} + +/** + * Where an Angular destination expression leads — the argument of + * `router.navigate(…)`, or a `[routerLink]` binding: + * + * - a command array, `['/article', slug]`; + * - a static string, `'/login'` or `'/' + $localize…`, and `'/editor/' + slug`; + * - a route constant, `internalRoutes.zen.routerLink`; + * - a property of the class it is written in, `this.routerLinkAbout`, + * holding one of those; + * - any of those `.concat(id)`, one more segment. + * + * `owner` is the class the expression is written in, for `this.` properties. + */ +export function angularDestination(expr: string, file: string, owner: Node | null, context: ResolutionContext, depth = 0): HrefLiteral | null { + return namesSomewhere(destinationOf(expr, file, owner, context, depth)); +} + +function destinationOf(expr: string, file: string, owner: Node | null, context: ResolutionContext, depth: number): HrefLiteral | null { + const text = expr.trim(); + if (text.length === 0 || depth > 4) return null; + if (text[0] === '[') return commandsHref(text); + const literal = staticString(text); + if (literal !== null) return toHref(literal); + // `'/editor/' + article.slug`: the literal head, a hole for the rest. + if (/^['"]/.test(text) && text.includes('+')) { + const head = staticString(text.slice(0, text.indexOf('+'))); + if (head !== null && head.startsWith('/')) return toHref(head.replace(/\/?$/, '/') + HOLE); + } + // `this.routerLinkAdminControlUsers.concat(userId)`: one segment more. + const concat = /^(.*?)\.concat\s*\((.*)\)$/s.exec(text); + if (concat) { + const base = angularDestination(concat[1]!, file, owner, context, depth + 1); + const added = concat[2]!.split(',').filter((a) => a.trim().length > 0).length; + return base && added > 0 ? toHref(base.path.replace(/\/$/, '') + ('/' + HOLE).repeat(added)) : base; + } + // `update.routerLink(dataSource, symbol)`: a route constant that builds its + // commands — its returned array, the arguments holes. + const call = /^((?:this\s*\.\s*)?[A-Za-z_$][\w$]*(?:\s*\.\s*[A-Za-z_$][\w$]*)+)\s*\((.*)\)$/s.exec(text); + if (call) { + const fn = constantText(call[1]!.replace(/\s+/g, '').replace(/^this\./, ''), file, context); + const returned = fn ? arrowReturn(fn) : null; + return returned ? commandsHref(returned) : null; + } + const chain = /^(?:this\s*\.\s*)?([A-Za-z_$][\w$]*(?:\s*\.\s*[A-Za-z_$][\w$]*)*)$/.exec(text); + if (!chain) return parseHrefExpression(text); + const parts = chain[1]!.replace(/\s+/g, '').split('.'); + const content = context.readFile(file); + if (content && owner && (text.startsWith('this') || parts.length === 1)) { + const init = propertyInitializer(owner, parts[0]!, content); + if (init !== null) { + if (parts.length === 1) return angularDestination(init, file, owner, context, depth + 1); + return angularDestination(`${init}.${parts.slice(1).join('.')}`, file, owner, context, depth + 1); + } + } + const constant = constantText(parts.join('.'), file, context); + return constant ? angularDestination(constant, file, owner, context, depth + 1) : null; +} + +// ============================================================================= +// The resolver +// ============================================================================= + +/** A route's lazily loaded component: `import:./x.component#default` / `#XComponent`. */ +const LAZY_COMPONENT_REF = /^import:([^#]+)#([\w$]+)$/; +/** A layout a screen renders inside: `layout:ProfileComponent`, `layout:import:./x#default`. */ +const LAYOUT_REF = /^layout:(.+)$/; + +export const angularRouterResolver: FrameworkResolver = { + name: 'angular-router', + languages: ['typescript', 'javascript'], + + detect(context: ResolutionContext): boolean { + return dependsOn(context, '@angular/router', '@angular/core'); + }, + + claimsReference(name: string): boolean { + return NAV_CALL.test(name) || LAZY_COMPONENT_REF.test(name) || LAYOUT_REF.test(name); + }, + + extract(filePath: string, content: string): FrameworkExtractionResult { + if (!/\.[cm]?ts$/.test(filePath) || filePath.endsWith('.d.ts')) return { nodes: [], references: [] }; + const { routes } = parseAngularRoutes(content); + if (routes.length === 0) return { nodes: [], references: [] }; + const now = Date.now(); + const nodes: Node[] = []; + const references: UnresolvedRef[] = []; + for (const route of routes) { + const node: Node = { + id: routeId(filePath, route.line, route.path), + kind: 'route', + name: route.path, + qualifiedName: `${filePath}::route:${route.path}`, + filePath, + startLine: route.line, + endLine: route.line, + startColumn: 0, + endColumn: 0, + language: 'typescript', + updatedAt: now, + }; + nodes.push(node); + const ref = (component: AngularComponentRef, layout: boolean): UnresolvedRef => ({ + fromNodeId: node.id, + referenceName: `${layout ? 'layout:' : ''}${component.spec === null ? component.name : `import:${component.spec}#${component.name ?? 'default'}`}`, + referenceKind: 'references', + line: route.line, + column: 0, + filePath, + language: 'typescript', + }); + if (route.component) references.push(ref(route.component, false)); + // The layouts around it: their links and buttons are on this screen too. + for (const layout of route.layouts) references.push(ref(layout, true)); + } + return { nodes, references }; + }, + + /** + * Put each routes file's routes under the path that lazy-loads it, and read + * constant path segments. Recomputed from each route's in-file path (its + * qualified name), so a second run changes nothing. + */ + postExtract(context: ResolutionContext): Node[] { + const routes = context.getNodesByKind('route').filter(isAngularRoute); + if (routes.length === 0) return []; + const routeFiles = new Set(routes.map((r) => r.filePath)); + // Every file that lazy-loads routes, whether or not it declares a screen of its own. + const mountsByFile = new Map(); + for (const file of context.getAllFiles()) { + if (!/\.[cm]?ts$/.test(file) || !(context.fileContains?.(file, 'loadChildren') ?? context.readFile(file)?.includes('loadChildren'))) continue; + const content = context.readFile(file); + if (!content) continue; + const { mounts } = parseAngularRoutes(content); + if (mounts.length > 0) mountsByFile.set(file, mounts); + } + const mountFiles = new Set(mountsByFile.keys()); + // file → the prefix it is mounted under (the parent file's own prefix + // plus the mount's), settled from the top down. + const loadedBy = new Map(); + for (const [file, mounts] of mountsByFile) { + for (const mount of mounts) { + for (const target of routeFilesLoadedBy(mount.spec, file, context, routeFiles, mountFiles)) { + // A file mounted from two places keeps its first mount. + if (target !== file && !loadedBy.has(target)) loadedBy.set(target, { parent: file, prefix: mount.prefix }); + } + } + } + const resolved = (pathText: string, file: string): string[] => resolvedSegments(pathText, file, context); + + const memo = new Map(); + const prefixOf = (file: string, seen: Set = new Set()): string[] => { + const hit = memo.get(file); + if (hit !== undefined) return hit; + const mount = loadedBy.get(file); + let prefix: string[] = []; + if (mount && !seen.has(file)) { + seen.add(file); + prefix = [...prefixOf(mount.parent, seen), ...resolved(mount.prefix, mount.parent)]; + } + memo.set(file, prefix); + return prefix; + }; + + const changed: Node[] = []; + for (const route of routes) { + const name = joinPath([...prefixOf(route.filePath), ...resolved(inFilePath(route), route.filePath)]); + if (name !== route.name) changed.push({ ...route, name }); + } + return changed; + }, + + resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + const layout = LAYOUT_REF.exec(ref.referenceName); + if (layout) { + if (!ref.fromNodeId.endsWith(':angular')) return null; + const component = routeComponent(layout[1]!, ref.filePath, context); + return component + ? { original: ref, targetNodeId: component.id, confidence: 0.95, resolvedBy: 'framework', metadata: { layout: true } } + : null; + } + const lazy = LAZY_COMPONENT_REF.exec(ref.referenceName); + if (lazy) { + if (!ref.fromNodeId.endsWith(':angular')) return null; + const component = lazyComponent(lazy[1]!, lazy[2] === 'default' ? null : lazy[2]!, ref.filePath, context); + return component ? { original: ref, targetNodeId: component.id, confidence: 0.95, resolvedBy: 'framework' } : null; + } + + const nav = NAV_CALL.exec(ref.referenceName); + if (!nav || ref.referenceKind !== 'calls') return null; + const verb = nav[1]!; + const routes = angularRoutesFor(angularRouteTable(context), ref.filePath); + if (!routes || routes.exact.size === 0) return null; + const lines = context.getFileLines?.(ref.filePath) ?? context.readFile(ref.filePath)?.split(/\r?\n/) ?? null; + if (!lines) return null; + const arg = firstArgumentText(lines, ref.line, ref.column, verb); + if (arg === null) return null; + // `navigate(['..', id], { relativeTo: this.route })` goes somewhere relative to here. + const extras = nthArgumentText(lines, ref.line, ref.column, verb, 1); + if (extras && /\brelativeTo\b/.test(extras)) return null; + const owner = context + .getNodesInFile(ref.filePath) + .filter((n) => n.kind === 'class' && n.startLine <= ref.line && n.endLine >= ref.line) + .reduce((inner, n) => (!inner || n.startLine >= inner.startLine ? n : inner), null); + const href = angularDestination(arg, ref.filePath, owner, context); + if (!href || !href.path.startsWith('/')) return null; + const targets = destinationsForHref(href, routes); + const target = targets[0]; + if (!target) return null; + return { + original: ref, + targetNodeId: target.node.id, + ...(targets.length > 1 + ? { alsoTargets: targets.slice(1).map((t) => ({ targetNodeId: t.node.id, metadata: { href: t.href.display, navMethod: verb } })) } + : {}), + confidence: 0.95, + resolvedBy: 'framework', + edgeKind: 'navigates', + metadata: { href: target.href.display, navMethod: verb }, + }; + }, +}; + +/** For the template synthesizer: a template file's path, from the component file that names it. */ +export function templatePathFor(componentFile: string, templateUrl: string): string { + return path.posix.normalize(path.posix.join(path.posix.dirname(componentFile), templateUrl)); +} diff --git a/src/resolution/frameworks/index.ts b/src/resolution/frameworks/index.ts index 4c96e186e0..97af4183fb 100644 --- a/src/resolution/frameworks/index.ts +++ b/src/resolution/frameworks/index.ts @@ -15,6 +15,7 @@ import { nextjsResolver } from './nextjs'; import { reactRouterResolver } from './react-router'; import { tanstackRouterResolver } from './tanstack-router'; import { vueRouterResolver } from './vue-router'; +import { angularRouterResolver } from './angular-router'; import { svelteKitRouterResolver } from './sveltekit-router'; import { svelteResolver } from './svelte'; import { vueResolver } from './vue'; @@ -59,6 +60,7 @@ const FRAMEWORK_RESOLVERS: FrameworkResolver[] = [ vueResolver, // Vue Router — `createRouter({ routes })` → route nodes; `router.push({ name })` / `router.push('/x')` → navigates edges vueRouterResolver, + angularRouterResolver, astroResolver, // Python djangoResolver, @@ -157,6 +159,7 @@ export { reactResolver } from './react'; export { reactRouterResolver } from './react-router'; export { tanstackRouterResolver } from './tanstack-router'; export { vueRouterResolver } from './vue-router'; +export { angularRouterResolver } from './angular-router'; export { svelteKitRouterResolver } from './sveltekit-router'; export { svelteResolver } from './svelte'; export { vueResolver } from './vue'; diff --git a/src/ui-server/api/route-roots.ts b/src/ui-server/api/route-roots.ts index 78412b8497..f9ab4c1296 100644 --- a/src/ui-server/api/route-roots.ts +++ b/src/ui-server/api/route-roots.ts @@ -52,6 +52,23 @@ export function looksLikeComponent(node: Node): boolean { return JS_FAMILY.has(node.language) && /^[A-Z]/.test(node.name); } +/** + * Route id → the layouts it renders inside, outermost first: an Angular + * parent route's component around its ``. What happens in a + * layout — its tabs, its buttons — happens on every screen nested in it. + */ +export function routeLayouts(cg: CodeGraph, routes: readonly Node[]): Map { + const out = new Map(); + if (routes.length === 0) return out; + for (const e of cg.getOutgoingEdgesFrom(routes.map((r) => r.id), ['references'])) { + if ((e.metadata as Record | undefined)?.layout !== true) continue; + const list = out.get(e.source) ?? []; + if (!list.includes(e.target)) list.push(e.target); + out.set(e.source, list); + } + return out; +} + /** Route id → where its code starts, for every route that has an answer. */ export function routeRoots(cg: CodeGraph, routes: readonly Node[]): Map { const out = new Map(); @@ -70,9 +87,10 @@ export function routeRoots(cg: CodeGraph, routes: readonly Node[]): Map e.kind === 'references') + .filter((e) => e.kind === 'references' && (e.metadata as Record | undefined)?.layout !== true) .map((e) => targets.get(e.target)) .filter((n): n is Node => !!n && HANDLER_KINDS.has(n.kind) && n.id !== route.id) .sort((a, b) => rank(a) - rank(b) || a.startLine - b.startLine); diff --git a/src/ui-server/api/screens.ts b/src/ui-server/api/screens.ts index 0099991cb7..94b70761a9 100644 --- a/src/ui-server/api/screens.ts +++ b/src/ui-server/api/screens.ts @@ -30,7 +30,7 @@ import * as fs from 'fs'; import type CodeGraph from '../../index'; import type { Edge, Node } from '../../types'; -import { routeRoots } from './route-roots'; +import { routeLayouts, routeRoots } from './route-roots'; import { resolveProjectFile } from '../security'; import { findIndexedFile, hasDriftedOnDisk } from './source'; import { createWhenReader } from './when'; @@ -194,6 +194,9 @@ const SHARED_CHROME_MIN = 3; /** True when the edge's destination is written at the line the edge points to. */ function writtenHere(edge: Edge, holder: Node): boolean { + // An Angular template names its destination where it is written — in a + // file of its own, beside the component the edge leaves from. + if ((edge.metadata as Record | undefined)?.template === true) return true; const at = (edge.metadata as Record | undefined)?.registeredAt; if (typeof at !== 'string') return edge.provenance !== 'heuristic'; return at === `${holder.filePath}:${edge.line}`; @@ -270,6 +273,15 @@ export async function buildScreens(cg: CodeGraph, projectRoot: string): Promise< if (serves) serves.push(routeId); else screenOfComponent.set(root.node.id, [routeId]); } + // A layout serves every screen nested in it: ProfileComponent's tabs are on + // both `/profile/:username` and `/profile/:username/favorites`. + for (const [routeId, layouts] of routeLayouts(cg, routes)) { + for (const layout of layouts) { + const serves = screenOfComponent.get(layout); + if (!serves) screenOfComponent.set(layout, [routeId]); + else if (!serves.includes(routeId)) serves.push(routeId); + } + } const nodesById = cg.getNodesByIds([...componentOf.values()].map((n) => n.id).concat(navEdges.map((e) => e.source))); const readWhen = createWhenReader(cg, projectRoot, MAX_WHEN_SITES); From c3efbbf4e735a69bd5feedec3e0749b43b410c29 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 02:51:30 +0000 Subject: [PATCH 002/259] feat(angular): a template's event bindings call the component's methods (#2114) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An Angular handler's only caller is its template: `(click)="toggleFavorite()"`, `(ngSubmit)="submitForm()"`. With no edge, the method showed no callers, impact stopped at it, and the Steps picture of a screen reached its handlers only through class containment, with nothing to say what fires them. The template pass now reads each `(event)="…"` binding — not `[(ngModel)]`'s two-way half — and links the component to every method of its own the statement calls (`save()`, `this.toggle(x)`; not `closed.emit()` or `form.reset()`), as a `calls` edge (`synthesizedBy: 'angular-event'`) carrying the binding as `metadata.trigger`: `{ kind: 'prop', name: '(click)', of: 'button' }`. The binding is in the template, so it cannot be read back from the source at the edge's line; Steps now takes a trigger an edge carries before reading one at the site. angular-realworld: 21 binding sites → 17 edges (3 repeats of a method and event, 1 `delete.emit(true)`); Steps for /article/:slug draws deleteArticle on (click) · `, ` save() { this.router.navigate(internalRoutes.account.subRoutes.access.routerLink('me')); }`).replace( + 'src/app/shared/card.component.ts': component( + 'CardComponent', + 'app-card', + `x`, + ` name = ''; + save() { this.router.navigate(internalRoutes.account.subRoutes.access.routerLink('me')); }` + ).replace( "import { Router } from '@angular/router';", "import { Router } from '@angular/router';\nimport { internalRoutes } from '../routes.constants';" ), @@ -270,6 +277,27 @@ export class AdminRoutingModule {} expect(renders).toEqual(['HomeComponent -> CardComponent ']); }); + it("links a template's event bindings to the component's own methods, the binding as the trigger", async () => { + const bindings = cg + .getNodesByKind('class') + .flatMap((c) => + cg + .getOutgoingEdgesFrom([c.id], ['calls']) + .filter((e) => (e.metadata as Record | undefined)?.synthesizedBy === 'angular-event') + .map((e) => { + const meta = e.metadata as Record; + return `${c.name} -> ${cg.getNode(e.target)?.name} ${JSON.stringify(meta.trigger)}`; + }) + ); + // `[(ngModel)]` is a two-way binding and `closed.emit()` raises the component's own output: neither calls a method. + expect(bindings).toEqual(['CardComponent -> save {"kind":"prop","name":"(click)","of":"button"}']); + + const home = cg.getNodesByKind('route').find((r) => r.name === '/home')!; + const payload = await buildSteps(cg, root, new URLSearchParams({ anchor: home.id })); + const save = payload.steps.find((st) => st.label === 'save'); + expect(save?.trigger).toMatchObject({ kind: 'prop', name: '(click)', of: 'button', in: 'CardComponent' }); + }); + it("draws the Screens picture: a child component's navigation on its screen, a layout's on each screen inside it", async () => { const payload = await buildScreens(cg, root); expect(payload.routed).toBe(true); diff --git a/docs/design/framework-coverage.md b/docs/design/framework-coverage.md index 8e71012c9d..febc60a301 100644 --- a/docs/design/framework-coverage.md +++ b/docs/design/framework-coverage.md @@ -56,7 +56,10 @@ Angular is the one whose markup is not indexed: a component's template is a which also yields the component tree (`` by element selector) — the edge a navigation in a child component rides to its screen. A `routerLink:` field written in a component's class (a tab bar's or a menu's config, bound -in a loop elsewhere) counts as a link from that component. A route with +in a loop elsewhere) counts as a link from that component. An event binding +(`(click)="save()"`) is a `calls` edge from the component to its own method +carrying `metadata.trigger`, which Steps uses in place of reading a trigger +at the edge's line (the binding is in the template, not the source there). A route with `children` is a layout: its component carries a `references` edge marked `layout: true` from each screen nested in it, and `routeLayouts` in `route-roots.ts` gives Screens every screen a layout serves. Known limits: a diff --git a/docs/viewer-launch-changelog.md b/docs/viewer-launch-changelog.md index d4f6de4624..79a50d451c 100644 --- a/docs/viewer-launch-changelog.md +++ b/docs/viewer-launch-changelog.md @@ -15,6 +15,8 @@ These describe `codegraph ui` and its screens. They were taken out of `## [Unrel ## New Features +- **The Steps tab draws what an Angular screen does.** Each handler a template binds is drawn with the event that fires it, like `submitForm` on the form's `(ngSubmit)` or `toggleFavorite` on a button's `(click)`, followed by the requests it sends and the screen it opens. That includes handlers in child components the screen renders. Re-index Angular projects after upgrading. + - **The Screens tab draws Angular apps.** Every screen in an Angular app's routes, with the navigation between them: `router.navigate(…)`, a guard's redirect and each template's `routerLink`, under the condition the code checks first. A button in a child component, like an article's favorite button, is drawn from the screen that renders it. A layout's tabs and buttons are drawn from every screen inside that layout. Re-index Angular projects after upgrading. - **A big screen's picture stops wrapping into a column.** How wide a screen's lines run before they wrap was worked out with a formula, and the formula was wrong for the way these pictures are actually drawn: a part of a screen spends lines on its own structure — a step that fires things gets a line to itself, and what it fires starts another — so estimating the lines from the boxes alone badly undercounted them, and one screen's 98 boxes wrapped into a 4,356px column. Laying a picture out is cheap and exact, so the widths are now simply tried and the one that comes out closest to the shape of a window is kept. Across one app's 51 screens the tallest picture went from 4,356px to 3,796px, total height fell 8%, and — because a shorter picture is also a picture whose lines have less far to go — lines running over other boxes fell by a third and lines crossing each other went from 13 to 5. diff --git a/src/resolution/angular-template-synthesizer.ts b/src/resolution/angular-template-synthesizer.ts index 936f340986..9ec76c1e5d 100644 --- a/src/resolution/angular-template-synthesizer.ts +++ b/src/resolution/angular-template-synthesizer.ts @@ -2,7 +2,7 @@ * Angular templates: what a component renders and where it links. * * An Angular component's markup is a template — a `templateUrl` file beside - * it, or an inline `template:` string — that the index never parses, so two + * it, or an inline `template:` string — that the index never parses, so three * things a reader relies on were missing from the graph: * * - **The component tree.** `` in the @@ -18,6 +18,12 @@ * `routerLinkAbout = publicRoutes.about.routerLink`) each become a * `navigates` edge from the component to the route it names. * + * - **Event bindings.** `(click)="toggleFavorite()"` is the only caller a + * handler method has. A `calls` edge from the component to its own method + * (`synthesizedBy: 'angular-event'`) carries the binding as its + * `trigger`, the label Steps puts on the hop — read from the template, so + * it cannot be read back from the source at the edge's line. + * * A child is matched by its element selector (`selector: 'app-article-list'`), * in the same app first; a selector two components share there is left * unmatched. A relative link and a destination no route serves draw nothing. @@ -132,6 +138,19 @@ export function angularComponents(ctx: ResolutionContext): AngularComponent[] { /** `routerLink="…"` (plain) and `[routerLink]="…"` (bound), either quote. `routerLinkActive` is neither. */ const ROUTER_LINK = /(\[routerLink\]|\brouterLink)\s*=\s*(?:"([^"]*)"|'([^']*)')/g; +/** `(click)="save()"` / `(ngSubmit)="submitForm()"` — an event binding, not `[(ngModel)]`'s two-way half. */ +const EVENT_BINDING = /(? n.kind === 'method' && n.qualifiedName.startsWith(`${component.node.qualifiedName}::`)) + .map((n) => [n.name, n]) + ); + let handlers = 0; + EVENT_BINDING.lastIndex = 0; + let ev: RegExpExecArray | null; + while ((ev = EVENT_BINDING.exec(text)) !== null && handlers < MAX_CHILDREN_PER_COMPONENT) { + const statement = ev[2] ?? ev[3] ?? ''; + const event = `(${ev[1]!})`; + const element = elementAt(text, ev.index); + OWN_CALL.lastIndex = 0; + let c: RegExpExecArray | null; + while ((c = OWN_CALL.exec(statement)) !== null) { + const method = members.get(c[1]!); + if (!method) continue; + const key = `${component.node.id}>${method.id}>${event}`; + if (seen.has(key)) continue; + seen.add(key); + handlers++; + edges.push({ + source: component.node.id, + target: method.id, + kind: 'calls', + line: siteLine(ev.index), + provenance: 'heuristic', + metadata: { + synthesizedBy: 'angular-event', + via: event, + registeredAt: registeredAt(ev.index), + trigger: { kind: 'prop', name: event, of: element }, + }, + }); + } + } + const routes = angularRoutesFor(table, component.file); if (!routes || routes.exact.size === 0) continue; let links = 0; diff --git a/src/ui-server/api/steps.ts b/src/ui-server/api/steps.ts index 24c3724dc6..7dea2f7d39 100644 --- a/src/ui-server/api/steps.ts +++ b/src/ui-server/api/steps.ts @@ -394,6 +394,13 @@ interface StepRecord extends WireStep { root: Node | null; } +/** A trigger an edge carries in its metadata (a template binding), when it has one. */ +function siteTriggerOf(meta: Record): SiteTrigger | null { + const t = meta.trigger as Partial | undefined; + if (!t || typeof t !== 'object' || t.kind !== 'prop' || typeof t.name !== 'string') return null; + return { kind: 'prop', name: t.name, of: typeof t.of === 'string' ? t.of : null }; +} + /** * The string a Java / Kotlin return expression builds, when it starts with a * literal: `"owners/ownerDetails"`, `"redirect:/owners/" + owner.getId()` → @@ -1296,7 +1303,10 @@ export async function buildSteps(cg: CodeGraph, projectRoot: string, query: URLS // every call-shaped hop, so a store action or an effect fired by // a tap says so on its link too. const isCall = e.kind === 'calls' || e.kind === 'instantiates' || (e.kind === 'references' && meta.fnRef === true); - const trigger = isCall ? await triggerAt(fold.node, { line: e.line, column: e.column }) : null; + // An Angular template's `(click)` binding rides on its edge: the + // template is a file of its own, not the source at `e.line`. + const carried = siteTriggerOf(meta); + const trigger = carried ? { ...carried, in: fold.node.name } : isCall ? await triggerAt(fold.node, { line: e.line, column: e.column }) : null; // A server action, by its directive: a function in a `'use server'` // file (or opening with the directive) called from a file that is From e845b1cd1092d04d87386df5e9b91795c64a7465 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 03:17:35 +0000 Subject: [PATCH 003/259] fix(nestjs): routes carry the app's global prefix and URI version (#2117) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `app.setGlobalPrefix('api')` and `app.enableVersioning({ type: VersioningType.URI, defaultVersion: '1' })` put every controller under `/api/v1`, but routes were named by the controller path alone (`GET /user`). A front end's `this.http.get('/api/v1/user')` then linked to nothing: Ghostfolio's Angular client had no cross-tier edge into its own NestJS API. postExtract now reads each `NestFactory.create` bootstrap and renames the routes of controllers in that app, in NestJS's order: `/{prefix unless excluded}/{v}{version}/{RouterModule prefix}/{path}`. - `exclude` paths (plain, `{ path, method }`, `{/*wildcard}`, `(.*)`, `*`) skip the prefix and keep their version, as NestJS does. - A route's version: its own `@Version` (found past decorators with object arguments), else the class's `@Version` or `@Controller({ version })`, else `defaultVersion`. `VERSION_NEUTRAL` has none; header/media-type versioning adds no segment. - A computed prefix (`config.get(...)`) is not guessed. - Each app uses its own bootstrap (the longest root above it). A seed script that creates the same app without serving it doesn't displace the one that configures it, and an app with no config doesn't borrow another app's. - Only routes whose decorator sits in a `@Controller` are touched, so an Express route in the same repo keeps its name. Ghostfolio: 0 → 103 cross-tier edges (82 of 85 client call sites), no mismatches. koel, proshop, IceCubes and nest-samples are byte-identical to main. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + README.md | 2 +- __tests__/nestjs-global-prefix.test.ts | 217 ++++++++++++++++++ src/resolution/frameworks/nestjs.ts | 301 +++++++++++++++++++++---- 4 files changed, 481 insertions(+), 40 deletions(-) create mode 100644 __tests__/nestjs-global-prefix.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 83d8e34229..2ab730c2d1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- NestJS routes are now named by the path a request takes: the global prefix from `app.setGlobalPrefix('api')` is applied, along with its `exclude` list, and so is URI versioning from `app.enableVersioning(...)`, with `@Version`, `VERSION_NEUTRAL` and a controller's own `version` taken into account. A route that used to appear as `GET /user` is now `GET /api/v1/user`, so a front end's `this.http.get('/api/v1/user')` or `fetch` call connects to the controller method that serves it. Re-index NestJS projects after upgrading. - Laravel routes are now named by the path a request takes: a leading `/` is added where the routes file leaves it out, `Route::prefix()` and `Route::group(['prefix' => …])` groups are applied, and routes in `routes/api.php` carry the `/api` prefix Laravel serves them under, read from your `RouteServiceProvider`, `bootstrap/app.php` or any file that mounts a routes file. A front-end `fetch('/api/…')` now connects to the Laravel route that serves it. Re-index Laravel projects after upgrading. - Controllers that Spring or Laravel tests exercise by URL now count as tested. That covers MockMvc's `perform(post("/owners/new"))`, WebTestClient, TestRestTemplate, RestAssured, Laravel's `$this->postJson('api/me')`, Pest's `get('/about')`, and a project's own request helpers built on them. `codegraph_explore` now names the test suite that reaches such an endpoint, where before every one of them looked untested. Re-index Spring and Laravel projects after upgrading. - In an Angular app, a method a template calls, like `(click)="toggleFavorite()"` or `(ngSubmit)="submitForm()"`, is now linked from its component. These methods used to show no callers at all, so impact and callers questions stopped at them, and dead-code checks could list them as unused. Re-index Angular projects after upgrading. diff --git a/README.md b/README.md index fc5d341bcc..e0aae08860 100644 --- a/README.md +++ b/README.md @@ -322,7 +322,7 @@ CodeGraph detects web-framework routing files and emits `route` nodes linked by | **Flask** | `@app.route('/path', methods=[...])`, blueprint routes | | **FastAPI** | `@app.get(...)`, `@router.post(...)`, all standard methods | | **Express** | `app.get(...)`, `router.post(...)` with middleware chains | -| **NestJS** | `@Controller` + `@Get/@Post/...`, GraphQL `@Resolver` + `@Query/@Mutation`, `@MessagePattern`/`@EventPattern`, `@SubscribeMessage` | +| **NestJS** | `@Controller` + `@Get/@Post/...` (with `RouterModule` prefixes, `setGlobalPrefix` and URI versioning), GraphQL `@Resolver` + `@Query/@Mutation`, `@MessagePattern`/`@EventPattern`, `@SubscribeMessage` | | **Laravel** | `Route::get()`, `Route::resource()`, `Controller@action`, tuple syntax | | **Drupal** | `*.routing.yml` routes (`_controller`, `_form`, entity handlers); `hook_*` implementations in `.module`/`.theme`/`.install`/`.inc` | | **Rails** | `get '/x', to: 'users#index'`, hash-rocket `=>` syntax | diff --git a/__tests__/nestjs-global-prefix.test.ts b/__tests__/nestjs-global-prefix.test.ts new file mode 100644 index 0000000000..fcb554a538 --- /dev/null +++ b/__tests__/nestjs-global-prefix.test.ts @@ -0,0 +1,217 @@ +/** + * A NestJS route is named by the path a request takes to it — the app's + * global prefix and URI version included. + * + * `app.setGlobalPrefix('api')` and `app.enableVersioning({ type: + * VersioningType.URI, defaultVersion: '1' })` in `main.ts` put every + * controller under `/api/v1`, but the routes were named `GET /user`. A front + * end's `this.http.get('/api/v1/user')` then connected to nothing: Ghostfolio's + * Angular client had no link to its own NestJS API. + */ +import { describe, it, expect, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +const projects: string[] = []; +afterAll(() => { + for (const p of projects.splice(0)) fs.rmSync(p, { recursive: true, force: true }); +}); + +async function project(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-nest-prefix-')); + projects.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); +} + +const MAIN = `import { NestFactory } from '@nestjs/core'; +import { VersioningType } from '@nestjs/common'; +import { AppModule } from './app/app.module'; +async function bootstrap() { + const app = await NestFactory.create(AppModule); + app.enableVersioning({ defaultVersion: '1', type: VersioningType.URI }); + app.setGlobalPrefix('api', { exclude: ['sitemap.xml', 'health{/*wildcard}'] }); + await app.listen(3333); +} +bootstrap(); +`; + +describe('NestJS: the global prefix and URI versioning', () => { + it('names each route under the prefix and its version — or neither, where the app says so', async () => { + const cg = await project({ + 'package.json': JSON.stringify({ dependencies: { '@nestjs/core': '^11.0.0', '@nestjs/common': '^11.0.0' } }), + 'apps/api/src/main.ts': MAIN, + // A seed script creates the same app without serving it — and is read first. + 'apps/api/src/database/run-seed.ts': `import { NestFactory } from '@nestjs/core'; +const runSeed = async () => { + const app = await NestFactory.create(SeedModule); + await app.close(); +}; +void runSeed(); +`, + 'apps/api/src/app/user.controller.ts': `import { Controller, Get, Post, Version, VERSION_NEUTRAL } from '@nestjs/common'; +@Controller('user') +export class UserController { + @Get() + public getUser() {} + + @Post() + @UseGuards(AuthGuard('jwt')) + @Version('2') + public createUser() {} + + @Get('oidc/callback') + @Version(VERSION_NEUTRAL) + public callback() {} +} +`, + 'apps/api/src/app/sitemap.controller.ts': `import { Controller, Get, Version, VERSION_NEUTRAL } from '@nestjs/common'; +@Controller() +@Version(VERSION_NEUTRAL) +export class SitemapController { + @Get('sitemap.xml') + public sitemap() {} +} +`, + 'apps/api/src/app/health.controller.ts': `import { Controller, Get } from '@nestjs/common'; +@Controller({ path: 'health', version: '3' }) +export class HealthController { + @Get('db') + public db() {} +} +`, + // A second app that sets nothing: its routes stay as written. + 'apps/jobs/src/main.ts': `import { NestFactory } from '@nestjs/core'; +NestFactory.create(JobsModule).then((app) => app.listen(3335)); +`, + 'apps/jobs/src/jobs.controller.ts': `import { Controller, Get } from '@nestjs/common'; +@Controller('jobs') +export class JobsController { + @Get() + public list() {} +} +`, + }); + try { + expect( + cg + .getNodesByKind('route') + .map((r) => r.name) + .sort() + ).toEqual([ + // VERSION_NEUTRAL: prefixed, not versioned. + 'GET /api/user/oidc/callback', + 'GET /api/v1/user', + 'GET /jobs', + 'GET /sitemap.xml', + // Excluded from the prefix, still versioned by its controller. + 'GET /v3/health/db', + 'POST /api/v2/user', + ]); + } finally { + cg.close(); + } + }); + + it('reads a version past decorators with object arguments, and each app its own bootstrap', async () => { + const cg = await project({ + 'package.json': JSON.stringify({ dependencies: { '@nestjs/core': '^11.0.0', '@nestjs/common': '^11.0.0' } }), + 'apps/api/src/main.ts': MAIN, + 'apps/api/src/app/stats.controller.ts': `import { Controller, Get, Version } from '@nestjs/common'; +@Controller('stats') +export class StatsController { + @Version('2') + @ApiResponse({ status: 200, description: 'The stats' }) + @Get('daily') + public daily() {} + + @Get('weekly') + public weekly() {} +} +`, + 'apps/api/src/app/reports.controller.ts': `import { Controller, Get, Version } from '@nestjs/common'; +@Version('4') +@ApiTags({ name: 'reports' }) +@Controller('reports') +export class ReportsController { + @Get() + public list() {} +} +`, + // A second app whose prefix is read from config: no prefix is claimed for + // it, and the other app's `/api` is not borrowed. + 'apps/admin/src/main.ts': `import { NestFactory } from '@nestjs/core'; +async function bootstrap() { + const app = await NestFactory.create(AdminModule); + app.setGlobalPrefix(config.get('app.prefix')); + app.enableVersioning({ defaultVersion: '1' }); + await app.listen(3334); +} +bootstrap(); +`, + 'apps/admin/src/users.controller.ts': `import { Controller, Get } from '@nestjs/common'; +@Controller('users') +export class UsersController { + @Get() + public list() {} +} +`, + }); + try { + expect( + cg + .getNodesByKind('route') + .map((r) => r.name) + .sort() + ).toEqual(['GET /api/v1/stats/weekly', 'GET /api/v2/stats/daily', 'GET /api/v4/reports', 'GET /v1/users']); + } finally { + cg.close(); + } + }); + + it("connects a front end's request to the route it reaches, and leaves an Express route in the same repo alone", async () => { + const cg = await project({ + 'package.json': JSON.stringify({ + dependencies: { '@nestjs/core': '^11.0.0', '@nestjs/common': '^11.0.0', '@angular/core': '^19.0.0', express: '^4.0.0' }, + }), + 'apps/api/src/main.ts': MAIN, + 'apps/api/src/app/user.controller.ts': `import { Controller, Get } from '@nestjs/common'; +@Controller('user') +export class UserController { + @Get() + public getUser() {} +} +`, + 'apps/client/src/app/user.service.ts': `import { HttpClient } from '@angular/common/http'; +export class UserService { + constructor(private http: HttpClient) {} + public fetchUser() { + return this.http.get('/api/v1/user'); + } +} +`, + 'tools/mock-server/src/server.ts': `import express from 'express'; +const app = express(); +app.get('/status', (req, res) => res.json({ ok: true })); +`, + }); + try { + const names = cg.getNodesByKind('route').map((r) => r.name).sort(); + expect(names).toContain('GET /api/v1/user'); + expect(names).toContain('GET /status'); + const fetchUser = cg.getNodesByName('fetchUser')[0]!; + const reached = cg + .getOutgoingEdgesFrom([fetchUser.id], ['calls']) + .filter((e) => (e.metadata as Record | undefined)?.channel === 'http') + .map((e) => cg.getNode(e.target)?.name); + expect(reached).toEqual(['GET /api/v1/user']); + } finally { + cg.close(); + } + }); +}); diff --git a/src/resolution/frameworks/nestjs.ts b/src/resolution/frameworks/nestjs.ts index 6ea13f5daa..dbadbbc9e6 100644 --- a/src/resolution/frameworks/nestjs.ts +++ b/src/resolution/frameworks/nestjs.ts @@ -208,14 +208,22 @@ export const nestjsResolver: FrameworkResolver = { postExtract(context: ResolutionContext): Node[] { const moduleToPrefix = new Map(); const controllerToModule = new Map(); + const bootstraps: NestBootstrap[] = []; for (const filePath of context.getAllFiles()) { - if (!/\.module\.(m?[jt]s|cjs)$/.test(filePath)) continue; + if (!/\.(m?[jt]s|cjs)$/.test(filePath)) continue; + const isModule = /\.module\.(m?[jt]s|cjs)$/.test(filePath); + if (!isModule && !(context.fileContains?.(filePath, 'NestFactory') ?? true)) continue; const content = context.readFile(filePath); if (!content) continue; const safe = stripCommentsForRegex(content, detectLanguage(filePath)); - collectRouterModuleRegistrations(safe, moduleToPrefix); - collectModuleControllers(safe, controllerToModule); + if (isModule) { + collectRouterModuleRegistrations(safe, moduleToPrefix); + collectModuleControllers(safe, controllerToModule); + } + // Every app counts, even one that adds nothing: its routes must not + // borrow another app's prefix. + if (/\bNestFactory\s*\.\s*create/.test(safe)) bootstraps.push({ root: appRootOf(filePath), config: readBootstrap(safe) }); } const controllerToPrefix = new Map(); @@ -228,35 +236,243 @@ export const nestjsResolver: FrameworkResolver = { } } - if (controllerToPrefix.size === 0) return []; + const adds = bootstraps.some((b) => b.config.prefix !== '' || b.config.uri); + if (controllerToPrefix.size === 0 && !adds) return []; - const updates: Node[] = []; + // Route → what the app adds to it. A RouterModule prefix binds through the + // controller class's line range; the bootstrap's prefix and versioning + // only to a route its source shows is in a `@Controller` — an Express + // route has the same qualified-name shape. + const plan = new Map(); for (const [controllerName, prefix] of controllerToPrefix) { - const classes = context - .getNodesByName(controllerName) - .filter((n) => n.kind === 'class'); - for (const cls of classes) { - const routes = context - .getNodesInFile(cls.filePath) - .filter((n) => n.kind === 'route'); + for (const cls of context.getNodesByName(controllerName).filter((n) => n.kind === 'class')) { + for (const route of context.getNodesInFile(cls.filePath)) { + if (route.kind !== 'route' || route.startLine < cls.startLine || route.startLine > cls.endLine) continue; + plan.set(route.id, { route, modulePrefix: prefix, config: null, version: undefined }); + } + } + } + if (adds) { + for (const filePath of context.getAllFiles()) { + if (!/\.(m?[jt]s|cjs)$/.test(filePath)) continue; + if (!(context.fileContains?.(filePath, '@Controller') ?? true)) continue; + const config = bootstrapFor(bootstraps, filePath); + if (!config || (config.prefix === '' && !config.uri)) continue; + const routes = context.getNodesInFile(filePath).filter((n) => n.kind === 'route' && HTTP_ROUTE_QN.test(n.qualifiedName)); + if (routes.length === 0) continue; + const content = context.readFile(filePath); + if (!content) continue; + const safe = stripCommentsForRegex(content, detectLanguage(filePath)); + const scopes = buildClassScopes(safe); + const hits = findDecorators(safe, HTTP_METHODS); for (const route of routes) { - // Multiple controllers can live in one file (covered by the - // existing "attributes methods to the right controller" test); - // each route must be associated with the controller whose line - // range contains it. - if (route.startLine < cls.startLine || route.startLine > cls.endLine) { - continue; - } - const updated = applyModulePrefix(route, prefix); - if (updated && updated.name !== route.name) updates.push(updated); + const hit = hits.find((h) => lineAt(safe, h.index) === route.startLine); + const scope = hit ? scopeFor(scopes, hit.index) : null; + if (!hit || !scope || scope.kind !== 'controller') continue; + const planned = plan.get(route.id); + plan.set(route.id, { route, modulePrefix: planned?.modulePrefix ?? '', config, version: routeVersion(safe, hit, scope) }); } } } + const updates: Node[] = []; + for (const { route, modulePrefix, config, version } of plan.values()) { + const updated = applyAppConfig(route, modulePrefix, config, version); + if (updated && updated.name !== route.name) updates.push(updated); + } return updates; }, }; +// --------------------------------------------------------------------------- +// The app's bootstrap: global prefix and URI versioning +// --------------------------------------------------------------------------- + +/** A route's in-file shape in its qualified name: `file::GET:/users`. */ +const HTTP_ROUTE_QN = /::(?:GET|POST|PUT|PATCH|DELETE|HEAD|OPTIONS|ALL):/; + +/** The version marker for `VERSION_NEUTRAL`: served without a version segment. */ +const NEUTRAL = '\u0000neutral'; + +interface NestAppConfig { + /** `setGlobalPrefix('api')` — '' when none, or not a literal. */ + prefix: string; + /** Paths the prefix skips: `exclude: ['health', { path: 'metrics', method }]`, without a leading `/`. */ + excludes: string[]; + /** URI versioning is on (`enableVersioning({ type: VersioningType.URI })`, the default type). */ + uri: boolean; + /** `defaultVersion: '1'` — the version a route without its own is served at; null for none. */ + defaultVersion: string | null; + /** The segment's prefix: `v` unless `prefix:` says otherwise (`false` is none). */ + versionPrefix: string; +} + +interface NestBootstrap { + root: string; + config: NestAppConfig; +} + +/** The app a file belongs to: the directory above its `src/` (`apps/api/`), else the repository. */ +function appRootOf(filePath: string): string { + const m = /^((?:[^/]+\/)*?)src\//.exec(filePath); + return m ? m[1]! : ''; +} + +/** + * The config of the app that holds `filePath`: the bootstrap with the longest + * root above it, else — for a file outside every app, like an Nx `libs/` + * controller — the app, when there is only one. + */ +function bootstrapFor(bootstraps: readonly NestBootstrap[], filePath: string): NestAppConfig | null { + let root: string | null = null; + for (const b of bootstraps) { + if (filePath.startsWith(b.root) && (root === null || b.root.length > root.length)) root = b.root; + } + if (root === null) { + if (new Set(bootstraps.map((b) => b.root)).size !== 1) return null; + root = bootstraps[0]!.root; + } + // A seed or migration script creates the same app without serving it: the + // bootstrap that configures the app is the one that serves it. + const here = bootstraps.filter((b) => b.root === root); + return (here.find((b) => b.config.prefix !== '' || b.config.uri) ?? here[0])?.config ?? null; +} + +/** One version value: `'2'`, `['1', '2']` (the first), `VERSION_NEUTRAL`. */ +function versionValue(text: string): string | null { + const t = text.trim(); + if (/^VERSION_NEUTRAL\b/.test(t)) return NEUTRAL; + const lit = /^\[?\s*(['"`])([^'"`]*)\1/.exec(t); + return lit ? lit[2]! : null; +} + +/** `setGlobalPrefix` and `enableVersioning` from the file that calls `NestFactory.create`. */ +function readBootstrap(safe: string): NestAppConfig { + const config: NestAppConfig = { prefix: '', excludes: [], uri: false, defaultVersion: null, versionPrefix: 'v' }; + const prefixCall = /\.setGlobalPrefix\s*\(/.exec(safe); + if (prefixCall) { + const open = prefixCall.index + prefixCall[0].length - 1; + const close = matchingClose(safe, open); + const args = close > open ? safe.slice(open + 1, close) : ''; + const lit = /^\s*(['"`])([^'"`$]*)\1/.exec(args); + if (lit) { + config.prefix = lit[2]!.replace(/^\/+|\/+$/g, ''); + const ex = /\bexclude\s*:\s*\[/.exec(args); + if (ex) { + const exOpen = ex.index + ex[0].length - 1; + const exClose = matchingClose(args, exOpen); + const list = exClose > exOpen ? args.slice(exOpen + 1, exClose) : ''; + for (const item of splitTopLevel(list)) { + const text = item.trim(); + const path = /^(['"])([^'"]*)\1$/.exec(text)?.[2] ?? /\bpath\s*:\s*(['"])([^'"]*)\1/.exec(text)?.[2]; + if (path !== undefined) config.excludes.push(path.replace(/^\/+/, '')); + } + } + } + } + const versioning = /\.enableVersioning\s*\(/.exec(safe); + if (versioning) { + const open = versioning.index + versioning[0].length - 1; + const close = matchingClose(safe, open); + const args = close > open ? safe.slice(open + 1, close) : ''; + const type = /\btype\s*:\s*VersioningType\s*\.\s*(\w+)/.exec(args)?.[1]; + config.uri = type === undefined || type === 'URI'; + const def = /\bdefaultVersion\s*:\s*/.exec(args); + if (def) config.defaultVersion = versionValue(args.slice(def.index + def[0].length)); + const pre = /\bprefix\s*:\s*(false|(['"`])([^'"`]*)\2)/.exec(args); + if (pre) config.versionPrefix = pre[1] === 'false' ? '' : pre[3]!; + } + return config; +} + +/** A route excluded from the global prefix: `health`, `health/(.*)`, `health{/*wildcard}`, `health/*`. */ +function isExcluded(path: string, excludes: readonly string[]): boolean { + const route = path.replace(/^\/+/, ''); + for (const pattern of excludes) { + const wild = /(?:\{\/?\*\w*\}|\/?\(\.\*\)|\/?\*\w*)$/.exec(pattern); + const base = wild ? pattern.slice(0, wild.index).replace(/\/+$/, '') : pattern.replace(/\/+$/, ''); + if (route === base || (wild && route.startsWith(`${base}/`))) return true; + } + return false; +} + +/** + * The version a route is served at: `@Version(…)` among its own decorators, + * else its controller's (`@Version(…)` on the class, `@Controller({ version })`). + * Undefined when neither says. + */ +function routeVersion(safe: string, hit: DecoratorHit, scope: ClassScope): string | null | undefined { + // The method's decorators: back through the ones stacked above this one… + const from = decoratorChainStart(safe, hit.index, scope.start); + // …and on to the handler itself — past the decorators between (`@UseGuards(…)`, `@Version('2')`). + const handler = methodNameAfter(safe, hit.end); + const at = handler ? new RegExp(String.raw`(? floor && /\s/.test(safe[i - 1]!)) i--; + if (safe[i - 1] === ')') { + let depth = 0; + let j = i - 1; + for (; j >= floor; j--) { + if (safe[j] === ')') depth++; + else if (safe[j] === '(' && --depth === 0) break; + } + if (j < floor) return start; + i = j; + } + let name = i; + while (name > floor && /[\w$.]/.test(safe[name - 1]!)) name--; + if (name === i || safe[name - 1] !== '@') return start; + start = name - 1; + } +} + +/** + * A route's name with everything the app adds to it, in NestJS's order: + * `/{globalPrefix}/{v}{version}/{RouterModule prefix}/{controller}/{method}`. + * Recomputed from the in-file `method:path` in the qualified name, so the pass + * is idempotent. + */ +function applyAppConfig(route: Node, modulePrefix: string, config: NestAppConfig | null, version: string | null | undefined): Node | null { + const sep = route.qualifiedName.indexOf('::'); + if (sep < 0) return null; + const tail = route.qualifiedName.slice(sep + 2); + const colon = tail.indexOf(':'); + if (colon < 0) return null; + const method = tail.slice(0, colon); + const inner = joinHttpPath(modulePrefix, tail.slice(colon + 1)); + let path = inner; + if (config) { + const v = version === undefined ? config.defaultVersion : version; + const versionSeg = config.uri && v !== null && v !== NEUTRAL ? `${config.versionPrefix}${v}` : ''; + const prefix = config.prefix && !isExcluded(inner, config.excludes) ? config.prefix : ''; + path = joinHttpPath(joinHttpPath(prefix, versionSeg), inner); + } + return { ...route, name: `${method} ${path}`, updatedAt: Date.now() }; +} + // --------------------------------------------------------------------------- // Provider resolution conventions // --------------------------------------------------------------------------- @@ -646,29 +862,36 @@ function classNameAfter(safe: string, start: number): string | null { return m ? m[1]! : null; } -/** - * Recompute a route node's `name` by prepending `prefix` to the *original* - * in-file path. The original is recovered from `qualifiedName`, which the - * per-file extract emits as `${filePath}::${method}:${path}` and which this - * pass deliberately never mutates — that's what keeps the update idempotent. - */ -function applyModulePrefix(route: Node, prefix: string): Node | null { - const sep = '::'; - const idx = route.qualifiedName.indexOf(sep); - if (idx < 0) return null; - const tail = route.qualifiedName.slice(idx + sep.length); - const colon = tail.indexOf(':'); - if (colon < 0) return null; - const method = tail.slice(0, colon); - const original = tail.slice(colon + 1); - const newName = `${method} ${joinHttpPath(prefix, original)}`; - return { ...route, name: newName, updatedAt: Date.now() }; -} // --------------------------------------------------------------------------- // Small string utilities (object/array literal splitters) // --------------------------------------------------------------------------- +/** A list's comma-separated items at depth 0, strings and brackets stepped over. */ +function splitTopLevel(list: string): string[] { + const out: string[] = []; + let depth = 0; + let inStr: string | null = null; + let start = 0; + for (let i = 0; i < list.length; i++) { + const ch = list[i]!; + if (inStr) { + if (ch === '\\') { i++; continue; } + if (ch === inStr) inStr = null; + continue; + } + if (ch === '"' || ch === "'" || ch === '`') { inStr = ch; continue; } + if (ch === '[' || ch === '{' || ch === '(') depth++; + else if (ch === ']' || ch === '}' || ch === ')') depth--; + else if (ch === ',' && depth === 0) { + out.push(list.slice(start, i)); + start = i + 1; + } + } + out.push(list.slice(start)); + return out.filter((item) => item.trim().length > 0); +} + /** Return the index of the bracket that closes the one at `open`, or -1. */ function matchingClose(s: string, open: number): number { const opener = s[open]; From b50c7c0a70bc666a7855f531b62899500074c8b4 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 03:59:28 +0000 Subject: [PATCH 004/259] fix(swift): a call through a type path lands on the type it names (#2118) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `API.PackageController.GetRoute.query(on:)` kept only `query` — the receiver was a navigation_expression, not an identifier — and the bare name exact-matched whichever type's `query` came first. On SwiftPackageIndex-Server every route type has one, so health checks and package-controller tests "called" the dependency controller (or a test's own `query`). Extraction (TS and the kernel's swift walker, byte-parity): a receiver that is a path of capitalized type names, two segments or more, not led by `Self.`, is kept: `API.PackageController.GetRoute.query`. Instance chains (`self.store.load()`) and mixed paths (`URLSession.shared.data`) are unchanged. Resolution (swift-type-visibility.ts, called decisively from matchReferenceInner so nothing falls through to name guessing): the member on the type the path names — a method, an enum case with associated values, or a nested type's initializer. A member's owner path is its qualified name with the outermost extension's written path read back from its line (`extension API.PackageController { enum ShowRoute … }` → `API::PackageController::ShowRoute`), so twin `Feature::State`s declared in `extension BasicsView.Feature` and `extension ObservableBasicsView.Feature` are told apart. A path shortened inside its namespace or behind a module qualifier fits; two owners, or none, leave the call unresolved. TCA's `@Reducer enum Path { case detail }` generates `Path.State`/`Path.Action`, so `Path.State.detail(…)` lands on the written case. A/B vs main (edge diffs, every change classified): SPI 76 retargeted, all to the type the line names, 7 dropped (Fluent `App.Version.query`, `Plot.Node.*` — SDK members main pinned on project methods); TCA 61 retargeted, 1 dropped (`State.StateReducer.scope` → an unrelated `Store::scope`); mastodon-ios 177 retargeted (`Mastodon.API.V2.Instance.instance`, `L10n.Plural.Count.vote`), 11 SDK drops; swift-nio 6 calls into a deprecated alias became instantiates of `ChannelOptions.Types.*`; IceCubes 1/1, Alamofire 2 SDK drops. Kernel/wasm parity 0 diffs over 2,752 files on six repos. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + .../fixtures/kernel-parity/torture.swift | 7 + __tests__/swift-type-path-calls.test.ts | 135 ++++++++++++++++++ codegraph-kernel/src/swift.rs | 28 +++- src/extraction/tree-sitter.ts | 17 +++ src/resolution/name-matcher.ts | 8 ++ src/resolution/swift-type-visibility.ts | 120 +++++++++++++++- 7 files changed, 314 insertions(+), 2 deletions(-) create mode 100644 __tests__/swift-type-path-calls.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 2ab730c2d1..4b16268308 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- A Swift call through a nested type, like `API.PackageController.GetRoute.query(on: db)`, now links to that type's method. Only `query` used to be kept, so the call linked to whichever type's `query` came first; in a Vapor app every route type has one, so callers, impact and `codegraph_explore` answers pointed at another endpoint's code. The same applies to nested types' initializers (`API.PackageController.Model(name:)`), to enum cases (`Gitlab.Error.requestFailed(status:)`), to types declared in an `extension API.PackageController { … }`, and to The Composable Architecture's `Path.State.detail(…)`. A call on a type path that names no type in the project, such as an SDK's `Plot.Node.footer(…)`, is left unlinked. Re-index Swift projects after upgrading. - NestJS routes are now named by the path a request takes: the global prefix from `app.setGlobalPrefix('api')` is applied, along with its `exclude` list, and so is URI versioning from `app.enableVersioning(...)`, with `@Version`, `VERSION_NEUTRAL` and a controller's own `version` taken into account. A route that used to appear as `GET /user` is now `GET /api/v1/user`, so a front end's `this.http.get('/api/v1/user')` or `fetch` call connects to the controller method that serves it. Re-index NestJS projects after upgrading. - Laravel routes are now named by the path a request takes: a leading `/` is added where the routes file leaves it out, `Route::prefix()` and `Route::group(['prefix' => …])` groups are applied, and routes in `routes/api.php` carry the `/api` prefix Laravel serves them under, read from your `RouteServiceProvider`, `bootstrap/app.php` or any file that mounts a routes file. A front-end `fetch('/api/…')` now connects to the Laravel route that serves it. Re-index Laravel projects after upgrading. - Controllers that Spring or Laravel tests exercise by URL now count as tested. That covers MockMvc's `perform(post("/owners/new"))`, WebTestClient, TestRestTemplate, RestAssured, Laravel's `$this->postJson('api/me')`, Pest's `get('/about')`, and a project's own request helpers built on them. `codegraph_explore` now names the test suite that reaches such an endpoint, where before every one of them looked untested. Re-index Spring and Laravel projects after upgrading. diff --git a/__tests__/fixtures/kernel-parity/torture.swift b/__tests__/fixtures/kernel-parity/torture.swift index d0c4a3e06b..6cb31f010d 100644 --- a/__tests__/fixtures/kernel-parity/torture.swift +++ b/__tests__/fixtures/kernel-parity/torture.swift @@ -86,6 +86,13 @@ func freeFn(a: Int, cb: @escaping (Int) -> Void) -> Session? { x?.optCall() y!.forced() Foo.make().draw() + API.PackageController.GetRoute.query(on: db) + API.PackageController + .GetRoute.query(on: db) + API.Model(name: "x") + Self.Inner.make() + App.lower.call() + API.Ünicode.query() foo.bar().baz() "lit".upper() arr.map { $0.name } diff --git a/__tests__/swift-type-path-calls.test.ts b/__tests__/swift-type-path-calls.test.ts new file mode 100644 index 0000000000..3e9d8c0e12 --- /dev/null +++ b/__tests__/swift-type-path-calls.test.ts @@ -0,0 +1,135 @@ +/** + * A Swift call through a type path lands on the type the path names. + * + * `API.PackageController.GetRoute.query(on:)` kept only `query`, which + * exact-matched whichever type's `query` came first — in a Vapor app every + * route type has one, so a health check "called" the dependency controller's + * query. The path is now kept and resolved on its type: declared nested, + * in `extension API.PackageController { … }`, in `extension A.B.C { … }`, + * shortened inside its namespace, or module-qualified. A path that names no + * project type is left unresolved. + */ +import { describe, it, expect, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +const roots: string[] = []; +afterAll(() => { + for (const r of roots.splice(0)) fs.rmSync(r, { recursive: true, force: true }); +}); + +const FILES: Record = { + 'Sources/App/API.swift': `enum API {} +extension API { + enum PackageController { + enum GetRoute { + static func query(on db: Int) -> Int { db } + static func query(on db: Int, limit: Int) -> Int { db + limit } + } + struct Model { let name: String } + } +} +extension API.PackageController { + enum ShowRoute { + static func query(on db: Int) -> Int { db } + } +} +extension API.DependencyController.GetRoute { + static func query(on db: Int) -> Int { db } +} +extension API { + enum DependencyController { + enum GetRoute {} + static func inside() -> Int { PackageController.GetRoute.query(on: 0) } + } +} +`, + 'Sources/App/Other.swift': `enum Other { static func query(on db: Int) -> Int { db } } +enum Gitlab { enum Error: Swift.Error { case requestFailed(status: Int) } } +enum Remote { enum Error: Swift.Error { case requestFailed(status: Int) } } +struct Detail {} +enum Screens { + @Reducer + enum Path { case detail(Detail) } +} +enum Zed { enum GetRoute { static func query(on db: Int) -> Int { db } } } +enum Yon { enum GetRoute { static func query(on db: Int) -> Int { db } } } +`, + 'Sources/App/Basics.swift': `struct BasicsView {} +struct ObservableBasicsView {} +extension BasicsView { struct Feature {} } +extension ObservableBasicsView { struct Feature {} } +extension BasicsView.Feature { struct State {} } +extension ObservableBasicsView.Feature { struct State {} } +`, + 'Sources/App/Health.swift': `import Foundation +enum Health { + static func one() -> Int { API.PackageController.GetRoute.query(on: 1) } + static func two() -> Int { + API.PackageController.GetRoute + .query(on: 2) + } + static func three() -> Int { Other.query(on: 3) } + static func four() -> Int { API.PackageController.ShowRoute.query(on: 4) } + static func five() -> Int { API.DependencyController.GetRoute.query(on: 5) } + static func six() -> API.PackageController.Model { API.PackageController.Model(name: "x") } + static func seven() -> Int { App.Zed.GetRoute.query(on: 7) } + static func eight() -> Int { Missing.Thing.query(on: 8) } + static func nine() -> Int { API.PackageController.GetRoute.nothing(on: 9) } + static func ten() throws { throw Gitlab.Error.requestFailed(status: 10) } + static func eleven() { _ = Screens.Path.State.detail(Detail()) } + static func twelve() { _ = ObservableBasicsView.Feature.State() } +} +`, +}; + +describe('Swift: a call through a type path', () => { + it('lands on the type the path names, or nowhere', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-swift-type-path-')); + roots.push(root); + for (const [rel, content] of Object.entries(FILES)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + const cg = await CodeGraph.init(root, { index: true }); + try { + const reached = (caller: string): string[] => { + const from = cg.getNodesByName(caller).find((n) => n.kind === 'method'); + expect(from, caller).toBeDefined(); + return cg + .getOutgoingEdgesFrom([from!.id], ['calls', 'instantiates']) + .map((e) => cg.getNode(e.target)) + .filter((n) => n && n.language === 'swift') + .map((n) => `${n!.qualifiedName}:${n!.startLine}`) + .sort(); + }; + // The first declared of GetRoute's two overloads. + expect(reached('one')).toEqual(['API::PackageController::GetRoute::query:5']); + expect(reached('two')).toEqual(['API::PackageController::GetRoute::query:5']); + expect(reached('three')).toEqual(['Other::query:1']); + // Declared in `extension API.PackageController { enum ShowRoute … }`. + expect(reached('four')).toEqual(['PackageController::ShowRoute::query:13']); + // Declared in `extension API.DependencyController.GetRoute { … }`. + expect(reached('five')).toEqual(['GetRoute::query:17']); + // A nested type's initializer. + expect(reached('six')).toEqual(['API::PackageController::Model:8']); + // A module qualifier. + expect(reached('seven')).toEqual(['Zed::GetRoute::query:9']); + // No such type, and no such member: nothing, not some other type's `query`. + expect(reached('eight')).toEqual([]); + expect(reached('nine')).toEqual([]); + // A case with associated values. + expect(reached('ten')).toEqual(['Gitlab::Error::requestFailed:2']); + // The case a TCA `@Reducer enum` generates its State from. + expect(reached('eleven')).toContain('Screens::Path::detail:7'); + // Two types share the qualified name `Feature::State`; the extension each is declared in tells them apart. + expect(reached('twelve')).toEqual(['Feature::State:6']); + // Shortened inside its namespace. + expect(reached('inside')).toEqual(['API::PackageController::GetRoute::query:5']); + } finally { + cg.close(); + } + }); +}); diff --git a/codegraph-kernel/src/swift.rs b/codegraph-kernel/src/swift.rs index 1613888c3e..f1d3ae4ac9 100644 --- a/codegraph-kernel/src/swift.rs +++ b/codegraph-kernel/src/swift.rs @@ -110,6 +110,24 @@ fn strip_js_ws(s: &str) -> String { s.chars().filter(|c| !is_js_space(*c)).collect() } +/// SWIFT_TYPE_PATH_RECEIVER (tree-sitter.ts): `API.PackageController.GetRoute` — +/// two segments or more, each an ASCII capital then `[A-Za-z0-9_]*` (JS `\w`), +/// not led by `Self.`. +fn is_type_path(s: &str) -> bool { + if s.starts_with("Self.") { + return false; + } + let mut segments = 0; + for seg in s.split('.') { + let b = seg.as_bytes(); + if b.is_empty() || !b[0].is_ascii_uppercase() || !b[1..].iter().all(|c| c.is_ascii_alphanumeric() || *c == b'_') { + return false; + } + segments += 1; + } + segments >= 2 +} + struct Scope { row: u32, kind: &'static str, @@ -1061,8 +1079,16 @@ impl<'t> Walker<'t> { } else { method_name.to_string() }; + } else if let Some(path) = receiver + .filter(|r| r.kind() == "navigation_expression") + .map(|r| strip_js_ws(self.text(r))) + .filter(|p| is_type_path(p)) + { + // A type path, `API.PackageController.GetRoute.query(on:)`: + // keep it for the resolver (tree-sitter.ts SWIFT_TYPE_PATH_RECEIVER). + callee_name = format!("{path}.{method_name}"); } else { - // self_expression / super_expression / inner nav / + // self_expression / super_expression / instance nav / // postfix / multi_line_string_literal → bare method name. callee_name = method_name.to_string(); } diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index 19c16471a2..6522bcfbd0 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -411,6 +411,8 @@ const TS_JS_CHAIN_LANGUAGES = new Set(['typescript', 'tsx', 'javascript', 'jsx'] const TS_JS_CHAIN_RECEIVER_TYPES = new Set(['member_expression', 'subscript_expression']); /** The field of a `this..()` receiver: public or ES private (#1496, #1987). */ const THIS_FIELD_PROPERTY_TYPES = new Set(['property_identifier', 'private_property_identifier']); +/** A Swift receiver that is a path of types, `API.PackageController.GetRoute` — two segments or more, each capitalized. */ +const SWIFT_TYPE_PATH_RECEIVER = /^(?!Self\.)[A-Z]\w*(?:\.[A-Z]\w*)+$/; /** * Identifier-rooted member chains have no inferred property type (#1566), @@ -4902,6 +4904,21 @@ export class TreeSitterExtractor { else reencode = !!innerCallee; } calleeName = reencode ? `${innerCallee}().${methodName}` : methodName; + } else if ( + this.language === 'swift' && + receiver && + receiver.type === 'navigation_expression' && + SWIFT_TYPE_PATH_RECEIVER.test(getNodeText(receiver, this.source).replace(/\s+/g, '')) + ) { + // Swift call through a type path — `API.PackageController.GetRoute.query(on:)`, + // on one line or split before the `.query`. Keep the path: the + // bare method name this used to emit exact-matched whichever + // type's `query` came first (every route in a Vapor app has one). + // The resolver finds the member on the type the path names, or + // leaves the call unresolved. An instance chain (`self.store.load()`, + // `viewModel.state.reset()`) is not a type path and stays bare. + // Mirrored in the kernel's extract_call (swift.rs). + calleeName = `${getNodeText(receiver, this.source).replace(/\s+/g, '')}.${methodName}`; } else if ( this.language === 'cfscript' && receiver && diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index d2675bddd8..878b820a7d 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -9,6 +9,7 @@ import { Language, Node } from '../types'; import { UnresolvedRef, ResolvedRef, ResolutionContext, isSupertypeTarget, CPP_DEFINE_SIGNATURE, isInheritanceRef, isImportableKind } from './types'; import { blankStringContents, stripCommentsForRegex } from './strip-comments'; import { JS_BUILT_INS, JS_BUILTIN_METHODS, TS_PRIMITIVE_TYPES } from './js-builtins'; +import { SWIFT_TYPE_PATH_CALL, resolveSwiftTypePathCall } from './swift-type-visibility'; /** * Ceiling on how many same-named definitions a FUZZY name-match strategy will * score. A name defined more times than this is "ubiquitous" — a method/symbol @@ -4383,6 +4384,13 @@ function matchReferenceInner( if (isUnresolvedJsMemberCall(ref)) return null; + // A Swift call through a type path (`API.PackageController.GetRoute.query`) + // resolves on the type the path names, or not at all: the strategies below + // would bind it by the member's name alone. + if (ref.language === 'swift' && ref.referenceKind === 'calls' && SWIFT_TYPE_PATH_CALL.test(ref.referenceName)) { + return nmTimed('swiftTypePath', ref, () => resolveSwiftTypePathCall(ref, context)); + } + // Try strategies in order of confidence let result: ResolvedRef | null; diff --git a/src/resolution/swift-type-visibility.ts b/src/resolution/swift-type-visibility.ts index 9c6442ce68..0e1446c183 100644 --- a/src/resolution/swift-type-visibility.ts +++ b/src/resolution/swift-type-visibility.ts @@ -39,13 +39,14 @@ interface Memo { conformances: Map; clauses: Map; inherited: Map>; + owners: Map; } const memos = new WeakMap(); function memoFor(context: ResolutionContext): Memo { let memo = memos.get(context); if (!memo) { - memo = { extension: new Map(), declaration: new Map(), conformances: new Map(), clauses: new Map(), inherited: new Map() }; + memo = { extension: new Map(), declaration: new Map(), conformances: new Map(), clauses: new Map(), inherited: new Map(), owners: new Map() }; memos.set(context, memo); } return memo; @@ -340,3 +341,120 @@ export function gateSwiftTypeTarget(result: ResolvedRef | null, ref: UnresolvedR if (!declared) return null; return declared.id === result.targetNodeId ? result : { ...result, targetNodeId: declared.id }; } + +/** + * A call through a type path, `API.PackageController.GetRoute.query` — at + * least two capitalized segments before the member (the extractor keeps the + * path only for those; see SWIFT_TYPE_PATH_RECEIVER in tree-sitter.ts). + */ +export const SWIFT_TYPE_PATH_CALL = /^(?!Self\.)[A-Z]\w*(?:\.[A-Z]\w*)+\.[A-Za-z_]\w*$/; + +/** What a type-path call can name: a method, or a case with associated values (`Gitlab.Error.requestFailed(status:)`). */ +const PATH_MEMBER_KINDS: ReadonlySet = new Set(['method', 'function', 'enum_member']); + +/** + * `API.PackageController.GetRoute.query(on:)` lands on the `query` of the + * type the path names, and `API.PackageController.Model(name:)` on that + * nested type — never on the member's name alone: in a Vapor app every + * route's type has a `query`. A path the call's own namespace lets it + * shorten (`PackageController.GetRoute` inside `extension API`) or a module + * qualifier (`Vapor.HTTPStatus`) still fits. No type on the path, or two, + * leaves the call unresolved. + */ +export function resolveSwiftTypePathCall(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + const name = ref.referenceName; + const dot = name.lastIndexOf('.'); + const member = name.slice(dot + 1); + const path = name.slice(0, dot).split('.').join('::'); + const onPath = (kinds: ReadonlySet): Node[] => { + const exact: Node[] = []; + const shortened: Node[] = []; + for (const n of context.getNodesByName(member)) { + if (n.language !== 'swift' || !kinds.has(n.kind)) continue; + if (TYPE_KINDS.has(n.kind) && isSwiftExtension(n, context)) continue; + const owner = ownerPath(n, context); + if (owner === null) continue; + if (owner === path) exact.push(n); + else if (owner.endsWith(`::${path}`) || moduleQualified(owner, path, context)) shortened.push(n); + } + return exact.length > 0 ? exact : shortened; + }; + // `API.PackageController.Model(name:)` constructs a nested type. + let fit = /^[A-Z]/.test(member) ? onPath(TYPE_KINDS) : []; + if (fit.length === 0) fit = onPath(PATH_MEMBER_KINDS); + // The Composable Architecture's `@Reducer enum Path { case detail(Detail) }` + // generates `Path.State` and `Path.Action` with the same cases, so + // `Path.State.detail(…)` constructs the case written in `Path`. + if (fit.length === 0) { + const reducer = /^(.+)::(?:State|Action)$/.exec(path)?.[1]; + if (reducer) { + fit = context.getNodesByName(member).filter((n) => { + if (n.language !== 'swift' || n.kind !== 'enum_member') return false; + const owner = ownerPath(n, context); + return owner !== null && (owner === reducer || owner.endsWith(`::${reducer}`)) && isReducerEnum(n, context); + }); + } + } + if (fit.length === 0 || new Set(fit.map((n) => ownerPath(n, context))).size > 1) return null; + // Of one type's overloads, the first declared. + const target = fit.reduce((a, b) => (a.filePath < b.filePath || (a.filePath === b.filePath && a.startLine <= b.startLine) ? a : b)); + return { original: ref, targetNodeId: target.id, confidence: 0.9, resolvedBy: 'qualified-name' }; +} + +/** Is this case declared in a `@Reducer enum`? */ +function isReducerEnum(member: Node, context: ResolutionContext): boolean { + const owner = context + .getNodesInFile(member.filePath) + .filter((t) => t.kind === 'enum' && t.startLine <= member.startLine && t.endLine >= member.endLine) + .reduce((inner, t) => (!inner || t.startLine >= inner.startLine ? t : inner), null); + if (!owner) return false; + const lines = linesOf(owner.filePath, context) ?? []; + // The node starts at its attributes; `@Reducer` may also sit on the line above. + return /@Reducer\b/.test(lines.slice(Math.max(0, owner.startLine - 2), owner.startLine + 1).join(' ')); +} + +/** `Vapor::HTTPStatus` for a type `HTTPStatus`: a qualifier that names no project type is a module. */ +function moduleQualified(owner: string, path: string, context: ResolutionContext): boolean { + if (!path.endsWith(`::${owner}`)) return false; + const head = path.slice(0, path.length - owner.length - 2); + return !head.includes('::') && !context.getNodesByName(head).some((n) => n.language === 'swift' && TYPE_KINDS.has(n.kind)); +} + +/** + * The full path of the type a member is declared on. Its qualified name + * starts at the outermost declaration in its file, and an extension is named + * by its last segment — the members of `extension API.PackageController { + * enum GetRoute { … } }` are `PackageController::GetRoute::…` — so the + * outermost extension's written path is read back from its line. Null for a + * top-level function. + */ +function ownerPath(member: Node, context: ResolutionContext): string | null { + const memo = memoFor(context).owners; + const hit = memo.get(member.id); + if (hit !== undefined) return hit; + const cut = member.qualifiedName.lastIndexOf('::'); + let path: string | null = cut < 0 ? null : member.qualifiedName.slice(0, cut); + if (path !== null) { + const first = path.split('::')[0]!; + const outer = context + .getNodesInFile(member.filePath) + .filter((t) => t.name === first && TYPE_KINDS.has(t.kind) && t.startLine <= member.startLine && t.endLine >= member.endLine) + .reduce((out, t) => (!out || t.startLine < out.startLine ? t : out), null); + const extended = outer ? extendedPath(outer, context) : null; + if (extended) path = extended + path.slice(first.length); + } + memo.set(member.id, path); + return path; +} + +/** `extension API.PackageController {` → `API::PackageController`; null for anything but an extension of a nested type. */ +function extendedPath(node: Node, context: ResolutionContext): string | null { + if (!isSwiftExtension(node, context)) return null; + const lines = linesOf(node.filePath, context); + if (!lines) return null; + const head = [(lines[node.startLine - 1] ?? '').slice(node.startColumn), ...lines.slice(node.startLine, node.startLine + 3)] + .join(' ') + .replace(/"(?:[^"\\\n]|\\.)*"/g, '""'); + const written = /\bextension\s+((?:[A-Za-z_]\w*\s*\.\s*)+[A-Za-z_]\w*)/.exec(head)?.[1]; + return written ? written.replace(/\s+/g, '').split('.').join('::') : null; +} From 7cc2edadce4363f30559782328e07f3d00f75d7e Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 04:17:34 +0000 Subject: [PATCH 005/259] fix(ui): the viewer server exits when the command it runs under is killed (#2119) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `codegraph ui` re-execs itself with `--liftoff-only`; the shim the user started blocks in spawnSync and cannot forward a signal. Killing it by pid — a process manager, an IDE task, `kill` — left the re-exec'd server serving its port forever (found during a sweep: servers from hours earlier were still answering on ports 4801–4808 against deleted indexes, holding their stdout pipes so the harness that started them never exited). Ctrl+C was unaffected (the signal reaches the whole process group). `index`/`init` already guard this with the #277 PPID watchdog. Factor it out of installCommandSupervision as `watchParent(onLost)` and use it in `ui` with the same shutdown as SIGINT/SIGTERM (close the index, then the socket). The liveness watchdog is deliberately not installed for `ui`: a long Steps computation can legitimately hold the event loop. Test: POSIX end-to-end through the real relaunch (CODEGRAPH_WASM_RELAUNCHED unset, 200ms poll) — SIGKILL the shim, the port stops answering; confirmed failing without the fix. Co-authored-by: Claude Opus 5.5 --- __tests__/cli-ui-command.test.ts | 23 ++++++++++++ docs/viewer-launch-changelog.md | 2 ++ src/bin/codegraph.ts | 6 +++- src/bin/command-supervision.ts | 62 +++++++++++++++++++------------- 4 files changed, 67 insertions(+), 26 deletions(-) diff --git a/__tests__/cli-ui-command.test.ts b/__tests__/cli-ui-command.test.ts index cb9a5edcac..d2bd848982 100644 --- a/__tests__/cli-ui-command.test.ts +++ b/__tests__/cli-ui-command.test.ts @@ -328,6 +328,29 @@ describe('codegraph ui — serving', () => { } }, 60_000); + // `codegraph ui` re-execs itself with `--liftoff-only`, and the command the + // user started blocks in spawnSync, unable to forward a signal: killing it by + // pid (a process manager, an IDE task, `kill`) used to leave the server + // serving the port forever. POSIX-only: it relies on reparenting. + it.runIf(process.platform !== 'win32')('stops serving when the command it was started as is killed', async () => { + const viewer = await startViewer(['--no-open', '--port', '0', projectDir], { + CODEGRAPH_WASM_RELAUNCHED: '', + CODEGRAPH_PPID_POLL_MS: '200', + }); + expect((await get(viewer.port, '/api/stats')).status).toBe(200); + viewer.child.kill('SIGKILL'); + const deadline = Date.now() + 15_000; + let serving = true; + while (serving && Date.now() < deadline) { + await new Promise((r) => setTimeout(r, 200)); + serving = await get(viewer.port, '/api/stats').then( + () => true, + () => false + ); + } + expect(serving).toBe(false); + }, 60_000); + it('refuses a foreign Host end-to-end', async () => { const viewer = await startViewer(['--no-open', '--port', '0', projectDir], {}); try { diff --git a/docs/viewer-launch-changelog.md b/docs/viewer-launch-changelog.md index 79a50d451c..4d6e75fac4 100644 --- a/docs/viewer-launch-changelog.md +++ b/docs/viewer-launch-changelog.md @@ -169,6 +169,8 @@ These describe `codegraph ui` and its screens. They were taken out of `## [Unrel ## Fixes — Symbols, tests and the viewer +- **Stopping `codegraph ui` stops the server.** Killing the command by its process id, as a process manager, an editor task or `kill` does, left the server running on its port with no way to reach it, until the machine restarted. The server now notices it has been left behind and shuts down, closing the index first. Ctrl+C was never affected. + - **A SwiftUI view with a preview is no longer listed as a file that runs something.** A `#Preview { … }` sits at the top level of a view's file, so every view with one showed up under entry points as if it ran code. A preview is Xcode's canvas, not code the app runs: its calls no longer count, and a file whose only top-level code is previews leaves the list. - **The Map opens a Maven or Gradle project on its packages.** A Java project keeps every file under `src/main/java/org///`, and those folders hold nothing but the next one, so the Map drew the whole program as one `src/main/java/org` box and no grouping option reached further. A folder with one subfolder and no files of its own no longer counts as a level: petclinic opens on `owner`, `vet`, `model` and `system`, each labelled `src/main/java/…/petclinic/owner`, with the full path on hover. diff --git a/src/bin/codegraph.ts b/src/bin/codegraph.ts index e0404edae0..38b4e55b56 100644 --- a/src/bin/codegraph.ts +++ b/src/bin/codegraph.ts @@ -63,7 +63,7 @@ import { ansiColorsEnabled } from '../ui/color'; import { buildNode25BlockBanner, buildNodeTooOldBanner, MIN_NODE_MAJOR } from './node-version-check'; import { installFatalHandlers } from './fatal-handler'; import { relaunchWithWasmRuntimeFlagsIfNeeded } from '../extraction/wasm-runtime-flags'; -import { installCommandSupervision } from './command-supervision'; +import { installCommandSupervision, watchParent } from './command-supervision'; import { EXTRACTION_VERSION } from '../extraction/extraction-version'; import { getTelemetry, TELEMETRY_DOCS, recordIndexEvent } from '../telemetry'; // Value import, but dependency-free by design so `--help` text can name the @@ -2140,6 +2140,10 @@ ${BROWSER_ENV}=none to never open one. }; process.once('SIGINT', shutdown); process.once('SIGTERM', shutdown); + // Killing the command the user started (its pid, not Ctrl+C's process + // group) leaves this re-exec'd server with no parent to forward the + // signal: shut down the same way instead of serving the port forever. + watchParent(shutdown); }); /** diff --git a/src/bin/command-supervision.ts b/src/bin/command-supervision.ts index 6031a49991..a55bbffb14 100644 --- a/src/bin/command-supervision.ts +++ b/src/bin/command-supervision.ts @@ -59,38 +59,50 @@ export function installCommandSupervision(label: string, watchdog: WatchdogOptio // PPID watchdog: detect that the parent (or the host threaded past the // relaunch shim) died and we've been orphaned, then exit instead of leaking. - // Baseline from the CLI entry's earliest-possible capture — reading - // process.ppid here would miss a launcher killed during startup (#1185). - const originalPpid = EARLY_PPID; - const hostPpid = parseHostPpid(process.env[HOST_PPID_ENV]); - const pollMs = parsePpidPollMs(process.env.CODEGRAPH_PPID_POLL_MS); - let ppidTimer: ReturnType | null = null; - if (pollMs > 0) { - ppidTimer = setInterval(() => { - const reason = supervisionLostReason({ - originalPpid, - currentPpid: process.ppid, - hostPpid, - isAlive: isProcessAlive, - }); - if (reason) { - try { - process.stderr.write(`[CodeGraph ${label}] Parent process exited (${reason}); aborting.\n`); - } catch { /* stderr gone with the parent — exit anyway */ } - process.exit(1); - } - }, pollMs); - // Never let the watchdog itself keep the process alive past its real work. - ppidTimer.unref(); - } + const stopParentWatch = watchParent((reason) => { + try { + process.stderr.write(`[CodeGraph ${label}] Parent process exited (${reason}); aborting.\n`); + } catch { /* stderr gone with the parent — exit anyway */ } + process.exit(1); + }); let stopped = false; return { stop(): void { if (stopped) return; stopped = true; - if (ppidTimer) clearInterval(ppidTimer); + stopParentWatch(); liveness?.stop(); }, }; } + +/** + * Call `onLost` once, when the parent — or the host threaded past the + * relaunch shim — goes away. The shim blocks in `spawnSync` and cannot forward + * a signal, so this is how a re-exec'd child learns it was orphaned. Returns a + * stop function. `CODEGRAPH_PPID_POLL_MS=0` disables it. + */ +export function watchParent(onLost: (reason: string) => void): () => void { + // Baseline from the CLI entry's earliest-possible capture — reading + // process.ppid here would miss a launcher killed during startup (#1185). + const originalPpid = EARLY_PPID; + const hostPpid = parseHostPpid(process.env[HOST_PPID_ENV]); + const pollMs = parsePpidPollMs(process.env.CODEGRAPH_PPID_POLL_MS); + if (pollMs <= 0) return () => {}; + const timer = setInterval(() => { + const reason = supervisionLostReason({ + originalPpid, + currentPpid: process.ppid, + hostPpid, + isAlive: isProcessAlive, + }); + if (reason) { + clearInterval(timer); + onLost(reason); + } + }, pollMs); + // Never let the watchdog itself keep the process alive past its real work. + timer.unref(); + return () => clearInterval(timer); +} From 7a51969fb0e7ac7453462d816eb94f56043a5bd3 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 04:39:30 +0000 Subject: [PATCH 006/259] fix(resolution): JS/TS calls never pick a class member by name alone when the code rules it out (#2120) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found by the README-wide sweep: the "most depended on" lists of popular repos were topped by members that nothing actually calls. - A member call on a NAMESPACE import (`import * as z from "zod/v4"`) calls one of the module's exports, never some class's method; a receiver imported from a package outside the repository names nothing in it. matchMethodCall's name/word-overlap strategies no longer run for either. zod: `z.string()` had bound 2,640 calls to a test helper's `string` getter (Mocker::string topped the hub list with 3,209), `z.toJSONSchema` 371 to ZodType::toJSONSchema; trpc `z.record` to a generated SDK class. - A receiver-less JS/TS or Go call cannot reach a PROPERTY, FIELD or enum case either (it already excluded methods, #1714/#1857): typeorm's mocha `it(…)` bound 2,879 calls to an interface's `it` property and `describe(…)` 1,368 to a command class's `describe` string field. - A bare call to a name the file imports from outside the repository (`import { useQuery } from '@tanstack/react-query'`, CommonJS `require('supertest')`, vitest's `test`) gets no name-matched candidate. - The React resolver leaves an imported component/hook/context to import resolution (framework resolution runs first and picked any same-named project hook — trpc tests' `useQuery` landed on a hook nested inside one of trpc's factories), and never targets a nested hook; same-file hook first. - "Outside the repository" is the coordinator's `isOutOfRepoImport`: isExternalImport (aliases, workspace members) AND not the repo's own package name AND unresolvable by resolveImportPath — so axios's own tests' `import { getAdapter } from 'axios'` now resolve to lib/adapters.js instead of the .d.ts interface member. A/B vs main (edge diffs, every change classified): zod −5,228, typeorm −4,318, express −700 (calls onto the `var request = require('supertest')` binding), trpc −350/+17, excalidraw −233/+3 (props callbacks onto Props interface members), hono −150, axios −52/+23 (the self-import upgrade), bulletproof-react: 156 sites that pointed at ANOTHER app's component/hook in the three-app monorepo now point at the importing app's own, 16 wrong cross-app links dropped, 8 imports the React heuristic had guessed right lost (per-app tsconfig paths don't resolve — separate follow-up); gin/cobra byte-identical, react-redux-realworld byte-identical. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/js-member-guessing.test.ts | 123 +++++++++++++++++++++++++++ src/resolution/frameworks/react.ts | 22 ++++- src/resolution/import-resolver.ts | 2 +- src/resolution/index.ts | 23 ++++- src/resolution/name-matcher.ts | 42 ++++++++- src/resolution/types.ts | 7 ++ 7 files changed, 212 insertions(+), 8 deletions(-) create mode 100644 __tests__/js-member-guessing.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 4b16268308..6207848626 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- JavaScript and TypeScript calls are no longer linked to a class member that merely shares their name. A call on a namespace import, like zod's `z.string()` (`import * as z from 'zod'`), or on something imported from an npm package, used to link to whichever class in the project had a `string` member, and a bare call like mocha's `it(…)` or `describe(…)` to an interface's `it` property or a class's `describe` field. On zod one test helper's getter had thousands of made-up callers and topped the most-depended-on list. A React hook or component imported from a package, like `useQuery` from `@tanstack/react-query`, no longer links to a same-named one in the project, and a hook nested inside another function is no longer a target at all. Re-index JavaScript and TypeScript projects after upgrading. - A Swift call through a nested type, like `API.PackageController.GetRoute.query(on: db)`, now links to that type's method. Only `query` used to be kept, so the call linked to whichever type's `query` came first; in a Vapor app every route type has one, so callers, impact and `codegraph_explore` answers pointed at another endpoint's code. The same applies to nested types' initializers (`API.PackageController.Model(name:)`), to enum cases (`Gitlab.Error.requestFailed(status:)`), to types declared in an `extension API.PackageController { … }`, and to The Composable Architecture's `Path.State.detail(…)`. A call on a type path that names no type in the project, such as an SDK's `Plot.Node.footer(…)`, is left unlinked. Re-index Swift projects after upgrading. - NestJS routes are now named by the path a request takes: the global prefix from `app.setGlobalPrefix('api')` is applied, along with its `exclude` list, and so is URI versioning from `app.enableVersioning(...)`, with `@Version`, `VERSION_NEUTRAL` and a controller's own `version` taken into account. A route that used to appear as `GET /user` is now `GET /api/v1/user`, so a front end's `this.http.get('/api/v1/user')` or `fetch` call connects to the controller method that serves it. Re-index NestJS projects after upgrading. - Laravel routes are now named by the path a request takes: a leading `/` is added where the routes file leaves it out, `Route::prefix()` and `Route::group(['prefix' => …])` groups are applied, and routes in `routes/api.php` carry the `/api` prefix Laravel serves them under, read from your `RouteServiceProvider`, `bootstrap/app.php` or any file that mounts a routes file. A front-end `fetch('/api/…')` now connects to the Laravel route that serves it. Re-index Laravel projects after upgrading. diff --git a/__tests__/js-member-guessing.test.ts b/__tests__/js-member-guessing.test.ts new file mode 100644 index 0000000000..d0d29895a9 --- /dev/null +++ b/__tests__/js-member-guessing.test.ts @@ -0,0 +1,123 @@ +/** + * A JS/TS call is never bound to a class member picked by name alone when the + * code rules the member out. + * + * - A namespace import (`import * as z from "zod/v4"`) is the module object: + * `z.string()` calls one of the module's exports. It used to bind to the one + * class in the project with a `string` member — on zod, 3,207 calls to a test + * helper's getter. + * - A binding from a package outside the repository (`import { z } from 'zod'`) + * names nothing in it. + * - A bare call (`it(…)`, `describe(…)`) resolves lexically: it cannot reach an + * interface's `it` property or a class's `describe` field. + * + * A default import of a local module's instance still resolves by method. + */ +import { describe, it, expect, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +const roots: string[] = []; +afterAll(() => { + for (const r of roots.splice(0)) fs.rmSync(r, { recursive: true, force: true }); +}); + +const FILES: Record = { + 'package.json': JSON.stringify({ name: 'app', dependencies: { zod: '^4.0.0', mocha: '^10.0.0', react: '^19.0.0', '@tanstack/react-query': '^5.0.0' } }), + 'src/test/Mocker.ts': `export class Mocker { + get string(): string { return 'x'; } + record(): number { return 1; } +} +`, + 'src/test/xfail.ts': `export interface XFailFunction { + it: (title: string) => void; +} +`, + 'src/commands/CacheClear.ts': `export class CacheClearCommand { + describe = "Clears all data stored in query runner cache."; + run() { return 1; } +} +`, + 'src/utils.ts': `export function helper() { return 1; } +`, + 'src/api.ts': `class ApiClient { + fetchUsers() { return []; } +} +export default new ApiClient(); +`, + 'src/schemas.ts': `import * as z from 'zod'; +import { z as zz } from 'zod'; +import * as utils from './utils'; +import api from './api'; + +export function schemas() { + const a = z.string(); + const b = zz.record(); + const c = utils.helper(); + const d = api.fetchUsers(); + return [a, b, c, d]; +} +`, + 'src/schemas.test.ts': `import { useQuery } from '@tanstack/react-query'; +describe('schemas', () => { + it('works', () => { useQuery({}); }); +}); +`, + 'src/hooks.ts': `export function useQuery(key: string) { return key; } +`, + 'src/factory.ts': `export function createRootHooks() { + function useQuery(input: unknown) { return input; } + return { useQuery }; +} +`, + 'src/useAuth.ts': `export function useAuth() { return { user: null }; } +`, + 'src/App.tsx': `import { useQuery } from '@tanstack/react-query'; +import { useAuth } from './useAuth'; +export function App() { + const q = useQuery({ queryKey: ['a'] }); + const auth = useAuth(); + return
{String(q)}{String(auth)}
; +} +`, +}; + +describe('JS/TS: a member picked by name alone', () => { + it('is not what a namespace import, an outside package or a bare call reaches', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-js-member-guess-')); + roots.push(root); + for (const [rel, content] of Object.entries(FILES)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + const cg = await CodeGraph.init(root, { index: true }); + try { + const callsFrom = (file: string): string[] => { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg + .getOutgoingEdgesFrom(ids, ['calls']) + .map((e) => cg.getNode(e.target)?.qualifiedName ?? '?') + .sort(); + }; + const fromSchemas = callsFrom('src/schemas.ts'); + expect(fromSchemas).not.toContain('Mocker::string'); + expect(fromSchemas).not.toContain('Mocker::record'); + expect(fromSchemas).toContain('helper'); + expect(fromSchemas).toContain('ApiClient::fetchUsers'); + const fromTest = callsFrom('src/schemas.test.ts'); + expect(fromTest).not.toContain('XFailFunction::it'); + expect(fromTest).not.toContain('CacheClearCommand::describe'); + // Imported from a package: not the project's own `useQuery`. + expect(fromTest).not.toContain('useQuery'); + // The same in a React file, where hooks resolve by framework rules — and a + // hook imported from the project still resolves. + const fromApp = callsFrom('src/App.tsx'); + expect(fromApp.filter((q) => q.endsWith('useQuery'))).toEqual([]); + expect(fromApp).toContain('useAuth'); + } finally { + cg.close(); + } + }); +}); diff --git a/src/resolution/frameworks/react.ts b/src/resolution/frameworks/react.ts index 863aa764ea..fe60ee17f8 100644 --- a/src/resolution/frameworks/react.ts +++ b/src/resolution/frameworks/react.ts @@ -28,6 +28,15 @@ export const reactResolver: FrameworkResolver = { }, resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + // A component, hook or context the file IMPORTS is the import's: the + // package's (`useQuery` from `@tanstack/react-query`, ` +`, + // Deno: the import map says `@std/assert` comes from JSR. + 'deno/deno.json': JSON.stringify({ imports: { '@std/assert': 'jsr:@std/assert@^1', 'app/': '../src/' } }), + 'deno/test/assert.ts': `export function assertEquals(a: unknown, b: unknown) { return a === b; } +`, + 'deno/app.test.ts': `import { assertEquals } from '@std/assert'; +assertEquals(1, 1); +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + const cg = await CodeGraph.init(root, { index: true }); + try { + const callsFrom = (file: string): string[] => { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg + .getOutgoingEdgesFrom(ids, ['calls']) + .map((e) => cg.getNode(e.target)?.qualifiedName ?? '?') + .sort(); + }; + const fromVue = callsFrom('frontend/pages/labels.vue'); + expect(fromVue).not.toContain('ProvidersContext::t'); + expect(fromVue).not.toContain('dayjs'); + expect(fromVue).toContain('useLabelStore'); + // A `workspace:*` dependency is in the repository. + expect(fromVue).toContain('formatUrl'); + expect(callsFrom('frontend/composables/use-labels.ts')).toContain('useLabelStore'); + const fromSvelte = callsFrom('docs/src/routes/+page.svelte'); + expect(fromSvelte).toContain('useDesignSystem'); + expect(fromSvelte).not.toContain('goto'); + expect(fromSvelte).not.toContain('fetch'); + expect(callsFrom('deno/app.test.ts')).not.toContain('assertEquals'); + // A built-in's name imported from the project is the import. + const svelteIds = cg.getNodesInFile('docs/src/routes/+page.svelte').map((n) => n.id); + const imported = cg.getOutgoingEdgesFrom(svelteIds, ['imports']).map((e) => cg.getNode(e.target)!.filePath); + expect(imported).toContain('docs/src/lib/Map.svelte'); + expect(callsFrom('docs/src/lib/use.ts')).toContain('useDesignSystem'); + } finally { + cg.close(); + } + }); }); diff --git a/src/resolution/index.ts b/src/resolution/index.ts index a5a90d3296..e5fa4f332d 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -41,6 +41,10 @@ import { lexicalPathWithinRoot } from '../utils'; import type { ReExport } from './types'; import { LRUCache } from './lru-cache'; import { JS_BUILT_INS } from './js-builtins'; +import { builtinModules } from 'module'; +import { parse as parseJsonc } from 'jsonc-parser'; + +const NODE_BUILTINS = new Set(builtinModules); /** Node kinds that can declare supertypes (extends/implements). */ const SUPERTYPE_BEARING_KINDS = new Set([ @@ -439,6 +443,7 @@ export class ReferenceResolver { this.supertypeGen++; this.nodesByKindCache.clear(); this.fileExistsMemo.clear(); + this.manifestScopes.clear(); this.knownNames = null; this.knownFiles = null; this.cachesWarmed = false; @@ -503,8 +508,8 @@ export class ReferenceResolver { resolveImport: (ref) => resolveViaImport(ref, this.context), isOutOfRepoImport: (source, fromFile, language) => isExternalImport(source, language, this.context) && - !this.isOwnPackage(source) && - resolveImportPath(source, fromFile, language, this.context) === null, + resolveImportPath(source, fromFile, language, this.context) === null && + this.isDeclaredOutsidePackage(source, fromFile), getNodesInFile: (filePath: string) => { if (!this.nodeCache.has(filePath)) { this.nodeCache.set(filePath, this.queries.getNodesByFile(filePath)); @@ -2343,10 +2348,13 @@ export class ReferenceResolver { private isBuiltInOrExternal(ref: UnresolvedRef): boolean { const name = ref.referenceName; const isJsTs = ref.language === 'typescript' || ref.language === 'javascript' - || ref.language === 'tsx' || ref.language === 'jsx' || ref.language === 'arkts'; + || ref.language === 'tsx' || ref.language === 'jsx' || ref.language === 'arkts' + || ref.language === 'vue' || ref.language === 'svelte' || ref.language === 'astro'; - // JavaScript/TypeScript built-ins - if (isJsTs && JS_BUILT_INS.has(name)) { + // JavaScript/TypeScript built-ins — unless the file imports its own + // binding of that name (`import Map from './Map.svelte'`). + if (isJsTs && JS_BUILT_INS.has(name) && + !this.context.getImportMappings(ref.filePath, ref.language).some((m) => m.localName === name)) { return true; } @@ -2873,20 +2881,67 @@ export class ReferenceResolver { } /** The repository's own package name, from its root package.json; null without one. */ - private ownPackageName: string | null | undefined; + /** Per directory: the package names its package.json and every enclosing one own and depend on. */ + private manifestScopes = new Map; deps: Set }>(); - /** `axios` (or `axios/unsafe/…`) inside the axios repository: a package importing itself. */ - private isOwnPackage(source: string): boolean { - if (this.ownPackageName === undefined) { - let name: string | null = null; - try { - const json = JSON.parse(this.context.readFile('package.json') ?? 'null') as { name?: unknown } | null; - if (json && typeof json.name === 'string' && json.name.length > 0) name = json.name; - } catch { /* no or unreadable package.json */ } - this.ownPackageName = name; + /** + * Is `source` a package from outside the repository? Only when the importing + * file's package.json (or an enclosing one) declares it, or it is a Node + * built-in or a runtime's virtual module. A specifier nothing declares is + * an alias this resolver does not know — SvelteKit's `$lib/…`, Nuxt's + * `~/…`, a nested app's own `@/…` — and stays the project's. + */ + private isDeclaredOutsidePackage(source: string, fromFile: string): boolean { + // Deno's standard library is `@std/…` from JSR. + if (/^(?:node|bun|jsr|npm|https?):/.test(source) || source.startsWith('@std/') || NODE_BUILTINS.has(source)) return true; + const name = source.startsWith('@') ? source.split('/').slice(0, 2).join('/') : source.split('/')[0]!; + const normalized = fromFile.replace(/\\/g, '/'); + const scope = this.manifestScope(normalized.includes('/') ? normalized.slice(0, normalized.lastIndexOf('/')) : ''); + if (scope.own.has(name)) return false; + if (scope.deps.has(name)) return true; + // SvelteKit's `$app/…` and Astro's `astro:…` belong to the framework — + // unless this repository is that framework. + const provider = source.startsWith('astro:') ? 'astro' : /^\$(?:app|env|service-worker)(?:\/|$)/.test(source) ? '@sveltejs/kit' : null; + return provider !== null && !scope.own.has(provider) && !this.context.getWorkspacePackages?.()?.byName.has(provider); + } + + private manifestScope(dir: string): { own: Set; deps: Set } { + const memo = this.manifestScopes.get(dir); + if (memo) return memo; + const parent = dir === '' ? null : this.manifestScope(dir.includes('/') ? dir.slice(0, dir.lastIndexOf('/')) : ''); + let scope = parent ?? { own: new Set(), deps: new Set() }; + try { + const json = JSON.parse(this.context.readFile(dir ? `${dir}/package.json` : 'package.json') ?? 'null') as Record | null; + if (json && typeof json === 'object') { + scope = { own: new Set(scope.own), deps: new Set(scope.deps) }; + if (typeof json.name === 'string' && json.name.length > 0) scope.own.add(json.name); + for (const field of ['dependencies', 'devDependencies', 'peerDependencies', 'optionalDependencies']) { + const deps = json[field]; + if (!deps || typeof deps !== 'object') continue; + for (const [dep, version] of Object.entries(deps)) { + // `workspace:*`, `file:../shared`, `link:…`: a package in this repository. + if (typeof version === 'string' && /^(?:workspace|file|link|portal):/.test(version)) scope.own.add(dep); + else scope.deps.add(dep); + } + } + } + } catch { /* no or unreadable package.json */ } + // A Deno import map names packages the same way — an entry that maps to a + // registry or a URL, not one that maps to a path in the repository. + for (const file of ['deno.json', 'deno.jsonc']) { + const raw = this.context.readFile(dir ? `${dir}/${file}` : file); + if (!raw) continue; + const json = parseJsonc(raw) as { imports?: Record } | undefined; + const imports = json && typeof json === 'object' ? json.imports : undefined; + if (!imports || typeof imports !== 'object') continue; + for (const [key, target] of Object.entries(imports)) { + if (typeof target !== 'string' || !/^(?:jsr|npm|https?):/.test(target)) continue; + if (scope === parent) scope = { own: new Set(scope.own), deps: new Set(scope.deps) }; + scope.deps.add(key.replace(/\/$/, '')); + } } - const own = this.ownPackageName; - return own !== null && (source === own || source.startsWith(`${own}/`)); + this.manifestScopes.set(dir, scope); + return scope; } /** The one supertype-kind node a TypeScript value shares its name and file with. */ diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index a290d30dab..ea09c6c01b 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -935,7 +935,13 @@ export function isVisibleAcrossFiles(candidate: Node, ref: UnresolvedRef, contex return isCrossFileReachable(candidate, ref, context); } -const JS_FAMILY = new Set(['typescript', 'tsx', 'javascript', 'jsx']); +/** + * Languages whose calls are JS/TS calls — Vue, Svelte and Astro components' + * scripts and template expressions included: a bare `t('key')` in a `.vue` + * file resolves lexically exactly as in a `.ts` one. + */ +const JS_FAMILY = new Set(['typescript', 'tsx', 'javascript', 'jsx', 'vue', 'svelte', 'astro']); +const JS_TS = new Set(['typescript', 'tsx', 'javascript', 'jsx']); /** * Whether a JS/TS `calls` ref is a RECEIVER-LESS call — `serialize(x)`, not @@ -4283,9 +4289,11 @@ export function matchJsStoreBindingCall(ref: UnresolvedRef, context: ResolutionC } /** A qualified untyped chain is useful source evidence, not permission to - * infer a property type. Framework resolution runs before this guard. */ + * infer a property type. Framework resolution runs before this guard. Vue, + * Svelte and Astro files keep resolving them: there `api.groupReports.getAll()` + * reaches its API client class far more often than a wrong namesake. */ export function isUnresolvedJsMemberCall(ref: UnresolvedRef): boolean { - return ref.referenceKind === 'calls' && JS_FAMILY.has(ref.language) && + return ref.referenceKind === 'calls' && JS_TS.has(ref.language) && !/^(?:this|window)\./.test(ref.referenceName) && /^[A-Za-z_$][\w$]*(?:\.[A-Za-z_$][\w$]*){2,}$/.test(ref.referenceName); } From 92e14eda40e98dbdf43ea1c69f089afaa99e783b Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 06:32:33 +0000 Subject: [PATCH 014/259] fix(extraction): a file-based router reads only its own app in a monorepo (#2129) create-t3-turbo, tamagui's starter and react-native-reusables keep an Expo app beside a Next.js app. Expo Router read the Next app's `app/` folder as its own, so `layout.tsx` became a `/layout` screen, `page.tsx` a `/page` one, `_components/posts.tsx` and `api/auth/[...all]/route.ts` screens too, and a shared `packages/app/` library's files became screens as well. A resolver can now name `appDependencies`; its extractor runs on a file only when that file's package.json or an enclosing one declares one of them. When no manifest in the project declares any, detection found the framework by other evidence and it runs everywhere, as before. Expo Router names `expo-router`, Next.js `next`. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/monorepo-app-frameworks.test.ts | 88 +++++++++++++++++++++++ src/extraction/index.ts | 73 +++++++++++++++++-- src/resolution/frameworks/expo-router.ts | 1 + src/resolution/frameworks/nextjs.ts | 1 + src/resolution/types.ts | 10 +++ 6 files changed, 170 insertions(+), 4 deletions(-) create mode 100644 __tests__/monorepo-app-frameworks.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index f85c403290..c549985a80 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- In a monorepo with an Expo app next to a Next.js app, like create-t3-turbo or tamagui's starter, each router now reads only its own app. Expo Router used to read the Next.js app's files too, so `layout.tsx`, `page.tsx`, a `_components` folder and an API route showed up as screens named `/layout`, `/page` or `/_components/posts`, and a shared package's files became screens as well. Re-index such monorepos after upgrading. - Calls in Vue, Svelte and Astro components now follow the same rules as JavaScript and TypeScript. A `t('…')` from `useI18n()` or a `ref(…)` from `vue` used to link to an interface property or a local variable of the same name elsewhere in the project, so on halo two unrelated symbols had over a thousand made-up callers each. A call imported from a package is also no longer mistaken for a project symbol when the package is declared in a nested app's `package.json` or a Deno import map. An import through an alias CodeGraph can't follow, like SvelteKit's `$lib/…` or a nested Nuxt app's `~/…`, is still treated as project code. Re-index Vue, Svelte, Astro and JavaScript/TypeScript projects after upgrading. - A bare Java call like `verify(mock)`, `assertThat(x)` or `hashCode()` now links only to a method of the class it's in, of that class's parents, or one the file imports statically. It used to link to any class's method of that name: in halo, Mockito's `verify` and `eq` gave one service's `verify` over a thousand made-up callers, and in retrofit Truth's `assertThat` did the same to a test helper. Calls to inherited methods and static imports from the project now link to the right one, like jsoup's `attr(…)` through `LeafNode` or halo's `and(…)` / `equal(…)` through `import static …Queries.*`. Re-index Java projects after upgrading. - Python calls like `User.objects.get(…)`, `self.client.login(…)` or `request.POST.get(…)` no longer link to whichever class in the project has a method of that name. Such a call now links only to a member of what its receiver names, like `helpers.slugify()` to `slugify` in `helpers.py`, and a bare `get(1)` never links to a method. A name the file imports from a package outside the project, like `from django.shortcuts import render`, no longer links to a same-named project function. In Django projects these made-up links had given single test or serializer methods thousands of callers, for example netbox's `.all()` calls on one `UserConfig.all`. Re-index Python projects after upgrading. diff --git a/__tests__/monorepo-app-frameworks.test.ts b/__tests__/monorepo-app-frameworks.test.ts new file mode 100644 index 0000000000..f7efbf8caa --- /dev/null +++ b/__tests__/monorepo-app-frameworks.test.ts @@ -0,0 +1,88 @@ +/** + * In a monorepo, a file-based router reads only its own app's files. + * + * create-t3-turbo, tamagui's starter and react-native-reusables keep an Expo + * app beside a Next.js app. Expo Router read the Next app's `app/` folder as + * its own: `app/layout.tsx` became a `/layout` screen, `app/page.tsx` a + * `/page` one, a `_components/posts.tsx` a `/_components/posts` screen, and + * `app/api/auth/[...all]/route.ts` a screen named after the file. + */ +import { describe, it, expect, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +const roots: string[] = []; +afterAll(() => { + for (const r of roots.splice(0)) fs.rmSync(r, { recursive: true, force: true }); +}); + +async function project(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-app-frameworks-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); +} + +const routesIn = (cg: CodeGraph, dir: string): string[] => + cg + .getNodesByKind('route') + .filter((n) => n.filePath.startsWith(dir)) + .map((n) => n.name) + .sort(); + +describe('file-based routers in a monorepo', () => { + it('Expo Router reads the Expo app, Next.js the Next app', async () => { + const cg = await project({ + 'package.json': JSON.stringify({ name: 'mono', private: true, workspaces: ['apps/*'] }), + 'apps/mobile/package.json': JSON.stringify({ name: 'mobile', dependencies: { expo: '*', 'expo-router': '*', react: '*' } }), + 'apps/mobile/app/_layout.tsx': `export default function Layout() { return null; } +`, + 'apps/mobile/app/index.tsx': `export default function Home() { return null; } +`, + 'apps/mobile/app/user/[id].tsx': `export default function User() { return null; } +`, + 'apps/web/package.json': JSON.stringify({ name: 'web', dependencies: { next: '*', react: '*' } }), + 'apps/web/app/layout.tsx': `export default function RootLayout({ children }: { children: unknown }) { return children; } +`, + 'apps/web/app/page.tsx': `export default function Page() { return null; } +`, + 'apps/web/app/user/[id]/page.tsx': `export default function UserPage() { return null; } +`, + 'apps/web/app/_components/posts.tsx': `export default function Posts() { return null; } +`, + 'apps/web/app/api/auth/[...all]/route.ts': `export async function GET() { return new Response('ok'); } +`, + }); + try { + expect(routesIn(cg, 'apps/mobile/')).toEqual(['/', '/user/[id]']); + const web = routesIn(cg, 'apps/web/'); + expect(web).toContain('/'); + expect(web).toContain('/user/:id'); + for (const bogus of ['/layout', '/page', '/_components/posts', '/api/auth/[...all]/route', '/user/[id]/page']) { + expect(web).not.toContain(bogus); + } + } finally { + cg.close(); + } + }); + + it('a single Expo app at the repository root keeps its routes', async () => { + const cg = await project({ + 'package.json': JSON.stringify({ name: 'demo', dependencies: { expo: '*', 'expo-router': '*' } }), + 'app/_layout.tsx': `export default function Layout() { return null; } +`, + 'app/settings.tsx': `export default function Settings() { return null; } +`, + }); + try { + expect(routesIn(cg, 'app/')).toEqual(['/settings']); + } finally { + cg.close(); + } + }); +}); diff --git a/src/extraction/index.ts b/src/extraction/index.ts index a4fec2ac3c..706bc0bd12 100644 --- a/src/extraction/index.ts +++ b/src/extraction/index.ts @@ -32,7 +32,8 @@ import { isCodeGraphDataDir } from '../directory'; import { logDebug, logWarn } from '../errors'; import { validatePathWithinRoot, normalizePath } from '../utils'; import ignore, { Ignore } from 'ignore'; -import { detectFrameworks } from '../resolution/frameworks'; +import { detectFrameworks, getFrameworkResolver } from '../resolution/frameworks'; +import { declaredDependencies } from '../resolution/frameworks/package-deps'; import type { ResolutionContext } from '../resolution/types'; import { createYielder, type MaybeYield } from '../resolution/cooperative-yield'; import { MAX_SOURCE_FILE_SIZE_BYTES, oversizeStamp, readBoundedSource, readBoundedSourceSync } from '../file-limits'; @@ -1908,9 +1909,72 @@ export class ExtractionOrchestrator { const fileList = files ?? scanDirectory(this.rootDir); const context = this.buildDetectionContext(fileList); this.detectedFrameworkNames = detectFrameworks(context).map((r) => r.name); + const declared = declaredDependencies(context); + this.gatedFrameworks = new Map(); + for (const name of this.detectedFrameworkNames) { + const deps = getFrameworkResolver(name)?.appDependencies; + // Gated only when some manifest names the framework: otherwise detection + // found it by other evidence and every file is its app's. + if (deps && deps.some((d) => declared.has(d))) this.gatedFrameworks.set(name, deps); + } + this.appFrameworkMemo.clear(); + this.manifestDependencies.clear(); return this.detectedFrameworkNames; } + /** Detected frameworks whose extractors run only inside their own apps, with the packages that mark one. */ + private gatedFrameworks = new Map(); + /** `|` → does the package.json at or above `dir` declare the framework. */ + private appFrameworkMemo = new Map(); + /** Directory → the dependency names its package.json declares, null without one. */ + private manifestDependencies = new Map | null>(); + + /** + * The detected frameworks whose extractors apply to `filePath`: all of them, + * except one with `appDependencies` when neither the file's package.json nor + * an enclosing one declares any of them. + */ + private frameworksForFile(filePath: string, names: string[]): string[] { + if (this.gatedFrameworks.size === 0) return names; + const slash = filePath.lastIndexOf('/'); + const dir = slash < 0 ? '' : filePath.slice(0, slash); + const kept = names.filter((name) => this.frameworkAppliesIn(dir, name)); + return kept.length === names.length ? names : kept; + } + + private frameworkAppliesIn(dir: string, name: string): boolean { + const deps = this.gatedFrameworks.get(name); + if (!deps) return true; + const key = `${dir}|${name}`; + const memo = this.appFrameworkMemo.get(key); + if (memo !== undefined) return memo; + let applies = false; + for (let d: string | null = dir; d !== null; d = d === '' ? null : d.includes('/') ? d.slice(0, d.lastIndexOf('/')) : '') { + const declared = this.dependenciesDeclaredIn(d); + if (declared && deps.some((dep) => declared.has(dep))) { + applies = true; + break; + } + } + this.appFrameworkMemo.set(key, applies); + return applies; + } + + private dependenciesDeclaredIn(dir: string): Set | null { + if (this.manifestDependencies.has(dir)) return this.manifestDependencies.get(dir)!; + let declared: Set | null = null; + try { + const pkg = JSON.parse(fs.readFileSync(path.join(this.rootDir, dir, 'package.json'), 'utf8')) as Record; + declared = new Set(); + for (const field of ['dependencies', 'devDependencies', 'peerDependencies']) { + const group = pkg[field]; + if (group && typeof group === 'object') for (const name of Object.keys(group)) declared.add(name); + } + } catch { /* no or unreadable manifest */ } + this.manifestDependencies.set(dir, declared); + return declared; + } + /** * Index all files in the project */ @@ -2099,8 +2163,9 @@ export class ExtractionOrchestrator { */ const parseFile = (filePath: string, content: string): Promise => { const language = detectLanguage(filePath, content, overrides); - if (!pool) return Promise.resolve(extractFromSource(filePath, content, language, frameworkNames)); - return pool.requestParse({ filePath, content, language, frameworkNames }); + const names = this.frameworksForFile(filePath, frameworkNames); + if (!pool) return Promise.resolve(extractFromSource(filePath, content, language, names)); + return pool.requestParse({ filePath, content, language, frameworkNames: names }); }; // --- Bounded rolling-window dispatch, ordered commit --- @@ -2722,7 +2787,7 @@ export class ExtractionOrchestrator { // Extract from source. Use cached framework names if indexAll has run, // otherwise detect on the spot so single-file re-index paths still emit // route nodes / middleware / etc. - const frameworkNames = this.ensureDetectedFrameworks(); + const frameworkNames = this.frameworksForFile(relativePath, this.ensureDetectedFrameworks()); const result = extractFromSource(relativePath, content, language, frameworkNames); // Store in database diff --git a/src/resolution/frameworks/expo-router.ts b/src/resolution/frameworks/expo-router.ts index 2f513e3fad..5d82b94e26 100644 --- a/src/resolution/frameworks/expo-router.ts +++ b/src/resolution/frameworks/expo-router.ts @@ -713,6 +713,7 @@ function scoreMatch(href: string[], route: string[]): number | null { export const expoRouterResolver: FrameworkResolver = { name: 'expo-router', languages: [...ROUTE_LANGUAGES], + appDependencies: ['expo-router'], detect(context: ResolutionContext): boolean { if (dependsOn(context, 'expo-router')) return true; diff --git a/src/resolution/frameworks/nextjs.ts b/src/resolution/frameworks/nextjs.ts index 7eb543dee9..22b85f7466 100644 --- a/src/resolution/frameworks/nextjs.ts +++ b/src/resolution/frameworks/nextjs.ts @@ -222,6 +222,7 @@ export function nextNavVerb(name: string): string | null { export const nextjsResolver: FrameworkResolver = { name: 'nextjs', languages: [...ROUTE_LANGUAGES], + appDependencies: ['next'], detect(context: ResolutionContext): boolean { if (dependsOn(context, 'next')) return true; diff --git a/src/resolution/types.ts b/src/resolution/types.ts index 7b76bb324e..65fd98cafe 100644 --- a/src/resolution/types.ts +++ b/src/resolution/types.ts @@ -239,6 +239,16 @@ export interface FrameworkResolver { name: string; /** Languages this framework applies to. If omitted, applies to all languages. */ languages?: Language[]; + /** + * Packages an app declares when it is built on this framework. When set, + * `extract()` runs only on files of an app whose package.json — the file's + * own or an enclosing one — declares one of them: in a monorepo with an Expo + * app beside a Next.js app, Expo Router must not read the Next app's + * `app/layout.tsx` as a `/layout` screen. When no manifest in the project + * declares any, detection found the framework by other evidence and the + * extractor runs on every file, as before. + */ + appDependencies?: readonly string[]; /** Detect if project uses this framework (project-level, called once at startup) */ detect(context: ResolutionContext): boolean; /** Resolve a reference using framework-specific patterns */ From 54a1698caa219563d136453ddcf4b573f9c170b0 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 06:46:26 +0000 Subject: [PATCH 015/259] fix(vue-router): admin-template route tables, children and layouts; Nuxt routes only in Nuxt (#2130) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit vue-element-admin, vue-admin-template and vben had no routes: their tables are named arrays (`export const constantRoutes = [...]`, `const routes: RouteRecordRaw[] = [...]`) or per-module objects (`const tableRouter = {…}`) handed to `new Router(...)`, and most of their screens are `children`. The Vue Router reader now reads those tables in any file that builds a router, imports vue-router or lives in a router/ directory; joins children onto their parent's path (a parent is a screen only when no child claims its address and it doesn't redirect); binds a lazy view by the FILE it imports (all of vue-element-admin's are `…/index.vue`), through an alias the resolver can't follow by the one file in the app with that path; links each screen to its parents' components as layouts; and takes `this.$router.push`. Nuxt's file routes move into their own `nuxt` resolver, detected only in a Nuxt app: halo's plain-Vue console got 30 made-up screens from its `pages/` folders. A Nuxt app at the repository root now gets its routes, and a top-level `pages/index.vue` is `/`, not `/index`. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + README.md | 2 +- __tests__/vue-router.test.ts | 164 ++++++++++++++++- docs/design/framework-coverage.md | 6 +- src/resolution/frameworks/index.ts | 6 +- src/resolution/frameworks/vue-router.ts | 229 +++++++++++++++++++----- src/resolution/frameworks/vue.ts | 38 +++- src/ui-server/api/screens.ts | 2 +- 8 files changed, 386 insertions(+), 62 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c549985a80..a7dd3a4bfd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- Vue Router apps built like vue-element-admin, vue-admin-template or vben now have their routes and screens. Routes declared in a named table (`export const constantRoutes = [...]`, `const routes: RouteRecordRaw[] = [...]`), in per-module route files, or through `new Router(...)` are now read. Child routes are joined onto their parent's path, and the parent's component is the layout around them. A lazily loaded view (`() => import('@/views/dashboard/index')`) is bound to the file it names, and `this.$router.push(...)` counts as navigation. Nuxt's file-based routes now come only from a Nuxt app, so a plain Vue app's `pages/` folder no longer turns into made-up screens, and a Nuxt app at the root of its repository now gets its routes. Re-index Vue projects after upgrading. - In a monorepo with an Expo app next to a Next.js app, like create-t3-turbo or tamagui's starter, each router now reads only its own app. Expo Router used to read the Next.js app's files too, so `layout.tsx`, `page.tsx`, a `_components` folder and an API route showed up as screens named `/layout`, `/page` or `/_components/posts`, and a shared package's files became screens as well. Re-index such monorepos after upgrading. - Calls in Vue, Svelte and Astro components now follow the same rules as JavaScript and TypeScript. A `t('…')` from `useI18n()` or a `ref(…)` from `vue` used to link to an interface property or a local variable of the same name elsewhere in the project, so on halo two unrelated symbols had over a thousand made-up callers each. A call imported from a package is also no longer mistaken for a project symbol when the package is declared in a nested app's `package.json` or a Deno import map. An import through an alias CodeGraph can't follow, like SvelteKit's `$lib/…` or a nested Nuxt app's `~/…`, is still treated as project code. Re-index Vue, Svelte, Astro and JavaScript/TypeScript projects after upgrading. - A bare Java call like `verify(mock)`, `assertThat(x)` or `hashCode()` now links only to a method of the class it's in, of that class's parents, or one the file imports statically. It used to link to any class's method of that name: in halo, Mockito's `verify` and `eq` gave one service's `verify` over a thousand made-up callers, and in retrofit Truth's `assertThat` did the same to a test helper. Calls to inherited methods and static imports from the project now link to the right one, like jsoup's `attr(…)` through `LeafNode` or halo's `and(…)` / `equal(…)` through `import static …Queries.*`. Re-index Java projects after upgrading. diff --git a/README.md b/README.md index e0aae08860..3d02481dd8 100644 --- a/README.md +++ b/README.md @@ -344,7 +344,7 @@ These frameworks additionally emit **`navigates`** edges: the function that send | **Next.js** | App Router `app/**/page.tsx` and Pages Router pages (`(group)` stripped, `[slug]` → `:slug`); `app/api/**/route.ts` exports and `pages/api/*` are endpoints, not screens | `router.push` / `replace` / `prefetch`, `redirect()` / `permanentRedirect()` in a server action or page, `NextResponse.redirect(new URL(…))` in middleware, `` and internal `` | | **React Router** | `` (v5 and v6) and `createBrowserRouter([{ path, element }])` | `history.push` / `replace`, `useNavigate`'s `navigate`, a loader's `redirect`, `` / `` / `` / react-router-bootstrap's `` | | **TanStack Router** | `createFileRoute('/posts/$postId')` (file-based) and `createRoute({ path, getParentRoute })` composed up its parent chain (code-based); `_pathless` segments, `(group)` folders, `__root` and `` layouts are not addresses | `navigate({ to })`, a thrown `redirect({ to })`, `` / `` — where `to` is the route PATTERN and the values ride beside it in `params` | -| **Vue Router** / **Nuxt** | `createRouter({ routes: [...] })` with the view each entry names, plus Nuxt `pages/` file-based routes, `server/api/` endpoints and route middleware | `router.push` / `replace`, `$router.push`, Nuxt's `navigateTo`, `` / `` / `` — **by route name** (`push({ name: 'profile' })`) as well as by path | +| **Vue Router** / **Nuxt** | `createRouter({ routes: [...] })` / `new Router(...)` and the route tables it's given (`export const constantRoutes = [...]`, per-module route files), with the view each entry names — a lazy `() => import(…)` bound to its file — and `children` joined onto their parent's path, the parent being the layout around them; plus Nuxt `pages/` file-based routes, `server/api/` endpoints and route middleware | `router.push` / `replace`, `this.$router.push`, Nuxt's `navigateTo`, `` / `` / `` — **by route name** (`push({ name: 'profile' })`) as well as by path | | **SvelteKit** | `src/routes/**/+page.svelte` (`[slug]` → `:slug`, `[[opt]]` → `:opt?`), joined to the `+page.server.js` beside it so a loader's guard belongs to its page | `goto('/x')`, `redirect(status, '/x')` from a load or form action, and the plain `` that is a link in a SvelteKit app | | **Angular** | `Routes` arrays (`RouterModule.forRoot` / `forChild`, `provideRouter`, a routes file's default export) with `component` or a lazy `loadComponent`; `children` and lazy `loadChildren` (an NgModule's through its routing module) joined into full paths; paths written as route constants or `$localize` strings; a route with children is a layout around the screens inside it | `router.navigate([...])`, `navigateByUrl`, a guard's `createUrlTree` / `parseUrl` — a command array, a route constant, or a component property holding one — `routerLink` / `[routerLink]` in the component's template, and `redirectTo`. Each template's child components (``) are linked to the component that renders them | diff --git a/__tests__/vue-router.test.ts b/__tests__/vue-router.test.ts index abe0871866..de94f2989e 100644 --- a/__tests__/vue-router.test.ts +++ b/__tests__/vue-router.test.ts @@ -65,15 +65,18 @@ describe('vue-router: parseVueRoutes', () => { ['home', '/'], ['login', '/login'], ['settings', '/settings'], + [null, '/profile/:username/favorites'], ['profile', '/profile/:username'], ]); }); it('reads the component from a lazy import and from an identifier', () => { - expect(entries.map((e) => e.component)).toEqual(['Home', 'Login', 'Settings', 'Profile']); + expect(entries.map((e) => e.component)).toEqual(['Home', 'Login', 'Settings', 'Favorites', 'Profile']); }); - it('skips a child route, whose path is relative to a parent this does not compose', () => { + it('joins a child route onto its parent, which is the layout around it', () => { + const child = entries.find((e) => e.path === '/profile/:username/favorites')!; + expect(child.layouts.map((l) => l.component)).toEqual(['Profile']); expect(entries.some((e) => e.path === 'favorites')).toBe(false); }); @@ -299,3 +302,160 @@ describe('vue-router: a routed app end to end', () => { expect(screens.dropped).toBe(0); }); }); + +// ============================================================================= +// Admin-template route tables, and Nuxt's own file routes +// ============================================================================= + +describe('vue-router: route tables beyond an inline createRouter', () => { + const roots: string[] = []; + afterAll(() => { + for (const r of roots.splice(0)) fs.rmSync(r, { recursive: true, force: true }); + }); + + async function project(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-vue-tables-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); + } + + const routeNames = (cg: CodeGraph): string[] => cg.getNodesByKind('route').map((n) => n.name).sort(); + const edgesFrom = (cg: CodeGraph, routeName: string, kind: 'calls' | 'references') => { + const r = cg.getNodesByKind('route').find((n) => n.name === routeName)!; + return cg.getOutgoingEdges(r.id).filter((e) => e.kind === kind).map((e) => ({ edge: e, target: cg.getNode(e.target)! })); + }; + + it('vue-element-admin: named tables, `new Router`, module files, children and layouts', async () => { + const view = (name: string) => `\n\n`; + const cg = await project({ + 'package.json': JSON.stringify({ name: 'admin', dependencies: { vue: '^2.6.0', 'vue-router': '^3.0.0' } }), + 'src/router/index.js': `import Vue from 'vue' +import Router from 'vue-router' +import Layout from '@/layout' +import tableRouter from './modules/table' +Vue.use(Router) + +export const constantRoutes = [ + { path: '/login', component: () => import('@/views/login/index'), hidden: true }, + { + path: '/', + component: Layout, + redirect: '/dashboard', + children: [ + { path: 'dashboard', component: () => import('@/views/dashboard/index'), name: 'Dashboard' } + ] + } +] + +export const asyncRoutes = [tableRouter, { path: '*', redirect: '/404', hidden: true }] + +const createRouter = () => new Router({ routes: constantRoutes }) +export default createRouter() +`, + 'src/router/modules/table.js': `import Layout from '@/layout' + +const tableRouter = { + path: '/table', + component: Layout, + redirect: '/table/complex-table', + name: 'Table', + children: [ + { path: 'complex-table', component: () => import('@/views/table/complex-table'), name: 'ComplexTable' } + ] +} +export default tableRouter +`, + 'src/layout/index.vue': view('Layout'), + 'src/views/login/index.vue': ` + +`, + 'src/views/dashboard/index.vue': view('Dashboard'), + 'src/views/table/complex-table.vue': view('ComplexTable'), + }); + try { + // `/` and `/table` redirect into their children: frames, not pages. + expect(routeNames(cg)).toEqual(['/dashboard', '/login', '/table/complex-table']); + // Each lazy view binds to ITS file's component, though every one is `index`. + expect(edgesFrom(cg, '/dashboard', 'calls').map((e) => e.target.filePath)).toEqual(['src/views/dashboard/index.vue']); + expect(edgesFrom(cg, '/login', 'calls').map((e) => e.target.filePath)).toEqual(['src/views/login/index.vue']); + // The parent's component is the layout around it. + const layouts = edgesFrom(cg, '/table/complex-table', 'references'); + expect(layouts.map((e) => e.target.filePath)).toEqual(['src/layout/index.vue']); + expect(layouts[0]!.edge.metadata).toMatchObject({ layout: true }); + // Navigation by name reaches the composed child. + const nav = cg + .getOutgoingEdgesFrom(cg.getNodesInFile('src/views/login/index.vue').map((n) => n.id), ['navigates' as never]) + .filter((e) => e.kind === 'navigates'); + expect(nav.map((e) => cg.getNode(e.target)!.name)).toEqual(['/dashboard']); + } finally { + cg.close(); + } + }); + + it('vben: a typed routes module with an alias the resolver cannot follow', async () => { + const cg = await project({ + 'package.json': JSON.stringify({ name: 'mono', private: true }), + 'apps/web/package.json': JSON.stringify({ name: 'web', imports: { '#/*': './src/*' }, dependencies: { vue: '*', 'vue-router': '*' } }), + 'apps/web/src/router/routes/modules/dashboard.ts': `import type { RouteRecordRaw } from 'vue-router'; + +const routes: RouteRecordRaw[] = [ + { + name: 'Dashboard', + path: '/dashboard', + children: [ + { name: 'Analytics', path: 'analytics', component: () => import('#/views/dashboard/analytics/index.vue') }, + ], + }, +]; + +export default routes; +`, + 'apps/web/src/views/dashboard/analytics/index.vue': ` + +`, + }); + try { + expect(routeNames(cg)).toEqual(['/dashboard/analytics']); + expect(edgesFrom(cg, '/dashboard/analytics', 'calls').map((e) => e.target.filePath)).toEqual([ + 'apps/web/src/views/dashboard/analytics/index.vue', + ]); + } finally { + cg.close(); + } + }); + + it('Nuxt pages at the repository root are routes; a plain Vue app’s pages/ folder is not', async () => { + const nuxt = await project({ + 'package.json': JSON.stringify({ name: 'site', dependencies: { nuxt: '*', vue: '*' } }), + 'pages/index.vue': '\n', + 'pages/users/[id].vue': '\n', + }); + try { + expect(routeNames(nuxt)).toEqual(['/', '/users/:id']); + } finally { + nuxt.close(); + } + const plain = await project({ + 'ui/package.json': JSON.stringify({ name: 'console', dependencies: { vue: '*', 'vue-router': '*' } }), + 'ui/src/modules/contents/pages/SinglePageList.vue': '\n', + }); + try { + expect(routeNames(plain)).toEqual([]); + } finally { + plain.close(); + } + }); +}); diff --git a/docs/design/framework-coverage.md b/docs/design/framework-coverage.md index febc60a301..b149ccb7d7 100644 --- a/docs/design/framework-coverage.md +++ b/docs/design/framework-coverage.md @@ -40,7 +40,7 @@ guessed. | Next.js | `frameworks/nextjs.ts` | `next-router-synthesizer.ts` | `nextjs.test.ts` | next-saas-starter | | React Router | `frameworks/react-router.ts` | `react-router-synthesizer.ts` | `react-router.test.ts` | proshop (44 edges) | | TanStack Router | `frameworks/tanstack-router.ts` | `tanstack-router-synthesizer.ts` | `tanstack-router.test.ts` | TanStack examples, fastapi-template frontend | -| Vue Router / Nuxt | `frameworks/vue-router.ts` | `vue-router-synthesizer.ts` | `vue-router.test.ts` | vue-realworld (23 edges) | +| Vue Router / Nuxt | `frameworks/vue-router.ts` (Nuxt file routes: `nuxtResolver` in `frameworks/vue.ts`) | `vue-router-synthesizer.ts` | `vue-router.test.ts` | vue-realworld (23 edges); vue-element-admin (62 routes), vue-admin-template (14), vben (192), halo console (34) — named tables, module files, `children` + layouts; Nuxt: mealie, elk, nuxt/movies | | SvelteKit | `frameworks/sveltekit-router.ts` | `sveltekit-synthesizer.ts` | `sveltekit-router.test.ts` | sveltekit-realworld (31 edges) | | Angular | `frameworks/angular-router.ts` | `angular-template-synthesizer.ts` | `angular-router.test.ts` | angular-realworld (31 edges, 18 renders), Ghostfolio (189 edges, 170 renders), ngx-admin (routes and renders; its menus are config) | @@ -191,6 +191,10 @@ Each of these cost real debugging time; they are not hypothetical. `name` is written above its `path`, so a text window handed every entry its predecessor's name — silently, for every route in the file. Use `frameworks/object-literal.ts`. + A lazy view binds by the FILE it imports, never by the import's last + segment: vue-element-admin's views are all `…/index.vue`. + Nuxt's `pages/` convention belongs to a Nuxt app only — a plain Vue app's + `pages/` folder (halo's console) holds components a router config names. 5. **A receiver is required for a generic verb.** `push` and `replace` are two of the most common method names in JavaScript; claiming a bare one puts every `paths.push('/tmp/x')` one string-match away from a route. diff --git a/src/resolution/frameworks/index.ts b/src/resolution/frameworks/index.ts index 97af4183fb..88221fdfa6 100644 --- a/src/resolution/frameworks/index.ts +++ b/src/resolution/frameworks/index.ts @@ -18,7 +18,7 @@ import { vueRouterResolver } from './vue-router'; import { angularRouterResolver } from './angular-router'; import { svelteKitRouterResolver } from './sveltekit-router'; import { svelteResolver } from './svelte'; -import { vueResolver } from './vue'; +import { vueResolver, nuxtResolver } from './vue'; import { astroResolver } from './astro'; import { djangoResolver, flaskResolver, fastapiResolver } from './python'; import { railsResolver } from './ruby'; @@ -58,6 +58,8 @@ const FRAMEWORK_RESOLVERS: FrameworkResolver[] = [ // SvelteKit — `src/routes/**/+page.svelte` routes are `svelteResolver`'s; `goto('/x')` / `redirect(303, '/x')` → navigates edges svelteKitRouterResolver, vueResolver, + // Nuxt — `pages/**` screens, `server/api/**` endpoints and `middleware/`, in a Nuxt app only + nuxtResolver, // Vue Router — `createRouter({ routes })` → route nodes; `router.push({ name })` / `router.push('/x')` → navigates edges vueRouterResolver, angularRouterResolver, @@ -162,7 +164,7 @@ export { vueRouterResolver } from './vue-router'; export { angularRouterResolver } from './angular-router'; export { svelteKitRouterResolver } from './sveltekit-router'; export { svelteResolver } from './svelte'; -export { vueResolver } from './vue'; +export { vueResolver, nuxtResolver } from './vue'; export { astroResolver } from './astro'; export { djangoResolver, flaskResolver, fastapiResolver } from './python'; export { railsResolver } from './ruby'; diff --git a/src/resolution/frameworks/vue-router.ts b/src/resolution/frameworks/vue-router.ts index 1a9884e8ba..84a1a9beb5 100644 --- a/src/resolution/frameworks/vue-router.ts +++ b/src/resolution/frameworks/vue-router.ts @@ -64,6 +64,7 @@ import { type RouteTable, } from './expo-router'; import { destinationsForHref } from './nextjs'; +import { resolveImportPath } from '../import-resolver'; const ROUTE_LANGUAGES: readonly Language[] = ['typescript', 'javascript', 'vue']; @@ -72,75 +73,137 @@ const ROUTE_LANGUAGES: readonly Language[] = ['typescript', 'javascript', 'vue'] // ============================================================================= export interface VueRouteEntry { - /** `/profile/:username` — the path, in the form every other framework's routes use. */ + /** `/profile/:username` — the full path, a child's joined onto its parents'. */ path: string; /** `profile` — what `router.push({ name })` names, when the entry has one. */ name: string | null; /** The component the entry names, by identifier or by the tail of its lazy import. */ component: string | null; + /** `@/views/dashboard/index` — the file a lazy `() => import(…)` component names. */ + spec: string | null; + /** The components of the parent routes it renders inside, outermost first. */ + layouts: VueLayout[]; line: number; } +export interface VueLayout { + component: string; + spec: string | null; +} + /** A file that builds a router — the cheap gate before parsing anything. */ -const ROUTER_FACTORY = /\b(?:createRouter|createWebHistory|createWebHashHistory|createMemoryHistory)\s*\(|\bnew\s+VueRouter\s*\(/; +const ROUTER_FACTORY = /\b(?:createRouter|createWebHistory|createWebHashHistory|createMemoryHistory)\s*\(|\bnew\s+(?:VueRouter|Router)\s*\(/; /** `routes: [` / `routes = [` — the array itself, for a file that only holds the table. */ const ROUTES_ARRAY = /\broutes\s*[:=]\s*\[/; +/** A file that imports from `vue-router`. */ +const VUE_ROUTER_IMPORT = /\bfrom\s*['"]vue-router['"]/; + +/** A file in a `router/` or `routes/` directory, or named `router.js` / `routes.ts`. */ +const ROUTE_FILE_PATH = /(?:^|\/)(?:router|routes)(?:\/|\.[cm]?[jt]s$)/; + +/** + * Where a routes table starts: a `routes: [` field, or a declaration named + * for what it holds — `export const constantRoutes = [`, `const routes: + * RouteRecordRaw[] = [`, vue-element-admin's per-module `const tableRouter = {`. + */ +const TABLE_OPENERS = /\broutes\s*[:=]\s*\[|\b(?:const|let|var)\s+(?:[A-Za-z_$][\w$]*)?(?:[Rr]outes?|[Rr]outers?)\s*(?::[^=;\n]*)?=\s*([[{])/g; + /** - * Every top-level entry of a `routes: [...]` array. + * Every route a file's routes tables declare. * - * The array is walked, not pattern-matched: a `name` is written ABOVE the + * The tables are walked, not pattern-matched: a `name` is written ABOVE the * `path` it belongs to, so reading fields out of a window around each `path` * hands an entry its PREDECESSOR's name — vue-realworld's `login` came out as - * `/register`, silently, for every route in the file. So each top-level `{…}` - * is matched as a unit and only its own depth-1 fields are read; a nested - * `children:` array, a `meta: {…}` and a lazy `component: () => import(…)` - * are stepped over rather than searched. + * `/register`, silently, for every route in the file. So each `{…}` is + * matched as a unit and only its own depth-1 fields are read; a `meta: {…}` + * and a lazy `component: () => import(…)` are stepped over rather than + * searched, and `children` are walked in turn with the parent's path in front + * of theirs. + * + * A route with children is a layout: its component renders the outlet they + * fill. It is a screen of its own only when no child claims its address and + * it does not `redirect` elsewhere — vue-element-admin's `{ path: '/table', + * component: Layout, redirect: '/table/complex-table', children }` is the + * frame around four tables, not a fifth page. * - * An entry whose path does not start with `/` is a child route, relative to a - * parent this does not compose, and is not a destination on its own. + * A top-level entry whose path does not start with `/` (`*`, a catch-all) is + * not an address. A table is read from a file that builds a router, imports + * `vue-router`, or lives in a `router/` directory. */ -export function parseVueRoutes(content: string): VueRouteEntry[] { - if (!ROUTER_FACTORY.test(content) && !ROUTES_ARRAY.test(content)) return []; +export function parseVueRoutes(content: string, filePath = ''): VueRouteEntry[] { + const routeFile = ROUTER_FACTORY.test(content) || ROUTES_ARRAY.test(content) || VUE_ROUTER_IMPORT.test(content) || ROUTE_FILE_PATH.test(filePath); + if (!routeFile) return []; const safe = stripCommentsForRegex(content, 'typescript'); const out: VueRouteEntry[] = []; const seen = new Set(); - const arrays = /\broutes\s*[:=]\s*\[/g; - let a: RegExpExecArray | null; - while ((a = arrays.exec(safe)) !== null) { - const open = a.index + a[0].length - 1; - const close = matchBracket(safe, open); - if (close < 0) continue; - for (const obj of topLevelObjects(safe, open + 1, close)) { - const fields = readFields(safe, obj.start, obj.end); - const pathField = fields.get('path'); - if (!pathField) continue; - const path = readStringAt(pathField.text.trimStart(), 0); - if (path === null || !path.startsWith('/')) continue; - const componentField = fields.get('component') ?? fields.get('components'); - if (!componentField) continue; // no component in the entry → not a route object - const component = componentName(componentField.text); - if (!component) continue; - const nameField = fields.get('name'); - const name = nameField ? readStringAt(nameField.text.trimStart(), 0) : null; - const line = safe.slice(0, pathField.at).split('\n').length; - const key = `${path} ${name ?? ''}`; - if (seen.has(key)) continue; - seen.add(key); - out.push({ path, name, component, line }); + const walked = new Set(); + const lineOf = (at: number) => safe.slice(0, at).split('\n').length; + + const visit = (obj: { start: number; end: number }, prefix: string | null, layouts: VueLayout[], depth: number): void => { + const fields = readFields(safe, obj.start, obj.end); + const pathField = fields.get('path'); + if (!pathField) return; + const own = readStringAt(pathField.text.trimStart(), 0); + if (own === null) return; + let path: string; + if (own.startsWith('/')) path = own; + else if (prefix === null) return; // a top-level `*` or relative path is no address + else path = own === '' ? prefix : `${prefix === '/' ? '' : prefix}/${own}`; + if (path.length > 1) path = path.replace(/\/+$/, ''); + + const componentField = fields.get('component') ?? fields.get('components'); + const component = componentField ? componentOf(componentField.text) : null; + + const children = fields.get('children'); + let childClaimsAddress = false; + if (children && depth < 12) { + const open = safe.indexOf('[', children.at); + const close = open < 0 ? -1 : matchBracket(safe, open); + if (open >= 0 && close > open) { + const inner = component ? [...layouts, component] : layouts; + for (const child of topLevelObjects(safe, open + 1, close)) { + const childPath = readFields(safe, child.start, child.end).get('path'); + const childOwn = childPath ? readStringAt(childPath.text.trimStart(), 0) : null; + if (childOwn === '' || childOwn === path) childClaimsAddress = true; + visit(child, path, inner, depth + 1); + } + } } - arrays.lastIndex = close; + if (!component) return; + if (children && (childClaimsAddress || fields.has('redirect'))) return; + const nameField = fields.get('name'); + const name = nameField ? readStringAt(nameField.text.trimStart(), 0) : null; + const key = `${path} ${name ?? ''}`; + if (seen.has(key)) return; + seen.add(key); + out.push({ path, name, component: component.component, spec: component.spec, layouts: [...layouts], line: lineOf(pathField.at) }); + }; + + TABLE_OPENERS.lastIndex = 0; + let m: RegExpExecArray | null; + while ((m = TABLE_OPENERS.exec(safe)) !== null) { + const open = m.index + m[0].length - 1; + const close = matchBracket(safe, open); + if (close < 0 || walked.has(open)) continue; + walked.add(open); + if (safe[open] === '{') visit({ start: open, end: close }, null, [], 0); + else for (const obj of topLevelObjects(safe, open + 1, close)) visit(obj, null, [], 0); + TABLE_OPENERS.lastIndex = close; } return out; } /** The component an entry names: an identifier, or the file a lazy import names. */ -function componentName(value: string): string | null { +function componentOf(value: string): VueLayout | null { const lazy = /\bimport\s*\(\s*['"`]([^'"`]+)['"`]/.exec(value); - if (lazy) return (lazy[1]!.split('/').pop() ?? '').replace(/\.\w+$/, '') || null; + if (lazy) { + const tail = (lazy[1]!.split('/').pop() ?? '').replace(/\.\w+$/, ''); + return tail ? { component: tail, spec: lazy[1]! } : null; + } const ident = /^\s*([A-Z][A-Za-z0-9_]*)\s*$/.exec(value); - return ident?.[1] ?? null; + return ident ? { component: ident[1]!, spec: null } : null; } function languageForFile(filePath: string): Language { @@ -175,7 +238,7 @@ function isVueConfigRoute(node: Node): boolean { function isNuxtPage(node: Node): boolean { return ( node.language === 'vue' && - node.filePath.includes('/pages/') && + `/${node.filePath}`.includes('/pages/') && node.id === `route:${node.filePath}:${node.name}:1` ); } @@ -212,7 +275,7 @@ export function vueRouteTable(context: ResolutionContext): VueRouteTable { if (!content) continue; const byName = tableAt(group.root).byName; const byPath = new Map(group.nodes.map((n) => [n.name, n])); - for (const entry of parseVueRoutes(content)) { + for (const entry of parseVueRoutes(content, filePath)) { if (!entry.name) continue; const node = byPath.get(entry.path); if (node && !byName.has(entry.name)) byName.set(entry.name, node); @@ -228,13 +291,13 @@ export function vueRouteTable(context: ResolutionContext): VueRouteTable { // ============================================================================= /** - * `router.push` / `.replace` (the Composition API), `$router.push` / - * `.replace` (the Options API and templates), and Nuxt's `navigateTo`. + * `router.push` / `.replace` (the Composition API), `this.$router.push` / + * `$router.push` (the Options API and templates), and Nuxt's `navigateTo`. * * As everywhere else, `push` and `replace` need a receiver that names a * router: an unqualified `push` is an array's. */ -const NAV_CALL = /^\$?router\.(?:push|replace)$|^navigateTo$/; +const NAV_CALL = /^(?:this\.)?\$?router\.(?:push|replace)$|^navigateTo$/; /** The verb a navigation call name stands for, or null. */ export function vueNavVerb(name: string): string | null { @@ -282,6 +345,56 @@ function vueComponentNamed(name: string, fromFile: string, context: ResolutionCo return near.length === 1 ? near[0]! : null; } +/** `import:@/views/Login#Login` for a lazy component, the bare name for an identifier. */ +function componentRefName(component: string, spec: string | null): string { + return spec === null ? component : `import:${spec}#${component}`; +} + +/** + * The component a route's reference names. A lazy import names a FILE, and + * the component is that file's: vue-element-admin's views are all + * `…/index.vue`, so the name alone (`index`) fits dozens. The import is + * resolved as the router file would resolve it, or — through an alias the + * resolver doesn't know, like vben's `#/views/…` — by the one file in the + * same app whose path ends the way the specifier does. + */ +function routeComponent(encoded: string, fromFile: string, context: ResolutionContext): Node | null { + const lazy = /^import:(.+)#([^#]+)$/.exec(encoded); + // `component: Layout` — the file's own import of it says which file that is + // (`import Layout from '@/layout'`, a `layout/index.vue`). + const spec = lazy ? lazy[1]! : (context.getImportMappings(fromFile, languageForFile(fromFile)).find((m) => m.localName === encoded)?.source ?? null); + const name = lazy ? lazy[2]! : encoded; + if (spec !== null) { + const file = resolveImportPath(spec, fromFile, 'vue', context) ?? fileBySuffix(spec, fromFile, context); + const component = file ? componentInFile(file, context) : null; + if (component) return component; + } + return name === 'index' ? null : vueComponentNamed(name, fromFile, context); +} + +function componentInFile(file: string, context: ResolutionContext): Node | null { + const nodes = context.getNodesInFile(file); + return nodes.find((n) => n.kind === 'component') ?? nodes.find((n) => n.kind === 'function' && /^[A-Z]/.test(n.name)) ?? null; +} + +const COMPONENT_EXTENSIONS = ['', '.vue', '.tsx', '.jsx', '.ts', '.js', '/index.vue', '/index.tsx', '/index.ts', '/index.js']; + +/** The one file under `fromFile`'s app whose path ends like `spec` minus its alias (`#/`, `@/`, `~/`). */ +function fileBySuffix(spec: string, fromFile: string, context: ResolutionContext): string | null { + const rest = spec.replace(/^(?:[@#~$][\w-]*\/|\.{1,2}\/)+/, ''); + if (rest === spec || rest.length === 0) return null; + const root = appRootFor(fromFile); + const hits = new Set(); + for (const file of context.getAllFiles()) { + if (!file.startsWith(root)) continue; + for (const ext of COMPONENT_EXTENSIONS) { + const tail = rest + ext; + if (file === tail || file.endsWith('/' + tail)) hits.add(file); + } + } + return hits.size === 1 ? [...hits][0]! : null; +} + // ============================================================================= // The resolver // ============================================================================= @@ -289,17 +402,18 @@ function vueComponentNamed(name: string, fromFile: string, context: ResolutionCo export const vueRouterResolver: FrameworkResolver = { name: 'vue-router', languages: [...ROUTE_LANGUAGES], + appDependencies: ['vue-router', 'nuxt', 'nuxt3'], detect(context: ResolutionContext): boolean { return dependsOn(context, 'vue-router', 'nuxt', 'nuxt3'); }, claimsReference(name: string): boolean { - return NAV_CALL.test(name); + return NAV_CALL.test(name) || name.startsWith('import:') || name.startsWith('layout:'); }, extract(filePath: string, content: string): FrameworkExtractionResult { - const entries = parseVueRoutes(content); + const entries = parseVueRoutes(content, filePath); if (entries.length === 0) return { nodes: [], references: [] }; const language = languageForFile(filePath); const now = Date.now(); @@ -329,7 +443,7 @@ export const vueRouterResolver: FrameworkResolver = { // a `calls` edge to a component as the page a screen renders. references.push({ fromNodeId: node.id, - referenceName: entry.component, + referenceName: componentRefName(entry.component, entry.spec), referenceKind: 'calls', line: entry.line, column: 0, @@ -338,11 +452,30 @@ export const vueRouterResolver: FrameworkResolver = { candidates: [entry.component], }); } + // The layouts around it: what they render — a sidebar, a navbar — is on + // this screen too. + for (const layout of entry.layouts) { + references.push({ + fromNodeId: node.id, + referenceName: `layout:${componentRefName(layout.component, layout.spec)}`, + referenceKind: 'references', + line: entry.line, + column: 0, + filePath, + language, + }); + } } return { nodes, references }; }, resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + if (ref.referenceKind === 'references' && isVueRouteRef(ref) && ref.referenceName.startsWith('layout:')) { + const layout = routeComponent(ref.referenceName.slice('layout:'.length), ref.filePath, context); + return layout + ? { original: ref, targetNodeId: layout.id, confidence: 0.95, resolvedBy: 'framework', metadata: { layout: true } } + : null; + } if (ref.referenceKind !== 'calls') return null; // A route naming the component it renders — this resolver's own reference, @@ -350,7 +483,7 @@ export const vueRouterResolver: FrameworkResolver = { // `Login` view AND a `login` action in a store, and only one of them is // the screen. if (isVueRouteRef(ref)) { - const component = vueComponentNamed(ref.referenceName, ref.filePath, context); + const component = routeComponent(ref.referenceName, ref.filePath, context); return component ? { original: ref, targetNodeId: component.id, confidence: 0.95, resolvedBy: 'framework' } : null; diff --git a/src/resolution/frameworks/vue.ts b/src/resolution/frameworks/vue.ts index c830885be3..03e50509ac 100644 --- a/src/resolution/frameworks/vue.ts +++ b/src/resolution/frameworks/vue.ts @@ -1,12 +1,13 @@ /** * Vue / Nuxt Framework Resolver * - * Handles Vue component references, compiler macros (defineProps, etc.), - * Nuxt auto-imports, and Nuxt file-based routing patterns. + * Handles Vue component references, compiler macros (defineProps, etc.) and + * Nuxt auto-imports; `nuxtResolver` reads Nuxt's file-based routes. */ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from '../types'; +import { dependsOn } from './package-deps'; /** * Vue 3 compiler macros — compiler-provided, not user code @@ -186,13 +187,34 @@ export const vueResolver: FrameworkResolver = { return null; }, +}; + +/** + * Nuxt's file-based routes: `pages/` screens, `server/api/` endpoints and + * `middleware/`. Its own resolver, detected only in a Nuxt app: a plain Vue + * app keeps its views in a `pages/` folder just as often (halo's console + * does), and those are components a router config names, not addresses. + */ +export const nuxtResolver: FrameworkResolver = { + name: 'nuxt', + appDependencies: ['nuxt', 'nuxt3', '@nuxt/kit'], + + detect(context: ResolutionContext): boolean { + if (dependsOn(context, 'nuxt', 'nuxt3', '@nuxt/kit')) return true; + return context.getAllFiles().some((f) => /(?:^|\/)nuxt\.config\.(?:[cm]?[jt]s)$/.test(f)); + }, + + resolve(): ResolvedRef | null { + return null; + }, extract(filePath: string, _content: string) { const nodes: Node[] = []; const now = Date.now(); - // Normalize to forward slashes - const normalized = filePath.replace(/\\/g, '/'); + // Forward slashes, and a leading `/` so an app at the repository root + // (`pages/index.vue`) is found by the same `/pages/` search as a nested one. + const normalized = '/' + filePath.replace(/\\/g, '/'); // Detect Nuxt page routes (pages/ directory) const pagesIndex = normalized.indexOf('/pages/'); @@ -221,8 +243,10 @@ export const vueResolver: FrameworkResolver = { const afterApi = normalized.substring(apiIndex + '/server/api/'.length); const routeName = afterApi .replace(/\.[^/.]+$/, '') // Remove extension - .replace(/\/index$/, ''); // index -> parent path - const apiRoute = '/api/' + routeName; + .replace(/(?:^|\/)index$/, '') // index -> parent path + .replace(/\[\.\.\.([^\]]+)\]/g, '*$1') // [...slug] -> *slug + .replace(/\[([^\]]+)\]/g, ':$1'); // [id] -> :id, as a page's params are + const apiRoute = routeName === '' ? '/api' : '/api/' + routeName; nodes.push({ id: `route:${filePath}:${apiRoute}:1`, @@ -317,7 +341,7 @@ function filePathToNuxtRoute(normalized: string, afterPagesStart: number): strin const withoutExt = afterPages.replace(/\.vue$/, ''); // Remove /index suffix (index.vue -> parent route) - const withoutIndex = withoutExt.replace(/\/index$/, ''); + const withoutIndex = withoutExt.replace(/(?:^|\/)index$/, ''); // Convert Nuxt param syntax [param] to :param let route = '/' + withoutIndex diff --git a/src/ui-server/api/screens.ts b/src/ui-server/api/screens.ts index 94b70761a9..8a79ff2d82 100644 --- a/src/ui-server/api/screens.ts +++ b/src/ui-server/api/screens.ts @@ -218,7 +218,7 @@ function writtenHere(edge: Edge, holder: Node): boolean { * Entry points, which is the list of what a request or a user can arrive at. */ function isScreenRoute(route: Node): boolean { - return route.name.startsWith('/') && !route.filePath.includes('/server/api/'); + return route.name.startsWith('/') && !`/${route.filePath}`.includes('/server/api/'); } export async function buildScreens(cg: CodeGraph, projectRoot: string): Promise { From b62da73550fea1e153d94116c902ffe7ee7d3af2 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 06:50:41 +0000 Subject: [PATCH 016/259] fix(sveltekit): route names drop (group) folders and parameter matchers (#2131) A `(group)` directory shares a layout and is never part of the URL, and a parameter's `=matcher` checks its value. Kept in the name, shadcn-svelte's `src/routes/(app)/(layout)/blocks/+page.svelte` was `/(app)/(layout)/blocks`, so no `goto('/blocks')` or `` ever reached it and its Screens showed no navigation at all. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + README.md | 2 +- __tests__/sveltekit-route-names.test.ts | 61 +++++++++++++++++++++++++ docs/design/framework-coverage.md | 2 +- src/resolution/frameworks/svelte.ts | 13 ++++-- 5 files changed, 72 insertions(+), 7 deletions(-) create mode 100644 __tests__/sveltekit-route-names.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index a7dd3a4bfd..7fb6d96024 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- SvelteKit routes are now named by the address a browser asks for. A `(group)` folder like `(app)` or `(marketing)` is no longer part of a route's path, and a parameter with a matcher, like `[id=integer]`, is just `:id`. So `goto('/blocks')` and `` now reach a page that lives in `src/routes/(app)/blocks/`, where before they reached nothing and the app's screens showed no navigation. Re-index SvelteKit projects after upgrading. - Vue Router apps built like vue-element-admin, vue-admin-template or vben now have their routes and screens. Routes declared in a named table (`export const constantRoutes = [...]`, `const routes: RouteRecordRaw[] = [...]`), in per-module route files, or through `new Router(...)` are now read. Child routes are joined onto their parent's path, and the parent's component is the layout around them. A lazily loaded view (`() => import('@/views/dashboard/index')`) is bound to the file it names, and `this.$router.push(...)` counts as navigation. Nuxt's file-based routes now come only from a Nuxt app, so a plain Vue app's `pages/` folder no longer turns into made-up screens, and a Nuxt app at the root of its repository now gets its routes. Re-index Vue projects after upgrading. - In a monorepo with an Expo app next to a Next.js app, like create-t3-turbo or tamagui's starter, each router now reads only its own app. Expo Router used to read the Next.js app's files too, so `layout.tsx`, `page.tsx`, a `_components` folder and an API route showed up as screens named `/layout`, `/page` or `/_components/posts`, and a shared package's files became screens as well. Re-index such monorepos after upgrading. - Calls in Vue, Svelte and Astro components now follow the same rules as JavaScript and TypeScript. A `t('…')` from `useI18n()` or a `ref(…)` from `vue` used to link to an interface property or a local variable of the same name elsewhere in the project, so on halo two unrelated symbols had over a thousand made-up callers each. A call imported from a package is also no longer mistaken for a project symbol when the package is declared in a nested app's `package.json` or a Deno import map. An import through an alias CodeGraph can't follow, like SvelteKit's `$lib/…` or a nested Nuxt app's `~/…`, is still treated as project code. Re-index Vue, Svelte, Astro and JavaScript/TypeScript projects after upgrading. diff --git a/README.md b/README.md index 3d02481dd8..a02bfc030b 100644 --- a/README.md +++ b/README.md @@ -345,7 +345,7 @@ These frameworks additionally emit **`navigates`** edges: the function that send | **React Router** | `` (v5 and v6) and `createBrowserRouter([{ path, element }])` | `history.push` / `replace`, `useNavigate`'s `navigate`, a loader's `redirect`, `` / `` / `` / react-router-bootstrap's `` | | **TanStack Router** | `createFileRoute('/posts/$postId')` (file-based) and `createRoute({ path, getParentRoute })` composed up its parent chain (code-based); `_pathless` segments, `(group)` folders, `__root` and `` layouts are not addresses | `navigate({ to })`, a thrown `redirect({ to })`, `` / `` — where `to` is the route PATTERN and the values ride beside it in `params` | | **Vue Router** / **Nuxt** | `createRouter({ routes: [...] })` / `new Router(...)` and the route tables it's given (`export const constantRoutes = [...]`, per-module route files), with the view each entry names — a lazy `() => import(…)` bound to its file — and `children` joined onto their parent's path, the parent being the layout around them; plus Nuxt `pages/` file-based routes, `server/api/` endpoints and route middleware | `router.push` / `replace`, `this.$router.push`, Nuxt's `navigateTo`, `` / `` / `` — **by route name** (`push({ name: 'profile' })`) as well as by path | -| **SvelteKit** | `src/routes/**/+page.svelte` (`[slug]` → `:slug`, `[[opt]]` → `:opt?`), joined to the `+page.server.js` beside it so a loader's guard belongs to its page | `goto('/x')`, `redirect(status, '/x')` from a load or form action, and the plain `` that is a link in a SvelteKit app | +| **SvelteKit** | `src/routes/**/+page.svelte` (`[slug]` → `:slug`, `[[opt]]` → `:opt?`, `[id=matcher]` → `:id`, `(group)` folders stripped), joined to the `+page.server.js` beside it so a loader's guard belongs to its page | `goto('/x')`, `redirect(status, '/x')` from a load or form action, and the plain `` that is a link in a SvelteKit app | | **Angular** | `Routes` arrays (`RouterModule.forRoot` / `forChild`, `provideRouter`, a routes file's default export) with `component` or a lazy `loadComponent`; `children` and lazy `loadChildren` (an NgModule's through its routing module) joined into full paths; paths written as route constants or `$localize` strings; a route with children is a layout around the screens inside it | `router.navigate([...])`, `navigateByUrl`, a guard's `createUrlTree` / `parseUrl` — a command array, a route constant, or a component property holding one — `routerLink` / `[routerLink]` in the component's template, and `redirectTo`. Each template's child components (``) are linked to the component that renders them | In a repository holding several apps, each app's routes are matched only against navigation written inside that app. diff --git a/__tests__/sveltekit-route-names.test.ts b/__tests__/sveltekit-route-names.test.ts new file mode 100644 index 0000000000..1f7ab5e465 --- /dev/null +++ b/__tests__/sveltekit-route-names.test.ts @@ -0,0 +1,61 @@ +/** + * SvelteKit route names are the URL a browser asks for. + * + * - A `(group)` directory organizes layouts and never appears in the URL: + * shadcn-svelte's `src/routes/(app)/(layout)/blocks/+page.svelte` is `/blocks`. + * - `[param=matcher]` is a parameter whose value a matcher checks: `/view/:view`. + * + * Named with the groups, no `goto('/blocks')` or `` ever + * matched its page. + */ +import { describe, it, expect, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +const roots: string[] = []; +afterAll(() => { + for (const r of roots.splice(0)) fs.rmSync(r, { recursive: true, force: true }); +}); + +describe('SvelteKit route names', () => { + it('drop (group) directories and parameter matchers', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-sveltekit-groups-')); + roots.push(root); + const files: Record = { + 'package.json': JSON.stringify({ name: 'site', devDependencies: { '@sveltejs/kit': '*', svelte: '*' } }), + 'src/routes/(app)/(layout)/blocks/+page.svelte': `

Blocks

+`, + 'src/routes/(app)/(layout)/+page.svelte': ` + +Card +`, + 'src/routes/(view)/view/[view=view]/+page.svelte': `

view

+`, + 'src/routes/docs/[...slug=doc]/+page.svelte': `

doc

+`, + 'src/params/view.ts': `export function match(p: string) { return p.length > 0; } +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + const cg = await CodeGraph.init(root, { index: true }); + try { + const routes = cg.getNodesByKind('route').map((n) => n.name).sort(); + expect(routes).toEqual(['/', '/blocks', '/docs/*slug', '/view/:view']); + const home = cg.getNodesByKind('route').find((n) => n.name === '/')!; + const navigates = cg + .getOutgoingEdgesFrom(cg.getNodesInFile(home.filePath).map((n) => n.id), ['navigates' as never]) + .map((e) => cg.getNode(e.target)!.name) + .sort(); + expect(navigates).toEqual(['/blocks', '/view/:view']); + } finally { + cg.close(); + } + }); +}); diff --git a/docs/design/framework-coverage.md b/docs/design/framework-coverage.md index b149ccb7d7..e0be9a92bc 100644 --- a/docs/design/framework-coverage.md +++ b/docs/design/framework-coverage.md @@ -41,7 +41,7 @@ guessed. | React Router | `frameworks/react-router.ts` | `react-router-synthesizer.ts` | `react-router.test.ts` | proshop (44 edges) | | TanStack Router | `frameworks/tanstack-router.ts` | `tanstack-router-synthesizer.ts` | `tanstack-router.test.ts` | TanStack examples, fastapi-template frontend | | Vue Router / Nuxt | `frameworks/vue-router.ts` (Nuxt file routes: `nuxtResolver` in `frameworks/vue.ts`) | `vue-router-synthesizer.ts` | `vue-router.test.ts` | vue-realworld (23 edges); vue-element-admin (62 routes), vue-admin-template (14), vben (192), halo console (34) — named tables, module files, `children` + layouts; Nuxt: mealie, elk, nuxt/movies | -| SvelteKit | `frameworks/sveltekit-router.ts` | `sveltekit-synthesizer.ts` | `sveltekit-router.test.ts` | sveltekit-realworld (31 edges) | +| SvelteKit | `frameworks/sveltekit-router.ts` | `sveltekit-synthesizer.ts` | `sveltekit-router.test.ts`, `sveltekit-route-names.test.ts` | sveltekit-realworld (31 edges); shadcn-svelte and skeleton (`(group)` layouts: 13 and 23 edges), svelte.dev (74), kit's test apps (47) | | Angular | `frameworks/angular-router.ts` | `angular-template-synthesizer.ts` | `angular-router.test.ts` | angular-realworld (31 edges, 18 renders), Ghostfolio (189 edges, 170 renders), ngx-admin (routes and renders; its menus are config) | Shared machinery all seven use, in `frameworks/expo-router.ts`: `RouteTable` / diff --git a/src/resolution/frameworks/svelte.ts b/src/resolution/frameworks/svelte.ts index d3271a6fc8..9598f98107 100644 --- a/src/resolution/frameworks/svelte.ts +++ b/src/resolution/frameworks/svelte.ts @@ -276,11 +276,14 @@ function filePathToSvelteKitRoute(filePath: string): string | null { const lastSlash = afterRoutes.lastIndexOf('/'); const dirPath = lastSlash === -1 ? '' : afterRoutes.substring(0, lastSlash); - // Convert SvelteKit param syntax [param] to :param - let route = '/' + dirPath - .replace(/\[\.\.\.([^\]]+)\]/g, '*$1') // [...rest] -> *rest - .replace(/\[{2}([^\]]+)\]{2}/g, ':$1?') // [[optional]] -> :optional? - .replace(/\[([^\]]+)\]/g, ':$1'); // [param] -> :param + // A `(group)` directory shares a layout and never appears in the URL: + // `(app)/(layout)/blocks` is `/blocks`. A parameter's `=matcher` checks + // its value and is no part of its name. + const segments = dirPath.split('/').filter((seg) => seg.length > 0 && !/^\(.*\)$/.test(seg)); + let route = '/' + segments.join('/') + .replace(/\[\.\.\.([^\]=]+)(?:=[^\]]*)?\]/g, '*$1') // [...rest] / [...rest=m] -> *rest + .replace(/\[{2}([^\]=]+)(?:=[^\]]*)?\]{2}/g, ':$1?') // [[optional]] -> :optional? + .replace(/\[([^\]=]+)(?:=[^\]]*)?\]/g, ':$1'); // [param] / [param=m] -> :param if (route === '/') return '/'; // Remove trailing slash From 4423891e76df05f2a2dcd6257992731e6bb64768 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 06:55:41 +0000 Subject: [PATCH 017/259] fix(react-router): v5 and styled(Link) wrappers navigate (#2132) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit react-boilerplate's header links are `export default styled(Link)`…``, used as ``; takenote's guards render v5's ``. Neither tag was read, so both apps had routes and no links between their screens. The link synthesizer now reads `` (navMethod `redirect`) and any styled-components / emotion wrapper of `Link` / `NavLink` — declared in the file, or imported from a file that declares or default-exports one. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + README.md | 2 +- __tests__/react-router.test.ts | 98 ++++++++++++++++++++++ docs/design/framework-coverage.md | 2 +- src/resolution/react-router-synthesizer.ts | 85 +++++++++++++++++-- 5 files changed, 177 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7fb6d96024..4c4bb38e8e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- React Router apps now show navigation written as React Router v5's `` and through a styled link, like react-boilerplate's `HeaderLink = styled(Link)` used as ``. Apps built this way had routes but no links between their screens. - SvelteKit routes are now named by the address a browser asks for. A `(group)` folder like `(app)` or `(marketing)` is no longer part of a route's path, and a parameter with a matcher, like `[id=integer]`, is just `:id`. So `goto('/blocks')` and `` now reach a page that lives in `src/routes/(app)/blocks/`, where before they reached nothing and the app's screens showed no navigation. Re-index SvelteKit projects after upgrading. - Vue Router apps built like vue-element-admin, vue-admin-template or vben now have their routes and screens. Routes declared in a named table (`export const constantRoutes = [...]`, `const routes: RouteRecordRaw[] = [...]`), in per-module route files, or through `new Router(...)` are now read. Child routes are joined onto their parent's path, and the parent's component is the layout around them. A lazily loaded view (`() => import('@/views/dashboard/index')`) is bound to the file it names, and `this.$router.push(...)` counts as navigation. Nuxt's file-based routes now come only from a Nuxt app, so a plain Vue app's `pages/` folder no longer turns into made-up screens, and a Nuxt app at the root of its repository now gets its routes. Re-index Vue projects after upgrading. - In a monorepo with an Expo app next to a Next.js app, like create-t3-turbo or tamagui's starter, each router now reads only its own app. Expo Router used to read the Next.js app's files too, so `layout.tsx`, `page.tsx`, a `_components` folder and an API route showed up as screens named `/layout`, `/page` or `/_components/posts`, and a shared package's files became screens as well. Re-index such monorepos after upgrading. diff --git a/README.md b/README.md index a02bfc030b..23a046f54c 100644 --- a/README.md +++ b/README.md @@ -342,7 +342,7 @@ These frameworks additionally emit **`navigates`** edges: the function that send |---|---|---| | **Expo Router** | Every screen file under `app/` (`app/item/[id].tsx` → `/item/[id]`, groups stripped), bound to its default-export component | `router.push` / `replace` / `navigate`, template hrefs, `{ pathname }` objects, and a helper's returned href | | **Next.js** | App Router `app/**/page.tsx` and Pages Router pages (`(group)` stripped, `[slug]` → `:slug`); `app/api/**/route.ts` exports and `pages/api/*` are endpoints, not screens | `router.push` / `replace` / `prefetch`, `redirect()` / `permanentRedirect()` in a server action or page, `NextResponse.redirect(new URL(…))` in middleware, `` and internal `` | -| **React Router** | `` (v5 and v6) and `createBrowserRouter([{ path, element }])` | `history.push` / `replace`, `useNavigate`'s `navigate`, a loader's `redirect`, `` / `` / `` / react-router-bootstrap's `` | +| **React Router** | `` (v5 and v6) and `createBrowserRouter([{ path, element }])` | `history.push` / `replace`, `useNavigate`'s `navigate`, a loader's `redirect`, `` / `` / `` / v5's `` / react-router-bootstrap's ``, and a `styled(Link)` wrapper | | **TanStack Router** | `createFileRoute('/posts/$postId')` (file-based) and `createRoute({ path, getParentRoute })` composed up its parent chain (code-based); `_pathless` segments, `(group)` folders, `__root` and `` layouts are not addresses | `navigate({ to })`, a thrown `redirect({ to })`, `` / `` — where `to` is the route PATTERN and the values ride beside it in `params` | | **Vue Router** / **Nuxt** | `createRouter({ routes: [...] })` / `new Router(...)` and the route tables it's given (`export const constantRoutes = [...]`, per-module route files), with the view each entry names — a lazy `() => import(…)` bound to its file — and `children` joined onto their parent's path, the parent being the layout around them; plus Nuxt `pages/` file-based routes, `server/api/` endpoints and route middleware | `router.push` / `replace`, `this.$router.push`, Nuxt's `navigateTo`, `` / `` / `` — **by route name** (`push({ name: 'profile' })`) as well as by path | | **SvelteKit** | `src/routes/**/+page.svelte` (`[slug]` → `:slug`, `[[opt]]` → `:opt?`, `[id=matcher]` → `:id`, `(group)` folders stripped), joined to the `+page.server.js` beside it so a loader's guard belongs to its page | `goto('/x')`, `redirect(status, '/x')` from a load or form action, and the plain `` that is a link in a SvelteKit app | diff --git a/__tests__/react-router.test.ts b/__tests__/react-router.test.ts index 70ea8266ce..fd4c9d7e90 100644 --- a/__tests__/react-router.test.ts +++ b/__tests__/react-router.test.ts @@ -593,3 +593,101 @@ describe('react-router: route declaration boundaries (#1348)', () => { expect(result).toEqual({ paths: ['/child', '/shell'], bindings: ['/child->Child', '/shell->Shell'] }); }); }); + +describe('react-router: v5 redirects and styled link wrappers', () => { + let root: string; + let cg: CodeGraph; + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-rr-wrappers-')); + const files: Record = { + 'package.json': JSON.stringify({ name: 'app', dependencies: { react: '*', 'react-router-dom': '^5.0.0', 'styled-components': '*' } }), + // react-boilerplate: the header's links are a styled Link, default-exported. + 'app/components/Header/HeaderLink.js': `import { Link } from 'react-router-dom'; +import styled from 'styled-components'; + +export default styled(Link)\` + color: #41addd; +\`; +`, + 'app/components/Header/index.js': `import React from 'react'; +import HeaderLink from './HeaderLink'; + +export default function Header() { + return ( + + ); +} +`, + 'app/components/Nav.js': `import React from 'react'; +import { NavLink } from 'react-router-dom'; +import styled from 'styled-components'; + +const MenuLink = styled(NavLink)\` + padding: 4px; +\`; + +export default function Nav() { + return Features; +} +`, + // takenote: a guard renders v5's . + 'app/router/PrivateRoute.js': `import React from 'react'; +import { Route, Redirect } from 'react-router-dom'; + +export default function PrivateRoute({ component: Component, ...rest }) { + return (rest.isAuthenticated ? : )} />; +} +`, + 'app/containers/App.js': `import React from 'react'; +import { Switch, Route } from 'react-router-dom'; +import HomePage from './HomePage'; +import FeaturePage from './FeaturePage'; + +export default function App() { + return ( + + + + + ); +} +`, + 'app/containers/HomePage.js': `import React from 'react'; +export default function HomePage() { return

Home

; } +`, + 'app/containers/FeaturePage.js': `import React from 'react'; +export default function FeaturePage() { return

Features

; } +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); + }); + afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); + }); + + const navsFrom = (name: string) => { + const from = cg.getNodesByName(name).find((n) => n.kind === 'function' || n.kind === 'component')!; + return cg + .getOutgoingEdges(from.id) + .filter((e) => e.kind === 'navigates') + .map((e) => `${(e.metadata as Record).navMethod} ${cg.getNode(e.target)!.name}`) + .sort(); + }; + + it('a styled(Link) wrapper, imported or local, is a link', () => { + expect(navsFrom('Header')).toEqual(['link /', 'link /features']); + expect(navsFrom('Nav')).toEqual(['link /features']); + }); + + it('v5’s navigates', () => { + expect(navsFrom('PrivateRoute')).toEqual(['redirect /']); + }); +}); diff --git a/docs/design/framework-coverage.md b/docs/design/framework-coverage.md index e0be9a92bc..fcb12619ce 100644 --- a/docs/design/framework-coverage.md +++ b/docs/design/framework-coverage.md @@ -38,7 +38,7 @@ guessed. |---|---|---|---|---| | Expo Router | `frameworks/expo-router.ts` | `expo-router-synthesizer.ts` | `expo-router.test.ts` | — | | Next.js | `frameworks/nextjs.ts` | `next-router-synthesizer.ts` | `nextjs.test.ts` | next-saas-starter | -| React Router | `frameworks/react-router.ts` | `react-router-synthesizer.ts` | `react-router.test.ts` | proshop (44 edges) | +| React Router | `frameworks/react-router.ts` | `react-router-synthesizer.ts` | `react-router.test.ts` | proshop (44 edges), proshop-v2 (28), react-redux-realworld (22), react-boilerplate (`styled(Link)`), takenote (v5 ``); bulletproof-react's `paths.x.path` constants and `lazy` routes are not read yet | | TanStack Router | `frameworks/tanstack-router.ts` | `tanstack-router-synthesizer.ts` | `tanstack-router.test.ts` | TanStack examples, fastapi-template frontend | | Vue Router / Nuxt | `frameworks/vue-router.ts` (Nuxt file routes: `nuxtResolver` in `frameworks/vue.ts`) | `vue-router-synthesizer.ts` | `vue-router.test.ts` | vue-realworld (23 edges); vue-element-admin (62 routes), vue-admin-template (14), vben (192), halo console (34) — named tables, module files, `children` + layouts; Nuxt: mealie, elk, nuxt/movies | | SvelteKit | `frameworks/sveltekit-router.ts` | `sveltekit-synthesizer.ts` | `sveltekit-router.test.ts`, `sveltekit-route-names.test.ts` | sveltekit-realworld (31 edges); shadcn-svelte and skeleton (`(group)` layouts: 13 and 23 edges), svelte.dev (74), kit's test apps (47) | diff --git a/src/resolution/react-router-synthesizer.ts b/src/resolution/react-router-synthesizer.ts index a0706219e7..3787b625a7 100644 --- a/src/resolution/react-router-synthesizer.ts +++ b/src/resolution/react-router-synthesizer.ts @@ -6,6 +6,8 @@ * * … // react-router-bootstrap * … // v5's object form + * // v5 + * // HeaderLink = styled(Link) * * A JSX attribute is not a call, so the extractor records no reference for it * and the resolver in `frameworks/react-router.ts` — which binds @@ -27,7 +29,7 @@ * attribute (`to`, not `href`) and the other table. */ -import type { Edge } from '../types'; +import type { Edge, Language } from '../types'; import type { ResolutionContext } from './types'; import type { MaybeYield } from './cooperative-yield'; import { stripCommentsForRegex } from './strip-comments'; @@ -37,14 +39,75 @@ import { matchBracket } from './frameworks/object-literal'; import { destinationsForHref } from './frameworks/nextjs'; import { reactRouterTable } from './frameworks/react-router'; import { enclosingFn, makeLineAt } from './synth-utils'; +import { resolveImportPath } from './import-resolver'; const JSX_FILE = /\.(?:[cm]?[jt]sx?|mdx)$/; -/** The tags that carry a route as a `to` attribute, the attribute anywhere in the tag. */ -const LINK_TAG = /<(Link|NavLink|Navigate|LinkContainer|IndexLinkContainer)\b([^>]*?)\bto\s*=\s*(?:"([^"]*)"|'([^']*)'|(?=\{))/g; +/** The tags that carry a route as a `to` attribute. */ +const LINK_TAGS = ['Link', 'NavLink', 'Navigate', 'Redirect', 'LinkContainer', 'IndexLinkContainer']; -/** A tag this pass could possibly match — the cheap prefilter before stripping comments. */ -const HAS_LINK_TAG = /<(?:Link|NavLink|Navigate|LinkContainer|IndexLinkContainer)\b/; +/** `` for any of `tags`, the attribute anywhere in the tag. */ +function linkTagPattern(tags: readonly string[]): RegExp { + return new RegExp(`<(${tags.map((t) => t.replace(/[$]/g, '\\$&')).join('|')})\\b([^>]*?)\\bto\\s*=\\s*(?:"([^"]*)"|'([^']*)'|(?=\\{))`, 'g'); +} + +const LINK_TAG = linkTagPattern(LINK_TAGS); + +/** A file that imports React Router's own components. */ +const ROUTER_IMPORT = /\bfrom\s*['"]react-router(?:-dom)?['"]/; + +/** `const HeaderLink = styled(Link)` — a styled wrapper is the link it wraps. */ +const STYLED_LINK_CONST = /\b(?:export\s+)?(?:const|let)\s+([A-Z][\w$]*)\s*=\s*styled\s*\(\s*(?:Link|NavLink)\s*\)/g; +const STYLED_LINK_DEFAULT = /\bexport\s+default\s+styled\s*\(\s*(?:Link|NavLink)\s*\)/; + +interface LinkWrappers { + /** File → the wrapper components it declares by name. */ + named: Map>; + /** Files whose default export is a wrapper. */ + defaults: Set; +} + +/** + * Every styled-components / emotion wrapper of `Link` or `NavLink` in the + * project — react-boilerplate's header links are `export default + * styled(Link)`…``, used as `` from another file. + */ +function linkWrappers(ctx: ResolutionContext): LinkWrappers { + const named = new Map>(); + const defaults = new Set(); + for (const file of ctx.getAllFiles()) { + if (!JSX_FILE.test(file) || isTestPath(file)) continue; + if (ctx.fileContains && !ctx.fileContains(file, 'styled')) continue; + const source = ctx.readFile(file); + if (!source || !/\bstyled\s*\(\s*(?:Link|NavLink)\s*\)/.test(source) || !ROUTER_IMPORT.test(source)) continue; + if (STYLED_LINK_DEFAULT.test(source)) defaults.add(file); + STYLED_LINK_CONST.lastIndex = 0; + let m: RegExpExecArray | null; + while ((m = STYLED_LINK_CONST.exec(source)) !== null) { + let set = named.get(file); + if (!set) named.set(file, (set = new Set())); + set.add(m[1]!); + } + } + return { named, defaults }; +} + +/** The link tags `file` can write: React Router's own and the wrappers in scope there. */ +function linkTagsFor(file: string, wrappers: LinkWrappers, ctx: ResolutionContext): string[] { + const tags = [...LINK_TAGS, ...(wrappers.named.get(file) ?? [])]; + if (wrappers.named.size > 0 || wrappers.defaults.size > 0) { + for (const m of ctx.getImportMappings(file, languageOf(file))) { + const target = resolveImportPath(m.source, file, languageOf(file), ctx); + if (!target) continue; + if ((m.isDefault && wrappers.defaults.has(target)) || wrappers.named.get(target)?.has(m.exportedName)) tags.push(m.localName); + } + } + return tags; +} + +function languageOf(file: string): Language { + return /\.tsx$/.test(file) ? 'tsx' : /\.[cm]?ts$/.test(file) ? 'typescript' : /\.jsx$/.test(file) ? 'jsx' : 'javascript'; +} /** Links a single component may carry before it is a navigation menu, not a decision. */ const MAX_LINKS_PER_COMPONENT = 24; @@ -55,6 +118,7 @@ export async function reactRouterLinkEdges(ctx: ResolutionContext, onYield: Mayb const edges: Edge[] = []; const seen = new Set(); const perComponent = new Map(); + const wrappers = linkWrappers(ctx); let scanned = 0; for (const file of ctx.getAllFiles()) { if (!JSX_FILE.test(file) || isTestPath(file)) continue; @@ -62,13 +126,16 @@ export async function reactRouterLinkEdges(ctx: ResolutionContext, onYield: Mayb if (!routes || routes.exact.size === 0) continue; if ((++scanned & 63) === 0) await onYield(); const source = ctx.readFile(file); - if (!source || !HAS_LINK_TAG.test(source)) continue; + if (!source || !/\bto\s*=/.test(source)) continue; + const tags = linkTagsFor(file, wrappers, ctx); + const pattern = tags.length === LINK_TAGS.length ? LINK_TAG : linkTagPattern(tags); + if (!new RegExp(`<(?:${tags.join('|')})\\b`).test(source)) continue; const safe = stripCommentsForRegex(source, 'typescript'); const nodes = ctx.getNodesInFile(file); const lineOf = makeLineAt(safe, 1); - LINK_TAG.lastIndex = 0; + pattern.lastIndex = 0; let m: RegExpExecArray | null; - while ((m = LINK_TAG.exec(safe)) !== null) { + while ((m = pattern.exec(safe)) !== null) { const tag = m[1]!; const quoted: string | null = m[3] ?? m[4] ?? null; let href: HrefLiteral | null; @@ -109,7 +176,7 @@ export async function reactRouterLinkEdges(ctx: ResolutionContext, onYield: Mayb metadata: { synthesizedBy: 'react-router-link', href: arm.display, - navMethod: tag === 'Navigate' ? 'navigate' : 'link', + navMethod: tag === 'Navigate' ? 'navigate' : tag === 'Redirect' ? 'redirect' : 'link', registeredAt: `${file}:${line}`, }, }); From cec041d241321060f17a612f034e1bcac36e0530 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 07:06:23 +0000 Subject: [PATCH 018/259] fix(angular): Nx workspaces, barrels, class-constant paths and root-relative navigate (#2133) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From the router sweep: - angular-spotify had 12 routes all named `/` and no navigation. Its lazy routes are `async () => (await import('@ws/home')).HomeModule` through an Nx library's `src/index.ts` barrel, which the mount pass never looked through; and every lib is its own `src/`, so each routes file got its own route table and no template's `routerLink` found the screen it names. - jira-clone had 3 routes and no navigation: its issue route is `issue/:${ProjectConst.IssueId}` (a class `static readonly`), its root redirect lives in `app.routes.ts`, which only mounts, and `navigate(['project', 'issue', id])` has no leading `/`. Now: lazy mounts follow `export … from` barrels (and an NgModule's routing imports); `(await import(x)).M` is read; route constants may be class statics (strings or objects), enum members, or live behind a barrel, and a template-literal path takes constant holes; a routes file that only mounts contributes its redirects at its mount prefix; the route table is keyed by the `angular.json` / `nx.json` workspace; and `router.navigate` / `createUrlTree` without `relativeTo` resolve a bare first command from the root, as Angular does (template `routerLink` stays relative). Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/angular-router.test.ts | 140 +++++++++ docs/design/framework-coverage.md | 2 +- src/resolution/frameworks/angular-router.ts | 313 ++++++++++++++------ 4 files changed, 370 insertions(+), 86 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4c4bb38e8e..037f61f834 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- Angular apps split across Nx libraries, like angular-spotify, now have their routes and navigation. A lazily loaded route written `async () => (await import('@app/home')).HomeModule` is followed through the library's `index.ts` to the module it re-exports, and a `routerLink` in one library now reaches a screen declared in another. Route paths built from a class constant (`RouterUtil.Configuration.Lyrics`, `` `issue/:${ProjectConst.IssueId}` ``) or an enum are now read. A redirect in a routes file that only lazy-loads others now counts, and `router.navigate(['project', 'issue', id])` without `relativeTo` is read from the root, as Angular does. Re-index Angular projects after upgrading. - React Router apps now show navigation written as React Router v5's `` and through a styled link, like react-boilerplate's `HeaderLink = styled(Link)` used as ``. Apps built this way had routes but no links between their screens. - SvelteKit routes are now named by the address a browser asks for. A `(group)` folder like `(app)` or `(marketing)` is no longer part of a route's path, and a parameter with a matcher, like `[id=integer]`, is just `:id`. So `goto('/blocks')` and `` now reach a page that lives in `src/routes/(app)/blocks/`, where before they reached nothing and the app's screens showed no navigation. Re-index SvelteKit projects after upgrading. - Vue Router apps built like vue-element-admin, vue-admin-template or vben now have their routes and screens. Routes declared in a named table (`export const constantRoutes = [...]`, `const routes: RouteRecordRaw[] = [...]`), in per-module route files, or through `new Router(...)` are now read. Child routes are joined onto their parent's path, and the parent's component is the layout around them. A lazily loaded view (`() => import('@/views/dashboard/index')`) is bound to the file it names, and `this.$router.push(...)` counts as navigation. Nuxt's file-based routes now come only from a Nuxt app, so a plain Vue app's `pages/` folder no longer turns into made-up screens, and a Nuxt app at the root of its repository now gets its routes. Re-index Vue projects after upgrading. diff --git a/__tests__/angular-router.test.ts b/__tests__/angular-router.test.ts index a5309abd60..52c13ca47d 100644 --- a/__tests__/angular-router.test.ts +++ b/__tests__/angular-router.test.ts @@ -317,3 +317,143 @@ export class AdminRoutingModule {} ]); }); }); + +describe('an Angular workspace split across libraries', () => { + const roots: string[] = []; + afterAll(() => { + for (const r of roots.splice(0)) fs.rmSync(r, { recursive: true, force: true }); + }); + + async function project(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-angular-ws-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); + } + const routeNames = (cg: CodeGraph) => cg.getNodesByKind('route').map((n) => n.name).sort(); + const navs = (cg: CodeGraph) => + cg + .getNodesByKind('route') + .flatMap((r) => cg.getIncomingEdges(r.id).filter((e) => e.kind === 'navigates').map((e) => `${cg.getNode(e.source)!.name} -> ${r.name}`)) + .sort(); + const component = (name: string, selector: string, template: string) => `import { Component } from '@angular/core'; +@Component({ selector: '${selector}', template: \`${template}\` }) +export class ${name} {} +`; + + it('angular-spotify: Nx libraries behind barrels, `(await import(…)).M`, a class-static path constant', async () => { + const cg = await project({ + 'package.json': JSON.stringify({ name: 'ws', dependencies: { '@angular/core': '*', '@angular/router': '*' } }), + 'nx.json': '{}', + 'tsconfig.base.json': JSON.stringify({ + compilerOptions: { + paths: { + '@ws/home': ['libs/home/src/index.ts'], + '@ws/lyrics': ['libs/lyrics/src/index.ts'], + '@ws/utils': ['libs/utils/src/index.ts'], + }, + }, + }), + 'libs/utils/src/index.ts': `export * from './lib/router-util';\n`, + 'libs/utils/src/lib/router-util.ts': `export class RouterUtil { + static Configuration = { Lyrics: 'lyrics' }; +} +`, + 'libs/shell/src/lib/shell.routes.ts': `import { Route } from '@angular/router'; +import { RouterUtil } from '@ws/utils'; + +export const shellRoutes: Route[] = [ + { path: '', loadChildren: async () => (await import('@ws/home')).HomeModule }, + { path: RouterUtil.Configuration.Lyrics, loadChildren: async () => (await import('@ws/lyrics')).LyricsModule }, +]; +`, + 'libs/home/src/index.ts': `export * from './lib/home.module';\n`, + 'libs/home/src/lib/home.module.ts': `import { NgModule } from '@angular/core'; +import { RouterModule } from '@angular/router'; +import { HomeComponent } from './home.component'; +@NgModule({ imports: [RouterModule.forChild([{ path: '', component: HomeComponent }])] }) +export class HomeModule {} +`, + 'libs/home/src/lib/home.component.ts': component('HomeComponent', 'as-home', 'Lyrics'), + 'libs/lyrics/src/index.ts': `export * from './lib/lyrics.module';\n`, + 'libs/lyrics/src/lib/lyrics.module.ts': `import { NgModule } from '@angular/core'; +import { RouterModule } from '@angular/router'; +import { LyricsComponent } from './lyrics.component'; +@NgModule({ imports: [RouterModule.forChild([{ path: '', component: LyricsComponent }])] }) +export class LyricsModule {} +`, + 'libs/lyrics/src/lib/lyrics.component.ts': component('LyricsComponent', 'as-lyrics', 'Home'), + }); + try { + expect(routeNames(cg)).toEqual(['/', '/lyrics']); + // A template in one library links to a screen another library declares. + expect(navs(cg)).toEqual(['HomeComponent -> /lyrics', 'LyricsComponent -> /']); + } finally { + cg.close(); + } + }); + + it('jira-clone: a template-literal path, a redirect in a file that only mounts, navigate() from the root', async () => { + const cg = await project({ + 'package.json': JSON.stringify({ name: 'jira', dependencies: { '@angular/core': '*', '@angular/router': '*' } }), + 'angular.json': '{}', + 'src/app/app.routes.ts': `import { Routes } from '@angular/router'; +export const appRoutes: Routes = [ + { path: 'project', loadChildren: () => import('./project/project.routes').then((m) => m.PROJECT_ROUTES) }, + { path: '', redirectTo: 'project', pathMatch: 'full' }, +]; +`, + 'src/app/project/config/const.ts': `export class ProjectConst { + static readonly IssueId = 'issueId'; +} +`, + 'src/app/project/project.routes.ts': `import { Routes } from '@angular/router'; +import { ProjectComponent } from './project.component'; +import { BoardComponent } from './board.component'; +import { IssueComponent } from './issue.component'; +import { ProjectConst } from './config/const'; +export const PROJECT_ROUTES: Routes = [ + { + path: '', + component: ProjectComponent, + children: [ + { path: 'board', component: BoardComponent }, + { path: \`issue/:\${ProjectConst.IssueId}\`, component: IssueComponent }, + { path: '', redirectTo: 'board', pathMatch: 'full' }, + ], + }, +]; +`, + 'src/app/project/project.component.ts': component('ProjectComponent', 'j-project', ''), + 'src/app/project/board.component.ts': `import { Component } from '@angular/core'; +import { Router } from '@angular/router'; +@Component({ selector: 'j-board', template: '
' }) +export class BoardComponent { + constructor(private _router: Router) {} + openIssuePage(issueId: string) { + this._router.navigate(['project', 'issue', issueId]); + } +} +`, + 'src/app/project/issue.component.ts': `import { Component } from '@angular/core'; +import { Router } from '@angular/router'; +@Component({ selector: 'j-issue', template: '
' }) +export class IssueComponent { + constructor(private _router: Router) {} + backHome() { + this._router.navigate(['/']); + } +} +`, + }); + try { + expect(routeNames(cg)).toEqual(['/project/board', '/project/issue/:issueId']); + expect(navs(cg)).toEqual(['backHome -> /project/board', 'openIssuePage -> /project/issue/:issueId']); + } finally { + cg.close(); + } + }); +}); diff --git a/docs/design/framework-coverage.md b/docs/design/framework-coverage.md index fcb12619ce..6e123c2f51 100644 --- a/docs/design/framework-coverage.md +++ b/docs/design/framework-coverage.md @@ -42,7 +42,7 @@ guessed. | TanStack Router | `frameworks/tanstack-router.ts` | `tanstack-router-synthesizer.ts` | `tanstack-router.test.ts` | TanStack examples, fastapi-template frontend | | Vue Router / Nuxt | `frameworks/vue-router.ts` (Nuxt file routes: `nuxtResolver` in `frameworks/vue.ts`) | `vue-router-synthesizer.ts` | `vue-router.test.ts` | vue-realworld (23 edges); vue-element-admin (62 routes), vue-admin-template (14), vben (192), halo console (34) — named tables, module files, `children` + layouts; Nuxt: mealie, elk, nuxt/movies | | SvelteKit | `frameworks/sveltekit-router.ts` | `sveltekit-synthesizer.ts` | `sveltekit-router.test.ts`, `sveltekit-route-names.test.ts` | sveltekit-realworld (31 edges); shadcn-svelte and skeleton (`(group)` layouts: 13 and 23 edges), svelte.dev (74), kit's test apps (47) | -| Angular | `frameworks/angular-router.ts` | `angular-template-synthesizer.ts` | `angular-router.test.ts` | angular-realworld (31 edges, 18 renders), Ghostfolio (189 edges, 170 renders), ngx-admin (routes and renders; its menus are config) | +| Angular | `frameworks/angular-router.ts` | `angular-template-synthesizer.ts` | `angular-router.test.ts` | angular-realworld (31 edges, 18 renders), Ghostfolio (189 edges, 170 renders), ngx-admin (routes and renders; its menus are config), angular-spotify (Nx libs behind barrels: 14 routes), jira-clone (class-constant paths, mount-only redirects), jhipster (60), ionic-conference (18), Angular-JumpStart (18) | Shared machinery all seven use, in `frameworks/expo-router.ts`: `RouteTable` / `RootedRouteTable`, `routesForFile`, `addRouteTo`, `matchRoute`, `appRootFor`, diff --git a/src/resolution/frameworks/angular-router.ts b/src/resolution/frameworks/angular-router.ts index 02f7dc9d81..0980c4a5c5 100644 --- a/src/resolution/frameworks/angular-router.ts +++ b/src/resolution/frameworks/angular-router.ts @@ -172,6 +172,16 @@ function pathSegments(text: string): string[] | null { return value.split('/').filter((s) => s.length > 0); } if (CONSTANT_PATH.test(trimmed)) return [`{${trimmed.replace(/\s+/g, '')}}`]; + // `` `issue/:${ProjectConst.IssueId}` ``: a template whose holes are constants. + if (trimmed.startsWith('`') && skipString(trimmed, 0) === trimmed.length - 1) { + let ok = true; + const body = trimmed.slice(1, -1).replace(/\$\{([^}]*)\}/g, (_all, hole: string) => { + if (!CONSTANT_PATH.test(hole.trim())) ok = false; + return `{${hole.replace(/\s+/g, '')}}`; + }); + if (ok && !body.includes('**')) return body.split('/').filter((seg) => seg.length > 0); + return null; + } // `internalRoutes.account.subRoutes.access.path + '/:id'`: each part a // string or a constant, the constant a whole segment of its own. if (trimmed.includes('+')) { @@ -225,8 +235,11 @@ function componentRef(field: 'component' | 'loadComponent', text: string): Angul function lazyImport(text: string): { spec: string; member: string | null } | null { const imp = /\bimport\s*\(\s*(['"`])([^'"`]+)\1\s*\)/.exec(text); if (!imp) return null; - const then = /\.then\s*\(\s*\(?\s*([A-Za-z_$][\w$]*)\s*\)?\s*=>\s*\(?\s*\1\s*\.\s*([A-Za-z_$][\w$]*)/.exec(text.slice(imp.index + imp[0].length)); - return { spec: imp[2]!, member: then ? then[2]! : null }; + const after = text.slice(imp.index + imp[0].length); + const then = /^\s*\.then\s*\(\s*\(?\s*([A-Za-z_$][\w$]*)\s*\)?\s*=>\s*\(?\s*\1\s*\.\s*([A-Za-z_$][\w$]*)/.exec(after); + // `async () => (await import('./x')).HomeModule` + const awaited = /^\s*\)\s*\.\s*([A-Za-z_$][\w$]*)/.exec(after); + return { spec: imp[2]!, member: then ? then[2]! : awaited ? awaited[1]! : null }; } function joinPath(segs: readonly string[]): string { @@ -395,9 +408,12 @@ function resolvedSegments(pathText: string, file: string, context: ResolutionCon .split('/') .filter((seg) => seg.length > 0) .flatMap((seg) => { - if (!seg.startsWith('{') || !seg.endsWith('}')) return [seg]; - const value = constantValue(seg.slice(1, -1), file, context); - return value === null ? [seg] : value.split('/').filter((v) => v.length > 0); + if (seg.startsWith('{') && seg.endsWith('}') && seg.indexOf('{', 1) < 0) { + const value = constantValue(seg.slice(1, -1), file, context); + return value === null ? [seg] : value.split('/').filter((v) => v.length > 0); + } + // `:{ProjectConst.IssueId}` — a constant inside a segment. + return [seg.replace(/\{([^{}]+)\}/g, (all, expr: string) => constantValue(expr, file, context) ?? all)]; }); } @@ -413,21 +429,48 @@ function routeComponent(encoded: string, fromFile: string, context: ResolutionCo // The cross-file pass: mounts and constant paths // ============================================================================= -/** The file a routes import names, and — for an NgModule — the routing modules it imports. */ +/** + * The file a routes import names, and — for an NgModule — the routing modules + * it imports. A barrel is looked through first: an Nx library is imported by + * its alias (`@angular-spotify/web/home/feature`), which names the library's + * `src/index.ts`, whose `export * from './lib/home.module'` names the module. + */ function routeFilesLoadedBy(spec: string, fromFile: string, context: ResolutionContext, routeFiles: ReadonlySet, mountFiles: ReadonlySet): string[] { const target = resolveImportPath(spec, fromFile, 'typescript', context); - if (!target) return []; - if (routeFiles.has(target) || mountFiles.has(target)) return [target]; - // `loadChildren: () => import('./layout/layout.module').then(m => m.LayoutModule)`: - // the module holds no routes; the routing module it imports does. - const content = context.readFile(target); - if (!content) return []; + const seen = new Set(); const out: string[] = []; - const imports = /\bimport\s+[^'"]*?from\s+(['"])([^'"]+)\1/g; - let m: RegExpExecArray | null; - while ((m = imports.exec(content)) !== null) { - const file = resolveImportPath(m[2]!, target, 'typescript', context); - if (file && (routeFiles.has(file) || mountFiles.has(file))) out.push(file); + // A barrel can re-export another barrel; a few hops settle it. + let barrels = target ? [target] : []; + for (let hop = 0; hop < 4 && barrels.length > 0; hop++) { + const next: string[] = []; + for (const file of barrels) { + if (seen.has(file)) continue; + seen.add(file); + if (routeFiles.has(file) || mountFiles.has(file)) { + out.push(file); + continue; + } + const content = context.readFile(file); + if (!content) continue; + // `loadChildren: () => import('./layout/layout.module').then(m => m.LayoutModule)`: + // the module holds no routes; the routing module it imports does. + if (/@NgModule\s*\(/.test(content)) { + const imports = /\bimport\s+[^'"]*?from\s+(['"])([^'"]+)\1/g; + let m: RegExpExecArray | null; + while ((m = imports.exec(content)) !== null) { + const imported = resolveImportPath(m[2]!, file, 'typescript', context); + if (imported && (routeFiles.has(imported) || mountFiles.has(imported)) && !out.includes(imported)) out.push(imported); + } + continue; + } + const reexports = /\bexport\s+(?:\*|\{[^}]*\})\s*(?:as\s+[\w$]+\s*)?from\s+(['"])([^'"]+)\1/g; + let m: RegExpExecArray | null; + while ((m = reexports.exec(content)) !== null) { + const reexported = resolveImportPath(m[2]!, file, 'typescript', context); + if (reexported && !seen.has(reexported)) next.push(reexported); + } + } + barrels = next; } return out; } @@ -458,29 +501,68 @@ export function constantText(expr: string, fromFile: string, context: Resolution const [root, ...chain] = expr.split('.').map((part) => part.trim()); let value: string | null = null; const file = root ? declaringFile(root, fromFile, context) : null; - const content = file ? context.readFile(file) : null; - if (content && chain.length > 0) { + const content = file && root ? declarationSource(root, file, context, 0) : null; + if (content && root && chain.length > 0) { const safe = stripCommentsForRegex(content, 'typescript'); - const decl = new RegExp(String.raw`\b(?:const|let)\s+${root}\s*(?::[^=]+)?=\s*\{`).exec(safe); - if (decl) { - let start = decl.index + decl[0].length - 1; + const readChain = (open: number, links: readonly string[]): string | null => { + let start = open; let end = matchBracket(safe, start); - for (let i = 0; i < chain.length && end > start; i++) { - const field = readFields(safe, start, end).get(chain[i]!); - if (!field) break; - if (i === chain.length - 1) { - value = field.text.trim(); - break; - } + for (let i = 0; i < links.length && end > start; i++) { + const field = readFields(safe, start, end).get(links[i]!); + if (!field) return null; + if (i === links.length - 1) return field.text.trim(); start = safe.indexOf('{', field.at); end = start < 0 ? -1 : matchBracket(safe, start); } + return null; + }; + const decl = new RegExp(String.raw`\b(?:const|let)\s+${root}\s*(?::[^=]+)?=\s*\{`).exec(safe); + const enumDecl = decl ? null : new RegExp(String.raw`\benum\s+${root}\s*\{`).exec(safe); + const classDecl = decl || enumDecl ? null : new RegExp(String.raw`\bclass\s+${root}\b[^{]*\{`).exec(safe); + if (decl) value = readChain(decl.index + decl[0].length - 1, chain); + else if (enumDecl && chain.length === 1) { + // `enum Paths { Board = 'board' }` + const open = enumDecl.index + enumDecl[0].length - 1; + const body = safe.slice(open, Math.max(open, matchBracket(safe, open))); + value = new RegExp(String.raw`\b${chain[0]}\s*=\s*(['"\`][^'"\`]*['"\`])`).exec(body)?.[1] ?? null; + } else if (classDecl) { + // `class ProjectConst { static readonly IssueId = 'issueId' }`, and + // `class RouterUtil { static Configuration = { Visualizer: 'visualizer' } }`. + const open = classDecl.index + classDecl[0].length - 1; + const close = matchBracket(safe, open); + const body = close > open ? safe.slice(open, close) : ''; + const field = new RegExp(String.raw`\bstatic\s+(?:readonly\s+)?${chain[0]}\s*(?::[^=;]+)?=\s*`).exec(body); + if (field) { + const at = open + field.index + field[0].length; + if (safe[at] === '{') value = chain.length > 1 ? readChain(at, chain.slice(1)) : null; + else if (chain.length === 1) value = /^(['"\`])[^'"\`]*\1/.exec(safe.slice(at))?.[0] ?? null; + } } } memo.set(key, value); return value; } +/** + * The source that declares `root`: `file` itself, or — when `file` is a + * barrel (an Nx library's `src/index.ts`) — the module it re-exports it from. + */ +function declarationSource(root: string, file: string, context: ResolutionContext, hops: number): string | null { + const content = context.readFile(file); + if (!content) return null; + if (new RegExp(String.raw`\b(?:const|let|class|enum)\s+${root}\b`).test(content)) return content; + if (hops >= 3) return null; + const reexports = /\bexport\s+(\*|\{[^}]*\})\s*from\s+(['"])([^'"]+)\2/g; + let m: RegExpExecArray | null; + while ((m = reexports.exec(content)) !== null) { + if (m[1] !== '*' && !new RegExp(String.raw`\b${root}\b`).test(m[1]!)) continue; + const target = resolveImportPath(m[3]!, file, 'typescript', context); + const found = target ? declarationSource(root, target, context, hops + 1) : null; + if (found) return found; + } + return null; +} + /** `create.path` as the chain its root was destructured from in `fromFile`, or null when the root is no local alias. */ function localAlias(expr: string, fromFile: string, context: ResolutionContext): string | null { const content = context.readFile(fromFile); @@ -522,6 +604,93 @@ function declaringFile(name: string, fromFile: string, context: ResolutionContex // Route table // ============================================================================= +interface AngularMounts { + /** Every file that lazy-loads routes, whether or not it declares a screen of its own. */ + mountFiles: ReadonlySet; + /** The path segments a file's routes sit under: its parent's prefix plus the mount's. */ + prefixOf(file: string): string[]; +} + +const mountMemo = new WeakMap(); + +/** + * Where every routes file is mounted — the `loadChildren` chain from the app + * down, settled from the top. Shared by the cross-file pass, which names the + * routes, and the route table, which reads the redirects in files that only + * mount others (jira-clone's `app.routes.ts` holds nothing but two mounts and + * `'' → 'project'`). + */ +function angularMounts(context: ResolutionContext, routes: readonly Node[]): AngularMounts { + const source = context.getNodesByKind('route'); + const cached = mountMemo.get(context); + if (cached && cached.source === source) return cached.mounts; + const routeFiles = new Set(routes.map((r) => r.filePath)); + const mountsByFile = new Map(); + for (const file of context.getAllFiles()) { + if (!/\.[cm]?ts$/.test(file) || !(context.fileContains?.(file, 'loadChildren') ?? context.readFile(file)?.includes('loadChildren'))) continue; + const content = context.readFile(file); + if (!content) continue; + const { mounts } = parseAngularRoutes(content); + if (mounts.length > 0) mountsByFile.set(file, mounts); + } + const mountFiles = new Set(mountsByFile.keys()); + const loadedBy = new Map(); + for (const [file, mounts] of mountsByFile) { + for (const mount of mounts) { + for (const target of routeFilesLoadedBy(mount.spec, file, context, routeFiles, mountFiles)) { + // A file mounted from two places keeps its first mount. + if (target !== file && !loadedBy.has(target)) loadedBy.set(target, { parent: file, prefix: mount.prefix }); + } + } + } + const memo = new Map(); + const prefixOf = (file: string, seen: Set = new Set()): string[] => { + const hit = memo.get(file); + if (hit !== undefined) return hit; + const mount = loadedBy.get(file); + let prefix: string[] = []; + if (mount && !seen.has(file)) { + seen.add(file); + prefix = [...prefixOf(mount.parent, seen), ...resolvedSegments(mount.prefix, mount.parent, context)]; + } + memo.set(file, prefix); + return prefix; + }; + const mounts: AngularMounts = { mountFiles, prefixOf: (file) => prefixOf(file) }; + mountMemo.set(context, { source, mounts }); + return mounts; +} + +const workspaceRoots = new WeakMap>(); + +/** + * The app a routes file belongs to. An Angular workspace — the directory an + * `angular.json` or an Nx `nx.json` sits in — is one app however its routes + * are split: angular-spotify declares its screens across a dozen `libs/*` + * packages, each with a `src/` of its own, and keyed by those every + * template's `routerLink` looked for its routes in the wrong table. Outside a + * workspace, the conventional app root. + */ +function angularAppRoot(filePath: string, context: ResolutionContext): string { + let memo = workspaceRoots.get(context); + if (!memo) workspaceRoots.set(context, (memo = new Map())); + const slash = filePath.lastIndexOf('/'); + const dir = slash < 0 ? '' : filePath.slice(0, slash); + let found = memo.get(dir); + if (found === undefined) { + found = null; + for (let d: string | null = dir; d !== null; d = d === '' ? null : d.includes('/') ? d.slice(0, d.lastIndexOf('/')) : '') { + const at = d === '' ? '' : `${d}/`; + if (context.fileExists(`${at}angular.json`) || context.fileExists(`${at}nx.json`)) { + found = at; + break; + } + } + memo.set(dir, found); + } + return found ?? appRootFor(filePath); +} + export type AngularRouteTable = RootedRouteTable; const tables = new WeakMap(); @@ -534,7 +703,7 @@ export function angularRouteTable(context: ResolutionContext): AngularRouteTable const byFile = new Map(); for (const node of all) { if (!isAngularRoute(node) || !node.name.startsWith('/')) continue; - const root = appRootFor(node.filePath); + const root = angularAppRoot(node.filePath, context); let t = byRoot.get(root); if (!t) byRoot.set(root, (t = { source: all, exact: new Map(), dynamic: [] })); addRouteTo(t, node.name, node); @@ -545,13 +714,21 @@ export function angularRouteTable(context: ResolutionContext): AngularRouteTable // in-file paths. A chain (`/` → `/pages` → `/pages/dashboard`) settles in // a few passes. const aliases: Array<{ table: RouteTable; from: string; to: string }> = []; + const mounts = byFile.size > 0 ? angularMounts(context, all.filter(isAngularRoute)) : null; + const redirectFiles = new Map(); for (const [file, sample] of byFile) { - const content = context.readFile(file); - if (!content || !content.includes('redirectTo')) continue; const own = resolvedSegments(inFilePath(sample), file, context); const full = sample.name.split('/').filter((seg) => seg.length > 0); - const prefix = full.slice(0, Math.max(0, full.length - own.length)); - const table = byRoot.get(appRootFor(file)); + redirectFiles.set(file, full.slice(0, Math.max(0, full.length - own.length))); + } + // A file that only mounts others can still redirect: its prefix is where it is mounted. + for (const file of mounts?.mountFiles ?? []) { + if (!redirectFiles.has(file)) redirectFiles.set(file, mounts!.prefixOf(file)); + } + for (const [file, prefix] of redirectFiles) { + const content = context.readFile(file); + if (!content || !content.includes('redirectTo')) continue; + const table = byRoot.get(angularAppRoot(file, context)) ?? (byRoot.size === 1 ? [...byRoot.values()][0]! : undefined); if (!table) continue; for (const r of parseAngularRoutes(content).redirects) { const from = joinPath([...prefix, ...resolvedSegments(r.from, file, context)]); @@ -601,7 +778,7 @@ export function angularRoutesFor(table: AngularRouteTable, filePath: string): Ro const NAV_CALL = /^(?:this\.)?_?[rR]outer\.(navigate|navigateByUrl|createUrlTree|parseUrl)$/; /** The destination a command array names: `['/article', slug]` is `/article/${…}`; null when it is relative or not a literal array. */ -export function commandsHref(text: string): HrefLiteral | null { +export function commandsHref(text: string, fromRoot = false): HrefLiteral | null { const arr = text.trim(); if (arr[0] !== '[') return null; const close = matchBracket(arr, 0); @@ -644,11 +821,13 @@ export function commandsHref(text: string): HrefLiteral | null { i++; } flush(); - // Only an absolute destination: `['/login']`. `['../', id]` and a bare - // `['edit']` are relative to wherever the component is. + // Only an absolute destination: `['/login']`. In a template, `['../', id]` + // and a bare `['edit']` are relative to wherever the component is; a + // `router.navigate(['project', 'issue', id])` with no `relativeTo` starts + // from the root. const firstElement = arr.slice(1, close).split(',')[0] ?? ''; const first = staticString(firstElement); - if (first === null || !first.startsWith('/')) return null; + if (first === null || (!first.startsWith('/') && !(fromRoot && !first.startsWith('.')))) return null; const pathText = '/' + parts.filter((p) => p.length > 0).join('/'); return namesSomewhere({ path: pathText, display: pathText.split(HOLE).join('${…}') }); } @@ -725,14 +904,14 @@ function arrowReturn(fn: string): string | null { * * `owner` is the class the expression is written in, for `this.` properties. */ -export function angularDestination(expr: string, file: string, owner: Node | null, context: ResolutionContext, depth = 0): HrefLiteral | null { - return namesSomewhere(destinationOf(expr, file, owner, context, depth)); +export function angularDestination(expr: string, file: string, owner: Node | null, context: ResolutionContext, depth = 0, fromRoot = false): HrefLiteral | null { + return namesSomewhere(destinationOf(expr, file, owner, context, depth, fromRoot)); } -function destinationOf(expr: string, file: string, owner: Node | null, context: ResolutionContext, depth: number): HrefLiteral | null { +function destinationOf(expr: string, file: string, owner: Node | null, context: ResolutionContext, depth: number, fromRoot: boolean): HrefLiteral | null { const text = expr.trim(); if (text.length === 0 || depth > 4) return null; - if (text[0] === '[') return commandsHref(text); + if (text[0] === '[') return commandsHref(text, fromRoot); const literal = staticString(text); if (literal !== null) return toHref(literal); // `'/editor/' + article.slug`: the literal head, a hole for the rest. @@ -743,7 +922,7 @@ function destinationOf(expr: string, file: string, owner: Node | null, context: // `this.routerLinkAdminControlUsers.concat(userId)`: one segment more. const concat = /^(.*?)\.concat\s*\((.*)\)$/s.exec(text); if (concat) { - const base = angularDestination(concat[1]!, file, owner, context, depth + 1); + const base = angularDestination(concat[1]!, file, owner, context, depth + 1, fromRoot); const added = concat[2]!.split(',').filter((a) => a.trim().length > 0).length; return base && added > 0 ? toHref(base.path.replace(/\/$/, '') + ('/' + HOLE).repeat(added)) : base; } @@ -753,7 +932,7 @@ function destinationOf(expr: string, file: string, owner: Node | null, context: if (call) { const fn = constantText(call[1]!.replace(/\s+/g, '').replace(/^this\./, ''), file, context); const returned = fn ? arrowReturn(fn) : null; - return returned ? commandsHref(returned) : null; + return returned ? commandsHref(returned, fromRoot) : null; } const chain = /^(?:this\s*\.\s*)?([A-Za-z_$][\w$]*(?:\s*\.\s*[A-Za-z_$][\w$]*)*)$/.exec(text); if (!chain) return parseHrefExpression(text); @@ -762,12 +941,12 @@ function destinationOf(expr: string, file: string, owner: Node | null, context: if (content && owner && (text.startsWith('this') || parts.length === 1)) { const init = propertyInitializer(owner, parts[0]!, content); if (init !== null) { - if (parts.length === 1) return angularDestination(init, file, owner, context, depth + 1); - return angularDestination(`${init}.${parts.slice(1).join('.')}`, file, owner, context, depth + 1); + if (parts.length === 1) return angularDestination(init, file, owner, context, depth + 1, fromRoot); + return angularDestination(`${init}.${parts.slice(1).join('.')}`, file, owner, context, depth + 1, fromRoot); } } const constant = constantText(parts.join('.'), file, context); - return constant ? angularDestination(constant, file, owner, context, depth + 1) : null; + return constant ? angularDestination(constant, file, owner, context, depth + 1, fromRoot) : null; } // ============================================================================= @@ -837,44 +1016,8 @@ export const angularRouterResolver: FrameworkResolver = { postExtract(context: ResolutionContext): Node[] { const routes = context.getNodesByKind('route').filter(isAngularRoute); if (routes.length === 0) return []; - const routeFiles = new Set(routes.map((r) => r.filePath)); - // Every file that lazy-loads routes, whether or not it declares a screen of its own. - const mountsByFile = new Map(); - for (const file of context.getAllFiles()) { - if (!/\.[cm]?ts$/.test(file) || !(context.fileContains?.(file, 'loadChildren') ?? context.readFile(file)?.includes('loadChildren'))) continue; - const content = context.readFile(file); - if (!content) continue; - const { mounts } = parseAngularRoutes(content); - if (mounts.length > 0) mountsByFile.set(file, mounts); - } - const mountFiles = new Set(mountsByFile.keys()); - // file → the prefix it is mounted under (the parent file's own prefix - // plus the mount's), settled from the top down. - const loadedBy = new Map(); - for (const [file, mounts] of mountsByFile) { - for (const mount of mounts) { - for (const target of routeFilesLoadedBy(mount.spec, file, context, routeFiles, mountFiles)) { - // A file mounted from two places keeps its first mount. - if (target !== file && !loadedBy.has(target)) loadedBy.set(target, { parent: file, prefix: mount.prefix }); - } - } - } + const { prefixOf } = angularMounts(context, routes); const resolved = (pathText: string, file: string): string[] => resolvedSegments(pathText, file, context); - - const memo = new Map(); - const prefixOf = (file: string, seen: Set = new Set()): string[] => { - const hit = memo.get(file); - if (hit !== undefined) return hit; - const mount = loadedBy.get(file); - let prefix: string[] = []; - if (mount && !seen.has(file)) { - seen.add(file); - prefix = [...prefixOf(mount.parent, seen), ...resolved(mount.prefix, mount.parent)]; - } - memo.set(file, prefix); - return prefix; - }; - const changed: Node[] = []; for (const route of routes) { const name = joinPath([...prefixOf(route.filePath), ...resolved(inFilePath(route), route.filePath)]); @@ -915,7 +1058,7 @@ export const angularRouterResolver: FrameworkResolver = { .getNodesInFile(ref.filePath) .filter((n) => n.kind === 'class' && n.startLine <= ref.line && n.endLine >= ref.line) .reduce((inner, n) => (!inner || n.startLine >= inner.startLine ? n : inner), null); - const href = angularDestination(arg, ref.filePath, owner, context); + const href = angularDestination(arg, ref.filePath, owner, context, 0, true); if (!href || !href.path.startsWith('/')) return null; const targets = destinationsForHref(href, routes); const target = targets[0]; From 67744468d2bdf473a70559e61e4cd20dcff9fd5f Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 07:28:28 +0000 Subject: [PATCH 019/259] fix(resolution): Expo module bindings, imported receivers, TS type-only imports (#2134) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Expo Modules: a JS call on a `requireNativeModule('N')` binding — typed (``) or not, local or imported (expo-camera's `import CameraManager from './ExpoCameraManager'`) — resolves to module N's declared function, Swift first and Kotlin beside it, else to the member on the binding's declared type. It went to the same-named static method making the call. - A JS/TS member call on an import binding never picks a method declared in the calling file, and ruling those out never manufactures a unique guess. - `import type { X }` was parsed as a default import named `type` (every type-only import bound `type`), and `{ a, type B }` bound `type B`. - A name the file binds from an out-of-repo package names nothing in the project, for every reference kind, and JS/TS fuzzy matching is exact-case: trpc's 402 `Record<…>` references went to a `record` property, element-plus's 121 `mount` imports from @vue/test-utils to a test helper. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 3 + README.md | 2 +- __tests__/expo-modules.test.ts | 77 ++++++++++++++++ __tests__/import-type-modifier.test.ts | 75 ++++++++++++++++ src/resolution/frameworks/expo-modules.ts | 105 ++++++++++++++++++++++ src/resolution/frameworks/index.ts | 6 +- src/resolution/import-resolver.ts | 10 ++- src/resolution/name-matcher.ts | 34 ++++++- 8 files changed, 303 insertions(+), 9 deletions(-) create mode 100644 __tests__/import-type-modifier.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 037f61f834..c0c959e44b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,9 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- Calls into an Expo module now reach the module's native functions even when the JavaScript names the module something else, like expo-camera's `CameraManager` for `requireNativeModule('ExpoCamera')`. Both the iOS and the Android implementations are linked. expo-camera's calls used to link back to the same-named method making the call, so a method looked like it called itself. +- A JavaScript or TypeScript call on something the file imports, like `fireEvent.click(…)` or `util.omit(…)`, no longer links to a same-named method declared in the calling file itself. +- TypeScript's `import type { X }` no longer reads as an import of something named `type`, and `import { a, type B }` binds `B`, not `type B`. So a type-only import now links to the module it names, where it used to link to whatever project symbol had that name. A type imported from a package, like `RsbuildConfig` from `@rsbuild/core`, no longer links to a project symbol, and JavaScript and TypeScript names no longer match a symbol whose name differs only in case (`Mock` is not `mock`). - Angular apps split across Nx libraries, like angular-spotify, now have their routes and navigation. A lazily loaded route written `async () => (await import('@app/home')).HomeModule` is followed through the library's `index.ts` to the module it re-exports, and a `routerLink` in one library now reaches a screen declared in another. Route paths built from a class constant (`RouterUtil.Configuration.Lyrics`, `` `issue/:${ProjectConst.IssueId}` ``) or an enum are now read. A redirect in a routes file that only lazy-loads others now counts, and `router.navigate(['project', 'issue', id])` without `relativeTo` is read from the root, as Angular does. Re-index Angular projects after upgrading. - React Router apps now show navigation written as React Router v5's `` and through a styled link, like react-boilerplate's `HeaderLink = styled(Link)` used as ``. Apps built this way had routes but no links between their screens. - SvelteKit routes are now named by the address a browser asks for. A `(group)` folder like `(app)` or `(marketing)` is no longer part of a route's path, and a parameter with a matcher, like `[id=integer]`, is just `:id`. So `goto('/blocks')` and `` now reach a page that lives in `src/routes/(app)/blocks/`, where before they reached nothing and the app's screens showed no navigation. Re-index SvelteKit projects after upgrading. diff --git a/README.md b/README.md index 23a046f54c..284027063a 100644 --- a/README.md +++ b/README.md @@ -363,7 +363,7 @@ Real iOS and React Native codebases live across multiple languages — a Swift c | **React Native legacy bridge** | JS `NativeModules.X.fn(...)` | ObjC `RCT_EXPORT_METHOD` / `RCT_REMAP_METHOD` · Java/Kotlin `@ReactMethod` | Parses macro/annotation declarations to build a JS-name → native-method map | | **React Native TurboModules** | JS `import M from './NativeM'; M.fn(...)` | Native impl matching the Codegen spec | Treats the `Native.ts` spec interface as ground truth | | **RN native → JS events** | JS `new NativeEventEmitter(...).addListener('e', cb)` | ObjC `[self sendEventWithName:@"e" body:...]` · Swift `sendEvent(withName: "e", ...)` · Java/Kotlin `.emit("e", ...)` | Synthesized cross-language event channel keyed by literal event name | -| **Expo Modules** | JS `requireNativeModule('X').fn(...)` | Swift / Kotlin `Module { Name("X"); AsyncFunction("fn") { ... } }` | Parses the Expo DSL literals; synthetic method nodes resolve via existing name-match | +| **Expo Modules** | JS `requireNativeModule('X').fn(...)`, directly or through a binding (`export default requireNativeModule('X')`) | Swift / Kotlin `Module { Name("X"); AsyncFunction("fn") { ... } }` | Parses the Expo DSL literals into method nodes; a call on a binding resolves to module `X`'s `fn` on both platforms, else to the method on the binding's declared type | | **Fabric view components** | JSX `` | TS Codegen spec + native impl class | Spec → `component` node; convention-based name+suffix lookup (`View`/`ComponentView`/`Manager`/`ViewManager`) bridges to native | | **Legacy Paper view managers** | JSX `` | ObjC `RCT_EXPORT_VIEW_PROPERTY` · Java/Kotlin `@ReactProp` | Same as Fabric — Paper-era declarations also produce `component` + `property` nodes | diff --git a/__tests__/expo-modules.test.ts b/__tests__/expo-modules.test.ts index eab16c1c33..4df758a2ea 100644 --- a/__tests__/expo-modules.test.ts +++ b/__tests__/expo-modules.test.ts @@ -205,3 +205,80 @@ class BatteryModule : Module() { expect(pair.c).toBeGreaterThanOrEqual(2); // swift->kotlin AND kotlin->swift }); }); + +describe('Expo Modules — a JS binding named for its role, not its module', () => { + let dir: string; + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'expo-modules-binding-')); + }); + afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('expo-camera: `CameraManager.fn()` on `export default requireNativeModule("ExpoCamera")` reaches the module’s functions', async () => { + const write = (rel: string, content: string) => { + fs.mkdirSync(path.dirname(path.join(dir, rel)), { recursive: true }); + fs.writeFileSync(path.join(dir, rel), content); + }; + write('package.json', '{"dependencies":{"expo-modules-core":"^1.0.0","expo":"*"}}'); + write('ios/CameraViewModule.swift', `import ExpoModulesCore +public final class CameraViewModule: Module { + public func definition() -> ModuleDefinition { + Name("ExpoCamera") + AsyncFunction("launchScanner") { (options: ScannerOptions) in } + AsyncFunction("dismissScanner") { } + } +} +`); + write('android/src/main/java/expo/modules/camera/CameraViewModule.kt', `package expo.modules.camera +class CameraViewModule : Module() { + override fun definition() = ModuleDefinition { + Name("ExpoCamera") + AsyncFunction("launchScanner") { options: ScannerOptions -> } + AsyncFunction("dismissScanner") { } + } +} +`); + write('src/Camera.types.ts', `import { NativeModule } from 'expo'; +export declare class CameraNativeModule extends NativeModule { + readonly isAvailableAsync: () => Promise; +} +`); + write('src/ExpoCameraManager.ts', `import { requireNativeModule } from 'expo'; +import type { CameraNativeModule } from './Camera.types'; + +export default requireNativeModule('ExpoCamera'); +`); + write('src/CameraView.tsx', `import CameraManager from './ExpoCameraManager'; + +export default class CameraView { + static async isAvailableAsync(): Promise { + return CameraManager.isAvailableAsync(); + } + static async launchScanner(options: object): Promise { + await CameraManager.launchScanner(options); + } +} +`); + const cg = await CodeGraph.init(dir, { index: true }); + try { + const view = cg.getNodesInFile('src/CameraView.tsx'); + const from = (name: string) => view.find((n) => n.name === name && n.kind === 'method')!; + const callees = (name: string) => + cg + .getOutgoingEdges(from(name).id) + .filter((e) => e.kind === 'calls') + .map((e) => cg.getNode(e.target)!) + .map((n) => `${n.language}:${n.qualifiedName.split('::').pop()}`) + .sort(); + // Both platforms' `launchScanner`, not CameraView's own. + expect(callees('launchScanner')).toEqual(['kotlin:ExpoCamera.launchScanner', 'swift:ExpoCamera.launchScanner']); + // Declared natively nowhere here: the type the binding is given, never the caller itself. + expect(callees('isAvailableAsync')).toEqual(['typescript:isAvailableAsync']); + const target = cg.getOutgoingEdges(from('isAvailableAsync').id).find((e) => e.kind === 'calls')!; + expect(cg.getNode(target.target)!.filePath).toBe('src/Camera.types.ts'); + } finally { + cg.close(); + } + }); +}); diff --git a/__tests__/import-type-modifier.test.ts b/__tests__/import-type-modifier.test.ts new file mode 100644 index 0000000000..a74b430703 --- /dev/null +++ b/__tests__/import-type-modifier.test.ts @@ -0,0 +1,75 @@ +/** + * TypeScript's `type` modifiers are not bindings. + * + * `import type { X } from './x'` used to read as a default import named + * `type` beside `X` — every type-only import line bound `type` — so zod's + * `type.innerType()` (a parameter named `type`) was taken for a call on an + * import. An inline `{ util, type objectUtil }` named its binding + * `type objectUtil`. + */ +import { describe, it, expect, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import { extractImportMappings } from '../src/resolution/import-resolver'; + +const bindings = (source: string) => + extractImportMappings('src/a.ts', source, 'typescript').map((m) => `${m.localName}<${m.exportedName}${m.isNamespace ? ' ns' : ''}`); + +describe('import mappings and TypeScript type modifiers', () => { + it('a type-only import binds its names, never `type`', () => { + expect(bindings(`import type { enumUtil } from './helpers/enumUtil.js';\n`)).toEqual(['enumUtil { + expect(bindings(`import { util, type objectUtil } from './helpers/util.js';\n`)).toEqual(['util { + expect(bindings(`import type from './type';\n`)).toEqual(['type { + const roots: string[] = []; + afterAll(() => { + for (const r of roots.splice(0)) fs.rmSync(r, { recursive: true, force: true }); + }); + + it('halo: `type RsbuildConfig` and `type Command` are not a local `rsbuildConfig` or `command`', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-import-type-')); + roots.push(root); + const files: Record = { + 'package.json': JSON.stringify({ name: 'console', dependencies: { '@rsbuild/core': '*', '@tiptap/core': '*' } }), + 'src/rsbuild.ts': `import { defineConfig, type RsbuildConfig } from '@rsbuild/core'; +export function rsbuildConfig(): RsbuildConfig { return defineConfig({}); } +`, + 'src/menu.ts': `export function command() { return 1; } +`, + 'src/gap.ts': `import type { Command } from '@tiptap/core'; +export function gapCursor(): Command { return () => true; } +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + const cg = await CodeGraph.init(root, { index: true }); + try { + const targets = (file: string) => + cg + .getOutgoingEdgesFrom(cg.getNodesInFile(file).map((n) => n.id), ['references', 'imports', 'type_of', 'returns']) + .map((e) => cg.getNode(e.target)!) + .filter((n) => n.kind !== 'file' && n.kind !== 'import') + .map((n) => n.name); + expect(targets('src/rsbuild.ts')).not.toContain('rsbuildConfig'); + expect(targets('src/gap.ts')).not.toContain('command'); + } finally { + cg.close(); + } + }); +}); diff --git a/src/resolution/frameworks/expo-modules.ts b/src/resolution/frameworks/expo-modules.ts index a9f60174ad..c70d1d5acd 100644 --- a/src/resolution/frameworks/expo-modules.ts +++ b/src/resolution/frameworks/expo-modules.ts @@ -42,7 +42,12 @@ import type { Node } from '../../types'; import { FrameworkExtractionResult, FrameworkResolver, + ResolutionContext, + ResolvedRef, + UnresolvedRef, } from '../types'; +import { resolveImportPath } from '../import-resolver'; +import { stripCommentsForRegex } from '../strip-comments'; /** * Match `Function("name")`, `AsyncFunction("name")`, or `Property("name")` @@ -196,3 +201,103 @@ export const expoModulesResolver: FrameworkResolver = { return null; }, }; + +// ============================================================================= +// The JS side: `ExpoCamera.scanFromURLAsync(…)` → the module's function +// ============================================================================= + +/** `requireNativeModule('ExpoCamera')`, typed or not, optional or not. */ +const REQUIRE_NATIVE = /\brequire(?:Optional)?NativeModule\s*(?:<\s*([A-Za-z_$][\w$]*)[^>]*>)?\s*\(\s*['"]([A-Za-z_$][\w$]*)['"]\s*\)/; + +interface NativeBinding { + /** The `Name("…")` the module declares. */ + module: string; + /** The declared JS type, from `requireNativeModule(…)`. */ + type: string | null; + /** The file the binding (and its type) is written in. */ + file: string; +} + +const bindingMemo = new WeakMap>(); + +/** + * What `name` is bound to in `file`: `const X = requireNativeModule('N')` + * here, or the import of a module whose default export (or a named export) + * is one — expo-camera's `import CameraManager from './ExpoCameraManager'`, + * whose whole body is `export default requireNativeModule('ExpoCamera')`. + */ +function nativeBinding(name: string, file: string, context: ResolutionContext): NativeBinding | null { + let memo = bindingMemo.get(context); + if (!memo) bindingMemo.set(context, (memo = new Map())); + const key = `${file}\0${name}`; + if (memo.has(key)) return memo.get(key)!; + let found: NativeBinding | null = null; + const own = context.readFile(file); + if (own && own.includes('NativeModule')) { + const local = new RegExp(String.raw`\b(?:const|let|var)\s+${name}\s*(?::[^=]+)?=\s*` + REQUIRE_NATIVE.source).exec(stripCommentsForRegex(own, 'typescript')); + if (local) found = { type: local[1] ?? null, module: local[2]!, file }; + } + if (!found) { + const language = /\.tsx$/.test(file) ? 'tsx' : /\.[cm]?ts$/.test(file) ? 'typescript' : /\.jsx$/.test(file) ? 'jsx' : 'javascript'; + const binding = context.getImportMappings(file, language).find((m) => m.localName === name); + const target = binding && binding.source.startsWith('.') ? resolveImportPath(binding.source, file, language, context) : null; + const content = target ? context.readFile(target) : null; + if (target && content && content.includes('NativeModule')) { + const safe = stripCommentsForRegex(content, 'typescript'); + const pattern = binding!.isDefault + ? new RegExp(String.raw`\bexport\s+default\s+` + REQUIRE_NATIVE.source) + : new RegExp(String.raw`\bexport\s+(?:const|let)\s+${binding!.exportedName}\s*(?::[^=]+)?=\s*` + REQUIRE_NATIVE.source); + const m = pattern.exec(safe); + if (m) found = { type: m[1] ?? null, module: m[2]!, file: target }; + } + } + memo.set(key, found); + return found; +} + +/** + * JS calls into an Expo module, bound by the module's NAME, not by the + * method name alone: expo-camera calls its module `CameraManager`, so + * `CameraManager.isAvailableAsync()` went to `CameraView`'s own static + * `isAvailableAsync` in the same file. The native functions the module + * declares are the targets — Swift first, Kotlin beside it — and when it + * declares none of that name, the method on the type the binding is given + * (`requireNativeModule`). + */ +export const expoModulesJsResolver: FrameworkResolver = { + name: 'expo-modules-js', + languages: ['typescript', 'tsx', 'javascript', 'jsx'], + + detect(context) { + return expoModulesResolver.detect(context); + }, + + resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + if (ref.referenceKind !== 'calls') return null; + const m = /^([A-Za-z_$][\w$]*)\.([A-Za-z_$][\w$]*)$/.exec(ref.referenceName); + if (!m) return null; + const binding = nativeBinding(m[1]!, ref.filePath, context); + if (!binding) return null; + const fn = m[2]!; + const declared = context + .getNodesByName(fn) + .filter((n) => n.id.startsWith('expo-module:') && n.qualifiedName.endsWith(`::${binding.module}.${fn}`)) + .sort((a, b) => (a.language === 'swift' ? 0 : 1) - (b.language === 'swift' ? 0 : 1)); + if (declared.length > 0) { + return { + original: ref, + targetNodeId: declared[0]!.id, + ...(declared.length > 1 ? { alsoTargets: declared.slice(1).map((n) => ({ targetNodeId: n.id })) } : {}), + confidence: 0.95, + resolvedBy: 'framework', + }; + } + if (binding.type) { + const typed = context + .getNodesByName(fn) + .find((n) => (n.kind === 'method' || n.kind === 'property' || n.kind === 'field') && n.qualifiedName.endsWith(`${binding.type}::${fn}`)); + if (typed) return { original: ref, targetNodeId: typed.id, confidence: 0.9, resolvedBy: 'framework' }; + } + return null; + }, +}; diff --git a/src/resolution/frameworks/index.ts b/src/resolution/frameworks/index.ts index 88221fdfa6..da76403c3f 100644 --- a/src/resolution/frameworks/index.ts +++ b/src/resolution/frameworks/index.ts @@ -31,7 +31,7 @@ import { aspnetResolver } from './csharp'; import { swiftUIResolver, uikitResolver, vaporResolver } from './swift'; import { swiftObjcBridgeResolver } from './swift-objc'; import { reactNativeBridgeResolver } from './react-native'; -import { expoModulesResolver } from './expo-modules'; +import { expoModulesResolver, expoModulesJsResolver } from './expo-modules'; import { expoRouterResolver } from './expo-router'; import { fabricViewResolver } from './fabric'; import { cicsResolver } from './cics'; @@ -90,6 +90,8 @@ const FRAMEWORK_RESOLVERS: FrameworkResolver[] = [ reactNativeBridgeResolver, // Expo Modules — Function/AsyncFunction/Property DSL on Swift/Kotlin expoModulesResolver, + // Expo Modules, JS side — `M.fn()` on a `requireNativeModule('N')` binding → module N's `fn` + expoModulesJsResolver, // Expo Router — `app/` screen files → route nodes; `router.push('/x')` → navigates edges expoRouterResolver, // React Native Fabric / Codegen view components — TS spec → component nodes @@ -177,6 +179,6 @@ export { aspnetResolver } from './csharp'; export { swiftUIResolver, uikitResolver, vaporResolver } from './swift'; export { swiftObjcBridgeResolver } from './swift-objc'; export { reactNativeBridgeResolver } from './react-native'; -export { expoModulesResolver } from './expo-modules'; +export { expoModulesResolver, expoModulesJsResolver } from './expo-modules'; export { expoRouterResolver } from './expo-router'; export { fabricViewResolver } from './fabric'; diff --git a/src/resolution/import-resolver.ts b/src/resolution/import-resolver.ts index 1d2e716301..01f3dc2449 100644 --- a/src/resolution/import-resolver.ts +++ b/src/resolution/import-resolver.ts @@ -933,8 +933,11 @@ export function extractImportMappings( function extractJSImports(content: string): ImportMapping[] { const mappings: ImportMapping[] = []; - // ES6 imports - const importRegex = /import\s+(?:(\w+)\s*,?\s*)?(?:\{([^}]+)\})?\s*(?:(\*)\s+as\s+(\w+))?\s*from\s*['"]([^'"]+)['"]/g; + // ES6 imports. `import type { X }` / `import type * as ns` is TypeScript's + // type-only form, not a default import named `type` — which every such + // line used to add, making `type.innerType()` a call on an import. + // (`import type from './x'` still binds `type`: backtracking gives it back.) + const importRegex = /import\s+(?:type\s+(?=[{*]|(?!from\b)\w))?(?:(\w+)\s*,?\s*)?(?:\{([^}]+)\})?\s*(?:(\*)\s+as\s+(\w+))?\s*from\s*['"]([^'"]+)['"]/g; let match; while ((match = importRegex.exec(content)) !== null) { @@ -953,7 +956,8 @@ function extractJSImports(content: string): ImportMapping[] { // Named imports if (namedImports) { - const names = namedImports.split(',').map((s) => s.trim()); + // `{ util, type objectUtil }`: an inline `type` modifier is not part of the name. + const names = namedImports.split(',').map((s) => s.trim().replace(/^type\s+(?=\w)/, '')); for (const name of names) { const aliasMatch = name.match(/(\w+)\s+as\s+(\w+)/); if (aliasMatch) { diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index ea09c6c01b..a35fcf7534 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -1509,6 +1509,12 @@ export function matchByExactName( const bareJs = isBareJsCall(ref, context); const bareGo = isBareGoCall(ref, context); const barePhp = isBarePhpCall(ref, context); + // A type, a value or an import the file binds from a package outside the + // repository names nothing in it, whatever kind of reference it is. + if (!bareJs && JS_FAMILY.has(ref.language) && ref.referenceKind !== 'calls' && + /^[A-Za-z_$][\w$]*$/.test(ref.referenceName) && isOutOfRepoBinding(ref.referenceName, ref, context)) { + return null; + } if (bareJs) { const storeAction = matchJsStoreBindingCall(ref, context); if (storeAction) return storeAction; @@ -3598,10 +3604,22 @@ export function matchMethodCall( // Filter to same-language candidates first const sameLanguageMethods = methods.filter(m => m.language === ref.language); - const targetMethods = sameLanguageMethods.length > 0 ? sameLanguageMethods : methods; + let targetMethods = sameLanguageMethods.length > 0 ? sameLanguageMethods : methods; + // A receiver the file imports is another module's value: never a method + // declared in the calling file. expo-camera's `CameraManager.isAvailableAsync()` + // (`import CameraManager from './ExpoCameraManager'`) went to `CameraView`'s + // own static `isAvailableAsync` — the method making the call. + // Ruling the caller's file out may reject a guess; it must never + // manufacture one — the one method left is then no likelier than before. + let narrowed = false; + if (JS_FAMILY.has(ref.language) && isImportBinding(objectOrClass!, ref, context)) { + const kept = targetMethods.filter((m) => m.filePath !== ref.filePath); + narrowed = kept.length !== targetMethods.length; + targetMethods = kept; + } // If only one same-language method with this name exists, use it - if (targetMethods.length === 1 && targetMethods[0]!.language === ref.language) { + if (targetMethods.length === 1 && !narrowed && targetMethods[0]!.language === ref.language) { return { original: ref, targetNodeId: targetMethods[0]!.id, @@ -3660,6 +3678,12 @@ function isImportedModuleReceiver(receiver: string, ref: UnresolvedRef, context: return binding.isNamespace || context.isOutOfRepoImport?.(binding.source, ref.filePath, ref.language) === true; } +/** Is the root of a member call's receiver (`CameraManager` in `CameraManager.x`) one of the file's imports? */ +function isImportBinding(receiver: string, ref: UnresolvedRef, context: ResolutionContext): boolean { + const root = receiver.split('.')[0]!; + return context.getImportMappings?.(ref.filePath, ref.language)?.some((m) => m.localName === root) === true; +} + /** Does the file bind `name` by importing it from outside the repository? */ function isOutOfRepoBinding(name: string, ref: UnresolvedRef, context: ResolutionContext): boolean { const binding = context.getImportMappings?.(ref.filePath, ref.language)?.find((m) => m.localName === name); @@ -4558,6 +4582,9 @@ export function matchFuzzy( const callableCandidates = candidates.filter((n) => callableKinds.has(n.kind) && !(typeRef && !canNameInTypePosition(n)) && !(rustBare && (n.name !== ref.referenceName || !isRustNameInScope(n, ref, context))) && !(ref.language === 'python' && n.name !== ref.referenceName) && + // JS and TS too: halo's `type RsbuildConfig` (imported from @rsbuild/core) + // is not its local `rsbuildConfig`, kit's vitest `Mock` not a `mock`. + !(JS_FAMILY.has(ref.language) && n.name !== ref.referenceName) && !(pythonShape && !fitsPythonCallShape(n, pythonShape, ref, context)) && !(javaBare && n.kind === 'method' && !isJavaMethodInScope(n, ref, context))) .filter((n) => (ref.referenceKind !== 'references' && ref.referenceKind !== 'function_ref') || @@ -4587,8 +4614,9 @@ export function matchFuzzy( isVisibleAcrossFiles(finalCandidates[0]!, ref, context) && isCrossFileReachable(finalCandidates[0]!, ref, context) && !(isBareJsCall(ref, context) && - (TYPE_MEMBER_KINDS.has(finalCandidates[0]!.kind) || isOutOfRepoBinding(ref.referenceName, ref, context) || + (TYPE_MEMBER_KINDS.has(finalCandidates[0]!.kind) || (finalCandidates[0]!.filePath !== ref.filePath && isLocallyBoundJsName(ref.referenceName, ref.filePath, context)))) && + !(JS_FAMILY.has(ref.language) && isOutOfRepoBinding(ref.referenceName, ref, context)) && !(finalCandidates[0]!.kind === 'method' && isBareGoCall(ref, context)) && // A bare PHP call is a function call (case-insensitive, so fuzzy may find // one) — never the class `View` for `view(…)`, never a method. From 72a9346b122be1f95a342c524fb6546105e57191 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 07:34:14 +0000 Subject: [PATCH 020/259] fix(expo-router): +api files are endpoints; the nearest router manifest decides (#2135) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - `app/blog/og-image/[post]+api.ts` was a screen named `/blog/og-image/[post]+api` (evanbacon.dev). An Expo API route is now one endpoint per exported HTTP method (`GET /blog/og-image/:post`), bound to its handler, the way Next.js `route.ts` files are. - The per-app framework gate (#2129) took ANY enclosing manifest's declaration: react-native-true-sheet's root declares expo-router for its example app, so its `docs/` Next.js app got 16 Expo screens (`/layout`, `/page`, `/api/search/route`, …). Walking up from a file, the first manifest that names any gated framework now decides. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + README.md | 2 +- __tests__/monorepo-app-frameworks.test.ts | 40 ++++++++++++++++ src/extraction/index.ts | 8 +++- src/resolution/frameworks/expo-router.ts | 56 +++++++++++++++++++++++ 5 files changed, 105 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c0c959e44b..25ff68733c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- Expo Router API routes (`app/hello+api.ts`) are now endpoints, one per exported method, like `GET /hello`, bound to the function that handles them. They used to show up as a screen named `/hello+api`. In a repository whose root declares Expo Router for an example app, a Next.js app in its own folder, like react-native-true-sheet's `docs/`, no longer gets Expo screens made from its files. - Calls into an Expo module now reach the module's native functions even when the JavaScript names the module something else, like expo-camera's `CameraManager` for `requireNativeModule('ExpoCamera')`. Both the iOS and the Android implementations are linked. expo-camera's calls used to link back to the same-named method making the call, so a method looked like it called itself. - A JavaScript or TypeScript call on something the file imports, like `fireEvent.click(…)` or `util.omit(…)`, no longer links to a same-named method declared in the calling file itself. - TypeScript's `import type { X }` no longer reads as an import of something named `type`, and `import { a, type B }` binds `B`, not `type B`. So a type-only import now links to the module it names, where it used to link to whatever project symbol had that name. A type imported from a package, like `RsbuildConfig` from `@rsbuild/core`, no longer links to a project symbol, and JavaScript and TypeScript names no longer match a symbol whose name differs only in case (`Mock` is not `mock`). diff --git a/README.md b/README.md index 284027063a..6e91a52f63 100644 --- a/README.md +++ b/README.md @@ -340,7 +340,7 @@ These frameworks additionally emit **`navigates`** edges: the function that send | Router | Routes from | Navigation from | |---|---|---| -| **Expo Router** | Every screen file under `app/` (`app/item/[id].tsx` → `/item/[id]`, groups stripped), bound to its default-export component | `router.push` / `replace` / `navigate`, template hrefs, `{ pathname }` objects, and a helper's returned href | +| **Expo Router** | Every screen file under `app/` (`app/item/[id].tsx` → `/item/[id]`, groups stripped), bound to its default-export component; `+api` files are endpoints (`GET /hello`) bound to their handlers | `router.push` / `replace` / `navigate`, template hrefs, `{ pathname }` objects, and a helper's returned href | | **Next.js** | App Router `app/**/page.tsx` and Pages Router pages (`(group)` stripped, `[slug]` → `:slug`); `app/api/**/route.ts` exports and `pages/api/*` are endpoints, not screens | `router.push` / `replace` / `prefetch`, `redirect()` / `permanentRedirect()` in a server action or page, `NextResponse.redirect(new URL(…))` in middleware, `` and internal `` | | **React Router** | `` (v5 and v6) and `createBrowserRouter([{ path, element }])` | `history.push` / `replace`, `useNavigate`'s `navigate`, a loader's `redirect`, `` / `` / `` / v5's `` / react-router-bootstrap's ``, and a `styled(Link)` wrapper | | **TanStack Router** | `createFileRoute('/posts/$postId')` (file-based) and `createRoute({ path, getParentRoute })` composed up its parent chain (code-based); `_pathless` segments, `(group)` folders, `__root` and `` layouts are not addresses | `navigate({ to })`, a thrown `redirect({ to })`, `` / `` — where `to` is the route PATTERN and the values ride beside it in `params` | diff --git a/__tests__/monorepo-app-frameworks.test.ts b/__tests__/monorepo-app-frameworks.test.ts index f7efbf8caa..c5a4664fc8 100644 --- a/__tests__/monorepo-app-frameworks.test.ts +++ b/__tests__/monorepo-app-frameworks.test.ts @@ -85,4 +85,44 @@ describe('file-based routers in a monorepo', () => { cg.close(); } }); + + it('the nearest manifest that names a router decides: a Next docs app under an Expo root', async () => { + const cg = await project({ + 'package.json': JSON.stringify({ name: 'true-sheet', devDependencies: { expo: '*', 'expo-router': '*', react: '*' } }), + 'docs/package.json': JSON.stringify({ name: 'docs', dependencies: { next: '*', react: '*' } }), + 'docs/app/layout.tsx': `export default function RootLayout({ children }: { children: unknown }) { return children; } +`, + 'docs/app/(docs)/[...slug]/page.tsx': `export default function DocPage() { return null; } +`, + 'example/app/_layout.tsx': `export default function Layout() { return null; } +`, + 'example/app/sheet.tsx': `export default function Sheet() { return null; } +`, + }); + try { + expect(routesIn(cg, 'docs/')).toEqual(['/:slug*']); + expect(routesIn(cg, 'example/')).toEqual(['/sheet']); + } finally { + cg.close(); + } + }); + + it('an Expo API route is an endpoint per exported method, not a screen', async () => { + const cg = await project({ + 'package.json': JSON.stringify({ name: 'site', dependencies: { expo: '*', 'expo-router': '*' } }), + 'app/index.tsx': `export default function Home() { return null; } +`, + 'app/blog/og-image/[post]+api.ts': `export async function GET(request: Request) { return new Response('png'); } +export const POST = async () => new Response('ok'); +`, + }); + try { + expect(routesIn(cg, 'app/')).toEqual(['/', 'GET /blog/og-image/:post', 'POST /blog/og-image/:post']); + const get = cg.getNodesByKind('route').find((n) => n.name.startsWith('GET '))!; + const handler = cg.getOutgoingEdges(get.id).map((e) => cg.getNode(e.target)!).find((n) => n.name === 'GET'); + expect(handler?.kind).toBe('function'); + } finally { + cg.close(); + } + }); }); diff --git a/src/extraction/index.ts b/src/extraction/index.ts index 706bc0bd12..bb5b9c0fab 100644 --- a/src/extraction/index.ts +++ b/src/extraction/index.ts @@ -1948,13 +1948,19 @@ export class ExtractionOrchestrator { const key = `${dir}|${name}`; const memo = this.appFrameworkMemo.get(key); if (memo !== undefined) return memo; + // The nearest manifest that names ANY gated framework decides: true-sheet's + // root package.json declares expo-router for its example app, and its + // `docs/` Next.js app — whose own package.json declares `next` — is not + // an Expo app for it. let applies = false; for (let d: string | null = dir; d !== null; d = d === '' ? null : d.includes('/') ? d.slice(0, d.lastIndexOf('/')) : '') { const declared = this.dependenciesDeclaredIn(d); - if (declared && deps.some((dep) => declared.has(dep))) { + if (!declared) continue; + if (deps.some((dep) => declared.has(dep))) { applies = true; break; } + if ([...this.gatedFrameworks].some(([other, otherDeps]) => other !== name && otherDeps.some((dep) => declared.has(dep)))) break; } this.appFrameworkMemo.set(key, applies); return applies; diff --git a/src/resolution/frameworks/expo-router.ts b/src/resolution/frameworks/expo-router.ts index 5d82b94e26..ef6f44d81c 100644 --- a/src/resolution/frameworks/expo-router.ts +++ b/src/resolution/frameworks/expo-router.ts @@ -70,6 +70,8 @@ export function routePathForFile(filePath: string): string | null { const segs = bare.split('/'); if (segs.includes('__tests__') || segs.includes('__mocks__')) return null; const base = segs[segs.length - 1]!; + // `hello+api.ts` is an endpoint (`apiRoutePathForFile`), not a screen. + if (base.endsWith('+api')) return null; // `_layout` (and any other `_`-prefixed file) is not navigable. `+not-found` // is a real screen; the other `+` files (`+html`, `+native-intent`) are not. if (base.startsWith('_')) return null; @@ -79,6 +81,26 @@ export function routePathForFile(filePath: string): string | null { return '/' + kept.join('/'); } +/** + * An API route's path — `app/blog/og-image/[post]+api.ts` is + * `/blog/og-image/:post`, written the way every server route is so a + * client's `fetch('/blog/og-image/…')` can find it. Null for any other file. + */ +export function apiRoutePathForFile(filePath: string): string | null { + const dir = APP_DIR.exec(filePath); + if (!dir) return null; + const rel = filePath.slice(dir.index + dir[0].length); + const ext = /\.(tsx|ts|jsx|js|mjs|cjs)$/.exec(rel); + if (!ext) return null; + const bare = rel.slice(0, ext.index); + if (!bare.endsWith('+api')) return null; + const segs = bare.slice(0, -'+api'.length).split('/').filter((seg) => seg.length > 0 && !(seg.startsWith('(') && seg.endsWith(')'))); + if (segs[segs.length - 1] === 'index') segs.pop(); + return '/' + segs.map((seg) => seg.replace(/^\[\.\.\.(.+)\]$/, ':$1*').replace(/^\[(.+)\]$/, ':$1')).join('/'); +} + +const API_METHODS = 'GET|POST|PUT|PATCH|DELETE|HEAD|OPTIONS'; + function languageForFile(filePath: string): Language { const ext = ROUTE_EXT.exec(filePath)?.[1]; switch (ext) { @@ -728,6 +750,40 @@ export const expoRouterResolver: FrameworkResolver = { }, extract(filePath: string, content: string) { + const apiPath = apiRoutePathForFile(filePath); + if (apiPath !== null) { + // `export async function GET(request) {…}` / `export const POST = …` — one endpoint per method. + const language = languageForFile(filePath); + const stripped = stripCommentsForRegex(content, 'typescript'); + const nodes: Node[] = []; + const references: UnresolvedRef[] = []; + const seen = new Set(); + const decl = new RegExp(`\\bexport\\s+(?:async\\s+)?function\\s+(${API_METHODS})\\b|\\bexport\\s+(?:const|let)\\s+(${API_METHODS})\\s*=`, 'g'); + let m: RegExpExecArray | null; + while ((m = decl.exec(stripped)) !== null) { + const method = (m[1] ?? m[2])!; + if (seen.has(method)) continue; + seen.add(method); + const line = stripped.slice(0, m.index).split('\n').length; + const node: Node = { + id: `route:${filePath}:${line}:${method}:${apiPath}`, + kind: 'route', + name: `${method} ${apiPath}`, + qualifiedName: `${filePath}::${method}:${apiPath}`, + filePath, + startLine: line, + endLine: line, + startColumn: 0, + endColumn: m[0].length, + language, + isExported: true, + updatedAt: Date.now(), + }; + nodes.push(node); + references.push({ fromNodeId: node.id, referenceName: method, referenceKind: 'references', line, column: 0, filePath, language, candidates: [method] }); + } + return { nodes, references }; + } const routePath = routePathForFile(filePath); if (routePath === null) return { nodes: [], references: [] }; const language = languageForFile(filePath); From a4f982023329371fa24b72d036edd5e597bfe97d Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 07:48:48 +0000 Subject: [PATCH 021/259] fix(resolution): a super call in an override is not a self-call (#2136) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extraction keeps `super.didMoveToWindow()` under the bare method name, so every strategy resolved it to the enclosing override itself. Across the bridge sweep that was 139 self-edges on Charts, 142 on commons-lang, 101 on BookStack, 92 on realm-swift, 82 on react-native-svg — lifecycle overrides and delegates drawn as recursion. `gateSuperSelfCall` declines a `calls` result that targets the calling node when the call site is written through `super` / `base` (C#) / `[super …]` (Objective-C) / `parent::` (PHP) / `super().` / `super(C, self).` (Python). Plain recursion keeps its edge. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/super-call-self-edge.test.ts | 96 ++++++++++++++++++++++++++ src/resolution/index.ts | 31 ++++++++- 3 files changed, 125 insertions(+), 3 deletions(-) create mode 100644 __tests__/super-call-self-edge.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 25ff68733c..dde38f8da8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- A `super` call in an overriding method, like `super.viewDidLoad()`, `[super init]`, `base.Handle()`, `parent::setUp()` or `super().dispatch(…)`, no longer makes the method look like it calls itself. The call goes to the parent's version, so an override stops showing up as recursive and no longer lists itself among its own callers. - Expo Router API routes (`app/hello+api.ts`) are now endpoints, one per exported method, like `GET /hello`, bound to the function that handles them. They used to show up as a screen named `/hello+api`. In a repository whose root declares Expo Router for an example app, a Next.js app in its own folder, like react-native-true-sheet's `docs/`, no longer gets Expo screens made from its files. - Calls into an Expo module now reach the module's native functions even when the JavaScript names the module something else, like expo-camera's `CameraManager` for `requireNativeModule('ExpoCamera')`. Both the iOS and the Android implementations are linked. expo-camera's calls used to link back to the same-named method making the call, so a method looked like it called itself. - A JavaScript or TypeScript call on something the file imports, like `fireEvent.click(…)` or `util.omit(…)`, no longer links to a same-named method declared in the calling file itself. diff --git a/__tests__/super-call-self-edge.test.ts b/__tests__/super-call-self-edge.test.ts new file mode 100644 index 0000000000..6067a54919 --- /dev/null +++ b/__tests__/super-call-self-edge.test.ts @@ -0,0 +1,96 @@ +/** + * A `super` call inside an override calls the parent's implementation, never + * the method making the call. + * + * Extraction keeps `super.didMoveToWindow()` under the bare method name, so + * every strategy resolved it to the enclosing override itself — a self-edge + * that made a view's lifecycle methods look recursive (eleven of one + * expo-camera view's overrides). Real recursion keeps its edge. + */ +import { describe, it, expect, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +const roots: string[] = []; +afterAll(() => { + for (const r of roots.splice(0)) fs.rmSync(r, { recursive: true, force: true }); +}); + +async function project(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-super-self-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); +} + +/** `Owner.method` names of every method whose calls include itself. */ +function selfCalling(cg: CodeGraph): string[] { + const out: string[] = []; + for (const kind of ['method', 'function'] as const) { + for (const n of cg.getNodesByKind(kind)) { + if (cg.getOutgoingEdges(n.id).some((e) => e.kind === 'calls' && e.target === n.id)) out.push(n.name); + } + } + return out.sort(); +} + +describe('a super call is not a self-call', () => { + it('Swift, Kotlin, Java, Python, TypeScript and C#', async () => { + const cg = await project({ + 'ios/CameraView.swift': `import UIKit +class CameraView: UIView { + override func didMoveToWindow() { + super.didMoveToWindow() + setup() + } + func setup() {} + func countdown(_ n: Int) { + if n > 0 { countdown(n - 1) } + } +} +`, + 'android/CameraView.kt': `package app +class CameraView(context: Context) : FrameLayout(context) { + override fun onAttachedToWindow() { + super.onAttachedToWindow() + } +} +`, + 'java/Service.java': `package app; +public class Service extends Base { + @Override + public void start() { + super.start(); + } +} +`, + 'py/views.py': `class ProfileView(BaseView): + def dispatch(self, request): + return super().dispatch(request) +`, + 'ts/list.ts': `export class List extends Base { + render(): string { + return super.render() + '!'; + } +} +`, + 'cs/Handler.cs': `public class Handler : BaseHandler { + public override void Handle() { + base.Handle(); + } +} +`, + }); + try { + // Only the real recursion is left. + expect(selfCalling(cg)).toEqual(['countdown']); + } finally { + cg.close(); + } + }); +}); diff --git a/src/resolution/index.ts b/src/resolution/index.ts index e5fa4f332d..1a9906d819 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -1000,9 +1000,10 @@ export class ReferenceResolver { this.context, ); const scoped = this.gateRustScope(candidate, ref); - const resolved = scoped?.resolvedBy === 'framework' - ? this.gateFrameworkLanguage(scoped, ref) - : this.gateLanguage(scoped, ref); + const resolved = this.gateSuperSelfCall( + scoped?.resolvedBy === 'framework' ? this.gateFrameworkLanguage(scoped, ref) : this.gateLanguage(scoped, ref), + ref, + ); if (!resolved || ref.referenceKind !== 'calls') return resolved; const target = this.nodeById(resolved.targetNodeId); @@ -2869,6 +2870,30 @@ export class ReferenceResolver { return result; } + /** + * `super.didMoveToWindow()` inside an override of `didMoveToWindow` calls + * the PARENT's implementation, never the method making the call. Extraction + * keeps a `super` call under the bare method name, so every strategy found + * the enclosing method itself: one expo-camera view had eleven of its + * overrides "calling" themselves. The parent's method is usually a + * framework's (UIKit, Android, React); a self-edge is never it. Real + * recursion keeps its edge — only a call written through `super` / `base` + * (C#) / `[super …]` (Objective-C) / `parent::` (PHP) / `super().` (Python) + * is declined. + */ + private gateSuperSelfCall(result: ResolvedRef | null, ref: UnresolvedRef): ResolvedRef | null { + if (!result || ref.referenceKind !== 'calls' || result.targetNodeId !== ref.fromNodeId) return result; + const name = ref.referenceName.slice(Math.max(ref.referenceName.lastIndexOf('.'), ref.referenceName.lastIndexOf(':')) + 1); + if (!/^[A-Za-z_$][\w$]*$/.test(name)) return result; + const line = (this.context.getFileLines?.(ref.filePath) ?? this.context.readFile(ref.filePath)?.split(/\r?\n/))?.[ref.line - 1]; + if (!line) return result; + const escaped = name.replace(/\$/g, '\\$'); + const viaSuper = new RegExp( + String.raw`(?:\b(?:super|base)\s*(?:\(\s*(?:[\w.]+\s*,\s*\w+)?\s*\))?\s*\??\.\s*|\[\s*super\s+|\bparent\s*::\s*)` + escaped + String.raw`\b`, + ); + return viaSuper.test(line) ? null : result; + } + /** * A bare Rust name reaches only what is in scope — every strategy's result, * a framework resolver's `Ok(x)` → `struct Ok` construction included (see From 6f798dce6b241c291d06c8cb1039b2f1b9a31c74 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 07:49:17 +0000 Subject: [PATCH 022/259] fix(react-native): Paper requireNativeComponent views reach native; JSX via default imports (#2137) - `requireNativeComponent('X')` (Paper) now emits the same JS `component` node a Codegen spec does, so the Fabric pass links it to the native view / manager classes; the extractor also runs on `.js` / `.jsx` spec modules (segmented-control's spec is Flow `.js`). react-native-maps' `AIRMap` gains its iOS implementation. - A JSX tag that names no node is the file's DEFAULT import of a module's one component: ``, and element-plus's test `` for `autocomplete.vue` (whose component node is `autocomplete`). A named import never takes a barrel's one component. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + README.md | 2 +- __tests__/fabric-view.test.ts | 52 ++++++++++++++++++++++++++ src/resolution/callback-synthesizer.ts | 25 ++++++++++++- src/resolution/frameworks/fabric.ts | 24 ++++++------ 5 files changed, 91 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index dde38f8da8..3336627c07 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- React Native views declared with `requireNativeComponent('X')`, the older Paper style that react-native-maps and segmented-control use, now link from JavaScript to their native iOS and Android implementations, as Codegen specs already did, including in plain `.js` files. A JSX tag written under a different name than its component, like `` for `import Autocomplete from './autocomplete.vue'`, now links to the component it renders. - A `super` call in an overriding method, like `super.viewDidLoad()`, `[super init]`, `base.Handle()`, `parent::setUp()` or `super().dispatch(…)`, no longer makes the method look like it calls itself. The call goes to the parent's version, so an override stops showing up as recursive and no longer lists itself among its own callers. - Expo Router API routes (`app/hello+api.ts`) are now endpoints, one per exported method, like `GET /hello`, bound to the function that handles them. They used to show up as a screen named `/hello+api`. In a repository whose root declares Expo Router for an example app, a Next.js app in its own folder, like react-native-true-sheet's `docs/`, no longer gets Expo screens made from its files. - Calls into an Expo module now reach the module's native functions even when the JavaScript names the module something else, like expo-camera's `CameraManager` for `requireNativeModule('ExpoCamera')`. Both the iOS and the Android implementations are linked. expo-camera's calls used to link back to the same-named method making the call, so a method looked like it called itself. diff --git a/README.md b/README.md index 6e91a52f63..cd6c7bb442 100644 --- a/README.md +++ b/README.md @@ -365,7 +365,7 @@ Real iOS and React Native codebases live across multiple languages — a Swift c | **RN native → JS events** | JS `new NativeEventEmitter(...).addListener('e', cb)` | ObjC `[self sendEventWithName:@"e" body:...]` · Swift `sendEvent(withName: "e", ...)` · Java/Kotlin `.emit("e", ...)` | Synthesized cross-language event channel keyed by literal event name | | **Expo Modules** | JS `requireNativeModule('X').fn(...)`, directly or through a binding (`export default requireNativeModule('X')`) | Swift / Kotlin `Module { Name("X"); AsyncFunction("fn") { ... } }` | Parses the Expo DSL literals into method nodes; a call on a binding resolves to module `X`'s `fn` on both platforms, else to the method on the binding's declared type | | **Fabric view components** | JSX `` | TS Codegen spec + native impl class | Spec → `component` node; convention-based name+suffix lookup (`View`/`ComponentView`/`Manager`/`ViewManager`) bridges to native | -| **Legacy Paper view managers** | JSX `` | ObjC `RCT_EXPORT_VIEW_PROPERTY` · Java/Kotlin `@ReactProp` | Same as Fabric — Paper-era declarations also produce `component` + `property` nodes | +| **Legacy Paper view managers** | JSX ``, through a `requireNativeComponent('X')` module | ObjC `RCT_EXPORT_VIEW_PROPERTY` · Java/Kotlin `@ReactProp` | Same as Fabric — `requireNativeComponent('X')` is a JS `component` node, and Paper-era declarations also produce `component` + `property` nodes | **Validated on real codebases** (small + medium + large for each bridge): diff --git a/__tests__/fabric-view.test.ts b/__tests__/fabric-view.test.ts index 84e6529404..857167d265 100644 --- a/__tests__/fabric-view.test.ts +++ b/__tests__/fabric-view.test.ts @@ -142,3 +142,55 @@ export function App() { // The full flow: App (TSX) → MyView (fabric-component) → MyViewView (ObjC native class) }); }); + +describe('Paper end-to-end: a requireNativeComponent module rendered under another name', () => { + let dir: string; + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'paper-fixture-')); + }); + afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('segmented-control: → requireNativeComponent("RNCSegmentedControl") → the native view', async () => { + const write = (rel: string, content: string) => { + fs.mkdirSync(path.dirname(path.join(dir, rel)), { recursive: true }); + fs.writeFileSync(path.join(dir, rel), content); + }; + write('package.json', '{"dependencies":{"react-native":"^0.73"}}'); + write('js/RNCSegmentedControlNativeComponent.js', `import { requireNativeComponent } from 'react-native'; +module.exports = requireNativeComponent('RNCSegmentedControl'); +`); + write('js/SegmentedControl.js', `import * as React from 'react'; +import RNCSegmentedControlNativeComponent from './RNCSegmentedControlNativeComponent'; +export default function SegmentedControl(props) { + return ; +} +`); + write('ios/RNCSegmentedControl.m', `@implementation RNCSegmentedControl +- (void)setValues:(NSArray *)values { } +@end +`); + write('ios/RNCSegmentedControlManager.m', `@implementation RNCSegmentedControlManager +RCT_EXPORT_MODULE() +RCT_EXPORT_VIEW_PROPERTY(values, NSArray) +@end +`); + const cg = await CodeGraph.init(dir, { index: true }); + try { + const control = cg.getNodesByName('SegmentedControl').find((n) => n.filePath === 'js/SegmentedControl.js')!; + const rendered = cg.getOutgoingEdges(control.id).filter((e) => e.kind === 'calls').map((e) => cg.getNode(e.target)!); + const component = rendered.find((n) => n.kind === 'component' && n.name === 'RNCSegmentedControl'); + expect(component?.filePath).toBe('js/RNCSegmentedControlNativeComponent.js'); + const native = cg + .getOutgoingEdges(component!.id) + .filter((e) => (e.metadata as Record | undefined)?.synthesizedBy === 'fabric-native-impl') + .map((e) => cg.getNode(e.target)!) + .filter((n) => n.language === 'objc') + .map((n) => n.name); + expect(native).toContain('RNCSegmentedControl'); + } finally { + cg.close(); + } + }); +}); diff --git a/src/resolution/callback-synthesizer.ts b/src/resolution/callback-synthesizer.ts index fbd34b3303..806978e842 100644 --- a/src/resolution/callback-synthesizer.ts +++ b/src/resolution/callback-synthesizer.ts @@ -1222,6 +1222,15 @@ const JSX_CHILD_KINDS = new Set(['component', 'function', 'class']); */ const JSX_CHILD_LANGUAGES = [...JS_FAMILY, 'vue', 'svelte']; +function languageForJsxFile(file: string): Language { + if (file.endsWith('.tsx')) return 'tsx'; + if (/\.[cm]?ts$/.test(file)) return 'typescript'; + if (file.endsWith('.jsx')) return 'jsx'; + if (file.endsWith('.vue')) return 'vue'; + if (file.endsWith('.svelte')) return 'svelte'; + return 'javascript'; +} + /** `localName` → the project file it is imported from, for one file's imports. */ function importedFrom(ctx: ResolutionContext, file: string, language: Language): Map { const out = new Map(); @@ -1259,7 +1268,21 @@ function jsxChild( importsOf: () => Map ): Node | undefined { const candidates = ctx.getNodesByName(name).filter((n) => JSX_CHILD_KINDS.has(n.kind)); - if (candidates.length <= 1) return candidates[0]; + if (candidates.length === 0) { + // A name nothing declares is the file's DEFAULT import of a module's one + // component under another name: segmented-control renders + // ``, the default export of a module + // that is `requireNativeComponent('RNCSegmentedControl')`; element-plus's + // tests render `` from `autocomplete.vue`. A named import + // names an export of its own, which a barrel's one component is not. + const isDefault = ctx + .getImportMappings(file, languageForJsxFile(file)) + .some((m) => m.localName === name && m.isDefault); + const from = isDefault ? importsOf().get(name) : undefined; + const components = from ? ctx.getNodesInFile(from).filter((n) => n.kind === 'component') : []; + return components.length === 1 ? components[0] : undefined; + } + if (candidates.length === 1) return candidates[0]; const local = candidates.find((n) => n.filePath === file); if (local) return local; const from = importsOf().get(name); diff --git a/src/resolution/frameworks/fabric.ts b/src/resolution/frameworks/fabric.ts index 308feca686..e5c15e835c 100644 --- a/src/resolution/frameworks/fabric.ts +++ b/src/resolution/frameworks/fabric.ts @@ -50,7 +50,7 @@ import { } from '../types'; const CODEGEN_DECL_RE = - /codegenNativeComponent\s*(?:<[^>]+>)?\s*\(\s*['"]([A-Za-z_][A-Za-z0-9_]*)['"]/g; + /\b(codegenNativeComponent|requireNativeComponent)\s*(?:<[^>]+>)?\s*\(\s*['"]([A-Za-z_][A-Za-z0-9_]*)['"]/g; /** * Legacy Paper view manager macros — older RN libs (still very common, @@ -95,7 +95,9 @@ function deriveComponentNameFromManager(className: string): string { * spec signal. */ function isFabricSpec(source: string): boolean { - return source.includes('codegenNativeComponent'); + // A Paper-era module names its native view the same way: + // segmented-control's `module.exports = requireNativeComponent('RNCSegmentedControl')`. + return source.includes('codegenNativeComponent') || source.includes('requireNativeComponent'); } /** @@ -299,7 +301,8 @@ function extractFabricNodes(filePath: string, source: string): Node[] { CODEGEN_DECL_RE.lastIndex = 0; let m: RegExpExecArray | null; while ((m = CODEGEN_DECL_RE.exec(source)) !== null) { - const componentName = m[1]!; + const declaredWith = m[1]!; + const componentName = m[2]!; const before = source.slice(0, m.index); const startLine = before.split('\n').length; const startColumn = before.length - before.lastIndexOf('\n') - 1; @@ -314,15 +317,14 @@ function extractFabricNodes(filePath: string, source: string): Node[] { name: componentName, qualifiedName: `${filePath}::${componentName}`, filePath, - // The spec file is .ts or .tsx; use the file's apparent language - // by extension. Trim to a known Language value. - language: filePath.endsWith('.tsx') ? 'tsx' : 'typescript', + // The spec file's language by extension (a Paper module is often Flow `.js`). + language: filePath.endsWith('.tsx') ? 'tsx' : /\.[cm]?ts$/.test(filePath) ? 'typescript' : filePath.endsWith('.jsx') ? 'jsx' : 'javascript', startLine, endLine: startLine, startColumn, - endColumn: startColumn + 'codegenNativeComponent'.length, - docstring: `Fabric/Codegen native component '${componentName}'`, - signature: `codegenNativeComponent('${componentName}')`, + endColumn: startColumn + declaredWith.length, + docstring: declaredWith === 'codegenNativeComponent' ? `Fabric/Codegen native component '${componentName}'` : `Native component '${componentName}' (requireNativeComponent)`, + signature: declaredWith === 'codegenNativeComponent' ? `codegenNativeComponent('${componentName}')` : `requireNativeComponent('${componentName}')`, isExported: true, updatedAt: now, }); @@ -363,7 +365,7 @@ function extractFabricNodes(filePath: string, source: string): Node[] { export const fabricViewResolver: FrameworkResolver = { name: 'fabric-view', - languages: ['typescript', 'tsx', 'objc', 'java', 'kotlin'], + languages: ['typescript', 'tsx', 'javascript', 'jsx', 'objc', 'java', 'kotlin'], detect(context) { // Root package.json is the common case. The indexer only tracks @@ -392,7 +394,7 @@ export const fabricViewResolver: FrameworkResolver = { // Pick the right extractor by file language. The framework registry // already filters by `languages` so we only see relevant files. let nodes: Node[] = []; - if (filePath.endsWith('.ts') || filePath.endsWith('.tsx')) { + if (/\.(?:[cm]?[jt]s|[jt]sx)$/.test(filePath)) { nodes = extractFabricNodes(filePath, source); } else if (filePath.endsWith('.m') || filePath.endsWith('.mm')) { nodes = extractLegacyViewManagerNodes(filePath, source); From 6a02dcaa76da7bc2b05132302a2d33f9d7c2fc05 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 07:57:43 +0000 Subject: [PATCH 023/259] =?UTF-8?q?fix(react-native):=20event=20names=20he?= =?UTF-8?q?ld=20in=20constants=20join=20the=20native=E2=86=94JS=20channel?= =?UTF-8?q?=20(#2138)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The rn-event-channel pairs a native emit with a JS listener on the event's LITERAL name. Most libraries name events through constants — NetInfo listens with `addListener(PrivateTypes.DEVICE_CONNECTIVITY_EVENT, …)`, many emit with `.emit(EVENT_NAME, …)` or `sendEventWithName:kEvent` — so their channels were empty. Identifier / member-chain event names are now read to the literal their declaration holds (JS/TS const & enum, Java `static final`, Kotlin `const val`, ObjC `NSString *const` / `#define`, Swift `let` / enum case), scoped as the language scopes them: a bare name in the same file or behind an import, `Owner.NAME` inside Owner or a namespace import's file. The handler of a constant-named listener is a function in the same file, else the enclosing one. A first cut that read constants by name alone turned a `.emit(eventName, …)` parameter into some test's `eventName = "pong"`. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/rn-event-channel.test.ts | 83 +++++++++++++++++++ src/resolution/callback-synthesizer.ts | 105 +++++++++++++++++++++++++ 3 files changed, 189 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3336627c07..ba364048bd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- React Native events whose name is held in a constant now link native code to the JavaScript that listens for them, like NetInfo's `addListener(PrivateTypes.DEVICE_CONNECTIVITY_EVENT, …)` or a Java `.emit(PROGRESS_EVENT, …)`. The constant is read the way the language scopes it, from the same file, an import or its owning class, so a parameter that happens to share a constant's name is never mistaken for it. - React Native views declared with `requireNativeComponent('X')`, the older Paper style that react-native-maps and segmented-control use, now link from JavaScript to their native iOS and Android implementations, as Codegen specs already did, including in plain `.js` files. A JSX tag written under a different name than its component, like `` for `import Autocomplete from './autocomplete.vue'`, now links to the component it renders. - A `super` call in an overriding method, like `super.viewDidLoad()`, `[super init]`, `base.Handle()`, `parent::setUp()` or `super().dispatch(…)`, no longer makes the method look like it calls itself. The call goes to the parent's version, so an override stops showing up as recursive and no longer lists itself among its own callers. - Expo Router API routes (`app/hello+api.ts`) are now endpoints, one per exported method, like `GET /hello`, bound to the function that handles them. They used to show up as a screen named `/hello+api`. In a repository whose root declares Expo Router for an example app, a Next.js app in its own folder, like react-native-true-sheet's `docs/`, no longer gets Expo screens made from its files. diff --git a/__tests__/rn-event-channel.test.ts b/__tests__/rn-event-channel.test.ts index ccf6e751e3..a5b01e58ed 100644 --- a/__tests__/rn-event-channel.test.ts +++ b/__tests__/rn-event-channel.test.ts @@ -226,3 +226,86 @@ function tick() {} expect(rows[1].registered_at).toBe('App.tsx:4'); }); }); + +describe('RN event channel — event names held in constants', () => { + let dir: string; + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'rn-event-const-')); + }); + afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); + }); + + const write = (rel: string, content: string) => { + fs.mkdirSync(path.dirname(path.join(dir, rel)), { recursive: true }); + fs.writeFileSync(path.join(dir, rel), content); + }; + const channel = (cg: CodeGraph) => + ((cg as any).db.db + .prepare( + `SELECT s.name s, t.name t, json_extract(e.metadata,'$.event') event FROM edges e + JOIN nodes s ON s.id = e.source JOIN nodes t ON t.id = e.target + WHERE json_extract(e.metadata,'$.synthesizedBy') = 'rn-event-channel' ORDER BY s, t` + ) + .all() as Array<{ s: string; t: string; event: string }>).map((r) => `${r.s} -> ${r.t} (${r.event})`); + + it('NetInfo: a JS listener on `PrivateTypes.DEVICE_CONNECTIVITY_EVENT`, native sends the literal', async () => { + write('package.json', '{"name":"netinfo","dependencies":{"react-native":"^0.73"}}'); + write('ios/RNCNetInfo.m', `@implementation RNCNetInfo +- (void)connectionChanged { + [self sendEventWithName:@"netInfo.networkStatusDidChange" body:@{}]; +} +@end +`); + write('src/internal/privateTypes.ts', `export const DEVICE_CONNECTIVITY_EVENT = 'netInfo.networkStatusDidChange'; +`); + write('src/internal/state.ts', `import * as PrivateTypes from './privateTypes'; +export default class State { + start(emitter: any) { + emitter.addListener( + PrivateTypes.DEVICE_CONNECTIVITY_EVENT, + this._handleNativeStateUpdate, + ); + } + _handleNativeStateUpdate(state: unknown) { return state; } +} +`); + const cg = await CodeGraph.init(dir, { index: true }); + try { + expect(channel(cg)).toEqual(['connectionChanged -> _handleNativeStateUpdate (netInfo.networkStatusDidChange)']); + } finally { + cg.close(); + } + }); + + it('a Java constant emit and a Kotlin const val meet a JS literal listener', async () => { + write('package.json', '{"name":"lib","dependencies":{"react-native":"^0.73"}}'); + write('android/src/main/java/app/Tracker.java', `package app; +public class Tracker { + private static final String PROGRESS_EVENT = "downloadProgress"; + void report(ReactContext ctx) { + ctx.getJSModule(RCTDeviceEventEmitter.class).emit(PROGRESS_EVENT, null); + } +} +`); + write('android/src/main/java/app/Finisher.kt', `package app +const val DONE_EVENT = "downloadDone" +class Finisher { + fun finish(ctx: ReactContext) { + sendEvent(ctx, DONE_EVENT, null) + } +} +`); + write('src/index.js', `export function onProgress(e) { return e; } +export function onDone(e) { return e; } +emitter.addListener('downloadProgress', onProgress); +emitter.addListener('downloadDone', onDone); +`); + const cg = await CodeGraph.init(dir, { index: true }); + try { + expect(channel(cg)).toEqual(['finish -> onDone (downloadDone)', 'report -> onProgress (downloadProgress)']); + } finally { + cg.close(); + } + }); +}); diff --git a/src/resolution/callback-synthesizer.ts b/src/resolution/callback-synthesizer.ts index 806978e842..d86ad94e1a 100644 --- a/src/resolution/callback-synthesizer.ts +++ b/src/resolution/callback-synthesizer.ts @@ -1513,8 +1513,79 @@ const RN_JVM_EMIT_RE = /\.emit\s*\(\s*"([^"]+)"\s*,/g; // is followed by `… ) {`) never matches. Multi-line tolerant. (java/kotlin/swift) const RN_NATIVE_SENDEVENT_RE = /\bsendEvent\s*\([^;{}]*?"([^"]+)"/g; +// The same calls with the event named by a CONSTANT — NetInfo listens with +// `addListener(PrivateTypes.DEVICE_CONNECTIVITY_EVENT, …)`, many libraries emit +// with `.emit(EVENT_NAME, …)` or `sendEventWithName:kLocationEvent`. +const RN_OBJC_SEND_CONST_RE = /\bsendEventWithName\s*:\s*([A-Za-z_]\w*)\b/g; +const RN_SWIFT_SEND_CONST_RE = /\bsendEvent\s*\(\s*withName\s*:\s*([A-Za-z_][\w.]*)/g; +const RN_JVM_EMIT_CONST_RE = /\.emit\s*\(\s*([A-Za-z_][\w.]*)\s*,/g; +const RN_NATIVE_SENDEVENT_CONST_RE = /\bsendEvent\s*\(\s*[A-Za-z_][\w.]*\s*,\s*([A-Za-z_][\w.]*)\s*,/g; +const RN_JS_LISTEN_CONST_RE = /\.(?:on|once|addListener)\(\s*((?:[A-Za-z_$][\w$]*\.)*[A-Za-z_$][\w$]*)\s*,\s*([A-Za-z_$][\w$.]*|(?:async\s*)?(?:\([^)]*\)\s*=>|[A-Za-z_$][\w$]*\s*=>|function\s*\())/g; + +const CONSTANT_KINDS = new Set(['constant', 'variable', 'field', 'property', 'enum_member']); + +/** + * The string an event-name constant holds, read off its declaration line: + * `export const DEVICE_CONNECTIVITY_EVENT = 'netInfo.networkStatusDidChange'`, + * `static final String EVENT = "x"`, `const val EVENT = "x"`, + * `NSString *const kEvent = @"x"`, `#define kEvent @"x"`, `static let event = "x"`, + * an enum case `Changed = 'changed'`. + * + * Scoped as the language scopes the name, never by the name alone — a + * parameter `eventName` is not some test's `eventName = "pong"`: + * - a bare `NAME` is a declaration in the same file, or (JS) the one an + * import of that name points at; + * - `Owner.NAME` is a declaration inside `Owner`, or (JS) `NAME` in the file + * a namespace import `Owner` points at. + * Null unless those declarations agree on one literal. + */ +function constantEventName(expr: string, file: string, ctx: ResolutionContext, memo: Map): string | null { + const key = `${file}\0${expr}`; + if (memo.has(key)) return memo.get(key)!; + const parts = expr.replace(/\.rawValue$/, '').split('.'); + const name = parts[parts.length - 1]!; + const owner = parts.length > 1 ? parts[parts.length - 2]! : null; + let value: string | null = null; + if (/^[A-Za-z_$][\w$]*$/.test(name) && name !== 'this' && (parts.length <= 2 || parts[0] !== 'this')) { + const read = (n: Node): string | null => { + const line = (ctx.getFileLines?.(n.filePath) ?? ctx.readFile(n.filePath)?.split(/\r?\n/))?.[n.startLine - 1] ?? ''; + const at = line.indexOf(name); + if (at < 0) return null; + const rest = line.slice(at + name.length); + const m = /^\s*(?::[^=;]*)?[=:]\s*@?(["'`])([^"'`]+)\1/.exec(rest) ?? /^\s+@?(")([^"]+)"/.exec(rest); + return m ? m[2]! : null; + }; + const declaredIn = (f: string) => ctx.getNodesInFile(f).filter((n) => n.name === name && CONSTANT_KINDS.has(n.kind)); + const js = /\.(?:[cm]?[jt]sx?)$/.test(file); + const language: Language = /\.tsx$/.test(file) ? 'tsx' : /\.[cm]?ts$/.test(file) ? 'typescript' : file.endsWith('.jsx') ? 'jsx' : 'javascript'; + const importedFile = (local: string): string | null => { + if (!js) return null; + const binding = ctx.getImportMappings(file, language).find((m) => m.localName === local); + return binding ? resolveImportPath(binding.source, file, language, ctx) : null; + }; + let decls: Node[] = []; + if (owner === null) { + decls = declaredIn(file); + if (decls.length === 0) { + const from = importedFile(name); + if (from) decls = declaredIn(from); + } + } else { + const from = importedFile(owner); + decls = from + ? declaredIn(from) + : ctx.getNodesByName(name).filter((n) => CONSTANT_KINDS.has(n.kind) && new RegExp(`(?:^|[.:])${owner}(?:[.:]|$)`).test(n.qualifiedName.slice(0, n.qualifiedName.lastIndexOf(name)))); + } + const values = new Set(decls.map(read).filter((v): v is string => v !== null)); + value = values.size === 1 ? [...values][0]! : null; + } + memo.set(key, value); + return value; +} + async function rnEventEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise { let scannedFiles = 0; + const constants = new Map(); // Native dispatchers (source = the native method whose body sends the // event) and JS handlers (target = the function/method registered as // the listener) keyed by event name. @@ -1544,6 +1615,11 @@ async function rnEventEdges(ctx: ResolutionContext, onYield: MaybeYield): Promis while ((m = RN_OBJC_SEND_RE.exec(content))) { if (m[1]) addDispatcher(m[1], lineOf(m.index)); } + RN_OBJC_SEND_CONST_RE.lastIndex = 0; + while ((m = RN_OBJC_SEND_CONST_RE.exec(content))) { + const event = constantEventName(m[1]!, file, ctx, constants); + if (event) addDispatcher(event, lineOf(m.index)); + } } // Swift side: same RCTEventEmitter method, parens/named-args syntax. @@ -1557,6 +1633,11 @@ async function rnEventEdges(ctx: ResolutionContext, onYield: MaybeYield): Promis while ((m = RN_NATIVE_SENDEVENT_RE.exec(content))) { if (m[1]) addDispatcher(m[1], lineOf(m.index)); } + RN_SWIFT_SEND_CONST_RE.lastIndex = 0; + while ((m = RN_SWIFT_SEND_CONST_RE.exec(content))) { + const event = constantEventName(m[1]!, file, ctx, constants); + if (event) addDispatcher(event, lineOf(m.index)); + } } // JVM side: `.emit("X", …)` in Java/Kotlin, plus the common @@ -1573,6 +1654,13 @@ async function rnEventEdges(ctx: ResolutionContext, onYield: MaybeYield): Promis while ((m = RN_NATIVE_SENDEVENT_RE.exec(content))) { if (m[1]) addDispatcher(m[1], lineOf(m.index)); } + for (const re of [RN_JVM_EMIT_CONST_RE, RN_NATIVE_SENDEVENT_CONST_RE]) { + re.lastIndex = 0; + while ((m = re.exec(content))) { + const event = constantEventName(m[1]!, file, ctx, constants); + if (event) addDispatcher(event, lineOf(m.index)); + } + } } // JS subscribers (.addListener("X", handler)). Restrict to JS-family @@ -1658,6 +1746,23 @@ async function rnEventEdges(ctx: ResolutionContext, onYield: MaybeYield): Promis if (!map.has(enclosing.id)) map.set(enclosing.id, `${file}:${lineOf(m.index)}`); jsHandlersByEvent.set(event, map); } + // A constant event name: the listener lands where a literal one would — + // the named handler when it is a node, else the enclosing function. + RN_JS_LISTEN_CONST_RE.lastIndex = 0; + while ((m = RN_JS_LISTEN_CONST_RE.exec(content))) { + const event = constantEventName(m[1]!, file, ctx, constants); + if (!event) continue; + const arg = m[2]!; + const bare = /^[A-Za-z_$][\w$.]*$/.test(arg) ? arg.slice(arg.lastIndexOf('.') + 1) : null; + const line = lineOf(m.index); + // The handler this file names — never a same-named function elsewhere. + const named = bare ? nodesInFile.find((n) => (n.kind === 'function' || n.kind === 'method') && n.name === bare) : undefined; + const target = named ?? enclosingFn(nodesInFile, line); + if (!target) continue; + const map = jsHandlersByEvent.get(event) ?? new Map(); + if (!map.has(target.id)) map.set(target.id, `${file}:${line}`); + jsHandlersByEvent.set(event, map); + } } } From 66d5dc36418a8b8becf4a0a7572e527dedf4a5a5 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 08:10:26 +0000 Subject: [PATCH 024/259] fix(resolution): fuzzy matching is exact-case outside case-insensitive languages (#2139) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The last-resort fuzzy match looks names up in a lowercase index — right for PHP, Pascal/Delphi, CFML, COBOL and VB.NET, whose identifiers ignore case, and wrong for every other language. jsoup's 1,844 `@Test` annotations decorated a `CharPredicate.test` method; commons-lang's `new BitSet()` instantiated a `bitSet` field accessor 64 times; mall's `new Info()` an `info()` endpoint; gson's `Method` type a test's `method()`; fmt's `#include ` gtest's `Optional`. This generalizes the per-language exact-case rules (Rust, Python, JS) into one: outside CASE_INSENSITIVE_LANGUAGES a fuzzy candidate must match the reference's exact name. The cross-language fuzzy-gate test now builds its winner from an exact-case name. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/cross-language-resolution.test.ts | 7 ++- __tests__/fuzzy-case-sensitivity.test.ts | 69 +++++++++++++++++++++ src/resolution/name-matcher.ts | 18 +++--- 4 files changed, 86 insertions(+), 9 deletions(-) create mode 100644 __tests__/fuzzy-case-sensitivity.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index ba364048bd..f35a343785 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- In Java, Kotlin, Swift, C#, C and C++, Go, Scala, Ruby, Dart and the other case-sensitive languages, a name no longer links to a symbol whose name differs only in case. On jsoup, JUnit's `@Test` had linked every test to a `test` method; `new CookieManager()` had linked to a `cookieManager()` getter, and a `Method` type to a `method()`. PHP, Pascal, CFML, COBOL and VB.NET, whose names really do ignore case, are unchanged. - React Native events whose name is held in a constant now link native code to the JavaScript that listens for them, like NetInfo's `addListener(PrivateTypes.DEVICE_CONNECTIVITY_EVENT, …)` or a Java `.emit(PROGRESS_EVENT, …)`. The constant is read the way the language scopes it, from the same file, an import or its owning class, so a parameter that happens to share a constant's name is never mistaken for it. - React Native views declared with `requireNativeComponent('X')`, the older Paper style that react-native-maps and segmented-control use, now link from JavaScript to their native iOS and Android implementations, as Codegen specs already did, including in plain `.js` files. A JSX tag written under a different name than its component, like `` for `import Autocomplete from './autocomplete.vue'`, now links to the component it renders. - A `super` call in an overriding method, like `super.viewDidLoad()`, `[super init]`, `base.Handle()`, `parent::setUp()` or `super().dispatch(…)`, no longer makes the method look like it calls itself. The call goes to the parent's version, so an override stops showing up as recursive and no longer lists itself among its own callers. diff --git a/__tests__/cross-language-resolution.test.ts b/__tests__/cross-language-resolution.test.ts index bfd40dbbde..9fd5925a46 100644 --- a/__tests__/cross-language-resolution.test.ts +++ b/__tests__/cross-language-resolution.test.ts @@ -132,8 +132,11 @@ describe('cross-language name resolution (#1986)', () => { expect(matchByQualifiedName(ref(method.qualifiedName), context)?.targetNodeId).toBe(method.id); expect(matchReference(ref(method.qualifiedName), context)).toBeNull(); expect(matchReference(ref('Foreign.act'), context)).toBeNull(); - expect(matchFuzzy(ref('FUZZYNAME'), context)).not.toBeNull(); - expect(matchReference(ref('FUZZYNAME'), context)).toBeNull(); + // Fuzzy is case-sensitive outside PHP/Pascal/CFML/COBOL/VB.NET, so the + // fuzzy winner here shares the exact name; the language gate still rejects it. + expect(matchFuzzy(ref('fuzzyName'), context)).not.toBeNull(); + expect(matchReference(ref('fuzzyName'), context)).toBeNull(); + expect(matchFuzzy(ref('FUZZYNAME'), context)).toBeNull(); // Exact qualified match selects Python even though a Swift type has the // same bare name. Rejection must not fall through to that Swift type. const foreign = graph.getNodesByName('Winner').find((n) => n.language === 'python')!; diff --git a/__tests__/fuzzy-case-sensitivity.test.ts b/__tests__/fuzzy-case-sensitivity.test.ts new file mode 100644 index 0000000000..fcfd301d4c --- /dev/null +++ b/__tests__/fuzzy-case-sensitivity.test.ts @@ -0,0 +1,69 @@ +/** + * The last-resort fuzzy match compares names without regard to case. That + * is right for PHP, Pascal, CFML, COBOL and VB.NET, whose identifiers + * resolve that way, and wrong everywhere else: jsoup's 1,844 `@Test` + * annotations decorated a `CharPredicate.test` method, `new CookieManager()` + * instantiated a `cookieManager()` getter, gson's `Method` type referenced a + * test's `method()`. + */ +import { describe, it, expect, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +const roots: string[] = []; +afterAll(() => { + for (const r of roots.splice(0)) fs.rmSync(r, { recursive: true, force: true }); +}); + +async function project(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-fuzzy-case-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); +} + +const targetsFrom = (cg: CodeGraph, file: string): string[] => + cg + .getOutgoingEdgesFrom(cg.getNodesInFile(file).map((n) => n.id)) + .filter((e) => e.kind !== 'contains') + .map((e) => cg.getNode(e.target)!) + .map((n) => `${n.kind}:${n.name}`) + .sort(); + +describe('fuzzy matching and case', () => { + it('Java: an annotation or a type is not a method whose name differs only in case', async () => { + const cg = await project({ + 'src/main/java/app/CharPredicate.java': `package app; +public interface CharPredicate { + boolean test(char c); +} +`, + 'src/main/java/app/Request.java': `package app; +public class Request { + public Object cookieManager() { return null; } +} +`, + 'src/test/java/app/ParserTest.java': `package app; +import org.junit.jupiter.api.Test; +import java.net.CookieManager; +public class ParserTest { + @Test void parses() { + Object manager = new CookieManager(); + } +} +`, + }); + try { + const targets = targetsFrom(cg, 'src/test/java/app/ParserTest.java'); + expect(targets).not.toContain('method:test'); + expect(targets).not.toContain('method:cookieManager'); + } finally { + cg.close(); + } + }); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index a35fcf7534..aa1db3deb3 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -943,6 +943,9 @@ export function isVisibleAcrossFiles(candidate: Node, ref: UnresolvedRef, contex const JS_FAMILY = new Set(['typescript', 'tsx', 'javascript', 'jsx', 'vue', 'svelte', 'astro']); const JS_TS = new Set(['typescript', 'tsx', 'javascript', 'jsx']); +/** Languages whose identifiers resolve regardless of case. */ +const CASE_INSENSITIVE_LANGUAGES = new Set(['php', 'pascal', 'cfml', 'cfscript', 'cfquery', 'cobol', 'vbnet']); + /** * Whether a JS/TS `calls` ref is a RECEIVER-LESS call — `serialize(x)`, not * `this.serialize(x)` / `obj.serialize(x)`. The extractor emits `this.m()` @@ -4577,14 +4580,15 @@ export function matchFuzzy( const rustBare = ref.language === 'rust' && /^[A-Za-z_]\w*$/.test(ref.referenceName); const pythonShape = pythonCallShape(ref, context); const javaBare = ref.language === 'java' && ref.referenceKind === 'calls' && /^[A-Za-z_$][\w$]*$/.test(ref.referenceName); - // Rust and Python names are case-sensitive: `Bytes` is not the method `bytes`, - // Python's builtin `dir(…)` not a class `Dir`. + // Names are case-sensitive in every language but a handful: Rust's + // `Bytes` is not the method `bytes`, Python's builtin `dir(…)` not a class + // `Dir`, halo's `type RsbuildConfig` not its local `rsbuildConfig`, a Java + // `Node` not a `node()`. Only PHP, Pascal/Delphi, CFML, COBOL and VB.NET + // resolve a name without regard to case, which is what this fallback's + // lowercase index is for. const callableCandidates = candidates.filter((n) => callableKinds.has(n.kind) && !(typeRef && !canNameInTypePosition(n)) && - !(rustBare && (n.name !== ref.referenceName || !isRustNameInScope(n, ref, context))) && - !(ref.language === 'python' && n.name !== ref.referenceName) && - // JS and TS too: halo's `type RsbuildConfig` (imported from @rsbuild/core) - // is not its local `rsbuildConfig`, kit's vitest `Mock` not a `mock`. - !(JS_FAMILY.has(ref.language) && n.name !== ref.referenceName) && + !(rustBare && !isRustNameInScope(n, ref, context)) && + !(!CASE_INSENSITIVE_LANGUAGES.has(ref.language) && n.name !== ref.referenceName) && !(pythonShape && !fitsPythonCallShape(n, pythonShape, ref, context)) && !(javaBare && n.kind === 'method' && !isJavaMethodInScope(n, ref, context))) .filter((n) => (ref.referenceKind !== 'references' && ref.referenceKind !== 'function_ref') || From a3d675ef4d7c0d014ee928e598bcacb8a632470d Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 08:22:09 +0000 Subject: [PATCH 025/259] fix(vue): Options API methods, computed, watchers and hooks are symbols (#2140) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A ` +` + ); + const cg = await CodeGraph.init(root, { index: true }); + try { + const nodes = cg.getNodesInFile('src/views/login/index.vue'); + const method = (name: string) => nodes.find((n) => n.kind === 'method' && n.name === name)!; + expect(nodes.filter((n) => n.kind === 'method').map((n) => n.name).sort()).toEqual(['data', 'handleLogin', 'validate']); + expect(method('handleLogin').qualifiedName).toBe('index::handleLogin'); + // The call inside handleLogin is handleLogin's, not the file's. + const calls = cg.getOutgoingEdges(method('handleLogin').id).filter((e) => e.kind === 'calls').map((e) => cg.getNode(e.target)!.name); + expect(calls).toContain('validate'); + // `@click="handleLogin"` in the template runs the method. + const component = nodes.find((n) => n.kind === 'component')!; + const handlers = cg.getOutgoingEdges(component.id).filter((e) => e.target === method('handleLogin').id); + expect(handlers.length).toBeGreaterThan(0); + } finally { + cg.close(); + } + }); +}); diff --git a/src/extraction/vue-extractor.ts b/src/extraction/vue-extractor.ts index 862f17c793..5429a90dea 100644 --- a/src/extraction/vue-extractor.ts +++ b/src/extraction/vue-extractor.ts @@ -2,6 +2,7 @@ import { Node, Edge, ExtractionResult, ExtractionError, UnresolvedReference, Lan import { generateNodeId } from './tree-sitter-helpers'; import { TreeSitterExtractor } from './tree-sitter'; import { isLanguageSupported } from './grammars'; +import { vueOptionsMembers } from './vue-options-api'; /** * Vue built-in components — skipped so a `` / `` in the @@ -114,6 +115,67 @@ export class VueExtractor { return node; } + /** + * Method nodes for an Options API component's members (see + * ./vue-options-api), and the references and edges written inside each — + * which the TS extractor attributed to the file — re-attributed to it. + * Lines are block-relative here; the caller offsets them with the rest. + */ + private addOptionsMembers( + block: { content: string; startLine: number }, + result: ExtractionResult, + componentNodeId: string + ): void { + const members = vueOptionsMembers(block.content); + if (members.length === 0) return; + const component = this.nodes.find((n) => n.id === componentNodeId); + const owner = component?.name ?? 'component'; + const lineAt = (offset: number) => block.content.slice(0, offset).split('\n').length; + const colAt = (offset: number) => offset - block.content.lastIndexOf('\n', offset - 1) - 1; + const now = Date.now(); + const created: Node[] = []; + for (const m of members) { + const startLine = lineAt(m.start); + const endLine = lineAt(m.end); + created.push({ + id: generateNodeId(this.filePath, 'method', `${owner}.${m.name}`, startLine + block.startLine), + kind: 'method', + name: m.name, + qualifiedName: `${owner}::${m.name}`, + filePath: this.filePath, + language: 'vue', + startLine, + endLine, + startColumn: colAt(m.start), + endColumn: colAt(m.end), + updatedAt: now, + }); + } + // Innermost member for a line: `computed: { x: { get() {…} } }` is one member. + const memberAt = (line: number): Node | undefined => { + let best: Node | undefined; + for (const n of created) { + if (n.startLine <= line && n.endLine >= line && (!best || n.startLine >= best.startLine)) best = n; + } + return best; + }; + // What the TS extractor attributed to the file (or to nothing narrower). + const fileNode = result.nodes.find((n) => n.kind === 'file'); + const narrower = new Set(result.nodes.filter((n) => n.kind !== 'file').map((n) => n.id)); + const isFileLevel = (id: string) => (fileNode ? id === fileNode.id : !narrower.has(id)); + for (const ref of result.unresolvedReferences) { + if (!isFileLevel(ref.fromNodeId)) continue; + const member = memberAt(ref.line); + if (member) ref.fromNodeId = member.id; + } + for (const edge of result.edges) { + if (edge.kind === 'contains' || !edge.line || !isFileLevel(edge.source)) continue; + const member = memberAt(edge.line); + if (member) edge.source = member.id; + } + result.nodes.push(...created); + } + /** * Extract +`, + 'backend/query.py': `class QueryFilterBuilder: + def __init__(self, raw): + self.raw = raw +`, + 'backend/test_query.py': `from mealie.services.query_filter.builder import QueryFilterBuilder + + +def test_builder(): + return QueryFilterBuilder("x") +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +describe('the Vue resolver', () => { + it('leaves a Python call to its own class alone', () => { + const ids = cg.getNodesInFile('backend/test_query.py').map((n) => n.id); + const targets = cg.getOutgoingEdgesFrom(ids).filter((e) => e.kind !== 'contains').map((e) => cg.getNode(e.target)!.filePath); + expect(targets).not.toContain('frontend/components/QueryFilterBuilder.vue'); + }); +}); diff --git a/src/resolution/frameworks/vue.ts b/src/resolution/frameworks/vue.ts index 03e50509ac..e894e9b468 100644 --- a/src/resolution/frameworks/vue.ts +++ b/src/resolution/frameworks/vue.ts @@ -9,6 +9,9 @@ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from '../types'; import { dependsOn } from './package-deps'; +/** The languages a Vue app's scripts are written in. */ +const VUE_SCRIPT_LANGUAGES: ReadonlySet = new Set(['vue', 'javascript', 'typescript', 'tsx', 'jsx']); + /** * Vue 3 compiler macros — compiler-provided, not user code */ @@ -102,6 +105,11 @@ export const vueResolver: FrameworkResolver = { }, resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + // Vue's macros, auto-imports and components are a script's, never a + // backend's: mealie's Python `QueryFilterBuilder(...)` is not the + // `QueryFilterBuilder.vue` component. + if (!VUE_SCRIPT_LANGUAGES.has(ref.language)) return null; + // Pattern 1: Vue compiler macros (defineProps, defineEmits, etc.) if (VUE_COMPILER_MACROS.has(ref.referenceName)) { return { From 5c01f017cf9dcf8372684cfb42808786b2e516e2 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 22:10:14 +0000 Subject: [PATCH 091/259] fix(react): the React resolver only resolves a script's references (#2211) The React resolver's `languages` gates its route extraction, not its resolve(): framework resolvers see every language's references. Its component rule was already JSX-only, but its hook (`use...`) and context (`...Context` / `...Provider`) rules were not: in halo - a Spring app whose UI package makes React detected - 209 Java import edges (`SecurityContext`, `ApplicationContext`, `ObjectProvider`, `DirtiesContext`...) were resolved by it onto import nodes. It now answers only JavaScript / TypeScript / TSX / JSX references. halo -209 +3 (a Java test's `responseContext()` helper now resolves to itself); trpc, excalidraw, react-redux-realworld, bulletproof-react, react-native-reusables unchanged. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/react-resolver-language.test.ts | 65 +++++++++++++++++++++++ src/resolution/frameworks/react.ts | 6 +++ 3 files changed, 72 insertions(+) create mode 100644 __tests__/react-resolver-language.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 77cd625e10..58a270a00c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- In a project with a React front end, the React rules for hooks (`use…`) and contexts (`…Context`, `…Provider`) no longer apply to backend code. In halo, a Spring app with a React UI, Java imports such as `SecurityContext` and `ObjectProvider` had been handled as React contexts. - In a project with a Vue front end, the backend's calls no longer land on Vue components. mealie's Python `QueryFilterBuilder(…)` had been linking to the `QueryFilterBuilder.vue` component. The Vue rules (compiler macros, Nuxt auto-imports, components by name) now apply only to the app's scripts. - Production code no longer links to a same-named symbol inside a test suite, since tests aren't built into the program. Before, typeorm's `Record` types reached a test entity called `Record`, and okhttp's sample `@Override` annotations reached a test's nested `Override` class. Test-support code a project ships, such as a `testing/` folder or a `*-test` module, stays in reach. - A Dart call like `ext.endsWith(".avi")` or `map.putIfAbsent(…)` on a value whose type isn't known now counts as Dart's own String, List or Map method. It no longer lands on a project method of the same name. A Dart extension is also matched by the type it extends, not by its own name, so getx's `ext.endsWith` on a String stopped reaching `extension RxStringExt on Rx`. diff --git a/__tests__/react-resolver-language.test.ts b/__tests__/react-resolver-language.test.ts new file mode 100644 index 0000000000..50bbf3bf36 --- /dev/null +++ b/__tests__/react-resolver-language.test.ts @@ -0,0 +1,65 @@ +/** + * The React resolver's hook (`use…`) and context (`…Context` / `…Provider`) + * rules are a script's, never another language's: in halo — a Spring app + * with a React-detected UI package — Java imports of `SecurityContext` or + * `ObjectProvider` were taken by the context rule, and a Java test's own + * `responseContext()` helper went unresolved. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-react-lang-')); + const files: Record = { + 'package.json': JSON.stringify({ name: 'app', dependencies: { react: '^18' } }), + 'ui/src/ThemeContext.tsx': `import { createContext } from 'react'; + +export const responseContext = createContext(null); +`, + 'src/main/java/app/Filter.java': `package app; + +import org.springframework.security.core.context.SecurityContext; + +public class Filter { + Object responseContext() { + return null; + } + + void filter() { + responseContext(); + } +} +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +describe('the React resolver', () => { + it('does not take a Java import of a …Context class', () => { + const ids = cg.getNodesInFile('src/main/java/app/Filter.java').map((n) => n.id); + const byReact = cg.getOutgoingEdgesFrom(ids).filter((e) => (e.metadata as { framework?: string } | undefined)?.framework === 'react'); + expect(byReact).toEqual([]); + }); + + it('leaves a Java call named like a context to Java', () => { + const filter = cg.getNodesInFile('src/main/java/app/Filter.java').find((n) => n.name === 'filter')!; + const targets = cg.getOutgoingEdges(filter.id).filter((e) => e.kind === 'calls').map((e) => cg.getNode(e.target)!); + expect(targets.map((t) => t.filePath)).not.toContain('ui/src/ThemeContext.tsx'); + expect(targets.map((t) => t.qualifiedName)).toContain('app::Filter::responseContext'); + }); +}); diff --git a/src/resolution/frameworks/react.ts b/src/resolution/frameworks/react.ts index f9180fa0be..c4fa6946c1 100644 --- a/src/resolution/frameworks/react.ts +++ b/src/resolution/frameworks/react.ts @@ -10,6 +10,9 @@ import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from import { dependsOn } from './package-deps'; import { resolveImportPath } from '../import-resolver'; +/** The languages React components, hooks and contexts are written and used in. */ +const REACT_SCRIPT_LANGUAGES: ReadonlySet = new Set(['typescript', 'javascript', 'tsx', 'jsx']); + export const reactResolver: FrameworkResolver = { name: 'react', // Includes 'tsx'/'jsx' so route extraction runs on JSX files (where @@ -34,6 +37,9 @@ export const reactResolver: FrameworkResolver = { }, resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + // Components, hooks and contexts are a script's: halo's Java + // `import org.springframework…SecurityContext` is no React context. + if (!REACT_SCRIPT_LANGUAGES.has(ref.language)) return null; if (ref.referenceName.startsWith(LAZY_ROUTE_PREFIX)) { const target = lazyRouteComponent(ref.referenceName.slice(LAZY_ROUTE_PREFIX.length), ref.filePath, context); return target ? { original: ref, targetNodeId: target, confidence: 0.9, resolvedBy: 'framework' } : null; From c20283e2ce8b07f5842bb6cd7c8dcfbc320536ab Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 22:21:17 +0000 Subject: [PATCH 092/259] fix(resolution): a call to an overload of the calling method reaches it (#2212) A method that delegates to another overload of itself - `toInstant(instant)` returning `toInstant(instant, Instant.EPOCH)`, `DeserializeXNode(value)` returning `DeserializeXNode(value, null)` - bound to itself, since every overload shares the name: the fuller overload never listed its convenience overloads among its callers, and its impact missed them. When a name match lands on the calling method itself, in a language that overloads by arity (C#, Java, Kotlin, Swift, C++, Scala, Dart, VB.NET), the call's argument count is read and compared with the method's parameters (defaults and varargs / `params` / `...` counted); if they cannot take it, the one same-owner overload they fit is the target. Real recursion (`depth(n - 1)`) keeps its self-edge. Self-loops -> overload edges: commons-lang 637 -> 457, Newtonsoft 303 -> 196, halo 173 -> 116, okhttp 248 -> 208, Alamofire 147 -> 120, gson 74 -> 55, AutoMapper 71 -> 57 (edge counts unchanged; overloads share a qualified name, so only the target moves). Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/self-overload-calls.test.ts | 82 +++++++++++++++++++++++++++ src/resolution/name-matcher.ts | 39 ++++++++++++- 3 files changed, 121 insertions(+), 1 deletion(-) create mode 100644 __tests__/self-overload-calls.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 58a270a00c..b0370b308e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- A method that calls another overload of itself now links to that overload rather than to itself. An example is `toInstant(instant)` returning `toInstant(instant, Instant.EPOCH)`. The fuller overload now lists its convenience overloads among its callers, so its impact includes them. This applies to Java, C#, Kotlin, Swift, C++, Scala, Dart and VB.NET. Real recursion is unchanged. - In a project with a React front end, the React rules for hooks (`use…`) and contexts (`…Context`, `…Provider`) no longer apply to backend code. In halo, a Spring app with a React UI, Java imports such as `SecurityContext` and `ObjectProvider` had been handled as React contexts. - In a project with a Vue front end, the backend's calls no longer land on Vue components. mealie's Python `QueryFilterBuilder(…)` had been linking to the `QueryFilterBuilder.vue` component. The Vue rules (compiler macros, Nuxt auto-imports, components by name) now apply only to the app's scripts. - Production code no longer links to a same-named symbol inside a test suite, since tests aren't built into the program. Before, typeorm's `Record` types reached a test entity called `Record`, and okhttp's sample `@Override` annotations reached a test's nested `Override` class. Test-support code a project ships, such as a `testing/` folder or a `*-test` module, stays in reach. diff --git a/__tests__/self-overload-calls.test.ts b/__tests__/self-overload-calls.test.ts new file mode 100644 index 0000000000..c0b121c5b9 --- /dev/null +++ b/__tests__/self-overload-calls.test.ts @@ -0,0 +1,82 @@ +/** + * A method calling its own name with arguments its own parameters cannot + * take calls another overload — `toInstant(instant)` delegating to + * `toInstant(instant, Instant.EPOCH)` — not itself: the self-edge hid the + * convenience overloads from the full overload's callers (commons-lang had + * 180 such, Newtonsoft 107). + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-self-overload-')); + const files: Record = { + 'src/main/java/app/Instants.java': `package app; + +public class Instants { + public static Object toInstant(final Object instant) { + return toInstant(instant, null); + } + + public static Object toInstant(final Object instant, final Object defaultInstant) { + return instant != null ? instant : defaultInstant; + } + + public static int depth(final int n) { + return n <= 0 ? 0 : depth(n - 1); + } +} +`, + 'src/Json/Convert.cs': `namespace App +{ + public static class Convert + { + public static string DeserializeXNode(string value) + { + return DeserializeXNode(value, null); + } + + public static string DeserializeXNode(string value, string root) + { + return value + root; + } + } +} +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +/** `callerLine -> targetLine` of the calls between same-named methods of a file. */ +const selfNamed = (file: string, name: string) => { + const nodes = cg.getNodesInFile(file).filter((n) => n.name === name); + const ids = new Set(nodes.map((n) => n.id)); + return cg.getOutgoingEdgesFrom([...ids]).filter((e) => e.kind === 'calls' && ids.has(e.target)) + .map((e) => `${cg.getNode(e.source)!.startLine} -> ${cg.getNode(e.target)!.startLine}`); +}; + +describe('a method calling its own name', () => { + it('Java: reaches the overload the arguments fit; real recursion stays', () => { + expect(selfNamed('src/main/java/app/Instants.java', 'toInstant')).toEqual(['4 -> 8']); + expect(selfNamed('src/main/java/app/Instants.java', 'depth')).toEqual(['12 -> 12']); + }); + + it('C#: reaches the overload the arguments fit', () => { + expect(selfNamed('src/Json/Convert.cs', 'DeserializeXNode')).toEqual(['5 -> 10']); + }); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 866cdb41d1..e91a20ec40 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -7962,7 +7962,44 @@ export function matchReference( ref: UnresolvedRef, context: ResolutionContext ): ResolvedRef | null { - return gateLanguageMatch(matchReferenceInner(ref, context), ref, context); + const result = gateLanguageMatch(matchReferenceInner(ref, context), ref, context); + return result ? retargetSelfOverload(result, ref, context) : result; +} + +/** Languages whose methods overload by arity. */ +const OVERLOADING_LANGUAGES: ReadonlySet = new Set(['csharp', 'java', 'kotlin', 'swift', 'cpp', 'scala', 'dart', 'vbnet']); + +/** + * A call a method makes to its own name, with arguments its own parameters + * cannot take, is to another overload of it: Newtonsoft's + * `DeserializeXNode(value)` body `return DeserializeXNode(value, null);` + * bound to itself, so the two-argument overload never saw the one-argument + * one among its callers. The same-owner overload the argument count fits. + */ +function retargetSelfOverload(result: ResolvedRef, ref: UnresolvedRef, context: ResolutionContext): ResolvedRef { + if (result.targetNodeId !== ref.fromNodeId || ref.referenceKind !== 'calls' || !OVERLOADING_LANGUAGES.has(ref.language)) return result; + const self = context.getNodeById?.(ref.fromNodeId); + if (!self || (self.kind !== 'method' && self.kind !== 'function')) return result; + const name = self.name; + const args = cppParenListAfter(ref.filePath, ref.line, Math.max(0, ref.column), name, context); + if (args === null) return result; + const argc = args.trim() === '' ? 0 : splitCppTopLevel(args).length; + const arity = (n: Node): { min: number; max: number } | null => { + const list = cppParenListAfter(n.filePath, n.startLine, 0, name, context); + if (list === null) return null; + const params = splitCppTopLevel(list).filter((p) => p !== '' && p !== 'void'); + const variadic = params.some((p) => /\.\.\.|\bparams\s|\bvararg\s/.test(p.replace(/<[^<>]*>/g, ''))); + const min = params.filter((p) => !/=/.test(p) && !/\.\.\.|\bparams\s|\bvararg\s/.test(p.replace(/<[^<>]*>/g, ''))).length; + return { min, max: variadic ? Infinity : params.length }; + }; + const own = arity(self); + if (!own || (argc >= own.min && argc <= own.max)) return result; + const owner = self.qualifiedName.slice(0, Math.max(0, self.qualifiedName.lastIndexOf('::'))); + const fits = context.getNodesInFile(self.filePath).filter((n) => + n.id !== self.id && n.name === name && (n.kind === 'method' || n.kind === 'function') && + n.qualifiedName.slice(0, Math.max(0, n.qualifiedName.lastIndexOf('::'))) === owner && + ((a) => a !== null && argc >= a.min && argc <= a.max)(arity(n))); + return fits.length === 1 ? { ...result, targetNodeId: fits[0]!.id } : result; } function matchReferenceInner( From 5f1a63571417799b9e57c05492f889e7a21dbc65 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 22:32:43 +0000 Subject: [PATCH 093/259] fix(resolution): any call reaches the overload its arguments fit (#2213) #2212 moved a method's call to its own name onto the sibling overload the arguments fit. The same holds for every caller: a match landing on an overload whose parameters cannot take the call's arguments moves to the one same-owner overload they fit - commons-lang's `HashCodeBuilder.reflectionHashCode(this)` from the three-parameter overload indexed first to the `(Object, String...)` one, Newtonsoft's `SetToken(token, value)` from `SetToken(token)`, halo's four-argument `menuItem(...)` from the one-argument `menuItem`. Overload sets and arities are read once per target node, and a target with no same-owner sibling costs one lookup, so indexing time is unchanged (commons-lang 4.67s vs 4.70s, Newtonsoft 4.04s vs 4.05s, wasm both arms). C++ keeps only the self-call case: its overload sets (templates, SFINAE priority tags, `data()` on any container) move targets unreliably. Targets moved between same-named overloads: commons-lang 2,334, Newtonsoft 697, halo 286, gson 232, okhttp 188, Alamofire 50; nlohmann json and fmt unchanged. Edge counts unchanged. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/self-overload-calls.test.ts | 37 +++++++++++++++--- src/resolution/name-matcher.ts | 56 +++++++++++++++++++-------- 3 files changed, 72 insertions(+), 22 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b0370b308e..13176aaf50 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- In Java, C#, Kotlin, Swift, Scala, Dart and VB.NET, a call now reaches the overload whose parameters fit its arguments, not whichever same-named overload was indexed first. For example, `HashCodeBuilder.reflectionHashCode(this)` now reaches the one-argument overload with varargs, not a three-parameter one. Each overload's callers are now its own. - A method that calls another overload of itself now links to that overload rather than to itself. An example is `toInstant(instant)` returning `toInstant(instant, Instant.EPOCH)`. The fuller overload now lists its convenience overloads among its callers, so its impact includes them. This applies to Java, C#, Kotlin, Swift, C++, Scala, Dart and VB.NET. Real recursion is unchanged. - In a project with a React front end, the React rules for hooks (`use…`) and contexts (`…Context`, `…Provider`) no longer apply to backend code. In halo, a Spring app with a React UI, Java imports such as `SecurityContext` and `ObjectProvider` had been handled as React contexts. - In a project with a Vue front end, the backend's calls no longer land on Vue components. mealie's Python `QueryFilterBuilder(…)` had been linking to the `QueryFilterBuilder.vue` component. The Vue rules (compiler macros, Nuxt auto-imports, components by name) now apply only to the app's scripts. diff --git a/__tests__/self-overload-calls.test.ts b/__tests__/self-overload-calls.test.ts index c0b121c5b9..d18b7c66e7 100644 --- a/__tests__/self-overload-calls.test.ts +++ b/__tests__/self-overload-calls.test.ts @@ -1,9 +1,9 @@ /** - * A method calling its own name with arguments its own parameters cannot - * take calls another overload — `toInstant(instant)` delegating to - * `toInstant(instant, Instant.EPOCH)` — not itself: the self-edge hid the - * convenience overloads from the full overload's callers (commons-lang had - * 180 such, Newtonsoft 107). + * A call that lands on an overload its arguments cannot fit is to the one + * sibling overload they do: `toInstant(instant)` delegating to + * `toInstant(instant, Instant.EPOCH)` bound to itself (commons-lang had 180 + * such self-edges, Newtonsoft 107), and `HashCodeBuilder.reflectionHashCode(this)` + * to the first overload indexed, a three-parameter one. */ import { describe, it, expect, afterAll, beforeAll } from 'vitest'; import * as fs from 'fs'; @@ -32,6 +32,26 @@ public class Instants { return n <= 0 ? 0 : depth(n - 1); } } +`, + 'src/main/java/app/HashCodeBuilder.java': `package app; + +public class HashCodeBuilder { + public static int reflectionHashCode(final int initial, final int multiplier, final Object object) { + return initial * multiplier; + } + + public static int reflectionHashCode(final Object object, final String... excludeFields) { + return 1; + } +} +`, + 'src/main/java/app/Point.java': `package app; + +public class Point { + public int hash() { + return HashCodeBuilder.reflectionHashCode(this); + } +} `, 'src/Json/Convert.cs': `namespace App { @@ -76,6 +96,13 @@ describe('a method calling its own name', () => { expect(selfNamed('src/main/java/app/Instants.java', 'depth')).toEqual(['12 -> 12']); }); + it('from another class: reaches the overload the arguments fit, not the first indexed', () => { + const hash = cg.getNodesInFile('src/main/java/app/Point.java').find((n) => n.name === 'hash')!; + const lines = cg.getOutgoingEdges(hash.id).filter((e) => e.kind === 'calls').map((e) => cg.getNode(e.target)!) + .filter((t) => t.name === 'reflectionHashCode').map((t) => t.startLine); + expect(lines).toEqual([8]); + }); + it('C#: reaches the overload the arguments fit', () => { expect(selfNamed('src/Json/Convert.cs', 'DeserializeXNode')).toEqual(['5 -> 10']); }); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index e91a20ec40..ac62250ce8 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -5257,6 +5257,7 @@ export function clearNameMatcherMemos(context: ResolutionContext): void { JVM_PACKAGES.delete(context); MINIFIED_SCRIPTS.delete(context); PY_LOCAL_BINDS.delete(context); + OVERLOAD_SETS.delete(context); PHP_FILE_SCOPES.delete(context); JAVA_STATIC_IMPORTS.delete(context); PY_IMPORTS.delete(context); @@ -7976,30 +7977,51 @@ const OVERLOADING_LANGUAGES: ReadonlySet = new Set(['csharp', 'java', 'k * bound to itself, so the two-argument overload never saw the one-argument * one among its callers. The same-owner overload the argument count fits. */ +type Arity = { min: number; max: number } | null; +/** Per context: node id → its same-owner overloads with their arities (null: none). */ +const OVERLOAD_SETS = new WeakMap } | null>>(); + function retargetSelfOverload(result: ResolvedRef, ref: UnresolvedRef, context: ResolutionContext): ResolvedRef { - if (result.targetNodeId !== ref.fromNodeId || ref.referenceKind !== 'calls' || !OVERLOADING_LANGUAGES.has(ref.language)) return result; - const self = context.getNodeById?.(ref.fromNodeId); - if (!self || (self.kind !== 'method' && self.kind !== 'function')) return result; - const name = self.name; - const args = cppParenListAfter(ref.filePath, ref.line, Math.max(0, ref.column), name, context); + if (ref.referenceKind !== 'calls' || !OVERLOADING_LANGUAGES.has(ref.language)) return result; + // C++ overload sets (templates, SFINAE tags, a `data()` on any container) are + // only trusted for a method's call to itself. + if (ref.language === 'cpp' && result.targetNodeId !== ref.fromNodeId) return result; + let memo = OVERLOAD_SETS.get(context); + if (!memo) OVERLOAD_SETS.set(context, (memo = new Map())); + let set = memo.get(result.targetNodeId); + if (set === undefined) { + set = overloadSetOf(result.targetNodeId, context); + memo.set(result.targetNodeId, set); + } + // Only an overload set has a sibling to move to. + if (!set || set.own === null) return result; + const args = cppParenListAfter(ref.filePath, ref.line, Math.max(0, ref.column), set.name, context); if (args === null) return result; const argc = args.trim() === '' ? 0 : splitCppTopLevel(args).length; - const arity = (n: Node): { min: number; max: number } | null => { + if (argc >= set.own.min && argc <= set.own.max) return result; + const fits = set.siblings.filter((s) => s.arity !== null && argc >= s.arity.min && argc <= s.arity.max); + return fits.length === 1 ? { ...result, targetNodeId: fits[0]!.id } : result; +} + +/** A method's same-owner overloads and every one's arity, read from its declaration. */ +function overloadSetOf(id: string, context: ResolutionContext): { name: string; own: Arity; siblings: Array<{ id: string; arity: Arity }> } | null { + const self = context.getNodeById?.(id); + if (!self || (self.kind !== 'method' && self.kind !== 'function')) return null; + const name = self.name; + const owner = self.qualifiedName.slice(0, Math.max(0, self.qualifiedName.lastIndexOf('::'))); + const siblings = (context.getNodesInFileNamed?.(self.filePath, name) ?? context.getNodesInFile(self.filePath).filter((n) => n.name === name)) + .filter((n) => n.id !== self.id && (n.kind === 'method' || n.kind === 'function') && + n.qualifiedName.slice(0, Math.max(0, n.qualifiedName.lastIndexOf('::'))) === owner); + if (siblings.length === 0) return null; + const arity = (n: Node): Arity => { const list = cppParenListAfter(n.filePath, n.startLine, 0, name, context); if (list === null) return null; const params = splitCppTopLevel(list).filter((p) => p !== '' && p !== 'void'); - const variadic = params.some((p) => /\.\.\.|\bparams\s|\bvararg\s/.test(p.replace(/<[^<>]*>/g, ''))); - const min = params.filter((p) => !/=/.test(p) && !/\.\.\.|\bparams\s|\bvararg\s/.test(p.replace(/<[^<>]*>/g, ''))).length; - return { min, max: variadic ? Infinity : params.length }; + const pack = (p: string) => /\.\.\.|\bparams\s|\bvararg\s/.test(p.replace(/<[^<>]*>/g, '')); + const min = params.filter((p) => !/=/.test(p) && !pack(p)).length; + return { min, max: params.some(pack) ? Infinity : params.length }; }; - const own = arity(self); - if (!own || (argc >= own.min && argc <= own.max)) return result; - const owner = self.qualifiedName.slice(0, Math.max(0, self.qualifiedName.lastIndexOf('::'))); - const fits = context.getNodesInFile(self.filePath).filter((n) => - n.id !== self.id && n.name === name && (n.kind === 'method' || n.kind === 'function') && - n.qualifiedName.slice(0, Math.max(0, n.qualifiedName.lastIndexOf('::'))) === owner && - ((a) => a !== null && argc >= a.min && argc <= a.max)(arity(n))); - return fits.length === 1 ? { ...result, targetNodeId: fits[0]!.id } : result; + return { name, own: arity(self), siblings: siblings.map((n) => ({ id: n.id, arity: arity(n) })) }; } function matchReferenceInner( From fa8d7284c5efe85ee86ede063967f95cd651ffb4 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 22:54:08 +0000 Subject: [PATCH 094/259] fix(rust): a bare name is in scope only through its module or a use (#2214) The Rust scope rule (#2124) only checked prelude names; every other bare name could mean any same-named project item, and a name after any path (`io::Result`) could mean any at all. The Rust framework resolver's PascalCase / handler / service patterns bound names with no scope check at 0.7-0.8. tokio's 1,221 `io::Result<...>` went to a private `runtime::task::Result` alias, 642 `Context<'_>` (from `use std::task::{Context, Poll}`) to `runtime::context::Context`. - A type or function from another file is in scope only through a `use` that binds it (its leaf, `as` alias, or `{self}`) or a glob over its module; methods and fields (reached through a value), import refs and module / file targets are unaffected. - A name a file imports from outside the project - `std` / `core` / `alloc`, or a crate its Cargo manifests declare as a dependency - is that outside item. (An inline `mod support { pub mod panic; }` is the project's.) - A path names its module: `seg::Name` reaches an item of module `seg`; a path rooted at std (`std::io::Error`) or at an outside import (`io::` under `use std::io`) reaches none; `crate::` / `self::` / `super::` and a project crate's name (`clap::Command`, re-exported) look it up as before. The path is read anywhere on the line, not only at the reference's column. - The framework resolver's patterns apply the same check. tokio -3008 +610, serde -792 +650, ripgrep -602 +247, axum -453 +186, clap -342 +32, bat -38 +9. Sampled losses: std / core / futures / syn / http / anyhow imports and paths, associated types (`S::Ok`, `S::Error`, `V::Value`, `Fut::Output`); gains are same-module items the old first match hid. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/rust-type-scope.test.ts | 90 +++++++++++++++++++++++ src/resolution/frameworks/rust.ts | 17 ++++- src/resolution/name-matcher.ts | 116 ++++++++++++++++++++++++++++-- 4 files changed, 217 insertions(+), 7 deletions(-) create mode 100644 __tests__/rust-type-scope.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 13176aaf50..219e0257b4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- Rust type and function names now follow Rust's scoping. A name from another file counts only when a `use` brings it in (or a `*` glob over its module). One imported from outside the project, like `use std::task::{Context, Poll}` or `use futures::Stream`, means that outside item. A path names its module: `io::Result` isn't an unrelated `Result` alias elsewhere, and `std::io::Error` is std's. On tokio, over 1,200 `io::Result` and 600 `Context` references had been landing on private project types. - In Java, C#, Kotlin, Swift, Scala, Dart and VB.NET, a call now reaches the overload whose parameters fit its arguments, not whichever same-named overload was indexed first. For example, `HashCodeBuilder.reflectionHashCode(this)` now reaches the one-argument overload with varargs, not a three-parameter one. Each overload's callers are now its own. - A method that calls another overload of itself now links to that overload rather than to itself. An example is `toInstant(instant)` returning `toInstant(instant, Instant.EPOCH)`. The fuller overload now lists its convenience overloads among its callers, so its impact includes them. This applies to Java, C#, Kotlin, Swift, C++, Scala, Dart and VB.NET. Real recursion is unchanged. - In a project with a React front end, the React rules for hooks (`use…`) and contexts (`…Context`, `…Provider`) no longer apply to backend code. In halo, a Spring app with a React UI, Java imports such as `SecurityContext` and `ObjectProvider` had been handled as React contexts. diff --git a/__tests__/rust-type-scope.test.ts b/__tests__/rust-type-scope.test.ts new file mode 100644 index 0000000000..766252fc65 --- /dev/null +++ b/__tests__/rust-type-scope.test.ts @@ -0,0 +1,90 @@ +/** + * A bare Rust type or function from another file is in scope only through a + * `use` that binds it (or a glob over its module), and one the file imports + * from outside the project — `use std::task::{Context, Poll}` — is that + * outside item: tokio's 1,221 `io::Result<…>` bound to a private + * `runtime::task::Result` alias, its 642 `Context<'_>` to + * `runtime::context::Context`. A path names its module (`io::Result` is no + * `task` alias; `std::io::Error` is std's), a project crate's name + * (`clap::Command`) re-exports as `crate::` does, and an inline test module + * (`mod support { pub mod panic; }`) is the project's own. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-rust-scope-')); + const files: Record = { + 'Cargo.toml': `[package]\nname = "app"\nversion = "0.1.0"\n\n[dependencies]\nfutures = "0.3"\n`, + 'src/lib.rs': `pub mod runtime;\npub mod io;\npub mod net;\n`, + 'src/runtime/mod.rs': `pub mod context;\npub mod task;\npub mod driver;\n`, + 'src/runtime/context.rs': `pub struct Context {\n pub depth: u8,\n}\n`, + 'src/runtime/task/mod.rs': `pub type Result = std::result::Result;\n`, + 'src/io/mod.rs': `pub mod copy;\n`, + 'src/io/copy.rs': `pub struct Error;\n`, + 'src/net.rs': `use std::io; +use std::task::{Context, Poll}; + +pub fn poll_ready(cx: &mut Context<'_>) -> Poll> { + let _ = cx; + Poll::Ready(Ok(())) +} + +pub fn fail() -> std::io::Error { + std::io::Error::other("x") +} +`, + 'src/runtime/driver.rs': `use crate::runtime::context::Context; + +pub fn enter(ctx: &Context) -> u8 { + ctx.depth +} +`, + 'tests/support/panic.rs': `pub fn test_panic() {}\n`, + 'tests/io_panic.rs': `mod support { + pub mod panic; +} +use support::panic::test_panic; + +#[test] +fn panics() { + test_panic(); +} +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +const targetsFrom = (file: string) => { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg.getOutgoingEdgesFrom(ids).filter((e) => e.kind !== 'contains').map((e) => cg.getNode(e.target)!.filePath); +}; + +describe('Rust names in scope', () => { + it('an outside import, or a path through one, is never a project item', () => { + const targets = targetsFrom('src/net.rs'); + expect(targets).not.toContain('src/runtime/context.rs'); + expect(targets).not.toContain('src/runtime/task/mod.rs'); + expect(targets).not.toContain('src/io/copy.rs'); + }); + + it('a crate-local use, and an inline test module, still reach the project item', () => { + expect(targetsFrom('src/runtime/driver.rs')).toContain('src/runtime/context.rs'); + expect(targetsFrom('tests/io_panic.rs')).toContain('tests/support/panic.rs'); + }); +}); diff --git a/src/resolution/frameworks/rust.ts b/src/resolution/frameworks/rust.ts index f4828a77ae..283c453521 100644 --- a/src/resolution/frameworks/rust.ts +++ b/src/resolution/frameworks/rust.ts @@ -8,6 +8,17 @@ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from '../types'; import { stripCommentsForRegex } from '../strip-comments'; import { getCargoWorkspaceCrateMap } from './cargo-workspace'; +import { isRustNameInScope } from '../name-matcher'; + +/** + * Whether the item a name heuristic found is one the reference can name: + * `Context<'_>` under `use std::task::{Context, Poll}` is std's, not tokio's + * `runtime::context::Context` (642 of them). + */ +function inRustScope(id: string, ref: UnresolvedRef, context: ResolutionContext): boolean { + const node = context.getNodeById?.(id); + return !node || isRustNameInScope(node, ref, context); +} const cargoWorkspaceMapCache = new WeakMap>(); @@ -32,7 +43,7 @@ export const rustResolver: FrameworkResolver = { // Pattern 1: Handler references if (ref.referenceName.endsWith('_handler') || ref.referenceName.startsWith('handle_')) { const result = resolveByNameAndKind(ref.referenceName, FUNCTION_KINDS, HANDLER_DIRS, context); - if (result) { + if (result && inRustScope(result, ref, context)) { return { original: ref, targetNodeId: result, @@ -45,7 +56,7 @@ export const rustResolver: FrameworkResolver = { // Pattern 2: Service/Repository trait implementations if (ref.referenceName.endsWith('Service') || ref.referenceName.endsWith('Repository')) { const result = resolveByNameAndKind(ref.referenceName, SERVICE_KINDS, SERVICE_DIRS, context); - if (result) { + if (result && inRustScope(result, ref, context)) { return { original: ref, targetNodeId: result, @@ -58,7 +69,7 @@ export const rustResolver: FrameworkResolver = { // Pattern 3: Struct references (PascalCase) if (/^[A-Z][a-zA-Z]+$/.test(ref.referenceName)) { const result = resolveByNameAndKind(ref.referenceName, STRUCT_KINDS, MODEL_DIRS, context); - if (result) { + if (result && inRustScope(result, ref, context)) { return { original: ref, targetNodeId: result, diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index ac62250ce8..18473aeda7 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -13,6 +13,7 @@ import { JS_BUILT_INS, JS_BUILTIN_METHODS, TS_PRIMITIVE_TYPES } from './js-built import { SWIFT_TYPE_PATH_CALL, resolveSwiftTypePathCall } from './swift-type-visibility'; import { isTestPath } from '../search/query-utils'; import { isMinifiedContent } from '../extraction/generated-detection'; +import { getCargoWorkspaceCrateMap } from './frameworks/cargo-workspace'; /** * Ceiling on how many same-named definitions a FUZZY name-match strategy will * score. A name defined more times than this is "ubiquitous" — a method/symbol @@ -3775,6 +3776,59 @@ interface RustUses { names: Set; /** `X` of each `use …::X::*` (`super` for `use super::*`). */ globs: Set; + /** Names the file imports from outside the project — `use std::task::{Context, Poll}`, `use futures::Stream`. */ + external: Set; + /** The items the file's project `use`s bind — their leaves, not the paths they walk. */ + bound: Set; +} + +const RUST_CRATES = new WeakMap>(); +const RUST_DEPENDENCIES = new WeakMap>(); + +/** + * The crates the project's manifests depend on (`[dependencies]`, + * `[dev-dependencies]`, `[build-dependencies]`, per-target ones), by the name + * code writes them (`futures_util`), less the project's own. + */ +function rustDependencyCrates(context: ResolutionContext): Set { + const hit = RUST_DEPENDENCIES.get(context); + if (hit) return hit; + const deps = new Set(); + const manifests = ['Cargo.toml', ...[...getCargoWorkspaceCrateMap(context).values()].map((dir) => `${dir}/Cargo.toml`)]; + for (const manifest of new Set(manifests)) { + const text = context.readFile(manifest) ?? ''; + let inDeps = false; + for (const raw of text.split(/\r?\n/)) { + const line = raw.replace(/#.*$/, '').trim(); + const header = /^\[([^\]]+)\]$/.exec(line); + if (header) { + const table = header[1]!.trim(); + // `[dependencies.tokio]` names one dependency in its header. + const named = /(?:^|\.)(?:dev-|build-)?dependencies\.([A-Za-z0-9_-]+)$/.exec(table); + if (named) deps.add(named[1]!.replace(/-/g, '_')); + inDeps = /(?:^|\.)(?:dev-|build-)?dependencies$/.test(table); + continue; + } + const key = inDeps ? /^([A-Za-z0-9_-]+)\s*=/.exec(line)?.[1] : undefined; + if (key) deps.add(key.replace(/-/g, '_')); + } + } + for (const own of rustProjectCrates(context)) deps.delete(own); + RUST_DEPENDENCIES.set(context, deps); + return deps; +} + +/** The project's own crate names (`tokio`, `tokio_util`), from its Cargo.toml files. */ +function rustProjectCrates(context: ResolutionContext): Set { + const hit = RUST_CRATES.get(context); + if (hit) return hit; + const crates = new Set(); + // The manifests are not indexed files: the root's package, and the workspace's members. + const root = /\[package\][^[]*?\bname\s*=\s*"([^"]+)"/.exec(context.readFile('Cargo.toml') ?? '')?.[1]; + if (root) crates.add(root.replace(/-/g, '_')); + for (const name of getCargoWorkspaceCrateMap(context).keys()) crates.add(name.replace(/-/g, '_')); + RUST_CRATES.set(context, crates); + return crates; } const RUST_USES = new WeakMap>(); @@ -3786,12 +3840,28 @@ function rustUsesOf(filePath: string, context: ResolutionContext): RustUses { } const hit = memo.get(filePath); if (hit) return hit; - const uses: RustUses = { names: new Set(), globs: new Set() }; + const uses: RustUses = { names: new Set(), globs: new Set(), external: new Set(), bound: new Set() }; + const leaves = (tree: string): string[] => [ + ...[...tree.matchAll(/([A-Za-z_]\w*)\s*(?=[,}]|$|\s+as\b)|\bas\s+([A-Za-z_]\w*)/g)] + .map((leaf) => leaf[2] ?? leaf[1]!).filter((id) => id !== 'self' && id !== 'as'), + // `use std::io::{self, Read}` binds `io` too. + ...[...tree.matchAll(/([A-Za-z_]\w*)\s*::\s*\{[^{}]*\bself\b/g)].map((m) => m[1]!), + ]; // Comments first: a doc comment's prose ("…use the Option…") is not a `use`. const text = stripCommentsForRegex(context.readFile(filePath) ?? '', 'rust'); + const dependencies = rustDependencyCrates(context); for (const m of text.matchAll(/(?:^|[;{}\s])use\s+([^;]{1,2000});/g)) { const tree = m[1]!; - if (/^\s*(?:::)?(?:std|core|alloc)\b/.test(tree)) continue; + const root = /^\s*(?:::)?([A-Za-z_]\w*)/.exec(tree)?.[1] ?? ''; + // Outside: the standard library or a crate the manifests depend on — not a + // module of the project's (`mod support { … }` inline in a test). + const outside = root === 'std' || root === 'core' || root === 'alloc' || (root !== '' && dependencies.has(root)); + // The items it binds: each leaf (`as` aliases by their alias), never the path it walks. + if (outside) { + for (const id of leaves(tree)) uses.external.add(id); + continue; + } + for (const id of leaves(tree)) uses.bound.add(id); for (const id of tree.matchAll(/[A-Za-z_]\w*/g)) uses.names.add(id[0]); for (const g of tree.matchAll(/(\w+)\s*::\s*(?:\{[^}]*)?\*/g)) uses.globs.add(g[1]!); } @@ -3829,7 +3899,27 @@ export function isRustNameInScope(candidate: Node, ref: UnresolvedRef, context: // Bare in the SOURCE: the index keeps `crate::error::Result` by its last // segment, and a path is not a prelude lookup. const line = context.getFileLines?.(ref.filePath)?.[ref.line - 1] ?? context.readFile(ref.filePath)?.split('\n')[ref.line - 1]; - if (line !== undefined && line.startsWith(name, ref.column) && /::\s*$/.test(line.slice(0, ref.column))) return true; + // Written through a path on its line (`jsont::SubMatch { … }`, `io::Result<…>`), + // wherever the reference's column points. + const pathed = line === undefined ? null + : (line.startsWith(name, ref.column) && /::\s*$/.test(line.slice(0, ref.column)) ? /((?:[A-Za-z_]\w*\s*::\s*)*)([A-Za-z_]\w*)?\s*::\s*$/.exec(line.slice(0, ref.column)) + : !new RegExp(`(?= 0 ? candidate.qualifiedName.slice(0, cut).split('::').pop()! : ''; return (owner !== '' && uses.globs.has(owner)) || (uses.names.has(name) && uses.names.has(owner)); } - if (!RUST_PRELUDE.has(name) || candidate.filePath === ref.filePath) return true; + if (candidate.filePath === ref.filePath) return true; const uses = rustUsesOf(ref.filePath, context); + // `use std::task::{Context, Poll}`: the file's `Context` is std's, not tokio's + // `runtime::context::Context`. (A method call `.env(…)` is no imported name.) + if (uses.external.has(name) && !uses.bound.has(name) && line !== undefined) { + // Not on its line at all: a later link of a chain written across lines (`Arg::new(…)\n.env(…)`). + const at = new RegExp(`(?` is + // not `runtime::task::trace`'s `Context` unless the file brings that one + // in. A method is reached through a value, never a `use`. + if (TYPE_MEMBER_KINDS.has(candidate.kind) || ref.referenceKind === 'imports' || + candidate.kind === 'file' || candidate.kind === 'module' || candidate.kind === 'namespace') return true; + return uses.bound.has(name) || rustGlobCovers(uses, candidate, ref); + } return uses.names.has(name) || rustGlobCovers(uses, candidate, ref); } @@ -5217,6 +5323,8 @@ export function clearNameMatcherMemos(context: ResolutionContext): void { C_STATIC_MEMO.delete(context); RUST_TRAIT_IMPL_MEMO.delete(context); RUST_USES.delete(context); + RUST_CRATES.delete(context); + RUST_DEPENDENCIES.delete(context); LEXICAL_SCOPE_MEMO.delete(context); KOTLIN_LAMBDA_RECEIVERS.delete(context); SCALA_IMPORTED_SUPERS.delete(context); From a8b578f7d8d4bd24502a9c10706a2657a21ba7d5 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 23:02:05 +0000 Subject: [PATCH 095/259] fix(csharp): a bare type name is one its namespaces can see (#2216) A bare C# type name could mean any same-named project type. Newtonsoft's `async Task` tests (`using System.Threading.Tasks;`) bound `Task` to a test class `Task` in `Newtonsoft.Json.Tests.Schema` 433 times; AutoMapper's DTOs in `OmmitedDTOModel3` extended `OmmitedDatabaseModel3`'s `BaseEntity`; Newtonsoft's `BindingFlags` went to its portable-build polyfill. A type in namespace N is now in reach of a file whose code runs in N or a namespace inside it, or that imports N: its `using N;`, any file's `global using N;`, the project files' ``, the SDK's implicit usings when `` is on (serilog's own `System.TimeProvider` polyfill is seen through them), or an alias naming the type (`using License = AutoMapper.Licensing.License;`). A type in the global namespace is seen everywhere; qualified names are not judged. Newtonsoft -529, AutoMapper -72, eShop -4; serilog, CleanArchitecture unchanged. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/csharp-namespace-visibility.test.ts | 86 +++++++++++++++++ src/resolution/name-matcher.ts | 96 +++++++++++++++++++ 3 files changed, 183 insertions(+) create mode 100644 __tests__/csharp-namespace-visibility.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 219e0257b4..f4dc07df9b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- C# type names now follow namespaces. A type is in reach from its own namespace or one nested inside it, or through a `using` of its namespace. Also counted: a project-wide `global using`, the project's `` entries, the SDK's implicit usings, or an alias that names it. Before, Newtonsoft's `async Task` tests linked `Task` to a test class of that name in another namespace, and AutoMapper's DTOs linked to another namespace's `BaseEntity`. - Rust type and function names now follow Rust's scoping. A name from another file counts only when a `use` brings it in (or a `*` glob over its module). One imported from outside the project, like `use std::task::{Context, Poll}` or `use futures::Stream`, means that outside item. A path names its module: `io::Result` isn't an unrelated `Result` alias elsewhere, and `std::io::Error` is std's. On tokio, over 1,200 `io::Result` and 600 `Context` references had been landing on private project types. - In Java, C#, Kotlin, Swift, Scala, Dart and VB.NET, a call now reaches the overload whose parameters fit its arguments, not whichever same-named overload was indexed first. For example, `HashCodeBuilder.reflectionHashCode(this)` now reaches the one-argument overload with varargs, not a three-parameter one. Each overload's callers are now its own. - A method that calls another overload of itself now links to that overload rather than to itself. An example is `toInstant(instant)` returning `toInstant(instant, Instant.EPOCH)`. The fuller overload now lists its convenience overloads among its callers, so its impact includes them. This applies to Java, C#, Kotlin, Swift, C++, Scala, Dart and VB.NET. Real recursion is unchanged. diff --git a/__tests__/csharp-namespace-visibility.test.ts b/__tests__/csharp-namespace-visibility.test.ts new file mode 100644 index 0000000000..8ffac2ff4d --- /dev/null +++ b/__tests__/csharp-namespace-visibility.test.ts @@ -0,0 +1,86 @@ +/** + * A bare C# type name means a type its file can see: one in its own + * namespace or an enclosing one, or in a namespace it imports (`using`, + * `global using`, the project's `` / implicit usings), or the one an + * alias names. Newtonsoft's `async Task` tests (`using + * System.Threading.Tasks;`) bound `Task` to a test class of that name in + * `Newtonsoft.Json.Tests.Schema` 433 times. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-cs-ns-')); + const files: Record = { + 'src/App.Tests/Schema/Generator.cs': `namespace App.Tests.Schema +{ + public class Task + { + } +} +`, + 'src/App.Tests/Reader/ReadTests.cs': `using System.Threading.Tasks; + +namespace App.Tests.Reader +{ + public class ReadTests + { + public async Task Read() + { + await System.Threading.Tasks.Task.Yield(); + } + } +} +`, + 'src/App.Tests/Schema/Planner.cs': `namespace App.Tests.Schema +{ + public class Planner + { + public Task Next() { return new Task(); } + } +} +`, + 'src/App.Tests/Other/Aliased.cs': `using Task = App.Tests.Schema.Task; + +namespace App.Tests.Other +{ + public class Aliased + { + public Task Make() { return new Task(); } + } +} +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +const reachesTask = (file: string) => { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg.getOutgoingEdgesFrom(ids).some((e) => cg.getNode(e.target)!.qualifiedName === 'App.Tests.Schema::Task'); +}; + +describe('C# type names and namespaces', () => { + it('a type another namespace does not import is not in reach', () => { + expect(reachesTask('src/App.Tests/Reader/ReadTests.cs')).toBe(false); + }); + + it('the same namespace, and an alias naming the type, reach it', () => { + expect(reachesTask('src/App.Tests/Schema/Planner.cs')).toBe(true); + expect(reachesTask('src/App.Tests/Other/Aliased.cs')).toBe(true); + }); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 18473aeda7..d9b6fb8148 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -1317,6 +1317,8 @@ export function isVisibleAcrossFiles(candidate: Node, ref: UnresolvedRef, contex // A test suite is not linked into the program: typeorm's `Record` is // not a test entity `Record`, tokio's `Output` not a `runtime/tests` type. if (isTestSuitePath(candidate.filePath) && !isTestPath(ref.filePath)) return false; + if (candidate.language === 'csharp' && ref.language === 'csharp' && CSHARP_TYPE_KINDS.has(candidate.kind) && + /^[A-Za-z_]\w*$/.test(ref.referenceName) && !isCsharpTypeVisible(candidate, ref, context)) return false; const lang = candidate.language as string; if (lang === 'c' || lang === 'cpp') { return ( @@ -3209,6 +3211,98 @@ function csharpProjectStaticUsings(dir: string, context: ResolutionContext, memo return owners; } +const CSHARP_NAMESPACE_SCOPES = new WeakMap; aliases: Map }>>(); +const CSHARP_PROJECT_USINGS = new WeakMap>>(); + +/** + * The namespaces a C# file's code runs in and the ones it imports: its + * `namespace` declarations, its `using X;`, every file's `global using X;` + * and the `` of the project files above it. + */ +function csharpNamespaceScope(file: string, context: ResolutionContext): { namespaces: string[]; usings: Set; aliases: Map } { + let memo = CSHARP_NAMESPACE_SCOPES.get(context); + if (!memo) CSHARP_NAMESPACE_SCOPES.set(context, (memo = new Map())); + const hit = memo.get(file); + if (hit) return hit; + const text = stripCommentsForRegex(context.readFile(file) ?? '', 'java'); + const namespaces = [...text.matchAll(/^\s*namespace\s+([\w.]+)/gm)].map((m) => m[1]!); + let projectMemo = CSHARP_PROJECT_USINGS.get(context); + if (!projectMemo) CSHARP_PROJECT_USINGS.set(context, (projectMemo = new Map())); + const usings = new Set(csharpProjectUsings(path.posix.dirname(file), context, projectMemo)); + const aliases = new Map(); + for (const m of text.matchAll(/^\s*(?:global\s+)?using\s+(?!static\b)(?:([A-Za-z_]\w*)\s*=\s*)?([\w.]+)\s*;/gm)) { + if (m[1]) aliases.set(m[1], m[2]!); + else usings.add(m[2]!); + } + const scope = { namespaces, usings, aliases }; + memo.set(file, scope); + return scope; +} + +/** `global using X;` from any file, and `` of the project files from `dir` up. */ +function csharpProjectUsings(dir: string, context: ResolutionContext, memo: Map>): Set { + const key = dir; + const hit = memo.get(key); + if (hit) return hit; + let usings: Set; + if (dir === '.' || dir === '' || dir === '/') { + usings = new Set(); + for (const f of context.getAllFiles()) { + if (!f.endsWith('.cs') || (context.fileContains && !context.fileContains(f, 'global using'))) continue; + for (const m of (context.readFile(f) ?? '').matchAll(/^\s*global\s+using\s+(?!static\b)([\w.]+)\s*;/gm)) usings.add(m[1]!); + } + } else { + usings = new Set(csharpProjectUsings(path.posix.dirname(dir), context, memo)); + } + let entries: string[] = []; + try { + entries = fs.readdirSync(path.join(context.getProjectRoot(), dir === '.' ? '' : dir)); + } catch { + entries = []; + } + for (const entry of entries) { + if (!/\.(?:csproj|props)$/i.test(entry)) continue; + let project = ''; + try { + project = fs.readFileSync(path.join(context.getProjectRoot(), dir === '.' ? '' : dir, entry), 'utf8'); + } catch { + continue; + } + for (const m of project.matchAll(/]*\bStatic\s*=\s*"true")[^>]*>/gi)) usings.add(m[1]!); + // The SDK's implicit usings (serilog's own `System.TimeProvider` polyfill is seen through them). + if (/\s*(?:enable|true)\s*<\/ImplicitUsings>/i.test(project)) { + for (const ns of CSHARP_IMPLICIT_USINGS) usings.add(ns); + } + } + memo.set(key, usings); + return usings; +} + +/** The namespaces `enable` imports into every file (Microsoft.NET.Sdk). */ +const CSHARP_IMPLICIT_USINGS: readonly string[] = [ + 'System', 'System.Collections.Generic', 'System.IO', 'System.Linq', 'System.Net.Http', 'System.Threading', 'System.Threading.Tasks', +]; + +/** + * Whether a bare C# type name can mean `n` — a type in namespace `N` is seen + * from `N` and the namespaces inside it, and through a `using N;` — + * Newtonsoft's `async Task` tests (`using System.Threading.Tasks;`) bound + * `Task` to a test class of that name in `Newtonsoft.Json.Tests.Schema`, 433 + * times. A type in the global namespace is seen everywhere. + */ +function isCsharpTypeVisible(n: Node, ref: UnresolvedRef, context: ResolutionContext): boolean { + const cut = n.qualifiedName.indexOf('::'); + if (cut < 0) return true; + const ns = n.qualifiedName.slice(0, cut); + // A nested type (`Outer::Inner`) is judged by its outermost type's namespace. + const scope = csharpNamespaceScope(ref.filePath, context); + // `using License = AutoMapper.Licensing.License;` names that type, whatever the file's usings. + const aliased = scope.aliases.get(ref.referenceName); + if (aliased !== undefined) return aliased === `${ns}.${n.qualifiedName.slice(cut + 2).replace(/::/g, '.')}`; + if (scope.namespaces.some((own) => own === ns || own.startsWith(ns + '.'))) return true; + return scope.usings.has(ns); +} + const OBJC_SUPERS = new WeakMap>(); const OBJC_MEMBER_KINDS: ReadonlySet = new Set(['method', 'property', 'field']); @@ -5357,6 +5451,8 @@ export function clearNameMatcherMemos(context: ResolutionContext): void { OBJC_SUPERS.delete(context); CSHARP_SUPERS.delete(context); CSHARP_STATIC_USINGS.delete(context); + CSHARP_NAMESPACE_SCOPES.delete(context); + CSHARP_PROJECT_USINGS.delete(context); SCALA_SUPERS.delete(context); SCALA_IMPORTS.delete(context); ESM_EXPORT_LISTS.delete(context); From fa43e259b2bb19cca8f2469130ffe8c8e406acdc Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 23:07:39 +0000 Subject: [PATCH 096/259] fix(scala): a symbolic name in a type is a type, never an operator (#2217) A kind-projector placeholder (`Align[Either[A, *]]`, `Functor[Map[K, ?]]`) reached the resolver as a type reference named `*`, and exact or fuzzy matching bound it to the one `*` in the project - algebra's `Signed.Sign.*` operator - 567 times in cats. A symbolic name in a Scala type now resolves only to a type or type alias of that name (cats' `F ~> G`), and otherwise to nothing, whatever strategy runs. cats -567; sttp unchanged. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/scala-kind-projector.test.ts | 53 ++++++++++++++++++++++++++ src/resolution/name-matcher.ts | 12 +++++- 3 files changed, 65 insertions(+), 1 deletion(-) create mode 100644 __tests__/scala-kind-projector.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index f4dc07df9b..02b0bd5273 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- In Scala, a kind-projector placeholder in a type, like `Align[Either[A, *]]`, is no longer linked to an operator method named `*`. A symbolic name in a type, like cats' `F ~> G`, now resolves only to a type. - C# type names now follow namespaces. A type is in reach from its own namespace or one nested inside it, or through a `using` of its namespace. Also counted: a project-wide `global using`, the project's `` entries, the SDK's implicit usings, or an alias that names it. Before, Newtonsoft's `async Task` tests linked `Task` to a test class of that name in another namespace, and AutoMapper's DTOs linked to another namespace's `BaseEntity`. - Rust type and function names now follow Rust's scoping. A name from another file counts only when a `use` brings it in (or a `*` glob over its module). One imported from outside the project, like `use std::task::{Context, Poll}` or `use futures::Stream`, means that outside item. A path names its module: `io::Result` isn't an unrelated `Result` alias elsewhere, and `std::io::Error` is std's. On tokio, over 1,200 `io::Result` and 600 `Context` references had been landing on private project types. - In Java, C#, Kotlin, Swift, Scala, Dart and VB.NET, a call now reaches the overload whose parameters fit its arguments, not whichever same-named overload was indexed first. For example, `HashCodeBuilder.reflectionHashCode(this)` now reaches the one-argument overload with varargs, not a three-parameter one. Each overload's callers are now its own. diff --git a/__tests__/scala-kind-projector.test.ts b/__tests__/scala-kind-projector.test.ts new file mode 100644 index 0000000000..54d195712d --- /dev/null +++ b/__tests__/scala-kind-projector.test.ts @@ -0,0 +1,53 @@ +/** + * A Scala kind-projector placeholder in a type — `Align[Either[A, *]]`, + * `Functor[Map[K, ?]]` — names no method: cats' 567 `*` type arguments went + * to an algebra `Sign`'s `*` operator. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-scala-kp-')); + const files: Record = { + 'src/main/scala/algebra/Signed.scala': `package algebra + +object Signed { + sealed abstract class Sign(val toInt: Int) { + def *(that: Sign): Sign = this + } +} +`, + 'src/main/scala/cats/Align.scala': `package cats + +trait Align[F[_]] + +object Align { + implicit def catsAlignForEither[A]: Align[Either[A, *]] = null +} +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +describe('Scala kind-projector placeholders', () => { + it('are not references to a `*` method', () => { + const ids = cg.getNodesInFile('src/main/scala/cats/Align.scala').map((n) => n.id); + const targets = cg.getOutgoingEdgesFrom(ids).filter((e) => e.kind === 'references').map((e) => cg.getNode(e.target)!.name); + expect(targets).not.toContain('*'); + }); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index d9b6fb8148..5777451ec6 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -4308,7 +4308,9 @@ export function matchByExactName( // A Scala type position (`Arbitrary[B]`) never names a method: an // `implicit def A: Order[A]` shares its name with half of cats' type // parameters. Scala's value references only read a file's own vals. - !(ref.language === 'scala' && ref.referenceKind === 'references' && /^[A-Z]/.test(ref.referenceName) && + // Nor does a kind-projector placeholder (`F[*]`, `G[?]`): cats' 567 `*` + // type arguments went to an algebra `Sign`'s `*` method. + !(ref.language === 'scala' && ref.referenceKind === 'references' && /^(?:[A-Z]|[^\w\s]+$)/.test(ref.referenceName) && (n.kind === 'method' || n.kind === 'function')) && // Type/value references retain same-family eligibility: a native namesake // must not hide the actual web type. Calls still gate only the winner. @@ -8278,6 +8280,14 @@ function matchReferenceInner( return null; } + // A symbolic name in a Scala type is a type (`F ~> G`) or a kind-projector + // placeholder (`Either[A, *]`) — never an operator method, by any strategy. + if (ref.language === 'scala' && ref.referenceKind === 'references' && /^[^\w\s]+$/.test(ref.referenceName)) { + const types = context.getNodesByName(ref.referenceName).filter((n) => n.language === 'scala' && (SCALA_TYPE_KINDS.has(n.kind) || n.kind === 'type_alias')); + const chosen = types.length > 1 ? preferCallSiteFile(types, ref.filePath) : types; + return chosen.length === 1 ? { original: ref, targetNodeId: chosen[0]!.id, confidence: 0.8, resolvedBy: 'exact-match' } : null; + } + // A bare Lua call through a `local` alias reaches what the alias names. if ((ref.language === 'lua' || ref.language === 'luau') && ref.referenceKind === 'calls' && /^[A-Za-z_]\w*$/.test(ref.referenceName)) { const aliased = luaAliasTarget(ref, context); From e62032e95a5d0e025c022b587f090c4057810a3f Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 23:18:21 +0000 Subject: [PATCH 097/259] fix(python): a member value through an untyped receiver is a method only its owner names (#2218) A lone project method now stands for an untyped receiver's member value only when the receiver is named after the method's class: netbox's `device=self.parent.device` model-field reads no longer reach a GraphQL filter's `device` method. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/member-function-ref-owner.test.ts | 61 +++++++++++++++++++++ src/resolution/name-matcher.ts | 10 +++- 3 files changed, 70 insertions(+), 2 deletions(-) create mode 100644 __tests__/member-function-ref-owner.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 02b0bd5273..ad3be10988 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- In Python, an attribute passed as a value through an object nothing types, like netbox's `device=self.parent.device`, is no longer linked to the project's only method of that name when the object isn't named after the method's class. Before, netbox's model-field reads went to a GraphQL filter's `device` method, and healthchecks' `check.last_ping` went to a notification transport's. - In Scala, a kind-projector placeholder in a type, like `Align[Either[A, *]]`, is no longer linked to an operator method named `*`. A symbolic name in a type, like cats' `F ~> G`, now resolves only to a type. - C# type names now follow namespaces. A type is in reach from its own namespace or one nested inside it, or through a `using` of its namespace. Also counted: a project-wide `global using`, the project's `` entries, the SDK's implicit usings, or an alias that names it. Before, Newtonsoft's `async Task` tests linked `Task` to a test class of that name in another namespace, and AutoMapper's DTOs linked to another namespace's `BaseEntity`. - Rust type and function names now follow Rust's scoping. A name from another file counts only when a `use` brings it in (or a `*` glob over its module). One imported from outside the project, like `use std::task::{Context, Poll}` or `use futures::Stream`, means that outside item. A path names its module: `io::Result` isn't an unrelated `Result` alias elsewhere, and `std::io::Error` is std's. On tokio, over 1,200 `io::Result` and 600 `Context` references had been landing on private project types. diff --git a/__tests__/member-function-ref-owner.test.ts b/__tests__/member-function-ref-owner.test.ts new file mode 100644 index 0000000000..697057da83 --- /dev/null +++ b/__tests__/member-function-ref-owner.test.ts @@ -0,0 +1,61 @@ +/** + * A member passed as a value through a receiver nothing types — + * `device=self.parent.device`, `self.check.last_ping` — is only the + * project's one method of that name when the receiver is named after its + * owner: netbox's model-field reads bound to a GraphQL filter's `device` + * method, healthchecks' `check.last_ping` to a transport's. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-member-fnref-')); + const files: Record = { + 'app/filters.py': `class FHRPGroupAssignmentFilter: + def device(self, queryset): + return queryset +`, + 'app/components.py': `def describe(parent): + return "{interface}".format(interface=parent, device=parent.device) +`, + 'app/recorder.py': `class HookRecorder: + def finish_recording(self): + return None + + +def register(request, hook_recorder): + request.addfinalizer(hook_recorder.finish_recording) +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +const refsFrom = (file: string) => { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg.getOutgoingEdgesFrom(ids).filter((e) => e.kind === 'references').map((e) => cg.getNode(e.target)!.qualifiedName); +}; + +describe('member values through an untyped receiver', () => { + it('are not a method whose owner the receiver is not named after', () => { + expect(refsFrom('app/components.py')).not.toContain('FHRPGroupAssignmentFilter::device'); + }); + + it('are a method whose owner the receiver is named after', () => { + expect(refsFrom('app/recorder.py')).toContain('HookRecorder::finish_recording'); + }); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 5777451ec6..888ac17ddb 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -325,8 +325,14 @@ function matchMemberFunctionRef(ref: UnresolvedRef, context: ResolutionContext): } } // Unknown receivers retain the old unique-or-drop discipline, across ALL - // files. Tests and abstract-looking bodies are candidates too. - return result(context.getNodesByName(member), 0.8); + // files. Tests and abstract-looking bodies are candidates too. A lone method + // stands only when the receiver is named after its owner: netbox's + // `device=self.parent.device` is a model field, not the project's one + // `device` method (a GraphQL filter's). A veto, never a filter — filtering + // first would promote some other lone match into a new guess. + const unique = result(context.getNodesByName(member), 0.8); + const target = unique ? context.getNodeById?.(unique.targetNodeId) : null; + return target && target.kind === 'method' && !sharesReceiverWord(receiverLink(receiver), target) ? null : unique; } function pythonRefClass(name: string, ref: UnresolvedRef, context: ResolutionContext): Node | null { From 03123335b0e3ecffb44e627dd6a2205862d66271 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 23:33:28 +0000 Subject: [PATCH 098/259] fix(resolution): a name passed as a value is what is in scope where it is written (#2219) A function nested in another function is reachable only from inside it; a Python name the enclosing function binds is that local's value; a pytest fixture is a test's parameter only in its own module or under its conftest. Unreachable same-named functions still count against a lone cross-file guess. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/python-function-value-scope.test.ts | 121 ++++++++++++++++++ src/resolution/name-matcher.ts | 61 ++++++++- 3 files changed, 176 insertions(+), 7 deletions(-) create mode 100644 __tests__/python-function-value-scope.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index ad3be10988..1b5dd83c83 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- A function or name passed as a value now resolves to what is in scope where it's written. A function nested inside another function is only reachable from inside it. In Python, a parameter or local of the same name is that local. A pytest fixture is a test's parameter only in its own module or under its `conftest.py`. Before, httpx's `self._build_auth(auth)` linked to an `auth` a test defines inside another function, and `auth_flow(self, request)` handing `request` on linked to the package's `request()` function. - In Python, an attribute passed as a value through an object nothing types, like netbox's `device=self.parent.device`, is no longer linked to the project's only method of that name when the object isn't named after the method's class. Before, netbox's model-field reads went to a GraphQL filter's `device` method, and healthchecks' `check.last_ping` went to a notification transport's. - In Scala, a kind-projector placeholder in a type, like `Align[Either[A, *]]`, is no longer linked to an operator method named `*`. A symbolic name in a type, like cats' `F ~> G`, now resolves only to a type. - C# type names now follow namespaces. A type is in reach from its own namespace or one nested inside it, or through a `using` of its namespace. Also counted: a project-wide `global using`, the project's `` entries, the SDK's implicit usings, or an alias that names it. Before, Newtonsoft's `async Task` tests linked `Task` to a test class of that name in another namespace, and AutoMapper's DTOs linked to another namespace's `BaseEntity`. diff --git a/__tests__/python-function-value-scope.test.ts b/__tests__/python-function-value-scope.test.ts new file mode 100644 index 0000000000..d6152c78a0 --- /dev/null +++ b/__tests__/python-function-value-scope.test.ts @@ -0,0 +1,121 @@ +/** + * A name passed as a value means what is in scope where it is written. A + * function nested in another function is in scope only in there — httpx's + * `self._build_auth(auth)` passes its parameter, not the `auth` a test + * defines inside `test_custom_auth`. A Python name the function around it + * binds is that local's — `auth_flow(self, request)` handing `request` on is + * not the package's `request()`. And a pytest fixture is a test's parameter + * only in its own module or under its `conftest.py`. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-py-fnvalue-')); + const files: Record = { + 'httpx/_client.py': `class Client: + @property + def auth(self): + return self._auth + + def _build_auth(self, auth): + return auth + + def set_auth(self, auth): + self._auth = self._build_auth(auth) +`, + 'httpx/_api.py': `def request(method, url): + return method, url +`, + 'httpx/_auth.py': `class Auth: + def auth_flow(self, request): + yield request + + def sync_auth_flow(self, request): + flow = self.auth_flow(request) + return flow + + +class Recorder: + def request(self): + return None +`, + 'tests/test_auth.py': `def test_custom_auth(): + def auth(request): + return request + + return register(auth) + + +def test_basic_auth(): + auth = ("user", "pass") + return register(auth) +`, + 'docs/example/test_order.py': `import pytest + + +@pytest.fixture +def func(): + return 1 +`, + 'tests/test_coverage.py': `import pytest + + +def _run_both(func): + return func() + + +@pytest.fixture( + scope="module", +) +def recipe(): + return {} + + +def test_recipe(recipe): + return use(recipe) +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +const targetsFrom = (file: string, qualifiedName?: string) => { + const ids = cg.getNodesInFile(file).filter((n) => !qualifiedName || n.qualifiedName === qualifiedName).map((n) => n.id); + return cg.getOutgoingEdgesFrom(ids).filter((e) => e.kind === 'references' || e.kind === 'calls') + .map((e) => cg.getNode(e.target)!).map((t) => `${t.filePath}:${t.qualifiedName}`); +}; + +describe('Python names passed as values', () => { + it('never reach a function nested in another function', () => { + expect(targetsFrom('httpx/_client.py')).not.toContain('tests/test_auth.py:test_custom_auth::auth'); + expect(targetsFrom('tests/test_auth.py', 'test_basic_auth')).not.toContain('tests/test_auth.py:test_custom_auth::auth'); + }); + + it('reach a nested function from inside its container', () => { + expect(targetsFrom('tests/test_auth.py', 'test_custom_auth')).toContain('tests/test_auth.py:test_custom_auth::auth'); + }); + + it('are the local a parameter binds', () => { + expect(targetsFrom('httpx/_auth.py')).not.toContain('httpx/_api.py:request'); + }); + + it('reach a fixture only from its own module', () => { + expect(targetsFrom('tests/test_coverage.py', '_run_both')).not.toContain('docs/example/test_order.py:func'); + expect(targetsFrom('tests/test_coverage.py', 'test_recipe')).toContain('tests/test_coverage.py:recipe'); + }); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 888ac17ddb..e4f45f48a7 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -523,7 +523,7 @@ export function matchFunctionRef( }; } - let candidates = context + const named = context .getNodesByName(ref.referenceName) .filter( (n) => @@ -533,7 +533,19 @@ export function matchFunctionRef( sameLanguageFamily(n.language, ref.language) && n.id !== ref.fromNodeId // a function registering itself is not a dependency edge ); + // A function declared inside another is in scope only in there: httpx's + // `self._build_auth(auth)` passes its own parameter, not the `auth` a test + // defines inside `test_custom_auth`. Those still count against a lone + // cross-file guess below — a name several functions use for themselves is + // as likely a local's. + let candidates = named.filter((n) => isLexicallyReachable(n, ref, context)); if (candidates.length === 0) return null; + // A Python name the function around it binds — a parameter, an assignment — + // is that local's value: httpx's `auth_flow(self, request)` handing `request` + // on is not the package's `request()` function. A pytest fixture is what a + // test's parameter of its name receives. + if (ref.language === 'python' && !candidates.some((n) => isFixtureInReach(n, ref.filePath, context)) && + isPythonLocallyBound(ref.referenceName, ref, context)) return null; // Swift implicit-self: a bare identifier can name a METHOD only of the // ENCLOSING type (`Button(action: handleTap)` written inside that type) — @@ -593,8 +605,11 @@ export function matchFunctionRef( } // Cross-file (imported names the import resolver didn't already claim): - // only an unambiguous match resolves. - if (candidates.length === 1) { + // only an unambiguous match resolves — or, in Python, the one in reach of + // a name the file imports (netbox's `sender=CustomField` beside a test's + // own nested `CustomField`). + if (candidates.length === 1 && (named.length === 1 || + (ref.language === 'python' && pythonFromImports(ref.filePath, context).has(ref.referenceName)))) { return { original: ref, targetNodeId: candidates[0]!.id, @@ -1557,7 +1572,7 @@ function fitsPythonCallShape(n: Node, shape: PythonCallShape, ref: UnresolvedRef if (isPythonNameImportedFromOutside(ref.referenceName, ref, context)) return false; // `view = UserView.as_view()` … `view(request)`: the file's own value — // unless it is a pytest fixture, which a test takes as a parameter of that name. - return isPytestFixture(n) || !isPythonLocallyBound(ref.referenceName, ref, context); + return isFixtureInReach(n, ref.filePath, context) || !isPythonLocallyBound(ref.referenceName, ref, context); } // A member of what the chain names: a method of a class of that name, or a // function / class in a module of that name (`helpers.slugify()`). @@ -1573,9 +1588,41 @@ function fitsPythonCallShape(n: Node, shape: PythonCallShape, ref: UnresolvedRef return stem === shape.owner || (stem === '__init__' && parts[parts.length - 2] === shape.owner); } -/** A pytest fixture: `@pytest.fixture` / `@fixture`, or anything a `conftest.py` defines. */ -function isPytestFixture(n: Node): boolean { - return /(?:^|\/)conftest\.py$/.test(n.filePath) || (n.decorators ?? []).some((d) => /(?:^|\.)fixture\b/.test(d)); +/** + * A pytest fixture: `@pytest.fixture` / `@fixture`, or anything a `conftest.py` + * defines. Python decorators are not kept on the node, so they are read from + * the lines above its `def` (a decorator's arguments may span lines). + */ +function isPytestFixture(n: Node, context: ResolutionContext): boolean { + if (/(?:^|\/)conftest\.py$/.test(n.filePath) || (n.decorators ?? []).some((d) => /(?:^|\.)fixture\b/.test(d))) return true; + return isDecoratedFixture(n, context); +} + +/** + * A fixture a test at `filePath` can take by name: one its own module defines, + * or one a `conftest.py` of its directory or a parent does. A test module's + * fixture is that module's alone — pytest's `_run_both(func)` is handing on its + * parameter, not a doc example's `func` fixture. + */ +function isFixtureInReach(n: Node, filePath: string, context: ResolutionContext): boolean { + if (n.filePath === filePath) return isPytestFixture(n, context); + const conftest = /^(.*?)(?:^|\/)conftest\.py$/.exec(n.filePath); + return conftest !== null && (conftest[1] === '' || filePath.startsWith(`${conftest[1]}/`)); +} + +function isDecoratedFixture(n: Node, context: ResolutionContext): boolean { + if (n.language !== 'python' || n.kind !== 'function') return false; + const lines = context.getFileLines?.(n.filePath) ?? context.readFile(n.filePath)?.split(/\r?\n/) ?? []; + // Upward through the decorator lines: each one starts with `@`, or sits inside one's parentheses. + let open = 0; + for (let i = n.startLine - 2; i >= 0 && i >= n.startLine - 16; i--) { + const text = lines[i]?.trim() ?? ''; + open += (text.match(/\)/g)?.length ?? 0) - (text.match(/\(/g)?.length ?? 0); + if (open > 0) continue; + if (!text.startsWith('@')) return false; + if (/^@(?:\w+\.)*fixture\b/.test(text)) return true; + } + return false; } const PY_LOCAL_BINDS = new WeakMap>(); From ff75a6ca5f6a3ccadae43ed55b1edc5304e80259 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 23:56:34 +0000 Subject: [PATCH 099/259] fix(csharp): namespaces, usings and nested types follow the language's scopes (#2220) A block namespace qualifies only its own body (a nested one is Outer.Inner), mirrored in the kernel; a using names the project's namespace, never another file's using; a global using is its own project's; a nested type is reached by bare name only from inside its owner or a type deriving from it, through any partial part; visible C# types are filtered before ranking. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/csharp-namespace-scope.test.ts | 172 ++++++++++++++++++++ __tests__/fixtures/kernel-parity/Torture.cs | 5 + codegraph-kernel/src/csharp.rs | 75 ++++++--- src/extraction/languages/csharp.ts | 8 +- src/extraction/tree-sitter.ts | 26 +++ src/resolution/name-matcher.ts | 155 +++++++++++++++--- 7 files changed, 391 insertions(+), 51 deletions(-) create mode 100644 __tests__/csharp-namespace-scope.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 1b5dd83c83..fbb2017ead 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- C# names now follow the language's scopes. A block `namespace X { … }` qualifies only the types inside it, so a type declared after the block, or in a file's second namespace, is no longer filed under the first. A namespace written inside another is `Outer.Inner`. A `using` links to the project's namespace instead of another file's `using` of the same name. A `global using` applies only within its own project. A nested type is reachable by bare name only from inside its owner or a type deriving from it, including through another partial part. Before, serilog's `Guard` was filed under `JetBrains.Annotations`, its tests' `Some.InformationEvent()` went to the performance tests' `Some`, and AutoMapper's same-file `new Source()` could reach another test class's nested `Source`. - A function or name passed as a value now resolves to what is in scope where it's written. A function nested inside another function is only reachable from inside it. In Python, a parameter or local of the same name is that local. A pytest fixture is a test's parameter only in its own module or under its `conftest.py`. Before, httpx's `self._build_auth(auth)` linked to an `auth` a test defines inside another function, and `auth_flow(self, request)` handing `request` on linked to the package's `request()` function. - In Python, an attribute passed as a value through an object nothing types, like netbox's `device=self.parent.device`, is no longer linked to the project's only method of that name when the object isn't named after the method's class. Before, netbox's model-field reads went to a GraphQL filter's `device` method, and healthchecks' `check.last_ping` went to a notification transport's. - In Scala, a kind-projector placeholder in a type, like `Align[Either[A, *]]`, is no longer linked to an operator method named `*`. A symbolic name in a type, like cats' `F ~> G`, now resolves only to a type. diff --git a/__tests__/csharp-namespace-scope.test.ts b/__tests__/csharp-namespace-scope.test.ts new file mode 100644 index 0000000000..b185d3f33d --- /dev/null +++ b/__tests__/csharp-namespace-scope.test.ts @@ -0,0 +1,172 @@ +/** + * C# names follow the scopes the language gives them: + * - a block `namespace X { … }` holds only its body — serilog's Guard.cs + * declares `static class Guard` after a `namespace JetBrains.Annotations` + * block, and a file's second namespace (or one nested inside another, + * `Outer.Inner`) is its own; + * - `using X;` names the project's namespace X, never another file's using; + * - a `global using` is its own project's — serilog's two test projects each + * `global using` their own `Support` namespace, and both define `Some`; + * - a nested type is reached by bare name only from inside its owner or a + * type deriving from it, through any partial part. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-csharp-ns-')); + const files: Record = { + 'src/Lib/Lib.csproj': '\n', + 'src/Lib/Guard.cs': `using JetBrains.Annotations; + +namespace JetBrains.Annotations +{ + sealed class NoEnumerationAttribute : System.Attribute { } +} + +static class Guard +{ + public static T AgainstNull(T value) where T : class => value; +} +`, + 'src/Lib/Multi.cs': `namespace Alpha +{ + class X { } +} + +namespace Beta +{ + class Y { } + + namespace Inner + { + class Z { } + } +} +`, + 'src/Lib/Uses.cs': `using Beta; + +namespace Gamma +{ + class Consumer { } +} +`, + 'src/Lib/Reader.cs': `namespace Lib +{ + public abstract class Reader + { + internal enum State { Start, Done } + } + + public partial class TextReader : Reader + { + } +} +`, + 'src/Lib/TextReader.Async.cs': `namespace Lib +{ + public partial class TextReader + { + object Peek() => State.Start; + } +} +`, + 'src/Lib/Writer.cs': `namespace Lib +{ + public abstract class Writer + { + internal enum State { Start, Done } + + object Current() => State.Done; + } +} +`, + 'test/Unit/Unit.csproj': '\n', + 'test/Unit/GlobalUsings.cs': 'global using Unit.Support;\n', + 'test/Unit/Support/Some.cs': `namespace Unit.Support; + +static class Some +{ + public static int InformationEvent() => 1; +} +`, + 'test/Unit/LogTests.cs': `namespace Unit.Context; + +class LogTests +{ + int Run() => Some.InformationEvent(); +} +`, + 'test/Unit/MappingTests.cs': `namespace Unit.Mapping; + +class FirstCase +{ + class Source { } + object Make() => new Source(); +} + +class SecondCase +{ + class Source { } + object Make() => new Source(); +} +`, + 'test/Perf/Perf.csproj': '\n', + 'test/Perf/GlobalUsings.cs': 'global using Perf.Support;\n', + 'test/Perf/Support/Some.cs': `namespace Perf.Support; + +static class Some +{ + public static int InformationEvent() => 2; +} +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +const qualifiedNames = (file: string) => cg.getNodesInFile(file).filter((n) => n.kind !== 'file').map((n) => `${n.kind} ${n.qualifiedName}`); +const targetsAt = (file: string, line: number, kind?: string) => { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg.getOutgoingEdgesFrom(ids).filter((e) => e.line === line && (!kind || e.kind === kind)) + .map((e) => cg.getNode(e.target)!).map((t) => `${t.kind} ${t.filePath}:${t.qualifiedName}`); +}; + +describe('C# namespaces, usings and nested types', () => { + it('a block namespace qualifies only its own body', () => { + expect(qualifiedNames('src/Lib/Guard.cs')).toEqual(expect.arrayContaining([ + 'class JetBrains.Annotations::NoEnumerationAttribute', 'class Guard', 'method Guard::AgainstNull', + ])); + expect(qualifiedNames('src/Lib/Multi.cs')).toEqual(expect.arrayContaining([ + 'class Alpha::X', 'class Beta::Y', 'class Beta.Inner::Z', + ])); + }); + + it('a using names the project’s namespace', () => { + expect(targetsAt('src/Lib/Uses.cs', 1, 'imports')).toEqual(['namespace src/Lib/Multi.cs:Beta']); + }); + + it('a global using is its own project’s', () => { + expect(targetsAt('test/Unit/LogTests.cs', 5, 'calls')).toEqual(['method test/Unit/Support/Some.cs:Unit.Support::Some::InformationEvent']); + }); + + it('a nested type is its owner’s, or a derived type’s through any partial part', () => { + expect(targetsAt('test/Unit/MappingTests.cs', 12, 'instantiates')).toEqual(['class test/Unit/MappingTests.cs:Unit.Mapping::SecondCase::Source']); + expect(targetsAt('src/Lib/TextReader.Async.cs', 5)).toContain('enum src/Lib/Reader.cs:Lib::Reader::State'); + expect(targetsAt('src/Lib/Writer.cs', 7)).toContain('enum src/Lib/Writer.cs:Lib::Writer::State'); + }); +}); diff --git a/__tests__/fixtures/kernel-parity/Torture.cs b/__tests__/fixtures/kernel-parity/Torture.cs index 520b1d4e32..39e7743dbe 100644 --- a/__tests__/fixtures/kernel-parity/Torture.cs +++ b/__tests__/fixtures/kernel-parity/Torture.cs @@ -179,3 +179,8 @@ namespace Torture.Beta { public class Other { } } + +static class TopLevelGuard +{ + public static T AgainstNull(T value) where T : class => value; +} diff --git a/codegraph-kernel/src/csharp.rs b/codegraph-kernel/src/csharp.rs index aec1656f82..7d481662a7 100644 --- a/codegraph-kernel/src/csharp.rs +++ b/codegraph-kernel/src/csharp.rs @@ -205,34 +205,21 @@ pub fn extract(file_path: &str, source: &str) -> Result { w.node_ids.push(ids::file_node_id(file_path)); w.stack.push(Scope { row: 0, kind: "file", name: base_name.to_string() }); - // extractFilePackage: the FIRST top-level namespace declaration mints ONE - // `namespace` node that stays pushed for the ENTIRE file — a second - // top-level namespace's types nest under the first's node/QN, nested - // namespaces leave no trace, and every import ref in a namespaced file - // hangs off this node (checklist §namespace). + // extractFilePackage: a top-level file-scoped `namespace X;` mints ONE + // `namespace` node that stays pushed for the ENTIRE file, so every import + // ref in the file hangs off it (checklist §namespace). A block namespace + // scopes only its own body — visit_node. let root = tree.root_node(); let mut pkg_pushed = false; for i in 0..root.named_child_count() { let Some(child) = root.named_child(i) else { continue }; - if child.kind() != "namespace_declaration" - && child.kind() != "file_scoped_namespace_declaration" - { + if child.kind() != "file_scoped_namespace_declaration" { continue; } - // csharpExtractor.extractPackage: `name` field ?? first - // qualified_name/identifier named child. No trim. - let name_node = child.child_by_field_name("name").or_else(|| { - (0..child.named_child_count()) - .filter_map(|j| child.named_child(j)) - .find(|c| matches!(c.kind(), "qualified_name" | "identifier")) - }); - if let Some(name_node) = name_node { - let pkg = w.text(name_node).to_string(); - if !pkg.is_empty() { - if let Some(row) = w.create_node("namespace", &pkg, child, Extra::default()) { - w.stack.push(Scope { row, kind: "namespace", name: pkg }); - pkg_pushed = true; - } + if let Some(pkg) = namespace_name(&w, child) { + if let Some(row) = w.create_node("namespace", &pkg, child, Extra::default()) { + w.stack.push(Scope { row, kind: "namespace", name: pkg }); + pkg_pushed = true; } } break; @@ -265,6 +252,18 @@ fn record_is_struct(node: Node) -> bool { .any(|c| c.kind() == "struct") } +/// csharpExtractor.extractPackage: `name` field ?? first qualified_name / +/// identifier named child. No trim; None when empty. +fn namespace_name(w: &Walker, node: Node) -> Option { + let name_node = node.child_by_field_name("name").or_else(|| { + (0..node.named_child_count()) + .filter_map(|j| node.named_child(j)) + .find(|c| matches!(c.kind(), "qualified_name" | "identifier")) + })?; + let name = w.text(name_node); + (!name.is_empty()).then(|| name.to_string()) +} + impl<'t> Walker<'t> { fn text(&self, node: Node) -> &'t str { &self.src[node.byte_range()] @@ -518,6 +517,38 @@ impl<'t> Walker<'t> { let kind = node.kind(); let mut skip_children = false; + // A block namespace scopes only its body (tree-sitter.ts visitNode): + // one written inside another is `Outer.Inner`, taking the outer's + // place on the stack while its body is walked. + if kind == "namespace_declaration" { + if let Some(ns_name) = namespace_name(self, node) { + let outer = match self.stack.last() { + Some(top) if top.kind == "namespace" => self.stack.pop(), + _ => None, + }; + let full = match &outer { + Some(o) => format!("{}.{}", o.name, ns_name), + None => ns_name, + }; + let row = self.create_node("namespace", &full, node, Extra::default()); + if let Some(row) = row { + self.stack.push(Scope { row, kind: "namespace", name: full }); + } + for i in 0..node.named_child_count() { + if let Some(c) = node.named_child(i) { + self.visit_node(c); + } + } + if row.is_some() { + self.stack.pop(); + } + if let Some(o) = outer { + self.stack.push(o); + } + return; + } + } + self.maybe_capture_fn_refs(node); if kind == "class_declaration" || kind == "record_declaration" { diff --git a/src/extraction/languages/csharp.ts b/src/extraction/languages/csharp.ts index a43539a7fe..a88b01f817 100644 --- a/src/extraction/languages/csharp.ts +++ b/src/extraction/languages/csharp.ts @@ -82,10 +82,10 @@ export const csharpExtractor: LanguageExtractor = { typeAliasTypes: [], // Namespaces qualify type names so same-named types in different namespaces are // distinguishable (e.g. `ApplicationCore.Entities.CatalogBrand` vs - // `BlazorShared.Models.CatalogBrand`). Both block (`namespace Foo { … }`, which - // nests its types) and file-scoped (`namespace Foo;`) forms — extractFilePackage - // pushes the namespace onto the scope so nested/top-level types pick it up. - packageTypes: ['namespace_declaration', 'file_scoped_namespace_declaration'], + // `BlazorShared.Models.CatalogBrand`). A file-scoped `namespace Foo;` covers the + // whole file (extractFilePackage); a block `namespace Foo { … }` only its body + // (the visitor scopes it). + packageTypes: ['file_scoped_namespace_declaration'], extractPackage: (node: SyntaxNode, source: string) => { const name = node.childForFieldName('name') ?? diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index 51086c3bfd..7c9baa69d0 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -1140,6 +1140,32 @@ export class TreeSitterExtractor { } } + // C# block namespaces scope only their own body: serilog's Guard.cs opens + // `namespace JetBrains.Annotations { … }` and then declares `static class + // Guard` at the top level, and a file's second namespace is its own. A + // namespace written inside another is `Outer.Inner` — the dotted name a + // type's qualifiedName leads with — so it takes the outer's place on the + // scope while its body is walked. (A file-scoped `namespace X;` covers + // the whole file: extractFilePackage.) Mirrored in the kernel (csharp.rs). + if (this.language === 'csharp' && nodeType === 'namespace_declaration') { + const nsName = this.extractor.extractPackage?.(node, this.source); + if (nsName) { + const topId = this.nodeStack[this.nodeStack.length - 1]; + const top = this.nodes.find((n) => n.id === topId); + const outer = top?.kind === 'namespace' ? top : null; + if (outer) this.nodeStack.pop(); + const ns = this.createNode('namespace', outer ? `${outer.name}.${nsName}` : nsName, node); + if (ns) this.nodeStack.push(ns.id); + for (let i = 0; i < node.namedChildCount; i++) { + const child = node.namedChild(i); + if (child) this.visitNode(child); + } + if (ns) this.nodeStack.pop(); + if (outer) this.nodeStack.push(outer.id); + return; + } + } + // Function-as-value capture (#756) — independent of the dispatch ladder // below (the captured container types have no other handler there), so it // can never shadow or be shadowed by an extraction branch. diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index e4f45f48a7..83dbda2453 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -1339,7 +1339,8 @@ export function isVisibleAcrossFiles(candidate: Node, ref: UnresolvedRef, contex // not a test entity `Record`, tokio's `Output` not a `runtime/tests` type. if (isTestSuitePath(candidate.filePath) && !isTestPath(ref.filePath)) return false; if (candidate.language === 'csharp' && ref.language === 'csharp' && CSHARP_TYPE_KINDS.has(candidate.kind) && - /^[A-Za-z_]\w*$/.test(ref.referenceName) && !isCsharpTypeVisible(candidate, ref, context)) return false; + /^[A-Za-z_]\w*$/.test(ref.referenceName) && + (!isCsharpTypeVisible(candidate, ref, context) || !isCsharpNestedTypeInScope(candidate, ref, context))) return false; const lang = candidate.language as string; if (lang === 'c' || lang === 'cpp') { return ( @@ -3265,12 +3266,13 @@ function csharpProjectStaticUsings(dir: string, context: ResolutionContext, memo } const CSHARP_NAMESPACE_SCOPES = new WeakMap; aliases: Map }>>(); -const CSHARP_PROJECT_USINGS = new WeakMap>>(); +const CSHARP_PROJECT_USINGS = new WeakMap; project: boolean }>>(); /** * The namespaces a C# file's code runs in and the ones it imports: its - * `namespace` declarations, its `using X;`, every file's `global using X;` - * and the `` of the project files above it. + * `namespace` declarations, its `using X;`, the `global using X;` of its + * project's files (every file's, outside any project) and the + * `` of the project files above it. */ function csharpNamespaceScope(file: string, context: ResolutionContext): { namespaces: string[]; usings: Set; aliases: Map } { let memo = CSHARP_NAMESPACE_SCOPES.get(context); @@ -3281,7 +3283,9 @@ function csharpNamespaceScope(file: string, context: ResolutionContext): { names const namespaces = [...text.matchAll(/^\s*namespace\s+([\w.]+)/gm)].map((m) => m[1]!); let projectMemo = CSHARP_PROJECT_USINGS.get(context); if (!projectMemo) CSHARP_PROJECT_USINGS.set(context, (projectMemo = new Map())); - const usings = new Set(csharpProjectUsings(path.posix.dirname(file), context, projectMemo)); + const project = csharpProjectUsings(path.posix.dirname(file), context, projectMemo); + const usings = new Set(project.usings); + if (!project.project) for (const u of csharpGlobalUsings('.', context)) usings.add(u); const aliases = new Map(); for (const m of text.matchAll(/^\s*(?:global\s+)?using\s+(?!static\b)(?:([A-Za-z_]\w*)\s*=\s*)?([\w.]+)\s*;/gm)) { if (m[1]) aliases.set(m[1], m[2]!); @@ -3292,42 +3296,65 @@ function csharpNamespaceScope(file: string, context: ResolutionContext): { names return scope; } -/** `global using X;` from any file, and `` of the project files from `dir` up. */ -function csharpProjectUsings(dir: string, context: ResolutionContext, memo: Map>): Set { +/** + * The `` of the project files from `dir` up, and the + * `global using X;` of each project's own files — a global using is its + * project's alone: serilog's Serilog.Tests and Serilog.PerformanceTests each + * `global using` their own `Support` namespace, and both define `Some`. + * `project` says whether a `.csproj` sits at `dir` or above it. + */ +function csharpProjectUsings(dir: string, context: ResolutionContext, memo: Map; project: boolean }>): { usings: Set; project: boolean } { const key = dir; const hit = memo.get(key); if (hit) return hit; - let usings: Set; - if (dir === '.' || dir === '' || dir === '/') { - usings = new Set(); - for (const f of context.getAllFiles()) { - if (!f.endsWith('.cs') || (context.fileContains && !context.fileContains(f, 'global using'))) continue; - for (const m of (context.readFile(f) ?? '').matchAll(/^\s*global\s+using\s+(?!static\b)([\w.]+)\s*;/gm)) usings.add(m[1]!); - } - } else { - usings = new Set(csharpProjectUsings(path.posix.dirname(dir), context, memo)); - } + const root = dir === '.' || dir === '' || dir === '/'; + const parent = root ? null : csharpProjectUsings(path.posix.dirname(dir), context, memo); + const usings = new Set(parent?.usings ?? []); + let project = parent?.project ?? false; let entries: string[] = []; try { - entries = fs.readdirSync(path.join(context.getProjectRoot(), dir === '.' ? '' : dir)); + entries = fs.readdirSync(path.join(context.getProjectRoot(), root ? '' : dir)); } catch { entries = []; } for (const entry of entries) { if (!/\.(?:csproj|props)$/i.test(entry)) continue; - let project = ''; + let text = ''; try { - project = fs.readFileSync(path.join(context.getProjectRoot(), dir === '.' ? '' : dir, entry), 'utf8'); + text = fs.readFileSync(path.join(context.getProjectRoot(), root ? '' : dir, entry), 'utf8'); } catch { continue; } - for (const m of project.matchAll(/]*\bStatic\s*=\s*"true")[^>]*>/gi)) usings.add(m[1]!); + if (/\.csproj$/i.test(entry) && !project) { + project = true; + for (const u of csharpGlobalUsings(root ? '.' : dir, context)) usings.add(u); + } + for (const m of text.matchAll(/]*\bStatic\s*=\s*"true")[^>]*>/gi)) usings.add(m[1]!); // The SDK's implicit usings (serilog's own `System.TimeProvider` polyfill is seen through them). - if (/\s*(?:enable|true)\s*<\/ImplicitUsings>/i.test(project)) { + if (/\s*(?:enable|true)\s*<\/ImplicitUsings>/i.test(text)) { for (const ns of CSHARP_IMPLICIT_USINGS) usings.add(ns); } } - memo.set(key, usings); + const result = { usings, project }; + memo.set(key, result); + return result; +} + +const CSHARP_GLOBAL_USINGS = new WeakMap>>(); + +/** The `global using X;` of the `.cs` files under `dir` (`.` = the whole repository). */ +function csharpGlobalUsings(dir: string, context: ResolutionContext): Set { + let memo = CSHARP_GLOBAL_USINGS.get(context); + if (!memo) CSHARP_GLOBAL_USINGS.set(context, (memo = new Map())); + const hit = memo.get(dir); + if (hit) return hit; + const usings = new Set(); + const prefix = dir === '.' ? '' : `${dir}/`; + for (const f of context.getAllFiles()) { + if (!f.endsWith('.cs') || !f.startsWith(prefix) || (context.fileContains && !context.fileContains(f, 'global using'))) continue; + for (const m of (context.readFile(f) ?? '').matchAll(/^\s*global\s+using\s+(?!static\b)([\w.]+)\s*;/gm)) usings.add(m[1]!); + } + memo.set(dir, usings); return usings; } @@ -3356,6 +3383,60 @@ function isCsharpTypeVisible(n: Node, ref: UnresolvedRef, context: ResolutionCon return scope.usings.has(ns); } +/** + * Whether a bare C# name can reach `n` as a nested type: only from inside the + * type that declares it (any partial part, any depth) or a class deriving from + * it. AutoMapper's tests each declare their own nested `Source`, and a + * same-file `new Source()` went to whichever test class came first. + */ +function isCsharpNestedTypeInScope(n: Node, ref: UnresolvedRef, context: ResolutionContext): boolean { + const cut = n.qualifiedName.lastIndexOf('::'); + if (cut < 0) return true; + const owner = n.qualifiedName.slice(0, cut); + const ownerType = context.getNodesInFile(n.filePath).find((p) => CSHARP_TYPE_KINDS.has(p.kind) && p.qualifiedName === owner); + // Declared in a namespace, not a type. + if (!ownerType) return true; + const enclosing = context.getNodesInFile(ref.filePath) + .filter((p) => CSHARP_TYPE_KINDS.has(p.kind) && p.startLine <= ref.line && p.endLine >= ref.line); + if (enclosing.some((p) => p.qualifiedName === owner || p.qualifiedName.startsWith(`${owner}::`))) return true; + // Inherited: a type around the ref derives from the owner (`class SourceA : + // Source`), through any partial part — Newtonsoft's JsonTextReader.Async.cs + // is `partial class JsonTextReader` with no base list, reading JsonReader's `State`. + return enclosing.some((p) => csharpAncestorNames(p.qualifiedName, context).has(ownerType.name)); +} + +const CSHARP_ANCESTORS = new WeakMap>>(); + +/** The simple names of the C# types `qn` derives from, through every partial part and base, a few levels up. */ +function csharpAncestorNames(qn: string, context: ResolutionContext, depth = 0): Set { + let memo = CSHARP_ANCESTORS.get(context); + if (!memo) CSHARP_ANCESTORS.set(context, (memo = new Map())); + const hit = memo.get(qn); + if (hit) return hit; + const names = new Set(); + memo.set(qn, names); // a cycle reads what is gathered so far + for (const decl of context.getNodesByQualifiedName(qn)) { + if (decl.language !== 'csharp' || !CSHARP_TYPE_KINDS.has(decl.kind)) continue; + const lines = context.getFileLines?.(decl.filePath) ?? context.readFile(decl.filePath)?.split(/\r?\n/) ?? []; + let header = ''; + for (let i = decl.startLine - 1; i < Math.min(lines.length, decl.startLine + 6) && !header.includes('{'); i++) header += `${lines[i] ?? ''} `; + const list = /:\s*([^{;]*)/.exec(header.split('{')[0]!.replace(/\bwhere\b[\s\S]*$/, ''))?.[1] ?? ''; + for (const base of splitCppTopLevel(list)) { + const name = /([A-Za-z_]\w*)\s*(?:<.*)?$/.exec(base.trim())?.[1]; + if (name) names.add(name); + } + } + if (depth < 4) { + for (const base of [...names]) { + for (const t of context.getNodesByName(base)) { + if (t.language !== 'csharp' || !CSHARP_TYPE_KINDS.has(t.kind) || t.qualifiedName === qn) continue; + for (const up of csharpAncestorNames(t.qualifiedName, context, depth + 1)) names.add(up); + } + } + } + return names; +} + const OBJC_SUPERS = new WeakMap>(); const OBJC_MEMBER_KINDS: ReadonlySet = new Set(['method', 'property', 'field']); @@ -4395,6 +4476,13 @@ export function matchByExactName( (!importRef || isImportableKind(n.kind)) && // Nested locals are only reachable from inside their container (#1230). isLexicallyReachable(n, ref, context) && + // A C# type name is a type its namespaces can see — ahead of the ranking, + // so a visible namesake wins where the veto after it would drop the + // ref: eShop's `WebhookType.OrderPaid` under `using Webhooks.API.Model;`. + // A nested type, only from inside its owner: AutoMapper's same-file `new + // Source()` in one test class is not the previous test class's `Source`. + !(ref.language === 'csharp' && n.language === 'csharp' && CSHARP_TYPE_KINDS.has(n.kind) && /^[A-Za-z_]\w*$/.test(ref.referenceName) && + (!isCsharpNestedTypeInScope(n, ref, context) || (n.filePath !== ref.filePath && !isCsharpTypeVisible(n, ref, context)))) && // Preserve import ranking; calls reject the winner without promoting another. (!importRef || n.filePath === ref.filePath || !ESM_FAMILY.has(n.language) || !isSealedModule(n.filePath, context)) && @@ -4474,7 +4562,14 @@ export function matchByQualifiedName( ) : nodes; - const candidates = keepForRef(context.getNodesByQualifiedName(ref.referenceName)); + let candidates = keepForRef(context.getNodesByQualifiedName(ref.referenceName)); + // A C# `using X.Y;` names a namespace: one the project declares, else it is + // the file's own (external) using — never another file's using of that name. + if (ref.language === 'csharp' && ref.referenceKind === 'imports') { + const namespaces = candidates.filter((n) => n.kind === 'namespace'); + candidates = namespaces.length > 0 ? preferCallSiteFile(namespaces, ref.filePath).slice(0, 1) + : candidates.filter((n) => n.kind !== 'import' || n.filePath === ref.filePath); + } if (candidates.length === 1) { return { @@ -5508,6 +5603,8 @@ export function clearNameMatcherMemos(context: ResolutionContext): void { CSHARP_STATIC_USINGS.delete(context); CSHARP_NAMESPACE_SCOPES.delete(context); CSHARP_PROJECT_USINGS.delete(context); + CSHARP_GLOBAL_USINGS.delete(context); + CSHARP_ANCESTORS.delete(context); SCALA_SUPERS.delete(context); SCALA_IMPORTS.delete(context); ESM_EXPORT_LISTS.delete(context); @@ -6756,10 +6853,18 @@ export function matchMethodCall( // own file first — otherwise the first-indexed class wins and a call in `b/` // resolves to `a/`'s method (#1079). const strat1 = nmTimedT('mc-class', ref, (): ResolvedRef | null => { - const classCandidates = preferCallSiteFile( + let classCandidates = preferCallSiteFile( context.getNodesByName(objectOrClass!).filter(isMethodOwnerKind), ref.filePath, ); + // A C# class the call's namespaces can see before one they can't: + // serilog's `Some.InformationEvent()` in Serilog.Tests is its own + // Support namespace's `Some`, not the performance tests'. + if (ref.language === 'csharp' && classCandidates.length > 1) { + const typeRef = { ...ref, referenceName: objectOrClass! }; + const visible = classCandidates.filter((c) => c.language !== 'csharp' || isCsharpTypeVisible(c, typeRef, context)); + classCandidates = [...visible, ...classCandidates.filter((c) => !visible.includes(c))]; + } for (const classNode of classCandidates) { // Skip cross-language class matches From ff190b30d8112f4aa6a5848138831468495cee0a Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 1 Oct 2026 00:09:37 +0000 Subject: [PATCH 100/259] fix(resolution): a method handing its call on to another object is not calling itself (#2221) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A TS/JS call written through this.… or window.… reaches the resolver by its bare name; it resolves to the calling method only when the field is declared as the caller's own class. A name-scored receiver guess never lands on the calling method. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/self-call-delegation.test.ts | 99 ++++++++++++++++++++++++++ src/resolution/name-matcher.ts | 56 ++++++++++++++- 3 files changed, 155 insertions(+), 1 deletion(-) create mode 100644 __tests__/self-call-delegation.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index fbb2017ead..e2b990520e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- A method that hands its call on to another object is no longer linked to itself. In TypeScript and JavaScript, a call through `this.` or `window.` counts as recursion only when the field is declared as the method's own class. A guess from a receiver's name alone never lands on the calling method. Before, BookStack's `toggle()` doing `this.container.classList.toggle('open')`, `listen()` doing `window.$events.listen(…)`, and `FileStorage::delete` doing `$storage->delete($path)` each pointed at themselves. - C# names now follow the language's scopes. A block `namespace X { … }` qualifies only the types inside it, so a type declared after the block, or in a file's second namespace, is no longer filed under the first. A namespace written inside another is `Outer.Inner`. A `using` links to the project's namespace instead of another file's `using` of the same name. A `global using` applies only within its own project. A nested type is reachable by bare name only from inside its owner or a type deriving from it, including through another partial part. Before, serilog's `Guard` was filed under `JetBrains.Annotations`, its tests' `Some.InformationEvent()` went to the performance tests' `Some`, and AutoMapper's same-file `new Source()` could reach another test class's nested `Source`. - A function or name passed as a value now resolves to what is in scope where it's written. A function nested inside another function is only reachable from inside it. In Python, a parameter or local of the same name is that local. A pytest fixture is a test's parameter only in its own module or under its `conftest.py`. Before, httpx's `self._build_auth(auth)` linked to an `auth` a test defines inside another function, and `auth_flow(self, request)` handing `request` on linked to the package's `request()` function. - In Python, an attribute passed as a value through an object nothing types, like netbox's `device=self.parent.device`, is no longer linked to the project's only method of that name when the object isn't named after the method's class. Before, netbox's model-field reads went to a GraphQL filter's `device` method, and healthchecks' `check.last_ping` went to a notification transport's. diff --git a/__tests__/self-call-delegation.test.ts b/__tests__/self-call-delegation.test.ts new file mode 100644 index 0000000000..d23bcae87d --- /dev/null +++ b/__tests__/self-call-delegation.test.ts @@ -0,0 +1,99 @@ +/** + * A method that hands its call on to another object — a wrapper — is not + * calling itself. TS/JS calls through `this.…` or `window.…` reach + * the resolver by their bare name, so BookStack's `toggle()` doing + * `this.container.classList.toggle('open')` and `listen()` doing + * `window.$events.listen(…)` bound to themselves; a guess from a receiver's + * name alone (`FileStorage::delete` doing `$storage->delete($path)`) did too. + * A recursion through a field the class declares as its own type stays. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-self-call-')); + const files: Record = { + 'resources/js/editor-toolbox.ts': `export class EditorToolbox { + container: HTMLElement; + + toggle(): void { + this.container.classList.toggle('open'); + this.toggle(); + } +} +`, + 'resources/js/tree.ts': `export class TreeNode { + left: TreeNode | null = null; + + insert(v: number): void { + this.left.insert(v); + } +} +`, + 'resources/js/common-events.js': `export function listen(editor) { + window.$events.listen('editor::replace', () => editor.reset()); +} +`, + 'app/Uploads/FileStorage.php': `getStorageDisk(); + $storage->delete($path); + } + + protected function getStorageDisk() + { + return null; + } +} +`, + 'app/Uploads/ImageStorage.php': ` { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +const selfCallLines = (file: string) => { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg.getOutgoingEdgesFrom(ids).filter((e) => e.kind === 'calls' && e.source === e.target).map((e) => e.line); +}; + +describe('a call a method hands on', () => { + it('through a field of another type is not the method itself', () => { + expect(selfCallLines('resources/js/editor-toolbox.ts')).toEqual([6]); + expect(selfCallLines('resources/js/common-events.js')).toEqual([]); + }); + + it('through a field of the class’s own type is a recursion', () => { + expect(selfCallLines('resources/js/tree.ts')).toEqual([5]); + }); + + it('through a receiver named like the caller’s class is not the method itself', () => { + expect(selfCallLines('app/Uploads/FileStorage.php')).toEqual([]); + }); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 83dbda2453..e12f46169f 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -1402,6 +1402,52 @@ export const CASE_INSENSITIVE_LANGUAGES = new Set(['php', 'pascal', 'cfm * enclosing method itself least of all, which the same-file proximity term * used to pick over the module-scope function the call actually means. */ +/** + * The links between `this` / `super` / `window` and the method of a TS/JS + * call the extractor recorded by its bare name — `this.container.classList + * .toggle()` reaches the resolver as `toggle`, with links `container`, + * `classList`. Null for a call not written that way, `this.toggle()` included. + */ +function collapsedJsChain(ref: UnresolvedRef, context: ResolutionContext): { root: string; links: string[] } | null { + if (!JS_FAMILY.has(ref.language) || ref.referenceKind !== 'calls' || !/^[A-Za-z_$][\w$]*$/.test(ref.referenceName)) return null; + const lines = context.getFileLines?.(ref.filePath) ?? context.readFile(ref.filePath)?.split(/\r?\n/); + if (!lines) return null; + const text = lines.slice(ref.line - 1, ref.line + 3).join('\n').slice(ref.column); + const name = ref.referenceName.replace(/\$/g, '\\$'); + const m = new RegExp(`^(this|super|window)((?:\\s*\\??\\.\\s*#?[\\w$]+(?:\\([^()]*\\))?)+?)\\s*\\??\\.\\s*${name}\\s*(?:<[^<>()]*>)?\\s*\\(`).exec(text); + if (!m) return null; + return { root: m[1]!, links: m[2]!.split('.').map((l) => l.replace(/[\s?]/g, '')).filter((l) => l !== '') }; +} + +/** + * Whether a collapsed `this..m()` inside `m` can be `m` itself: only + * when the class declares the field as its own type — a tree node's + * `this.left.insert(v)` — never through a field of another type + * (`this.editor.input.focus()`, `this.container.classList.toggle()`). + */ +function isCollapsedNonRecursion(ref: UnresolvedRef, context: ResolutionContext): boolean { + const chain = collapsedJsChain(ref, context); + return chain !== null && !isCollapsedSelfRecursion(chain, ref, context); +} + +function isCollapsedSelfRecursion(chain: { root: string; links: string[] }, ref: UnresolvedRef, context: ResolutionContext): boolean { + if (chain.root !== 'this' || chain.links.length !== 1 || chain.links[0]!.includes('(')) return false; + const field = chain.links[0]!.replace(/^#/, ''); + const caller = context.getNodeById?.(ref.fromNodeId); + const cut = caller ? caller.qualifiedName.lastIndexOf('::') : -1; + if (!caller || cut <= 0) return false; + const owner = caller.qualifiedName.slice(0, cut).split('::').pop()!; + const cls = context.getNodesInFile(ref.filePath).find((n) => + n.kind === 'class' && n.name === owner && n.startLine <= ref.line && n.endLine >= ref.line); + if (!cls) return false; + const lines = context.getFileLines?.(ref.filePath) ?? context.readFile(ref.filePath)?.split(/\r?\n/) ?? []; + const body = lines.slice(cls.startLine - 1, cls.endLine).join('\n'); + const f = field.replace(/\$/g, '\\$'); + const declared = new RegExp(`(?:^|[\\s(,])#?${f}\\s*[?!]?\\s*:\\s*([A-Za-z_$][\\w$]*)`, 'm').exec(body)?.[1] ?? + new RegExp(`\\bthis\\.${f}\\s*=\\s*new\\s+([A-Za-z_$][\\w$]*)`).exec(body)?.[1]; + return declared === owner; +} + function isBareJsCall(ref: UnresolvedRef, context: ResolutionContext): boolean { return JS_FAMILY.has(ref.language) && isReceiverLessCall(ref, context); } @@ -7101,7 +7147,11 @@ export function matchMethodCall( } } - if (bestMatch && bestScore >= 2) { + // A wrapper handing its call on — BookStack's `FileStorage::delete` doing + // `$storage->delete($path)`, `CommentRepo::delete` doing + // `$comment->delete()` — names the caller's own class only by a shared + // word. The guess is the caller itself, so there is no guess. + if (bestMatch && bestScore >= 2 && bestMatch.id !== ref.fromNodeId) { return { original: ref, targetNodeId: bestMatch.id, @@ -8328,6 +8378,10 @@ export function matchReference( context: ResolutionContext ): ResolvedRef | null { const result = gateLanguageMatch(matchReferenceInner(ref, context), ref, context); + // `this.container.classList.toggle()` inside `toggle()`, `window.$events + // .listen()` inside `listen()`: a member of what the chain reaches, which is + // the calling method only through a field of the caller's own type. + if (result && result.targetNodeId === ref.fromNodeId && isCollapsedNonRecursion(ref, context)) return null; return result ? retargetSelfOverload(result, ref, context) : result; } From d6f68648cb3c83712a66206a07118b975b9bf8fe Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 1 Oct 2026 00:16:23 +0000 Subject: [PATCH 101/259] fix(spring): name heuristics start from the reference's own file and package (#2222) The Spring resolver's entity/service/controller name heuristics took the first same-named class under a conventional folder; they now prefer the ref's own file, then its package, and never reach another file's nested class by its bare name. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/spring-model-name-scope.test.ts | 64 +++++++++++++++++++++++ src/resolution/frameworks/java.ts | 36 ++++++++++--- 3 files changed, 93 insertions(+), 8 deletions(-) create mode 100644 __tests__/spring-model-name-scope.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index e2b990520e..c6e97b3384 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- In Spring projects, a class name now resolves to the one in the reference's own file or package before any guess by folder convention. A class nested inside another file's class is never reached by its bare name. Before, every MyBatis `Example` class's `new Criteria()` in mall linked to the first `Example`'s nested `Criteria`, and halo's Lombok `@Builder` linked to a nested `Builder` class elsewhere. - A method that hands its call on to another object is no longer linked to itself. In TypeScript and JavaScript, a call through `this.` or `window.` counts as recursion only when the field is declared as the method's own class. A guess from a receiver's name alone never lands on the calling method. Before, BookStack's `toggle()` doing `this.container.classList.toggle('open')`, `listen()` doing `window.$events.listen(…)`, and `FileStorage::delete` doing `$storage->delete($path)` each pointed at themselves. - C# names now follow the language's scopes. A block `namespace X { … }` qualifies only the types inside it, so a type declared after the block, or in a file's second namespace, is no longer filed under the first. A namespace written inside another is `Outer.Inner`. A `using` links to the project's namespace instead of another file's `using` of the same name. A `global using` applies only within its own project. A nested type is reachable by bare name only from inside its owner or a type deriving from it, including through another partial part. Before, serilog's `Guard` was filed under `JetBrains.Annotations`, its tests' `Some.InformationEvent()` went to the performance tests' `Some`, and AutoMapper's same-file `new Source()` could reach another test class's nested `Source`. - A function or name passed as a value now resolves to what is in scope where it's written. A function nested inside another function is only reachable from inside it. In Python, a parameter or local of the same name is that local. A pytest fixture is a test's parameter only in its own module or under its `conftest.py`. Before, httpx's `self._build_auth(auth)` linked to an `auth` a test defines inside another function, and `auth_flow(self, request)` handing `request` on linked to the package's `request()` function. diff --git a/__tests__/spring-model-name-scope.test.ts b/__tests__/spring-model-name-scope.test.ts new file mode 100644 index 0000000000..d8d60c7c29 --- /dev/null +++ b/__tests__/spring-model-name-scope.test.ts @@ -0,0 +1,64 @@ +/** + * The Spring resolver's name heuristics (an entity under `/model/`, a + * `…Service`, a `…Controller`) start from what the reference's own scope + * declares — its file, then its package — and never reach a class nested in + * another file's class by its bare name: every MyBatis `XExample` declares its + * own nested `Criteria`, and mall's `new Criteria()` all went to the first. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +const example = (name: string) => `package com.macro.mall.model; + +public class ${name} { + public Criteria createCriteria() { + Criteria criteria = new Criteria(); + return criteria; + } + + public static class Criteria { + } +} +`; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-spring-model-')); + const files: Record = { + 'pom.xml': 'spring-boot-starter\n', + 'mall-mbg/src/main/java/com/macro/mall/model/CmsHelpCategoryExample.java': example('CmsHelpCategoryExample'), + 'mall-mbg/src/main/java/com/macro/mall/model/OmsOrderExample.java': example('OmsOrderExample'), + 'mall-admin/src/main/java/com/macro/mall/service/OrderService.java': `package com.macro.mall.service; + +@Service +public class OrderService { +} +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +describe('Spring name heuristics', () => { + it('resolve a name to its own file’s nested class, not another file’s', () => { + const file = 'mall-mbg/src/main/java/com/macro/mall/model/OmsOrderExample.java'; + const ids = cg.getNodesInFile(file).map((n) => n.id); + const targets = cg.getOutgoingEdgesFrom(ids).filter((e) => e.kind === 'instantiates' || e.kind === 'references') + .map((e) => cg.getNode(e.target)!).filter((t) => t.name === 'Criteria').map((t) => t.filePath); + expect(targets.length).toBeGreaterThan(0); + expect(new Set(targets)).toEqual(new Set([file])); + }); +}); diff --git a/src/resolution/frameworks/java.ts b/src/resolution/frameworks/java.ts index bd36b1e4ed..3e1b7d6053 100644 --- a/src/resolution/frameworks/java.ts +++ b/src/resolution/frameworks/java.ts @@ -137,7 +137,7 @@ export const springResolver: FrameworkResolver = { // Pattern 1: Service references (dependency injection) if (ref.referenceName.endsWith('Service')) { - const result = resolveByNameAndKind(ref.referenceName, SERVICE_KINDS, SERVICE_DIRS, context); + const result = resolveByNameAndKind(ref, SERVICE_KINDS, SERVICE_DIRS, context); if (result) { return { original: ref, @@ -150,7 +150,7 @@ export const springResolver: FrameworkResolver = { // Pattern 2: Repository references if (ref.referenceName.endsWith('Repository')) { - const result = resolveByNameAndKind(ref.referenceName, SERVICE_KINDS, REPO_DIRS, context); + const result = resolveByNameAndKind(ref, SERVICE_KINDS, REPO_DIRS, context); if (result) { return { original: ref, @@ -163,7 +163,7 @@ export const springResolver: FrameworkResolver = { // Pattern 3: Controller references if (ref.referenceName.endsWith('Controller')) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, CONTROLLER_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, CONTROLLER_DIRS, context); if (result) { return { original: ref, @@ -176,7 +176,7 @@ export const springResolver: FrameworkResolver = { // Pattern 4: Entity/Model references if (/^[A-Z][a-zA-Z]+$/.test(ref.referenceName)) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, ENTITY_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, ENTITY_DIRS, context); if (result) { return { original: ref, @@ -189,7 +189,7 @@ export const springResolver: FrameworkResolver = { // Pattern 5: Component references if (ref.referenceName.endsWith('Component') || ref.referenceName.endsWith('Config')) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, COMPONENT_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, COMPONENT_DIRS, context); if (result) { return { original: ref, @@ -555,19 +555,30 @@ function joinPath(prefix: string, sub: string): string { /** * Resolve a symbol by name using indexed queries instead of scanning all files. + * What the reference's own scope declares comes first — its file, then its + * package (directory) — and a class nested in another file's class is out of + * reach by its bare name: every MyBatis `XExample` declares its own nested + * `Criteria`, and mall's 8,747 `new Criteria()` went to the first one. */ function resolveByNameAndKind( - name: string, + ref: UnresolvedRef, kinds: Set, preferredDirPatterns: string[], context: ResolutionContext, ): string | null { - const candidates = context.getNodesByName(name); + const candidates = context.getNodesByName(ref.referenceName); if (candidates.length === 0) return null; - const kindFiltered = candidates.filter((n) => kinds.has(n.kind)); + const kindFiltered = candidates.filter((n) => kinds.has(n.kind) && + (n.filePath === ref.filePath || !isNestedType(n, context))); if (kindFiltered.length === 0) return null; + const sameFile = kindFiltered.find((n) => n.filePath === ref.filePath); + if (sameFile) return sameFile.id; + const dir = ref.filePath.slice(0, ref.filePath.lastIndexOf('/') + 1); + const samePackage = kindFiltered.find((n) => n.filePath.startsWith(dir) && !n.filePath.slice(dir.length).includes('/')); + if (samePackage) return samePackage.id; + // Prefer candidates in framework-conventional directories const preferred = kindFiltered.filter((n) => preferredDirPatterns.some((d) => n.filePath.includes(d)) @@ -578,3 +589,12 @@ function resolveByNameAndKind( // Fall back to any match return kindFiltered[0]!.id; } + +/** Whether a JVM type is declared inside another type of its file. */ +function isNestedType(n: Node, context: ResolutionContext): boolean { + const cut = n.qualifiedName.lastIndexOf('::'); + if (cut < 0) return false; + const owner = n.qualifiedName.slice(0, cut); + return context.getNodesInFile(n.filePath).some((p) => + p.qualifiedName === owner && (p.kind === 'class' || p.kind === 'interface' || p.kind === 'enum' || p.kind === 'struct' || p.kind === 'trait')); +} From ca1eba98bd52de3fdb321660f10006efd74f513c Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 1 Oct 2026 00:27:15 +0000 Subject: [PATCH 102/259] fix(resolution): a bare-recorded call on another expression's value is not the caller (#2223) Generalizes the collapsed-chain self veto beyond TS/JS this./window. roots: a call the extractor recorded by its bare name but the source writes as a member call on anything other than this/self/super is never the calling method; and a value's initializer never calls the value itself. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/self-call-delegation.test.ts | 43 +++++++++++++++++--- src/resolution/name-matcher.ts | 54 ++++++++++++++++---------- 3 files changed, 71 insertions(+), 27 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c6e97b3384..7108327cb0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- A method that calls a same-named method on another expression's result is no longer linked to itself in any language. Examples are Scala's `requestToArmeria(request).execute()` inside `execute()`, Rust's `self.0.into_route(state)` inside `into_route()`, and Kotlin's `this@Buffer.write(…)` from an inner object. A value whose initializer chain calls a same-named method, like sttp's `val response = basicRequest.get(…).response(…)`, no longer links to itself either. - In Spring projects, a class name now resolves to the one in the reference's own file or package before any guess by folder convention. A class nested inside another file's class is never reached by its bare name. Before, every MyBatis `Example` class's `new Criteria()` in mall linked to the first `Example`'s nested `Criteria`, and halo's Lombok `@Builder` linked to a nested `Builder` class elsewhere. - A method that hands its call on to another object is no longer linked to itself. In TypeScript and JavaScript, a call through `this.` or `window.` counts as recursion only when the field is declared as the method's own class. A guess from a receiver's name alone never lands on the calling method. Before, BookStack's `toggle()` doing `this.container.classList.toggle('open')`, `listen()` doing `window.$events.listen(…)`, and `FileStorage::delete` doing `$storage->delete($path)` each pointed at themselves. - C# names now follow the language's scopes. A block `namespace X { … }` qualifies only the types inside it, so a type declared after the block, or in a file's second namespace, is no longer filed under the first. A namespace written inside another is `Outer.Inner`. A `using` links to the project's namespace instead of another file's `using` of the same name. A `global using` applies only within its own project. A nested type is reachable by bare name only from inside its owner or a type deriving from it, including through another partial part. Before, serilog's `Guard` was filed under `JetBrains.Annotations`, its tests' `Some.InformationEvent()` went to the performance tests' `Some`, and AutoMapper's same-file `new Source()` could reach another test class's nested `Source`. diff --git a/__tests__/self-call-delegation.test.ts b/__tests__/self-call-delegation.test.ts index d23bcae87d..40e2a617aa 100644 --- a/__tests__/self-call-delegation.test.ts +++ b/__tests__/self-call-delegation.test.ts @@ -1,11 +1,13 @@ /** * A method that hands its call on to another object — a wrapper — is not - * calling itself. TS/JS calls through `this.…` or `window.…` reach - * the resolver by their bare name, so BookStack's `toggle()` doing - * `this.container.classList.toggle('open')` and `listen()` doing - * `window.$events.listen(…)` bound to themselves; a guess from a receiver's - * name alone (`FileStorage::delete` doing `$storage->delete($path)`) did too. - * A recursion through a field the class declares as its own type stays. + * calling itself. Calls through `this.…` / `window.…` (TS/JS) or + * through an expression's value (Scala `requestToArmeria(request).execute()`, + * Rust `self.0.into_route(state)`) reach the resolver by their bare name, so + * BookStack's `toggle()` doing `this.container.classList.toggle('open')` + * bound to itself; a guess from a receiver's name alone (`FileStorage::delete` + * doing `$storage->delete($path)`) did too, and a value's initializer chain + * (`val response = basicRequest.get(…).response(…)`) to the value. A recursion + * through a field the class declares as its own type stays. */ import { describe, it, expect, afterAll, beforeAll } from 'vitest'; import * as fs from 'fs'; @@ -64,6 +66,30 @@ class ImageStorage { } } +`, + 'core/src/main/scala/sttp/ArmeriaBackend.scala': `package sttp + +class ArmeriaBackend { + def requestToArmeria(request: String): Client = new Client + + def execute(request: String): Unit = { + val armeriaRes = requestToArmeria(request).execute() + } +} + +class CurlTest { + val response = basicRequest + .get("http://example.com") + .response(asString) +} +`, + 'src/routing/route.rs': `pub struct BoxedIntoRoute(Box); + +impl BoxedIntoRoute { + pub fn into_route(self, state: u32) -> u32 { + self.0.into_route(state) + } +} `, }; for (const [rel, content] of Object.entries(files)) { @@ -93,6 +119,11 @@ describe('a call a method hands on', () => { expect(selfCallLines('resources/js/tree.ts')).toEqual([5]); }); + it('through another expression’s value is not the method itself', () => { + expect(selfCallLines('core/src/main/scala/sttp/ArmeriaBackend.scala')).toEqual([]); + expect(selfCallLines('src/routing/route.rs')).toEqual([]); + }); + it('through a receiver named like the caller’s class is not the method itself', () => { expect(selfCallLines('app/Uploads/FileStorage.php')).toEqual([]); }); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index e12f46169f..640a2beeb0 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -1403,31 +1403,35 @@ export const CASE_INSENSITIVE_LANGUAGES = new Set(['php', 'pascal', 'cfm * used to pick over the module-scope function the call actually means. */ /** - * The links between `this` / `super` / `window` and the method of a TS/JS - * call the extractor recorded by its bare name — `this.container.classList - * .toggle()` reaches the resolver as `toggle`, with links `container`, - * `classList`. Null for a call not written that way, `this.toggle()` included. - */ -function collapsedJsChain(ref: UnresolvedRef, context: ResolutionContext): { root: string; links: string[] } | null { - if (!JS_FAMILY.has(ref.language) || ref.referenceKind !== 'calls' || !/^[A-Za-z_$][\w$]*$/.test(ref.referenceName)) return null; + * The receiver a call the extractor recorded by its bare name is written on, + * read from the source: TS/JS keeps `this.container.classList.toggle()` and + * `window.$events.listen()` bare, Scala `requestToArmeria(request).execute()` + * and `_.get.whenRequestMatchesPartial(…)`. `'self'` for `this.m()` / + * `self.m()` / `super.m()` / `super().m()`; null for a call written bare (or + * not found). `links` are the member names between `this` and the method. + */ +function bareCallReceiver(ref: UnresolvedRef, context: ResolutionContext): { receiver: string; links: string[] } | null { + if (ref.referenceKind !== 'calls' || !/^[A-Za-z_$][\w$]*$/.test(ref.referenceName)) return null; const lines = context.getFileLines?.(ref.filePath) ?? context.readFile(ref.filePath)?.split(/\r?\n/); if (!lines) return null; - const text = lines.slice(ref.line - 1, ref.line + 3).join('\n').slice(ref.column); + const text = lines.slice(ref.line - 1, ref.line + 7).join('\n').slice(Math.max(0, ref.column)); const name = ref.referenceName.replace(/\$/g, '\\$'); - const m = new RegExp(`^(this|super|window)((?:\\s*\\??\\.\\s*#?[\\w$]+(?:\\([^()]*\\))?)+?)\\s*\\??\\.\\s*${name}\\s*(?:<[^<>()]*>)?\\s*\\(`).exec(text); - if (!m) return null; - return { root: m[1]!, links: m[2]!.split('.').map((l) => l.replace(/[\s?]/g, '')).filter((l) => l !== '') }; + const at = new RegExp(`(?()]*>|\\[(?:[^\\[\\]]|\\[[^\\[\\]]*\\])*\\])?\\s*[({]`).exec(text); + if (!at) return null; + const before = text.slice(0, at.index).replace(/\s+$/, ''); + if (!/\??\.$/.test(before)) return null; + const head = before.replace(/\??\.$/, '').replace(/\s+$/, ''); + if (/(?:^|[^\w$.])(?:this|self|super|Self)$/.test(head) || /(?:^|[^\w$.])super\s*\([^()]*\)$/.test(head)) return { receiver: 'self', links: [] }; + const chain = /(?:^|[^\w$.#])((?:this|super)(?:\s*\??\.\s*#?[\w$]+)+)$/.exec(head); + const links = chain ? chain[1]!.split('.').slice(1).map((l) => l.replace(/[\s?]/g, '')) : []; + return { receiver: head.slice(-40), links }; } -/** - * Whether a collapsed `this..m()` inside `m` can be `m` itself: only - * when the class declares the field as its own type — a tree node's - * `this.left.insert(v)` — never through a field of another type - * (`this.editor.input.focus()`, `this.container.classList.toggle()`). - */ +/** Whether a call recorded by its bare name is written on something other than the caller's own object. */ function isCollapsedNonRecursion(ref: UnresolvedRef, context: ResolutionContext): boolean { - const chain = collapsedJsChain(ref, context); - return chain !== null && !isCollapsedSelfRecursion(chain, ref, context); + const written = bareCallReceiver(ref, context); + if (!written || written.receiver === 'self') return false; + return !(JS_FAMILY.has(ref.language) && isCollapsedSelfRecursion({ root: 'this', links: written.links }, ref, context)); } function isCollapsedSelfRecursion(chain: { root: string; links: string[] }, ref: UnresolvedRef, context: ResolutionContext): boolean { @@ -8379,12 +8383,20 @@ export function matchReference( ): ResolvedRef | null { const result = gateLanguageMatch(matchReferenceInner(ref, context), ref, context); // `this.container.classList.toggle()` inside `toggle()`, `window.$events - // .listen()` inside `listen()`: a member of what the chain reaches, which is - // the calling method only through a field of the caller's own type. + // .listen()` inside `listen()`, Scala's `requestToArmeria(request).execute()` + // inside `execute()`: a member of what the receiver is, which is the calling + // method only through a TS/JS field of the caller's own type. if (result && result.targetNodeId === ref.fromNodeId && isCollapsedNonRecursion(ref, context)) return null; + // Nor does a value's initializer call the value: sttp's `val response = + // basicRequest.get(…).response(asStringAlways)` is a request's `response`. + if (result && result.targetNodeId === ref.fromNodeId && ref.referenceKind === 'calls' && + VALUE_KINDS.has(context.getNodeById?.(ref.fromNodeId)?.kind ?? '')) return null; return result ? retargetSelfOverload(result, ref, context) : result; } +/** Node kinds that hold a value rather than run code. */ +const VALUE_KINDS: ReadonlySet = new Set(['variable', 'constant', 'field', 'property']); + /** Languages whose methods overload by arity. */ const OVERLOADING_LANGUAGES: ReadonlySet = new Set(['csharp', 'java', 'kotlin', 'swift', 'cpp', 'scala', 'dart', 'vbnet']); From 7365427d4f199ccfbfa25711449ca4378ce0570f Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 1 Oct 2026 00:34:57 +0000 Subject: [PATCH 103/259] fix(rust): framework name heuristics never reach a function-local item (#2224) The Rust resolver's handler/service/struct heuristics took the first same-named item and then checked scope; they now drop items declared inside a function body and out-of-scope names before choosing, and take the file's own item first. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/rust-framework-name-scope.test.ts | 83 +++++++++++++++++++++ src/resolution/frameworks/rust.ts | 27 +++++-- src/resolution/name-matcher.ts | 2 +- 4 files changed, 105 insertions(+), 8 deletions(-) create mode 100644 __tests__/rust-framework-name-scope.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 7108327cb0..1d1ee60c32 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- In Rust projects, a struct, handler or service name no longer resolves to an item declared inside some function body, and a file's own item comes before any other crate's. Before, axum's examples' `Uri` linked to a `struct Uri` declared inside a routing test, tokio's runtime `Handle` linked to `tokio-test`'s, and ripgrep's `Glob { … }` in globset linked to the CLI flags' `Glob`. - A method that calls a same-named method on another expression's result is no longer linked to itself in any language. Examples are Scala's `requestToArmeria(request).execute()` inside `execute()`, Rust's `self.0.into_route(state)` inside `into_route()`, and Kotlin's `this@Buffer.write(…)` from an inner object. A value whose initializer chain calls a same-named method, like sttp's `val response = basicRequest.get(…).response(…)`, no longer links to itself either. - In Spring projects, a class name now resolves to the one in the reference's own file or package before any guess by folder convention. A class nested inside another file's class is never reached by its bare name. Before, every MyBatis `Example` class's `new Criteria()` in mall linked to the first `Example`'s nested `Criteria`, and halo's Lombok `@Builder` linked to a nested `Builder` class elsewhere. - A method that hands its call on to another object is no longer linked to itself. In TypeScript and JavaScript, a call through `this.` or `window.` counts as recursion only when the field is declared as the method's own class. A guess from a receiver's name alone never lands on the calling method. Before, BookStack's `toggle()` doing `this.container.classList.toggle('open')`, `listen()` doing `window.$events.listen(…)`, and `FileStorage::delete` doing `$storage->delete($path)` each pointed at themselves. diff --git a/__tests__/rust-framework-name-scope.test.ts b/__tests__/rust-framework-name-scope.test.ts new file mode 100644 index 0000000000..614834f249 --- /dev/null +++ b/__tests__/rust-framework-name-scope.test.ts @@ -0,0 +1,83 @@ +/** + * The Rust resolver's struct/handler/service name heuristics never reach an + * item declared inside a function body — axum's examples' `Uri` (imported + * from `http`) went to a `struct Uri` a routing test declares inside itself — + * and a file's own item comes first: ripgrep's `Ok(Glob { … })` in + * globset's glob.rs is its own `Glob`, not the CLI flags' `Glob`. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-rust-fw-')); + const files: Record = { + 'Cargo.toml': '[workspace]\nmembers = ["crates/core", "crates/globset", "crates/web", "examples/tls"]\n', + 'crates/core/Cargo.toml': '[package]\nname = "rg-core"\n', + 'crates/web/Cargo.toml': '[package]\nname = "web"\n', + 'crates/web/src/lib.rs': 'pub mod http;\n', + 'crates/web/src/http.rs': 'pub use ::http::Uri;\n', + 'crates/web/src/routing/tests.rs': `fn outer_middleware_still_see_whole_url() { + struct Uri; + let _ = Uri; +} +`, + 'examples/tls/Cargo.toml': '[package]\nname = "example-tls"\n\n[dependencies]\nweb = { path = "../../crates/web" }\n', + 'examples/tls/src/main.rs': `use web::http::Uri; + +fn make_https(uri: Uri, port: u16) -> Uri { + uri +} +`, + 'crates/core/src/flags/defs.rs': `pub struct Glob; + +pub fn glob_flag() -> Glob { + Glob +} +`, + 'crates/globset/Cargo.toml': '[package]\nname = "globset"\n', + 'crates/globset/src/glob.rs': `pub struct Glob { + glob: String, +} + +impl Glob { + pub fn new(glob: &str) -> Result { + Ok(Glob { glob: glob.to_string() }) + } +} +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +const targetsFrom = (file: string) => { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg.getOutgoingEdgesFrom(ids).filter((e) => e.kind !== 'contains') + .map((e) => cg.getNode(e.target)!).map((t) => `${t.filePath}:${t.qualifiedName}`); +}; + +describe('Rust framework name heuristics', () => { + it('never reach an item declared inside a function body', () => { + expect(targetsFrom('examples/tls/src/main.rs').filter((t) => t.includes('outer_middleware'))).toEqual([]); + }); + + it('take the file’s own item first', () => { + const globs = targetsFrom('crates/globset/src/glob.rs').filter((t) => t.endsWith(':Glob')); + expect(globs.length).toBeGreaterThan(0); + expect(new Set(globs)).toEqual(new Set(['crates/globset/src/glob.rs:Glob'])); + }); +}); diff --git a/src/resolution/frameworks/rust.ts b/src/resolution/frameworks/rust.ts index 283c453521..f7e79927a0 100644 --- a/src/resolution/frameworks/rust.ts +++ b/src/resolution/frameworks/rust.ts @@ -8,7 +8,7 @@ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from '../types'; import { stripCommentsForRegex } from '../strip-comments'; import { getCargoWorkspaceCrateMap } from './cargo-workspace'; -import { isRustNameInScope } from '../name-matcher'; +import { isRustNameInScope, isLexicallyReachable } from '../name-matcher'; /** * Whether the item a name heuristic found is one the reference can name: @@ -42,7 +42,7 @@ export const rustResolver: FrameworkResolver = { resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { // Pattern 1: Handler references if (ref.referenceName.endsWith('_handler') || ref.referenceName.startsWith('handle_')) { - const result = resolveByNameAndKind(ref.referenceName, FUNCTION_KINDS, HANDLER_DIRS, context); + const result = resolveByNameAndKind(ref, FUNCTION_KINDS, HANDLER_DIRS, context); if (result && inRustScope(result, ref, context)) { return { original: ref, @@ -55,7 +55,7 @@ export const rustResolver: FrameworkResolver = { // Pattern 2: Service/Repository trait implementations if (ref.referenceName.endsWith('Service') || ref.referenceName.endsWith('Repository')) { - const result = resolveByNameAndKind(ref.referenceName, SERVICE_KINDS, SERVICE_DIRS, context); + const result = resolveByNameAndKind(ref, SERVICE_KINDS, SERVICE_DIRS, context); if (result && inRustScope(result, ref, context)) { return { original: ref, @@ -68,7 +68,7 @@ export const rustResolver: FrameworkResolver = { // Pattern 3: Struct references (PascalCase) if (/^[A-Z][a-zA-Z]+$/.test(ref.referenceName)) { - const result = resolveByNameAndKind(ref.referenceName, STRUCT_KINDS, MODEL_DIRS, context); + const result = resolveByNameAndKind(ref, STRUCT_KINDS, MODEL_DIRS, context); if (result && inRustScope(result, ref, context)) { return { original: ref, @@ -291,18 +291,31 @@ function findMatchingParen(s: string, openIdx: number): number { /** * Resolve a symbol by name using indexed queries instead of scanning all files. */ +/** + * The same-named item of these kinds that the reference can see — in scope + * by its `use`s and module paths, and not declared inside some function body + * (axum's examples' `Uri`, imported from `http`, went to a `struct Uri` a + * routing test declares inside itself) — its own file's first, then a + * framework-conventional directory's. + */ function resolveByNameAndKind( - name: string, + ref: UnresolvedRef, kinds: Set, preferredDirPatterns: string[], context: ResolutionContext, ): string | null { - const candidates = context.getNodesByName(name); + const candidates = context.getNodesByName(ref.referenceName); if (candidates.length === 0) return null; - const kindFiltered = candidates.filter((n) => kinds.has(n.kind)); + const kindFiltered = candidates.filter((n) => kinds.has(n.kind) && + isLexicallyReachable(n, ref, context) && isRustNameInScope(n, ref, context)); if (kindFiltered.length === 0) return null; + // The file is a module: its own items are in scope. A sibling file is + // another module, reached only through a `use` — no nearer than any other. + const sameFile = kindFiltered.find((n) => n.filePath === ref.filePath); + if (sameFile) return sameFile.id; + // Prefer candidates in framework-conventional directories const preferred = kindFiltered.filter((n) => preferredDirPatterns.some((d) => n.filePath.includes(d)) diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 640a2beeb0..9cb0459b6b 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -636,7 +636,7 @@ const LOCAL_TYPE_KINDS = new Set(['class', 'struct', 'enum', 'interface' * unaffected (their parent resolves to a class-like node), as are top-level * symbols and C++ namespace-prefixed names (the prefix has no node). */ -function isLexicallyReachable( +export function isLexicallyReachable( candidate: Node, ref: UnresolvedRef, context: ResolutionContext From 179939055c6d0e0bb094028825fcd8797daeaf30 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 1 Oct 2026 00:45:56 +0000 Subject: [PATCH 104/259] fix(frameworks): name heuristics pick only what the reference can see, own scope first (#2225) One shared pick behind the Django/FastAPI/Flask, ASP.NET, Gin, SwiftUI/Vapor, Spring and Rust name heuristics: never a function-local declaration, a type nested in another file's type, or a node the language keeps out of the reference's file; the reference's own file first, then its directory (not Rust), then a conventional folder. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + .../framework-name-heuristic-scope.test.ts | 68 ++++++++++++++++++ __tests__/frameworks.test.ts | 6 +- src/resolution/frameworks/csharp.ts | 33 +++------ src/resolution/frameworks/go.ts | 42 ++++------- src/resolution/frameworks/java.ts | 40 +---------- src/resolution/frameworks/name-heuristic.ts | 70 +++++++++++++++++++ src/resolution/frameworks/python.ts | 36 +++------- src/resolution/frameworks/rust.ts | 40 +++-------- src/resolution/frameworks/swift.ts | 44 ++++-------- 10 files changed, 198 insertions(+), 182 deletions(-) create mode 100644 __tests__/framework-name-heuristic-scope.test.ts create mode 100644 src/resolution/frameworks/name-heuristic.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d1ee60c32..0374fc2041 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- Framework name conventions for Django, FastAPI, Flask, ASP.NET, Gin, SwiftUI and Vapor now only pick a class the reference can see. Examples of these conventions are "a `…View` is a view", "a `…Service` is a service" and "a capitalized name is a model". The pick starts from the reference's own file and then its package, and never takes a class declared inside some function or nested in another file's class. Before, Django REST framework's tests linked each file's own `MockView` and `Serializer` to another file's, and netbox's tests linked `TestForm()` to a form declared inside a different test function. - In Rust projects, a struct, handler or service name no longer resolves to an item declared inside some function body, and a file's own item comes before any other crate's. Before, axum's examples' `Uri` linked to a `struct Uri` declared inside a routing test, tokio's runtime `Handle` linked to `tokio-test`'s, and ripgrep's `Glob { … }` in globset linked to the CLI flags' `Glob`. - A method that calls a same-named method on another expression's result is no longer linked to itself in any language. Examples are Scala's `requestToArmeria(request).execute()` inside `execute()`, Rust's `self.0.into_route(state)` inside `into_route()`, and Kotlin's `this@Buffer.write(…)` from an inner object. A value whose initializer chain calls a same-named method, like sttp's `val response = basicRequest.get(…).response(…)`, no longer links to itself either. - In Spring projects, a class name now resolves to the one in the reference's own file or package before any guess by folder convention. A class nested inside another file's class is never reached by its bare name. Before, every MyBatis `Example` class's `new Criteria()` in mall linked to the first `Example`'s nested `Criteria`, and halo's Lombok `@Builder` linked to a nested `Builder` class elsewhere. diff --git a/__tests__/framework-name-heuristic-scope.test.ts b/__tests__/framework-name-heuristic-scope.test.ts new file mode 100644 index 0000000000..7457c2ea71 --- /dev/null +++ b/__tests__/framework-name-heuristic-scope.test.ts @@ -0,0 +1,68 @@ +/** + * The framework resolvers' name heuristics ("a `…View` is a view under + * `/views/`", "a `…Form` is a form") pick only what the reference can see, + * starting from its own scope: never a class a test declares inside one of + * its functions (netbox's `TestForm`, DRF's `MockView`), its own file's + * first, then its package's, then a conventional folder's. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-fw-name-')); + const files: Record = { + 'manage.py': '', + 'utilities/tests/test_forms.py': `class GetFieldValueTestCase: + def setUpTestData(self): + class TestForm: + pass + return TestForm() +`, + 'utilities/tests/test_templatetags.py': `def test_any_required(): + return TestForm() +`, + 'tests/browsable_api/views.py': `class MockView: + pass +`, + 'tests/authentication/test_authentication.py': `class MockView: + pass + + +def test_auth(): + return MockView() +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +const targetsFrom = (file: string) => { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg.getOutgoingEdgesFrom(ids).filter((e) => e.kind === 'instantiates' || e.kind === 'calls' || e.kind === 'references') + .map((e) => cg.getNode(e.target)!).map((t) => `${t.filePath}:${t.qualifiedName}`); +}; + +describe('framework name heuristics', () => { + it('never pick a class declared inside another file’s function', () => { + expect(targetsFrom('utilities/tests/test_templatetags.py').filter((t) => t.includes('TestForm'))).toEqual([]); + }); + + it('take the reference’s own file first', () => { + expect(targetsFrom('tests/authentication/test_authentication.py').filter((t) => t.includes('MockView'))) + .toEqual(['tests/authentication/test_authentication.py:MockView']); + }); +}); diff --git a/__tests__/frameworks.test.ts b/__tests__/frameworks.test.ts index cc03c7702b..940c5a0066 100644 --- a/__tests__/frameworks.test.ts +++ b/__tests__/frameworks.test.ts @@ -998,11 +998,11 @@ describe('springResolver.resolve — DI heuristics are gated to Java/Kotlin non- // hijack a Scala `extends X` to a same-named class found via directory // heuristics — inheritance must resolve through imports/name matching. const decoyNode: Node = { - id: 'class:src/test/model/ExtCustomer.java:ExtCustomer:3', + id: 'class:src/main/model/ExtCustomer.java:ExtCustomer:3', kind: 'class', name: 'ExtCustomer', - qualifiedName: 'src/test/model/ExtCustomer.java::ExtCustomer', - filePath: 'src/test/model/ExtCustomer.java', + qualifiedName: 'src/main/model/ExtCustomer.java::ExtCustomer', + filePath: 'src/main/model/ExtCustomer.java', language: 'java', startLine: 3, endLine: 10, diff --git a/src/resolution/frameworks/csharp.ts b/src/resolution/frameworks/csharp.ts index 51040c20b4..a70daa4a99 100644 --- a/src/resolution/frameworks/csharp.ts +++ b/src/resolution/frameworks/csharp.ts @@ -7,6 +7,7 @@ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from '../types'; import { stripCommentsForRegex } from '../strip-comments'; +import { pickByNameAndKind } from './name-heuristic'; export const aspnetResolver: FrameworkResolver = { name: 'aspnet', @@ -67,7 +68,7 @@ export const aspnetResolver: FrameworkResolver = { resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { // Pattern 1: Controller references if (ref.referenceName.endsWith('Controller')) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, CONTROLLER_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, CONTROLLER_DIRS, context); if (result) { return { original: ref, @@ -80,7 +81,7 @@ export const aspnetResolver: FrameworkResolver = { // Pattern 2: Service references (dependency injection) if (ref.referenceName.endsWith('Service') || ref.referenceName.startsWith('I') && ref.referenceName.length > 1) { - const result = resolveByNameAndKind(ref.referenceName, SERVICE_KINDS, SERVICE_DIRS, context); + const result = resolveByNameAndKind(ref, SERVICE_KINDS, SERVICE_DIRS, context); if (result) { return { original: ref, @@ -93,7 +94,7 @@ export const aspnetResolver: FrameworkResolver = { // Pattern 3: Repository references if (ref.referenceName.endsWith('Repository')) { - const result = resolveByNameAndKind(ref.referenceName, SERVICE_KINDS, REPO_DIRS, context); + const result = resolveByNameAndKind(ref, SERVICE_KINDS, REPO_DIRS, context); if (result) { return { original: ref, @@ -106,7 +107,7 @@ export const aspnetResolver: FrameworkResolver = { // Pattern 4: Model/Entity references if (/^[A-Z][a-zA-Z]+$/.test(ref.referenceName)) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, MODEL_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, MODEL_DIRS, context); if (result) { return { original: ref, @@ -119,7 +120,7 @@ export const aspnetResolver: FrameworkResolver = { // Pattern 5: ViewModel references if (ref.referenceName.endsWith('ViewModel') || ref.referenceName.endsWith('Dto')) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, VIEWMODEL_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, VIEWMODEL_DIRS, context); if (result) { return { original: ref, @@ -410,28 +411,12 @@ const VIEWMODEL_DIRS = ['/ViewModels/', '/ViewModel/', '/DTOs/', '/Dto/']; const CLASS_KINDS = new Set(['class']); const SERVICE_KINDS = new Set(['class', 'interface']); -/** - * Resolve a symbol by name using indexed queries instead of scanning all files. - */ +/** A framework name heuristic's pick (see name-heuristic.ts), preferring these folders. */ function resolveByNameAndKind( - name: string, + ref: UnresolvedRef, kinds: Set, preferredDirPatterns: string[], context: ResolutionContext, ): string | null { - const candidates = context.getNodesByName(name); - if (candidates.length === 0) return null; - - const kindFiltered = candidates.filter((n) => kinds.has(n.kind)); - if (kindFiltered.length === 0) return null; - - // Prefer candidates in framework-conventional directories - const preferred = kindFiltered.filter((n) => - preferredDirPatterns.some((d) => n.filePath.includes(d)) - ); - - if (preferred.length > 0) return preferred[0]!.id; - - // Fall back to any match - return kindFiltered[0]!.id; + return pickByNameAndKind(ref, kinds, (f) => preferredDirPatterns.some((d) => f.includes(d)), context); } diff --git a/src/resolution/frameworks/go.ts b/src/resolution/frameworks/go.ts index 168d4b22c3..5e819060eb 100644 --- a/src/resolution/frameworks/go.ts +++ b/src/resolution/frameworks/go.ts @@ -7,6 +7,7 @@ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from '../types'; import { stripCommentsForRegex } from '../strip-comments'; +import { pickByNameAndKind } from './name-heuristic'; export const goResolver: FrameworkResolver = { name: 'go', @@ -27,7 +28,7 @@ export const goResolver: FrameworkResolver = { resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { // Pattern 1: Handler references if (ref.referenceName.endsWith('Handler') || ref.referenceName.startsWith('Handle')) { - const result = resolveByNameAndKind(ref.referenceName, 'function', HANDLER_DIRS, context); + const result = resolveByNameAndKind(ref, 'function', HANDLER_DIRS, context); if (result) { return { original: ref, @@ -40,7 +41,7 @@ export const goResolver: FrameworkResolver = { // Pattern 2: Service/Repository references if (ref.referenceName.endsWith('Service') || ref.referenceName.endsWith('Repository') || ref.referenceName.endsWith('Store')) { - const result = resolveByNameAndKind(ref.referenceName, null, SERVICE_DIRS, context, SERVICE_KINDS); + const result = resolveByNameAndKind(ref, null, SERVICE_DIRS, context, SERVICE_KINDS); if (result) { return { original: ref, @@ -53,7 +54,7 @@ export const goResolver: FrameworkResolver = { // Pattern 3: Middleware references if (ref.referenceName.endsWith('Middleware') || ref.referenceName.startsWith('Auth') || ref.referenceName.startsWith('Log')) { - const result = resolveByNameAndKind(ref.referenceName, 'function', MIDDLEWARE_DIRS, context); + const result = resolveByNameAndKind(ref, 'function', MIDDLEWARE_DIRS, context); if (result) { return { original: ref, @@ -66,7 +67,7 @@ export const goResolver: FrameworkResolver = { // Pattern 4: Model/Entity references (typically PascalCase structs) if (/^[A-Z][a-zA-Z]+$/.test(ref.referenceName)) { - const result = resolveByNameAndKind(ref.referenceName, 'struct', MODEL_DIRS, context); + const result = resolveByNameAndKind(ref, 'struct', MODEL_DIRS, context); if (result) { return { original: ref, @@ -172,36 +173,17 @@ const MIDDLEWARE_DIRS = ['middleware', 'middlewares']; const MODEL_DIRS = ['model', 'models', 'entity', 'entities', 'domain', 'pkg']; const SERVICE_KINDS = new Set(['struct', 'interface']); -/** - * Resolve a symbol by name using indexed queries instead of scanning all files. - * Uses getNodesByName (O(log n) indexed lookup) instead of iterating every file. - */ +/** A framework name heuristic's pick (see name-heuristic.ts), preferring these folders. */ function resolveByNameAndKind( - name: string, + ref: UnresolvedRef, kind: string | null, preferredDirs: string[], context: ResolutionContext, kinds?: Set ): string | null { - const candidates = context.getNodesByName(name); - if (candidates.length === 0) return null; - - // Filter by kind - const kindFiltered = candidates.filter((n) => { - if (kinds) return kinds.has(n.kind); - if (kind) return n.kind === kind; - return true; - }); - - if (kindFiltered.length === 0) return null; - - // Prefer candidates in framework-conventional directories - const preferred = kindFiltered.filter((n) => - preferredDirs.some((d) => n.filePath.includes(`/${d}/`)) - ); - - if (preferred.length > 0) return preferred[0]!.id; - - // Fall back to any match - return kindFiltered[0]!.id; + const allowed: ReadonlySet = kinds ?? (kind ? new Set([kind]) : GO_NAMED_KINDS); + return pickByNameAndKind(ref, allowed, (f) => preferredDirs.some((d) => f.includes(`/${d}/`)), context); } + +/** Any declaration a name can be (the heuristic with no kind). */ +const GO_NAMED_KINDS: ReadonlySet = new Set(['function', 'method', 'struct', 'interface', 'type_alias', 'variable', 'constant']); diff --git a/src/resolution/frameworks/java.ts b/src/resolution/frameworks/java.ts index 3e1b7d6053..1759b4ea59 100644 --- a/src/resolution/frameworks/java.ts +++ b/src/resolution/frameworks/java.ts @@ -7,6 +7,7 @@ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from '../types'; import { stripCommentsForRegex } from '../strip-comments'; +import { pickByNameAndKind } from './name-heuristic'; export const springResolver: FrameworkResolver = { name: 'spring', @@ -553,48 +554,13 @@ function joinPath(prefix: string, sub: string): string { return '/' + parts.join('/'); } -/** - * Resolve a symbol by name using indexed queries instead of scanning all files. - * What the reference's own scope declares comes first — its file, then its - * package (directory) — and a class nested in another file's class is out of - * reach by its bare name: every MyBatis `XExample` declares its own nested - * `Criteria`, and mall's 8,747 `new Criteria()` went to the first one. - */ +/** A framework name heuristic's pick (see name-heuristic.ts), preferring these folders. */ function resolveByNameAndKind( ref: UnresolvedRef, kinds: Set, preferredDirPatterns: string[], context: ResolutionContext, ): string | null { - const candidates = context.getNodesByName(ref.referenceName); - if (candidates.length === 0) return null; - - const kindFiltered = candidates.filter((n) => kinds.has(n.kind) && - (n.filePath === ref.filePath || !isNestedType(n, context))); - if (kindFiltered.length === 0) return null; - - const sameFile = kindFiltered.find((n) => n.filePath === ref.filePath); - if (sameFile) return sameFile.id; - const dir = ref.filePath.slice(0, ref.filePath.lastIndexOf('/') + 1); - const samePackage = kindFiltered.find((n) => n.filePath.startsWith(dir) && !n.filePath.slice(dir.length).includes('/')); - if (samePackage) return samePackage.id; - - // Prefer candidates in framework-conventional directories - const preferred = kindFiltered.filter((n) => - preferredDirPatterns.some((d) => n.filePath.includes(d)) - ); - - if (preferred.length > 0) return preferred[0]!.id; - - // Fall back to any match - return kindFiltered[0]!.id; + return pickByNameAndKind(ref, kinds, (f) => preferredDirPatterns.some((d) => f.includes(d)), context); } -/** Whether a JVM type is declared inside another type of its file. */ -function isNestedType(n: Node, context: ResolutionContext): boolean { - const cut = n.qualifiedName.lastIndexOf('::'); - if (cut < 0) return false; - const owner = n.qualifiedName.slice(0, cut); - return context.getNodesInFile(n.filePath).some((p) => - p.qualifiedName === owner && (p.kind === 'class' || p.kind === 'interface' || p.kind === 'enum' || p.kind === 'struct' || p.kind === 'trait')); -} diff --git a/src/resolution/frameworks/name-heuristic.ts b/src/resolution/frameworks/name-heuristic.ts new file mode 100644 index 0000000000..9b06843a21 --- /dev/null +++ b/src/resolution/frameworks/name-heuristic.ts @@ -0,0 +1,70 @@ +/** + * The shared pick behind the framework resolvers' name heuristics — "a + * `…Service` is a class under `/services/`", "a PascalCase name is a model + * under `/models/`", "`router` is a FastAPI router under `/routers/`". + * + * Each resolver used to take the FIRST same-named node of the right kind, + * preferring a conventional folder, and that heuristic outranks name matching + * on a tie. So every MyBatis `XExample`'s `new Criteria()` in mall went to the + * first `Example`'s nested `Criteria`, and axum's examples' `Uri` to a struct a + * routing test declares inside itself. The pick now considers only what the + * reference can see, and starts from its own scope. + */ +import { Node } from '../../types'; +import { UnresolvedRef, ResolutionContext } from '../types'; +import { isLexicallyReachable, isVisibleAcrossFiles } from '../name-matcher'; + +/** Node kinds a nested type can be declared in. */ +const TYPE_OWNER_KINDS: ReadonlySet = new Set(['class', 'struct', 'interface', 'enum', 'trait', 'protocol', 'record']); + +export interface NamePickOptions { + /** + * Whether a file in the reference's directory comes before the conventional + * folders — its package (JVM, Go, C#, Python). Not for Rust, where a sibling + * file is another module, reached only through a `use`. + */ + sameDirectory?: boolean; + /** A language's own scope rule over the candidates (Rust's `use` / module paths). */ + accept?: (n: Node) => boolean; +} + +/** + * The id of the node named `ref.referenceName`, of one of `kinds`, that the + * reference can see: never a declaration inside some function body, never a + * type nested in another file's type (a nested type is reached by its bare + * name only from inside its owner), never one the language keeps out of the + * reference's file (a `private` member, an unexported Go name, a C# namespace + * the file does not use, a test suite from production code). Its own file's + * first, then (by default) its directory's, then a conventional folder's. + */ +export function pickByNameAndKind( + ref: UnresolvedRef, + kinds: ReadonlySet, + inPreferredDir: (filePath: string) => boolean, + context: ResolutionContext, + options: NamePickOptions = {}, +): string | null { + const candidates = context.getNodesByName(ref.referenceName).filter((n) => + kinds.has(n.kind) && + isLexicallyReachable(n, ref, context) && + (n.filePath === ref.filePath || (!isNestedType(n, context) && isVisibleAcrossFiles(n, ref, context))) && + (!options.accept || options.accept(n))); + if (candidates.length === 0) return null; + + const sameFile = candidates.find((n) => n.filePath === ref.filePath); + if (sameFile) return sameFile.id; + if (options.sameDirectory !== false) { + const dir = ref.filePath.slice(0, ref.filePath.lastIndexOf('/') + 1); + const sameDir = candidates.find((n) => n.filePath.startsWith(dir) && !n.filePath.slice(dir.length).includes('/')); + if (sameDir) return sameDir.id; + } + return (candidates.find((n) => inPreferredDir(n.filePath)) ?? candidates[0]!).id; +} + +/** Whether a node is declared inside a type of its own file. */ +function isNestedType(n: Node, context: ResolutionContext): boolean { + const cut = n.qualifiedName.lastIndexOf('::'); + if (cut < 0) return false; + const owner = n.qualifiedName.slice(0, cut); + return context.getNodesInFile(n.filePath).some((p) => p.qualifiedName === owner && TYPE_OWNER_KINDS.has(p.kind)); +} diff --git a/src/resolution/frameworks/python.ts b/src/resolution/frameworks/python.ts index 7ef69f0dfb..777016f2ac 100644 --- a/src/resolution/frameworks/python.ts +++ b/src/resolution/frameworks/python.ts @@ -8,6 +8,7 @@ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolutionContext, FrameworkExtractionResult } from '../types'; import { stripCommentsForRegex } from '../strip-comments'; import { resolveImportPath } from '../import-resolver'; +import { pickByNameAndKind } from './name-heuristic'; export const djangoResolver: FrameworkResolver = { name: 'django', @@ -25,15 +26,15 @@ export const djangoResolver: FrameworkResolver = { resolve(ref, context) { if (ref.referenceName.endsWith('Model') || /^[A-Z][a-z]+$/.test(ref.referenceName)) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, MODEL_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, MODEL_DIRS, context); if (result) return { original: ref, targetNodeId: result, confidence: 0.8, resolvedBy: 'framework' }; } if (ref.referenceName.endsWith('View') || ref.referenceName.endsWith('ViewSet')) { - const result = resolveByNameAndKind(ref.referenceName, VIEW_KINDS, VIEW_DIRS, context); + const result = resolveByNameAndKind(ref, VIEW_KINDS, VIEW_DIRS, context); if (result) return { original: ref, targetNodeId: result, confidence: 0.8, resolvedBy: 'framework' }; } if (ref.referenceName.endsWith('Form')) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, FORM_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, FORM_DIRS, context); if (result) return { original: ref, targetNodeId: result, confidence: 0.8, resolvedBy: 'framework' }; } // ORM dynamic dispatch: QuerySet._fetch_all (and siblings) call @@ -202,7 +203,7 @@ export const flaskResolver: FrameworkResolver = { resolve(ref, context) { if (ref.referenceName.endsWith('_bp') || ref.referenceName.endsWith('_blueprint')) { - const result = resolveByNameAndKind(ref.referenceName, VARIABLE_KINDS, [], context); + const result = resolveByNameAndKind(ref, VARIABLE_KINDS, [], context); if (result) return { original: ref, targetNodeId: result, confidence: 0.8, resolvedBy: 'framework' }; } return null; @@ -265,11 +266,11 @@ export const fastapiResolver: FrameworkResolver = { resolve(ref, context) { if (ref.referenceName.endsWith('_router') || ref.referenceName === 'router') { - const result = resolveByNameAndKind(ref.referenceName, VARIABLE_KINDS, ROUTER_DIRS, context); + const result = resolveByNameAndKind(ref, VARIABLE_KINDS, ROUTER_DIRS, context); if (result) return { original: ref, targetNodeId: result, confidence: 0.8, resolvedBy: 'framework' }; } if (ref.referenceName.startsWith('get_') || ref.referenceName.startsWith('Depends')) { - const result = resolveByNameAndKind(ref.referenceName, FUNCTION_KINDS, DEP_DIRS, context); + const result = resolveByNameAndKind(ref, FUNCTION_KINDS, DEP_DIRS, context); if (result) return { original: ref, targetNodeId: result, confidence: 0.75, resolvedBy: 'framework' }; } return null; @@ -640,29 +641,12 @@ const VIEW_KINDS = new Set(['class', 'function']); const VARIABLE_KINDS = new Set(['variable']); const FUNCTION_KINDS = new Set(['function']); -/** - * Resolve a symbol by name using indexed queries instead of scanning all files. - */ +/** A framework name heuristic's pick (see name-heuristic.ts), preferring these folders. */ function resolveByNameAndKind( - name: string, + ref: UnresolvedRef, kinds: Set, preferredDirPatterns: string[], context: ResolutionContext, ): string | null { - const candidates = context.getNodesByName(name); - if (candidates.length === 0) return null; - - const kindFiltered = candidates.filter((n) => kinds.has(n.kind)); - if (kindFiltered.length === 0) return null; - - // Prefer candidates in framework-conventional directories - if (preferredDirPatterns.length > 0) { - const preferred = kindFiltered.filter((n) => - preferredDirPatterns.some((d) => n.filePath.includes(d)) - ); - if (preferred.length > 0) return preferred[0]!.id; - } - - // Fall back to any match - return kindFiltered[0]!.id; + return pickByNameAndKind(ref, kinds, (f) => preferredDirPatterns.some((d) => f.includes(d)), context); } diff --git a/src/resolution/frameworks/rust.ts b/src/resolution/frameworks/rust.ts index f7e79927a0..1fa4f79d19 100644 --- a/src/resolution/frameworks/rust.ts +++ b/src/resolution/frameworks/rust.ts @@ -8,7 +8,8 @@ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from '../types'; import { stripCommentsForRegex } from '../strip-comments'; import { getCargoWorkspaceCrateMap } from './cargo-workspace'; -import { isRustNameInScope, isLexicallyReachable } from '../name-matcher'; +import { isRustNameInScope } from '../name-matcher'; +import { pickByNameAndKind } from './name-heuristic'; /** * Whether the item a name heuristic found is one the reference can name: @@ -288,43 +289,18 @@ function findMatchingParen(s: string, openIdx: number): number { return -1; } -/** - * Resolve a symbol by name using indexed queries instead of scanning all files. - */ -/** - * The same-named item of these kinds that the reference can see — in scope - * by its `use`s and module paths, and not declared inside some function body - * (axum's examples' `Uri`, imported from `http`, went to a `struct Uri` a - * routing test declares inside itself) — its own file's first, then a - * framework-conventional directory's. - */ +/** A framework name heuristic's pick (see name-heuristic.ts): in scope by its `use`s and module paths. */ function resolveByNameAndKind( ref: UnresolvedRef, kinds: Set, preferredDirPatterns: string[], context: ResolutionContext, ): string | null { - const candidates = context.getNodesByName(ref.referenceName); - if (candidates.length === 0) return null; - - const kindFiltered = candidates.filter((n) => kinds.has(n.kind) && - isLexicallyReachable(n, ref, context) && isRustNameInScope(n, ref, context)); - if (kindFiltered.length === 0) return null; - - // The file is a module: its own items are in scope. A sibling file is - // another module, reached only through a `use` — no nearer than any other. - const sameFile = kindFiltered.find((n) => n.filePath === ref.filePath); - if (sameFile) return sameFile.id; - - // Prefer candidates in framework-conventional directories - const preferred = kindFiltered.filter((n) => - preferredDirPatterns.some((d) => n.filePath.includes(d)) - ); - - if (preferred.length > 0) return preferred[0]!.id; - - // Fall back to any match - return kindFiltered[0]!.id; + return pickByNameAndKind(ref, kinds, (f) => preferredDirPatterns.some((d) => f.includes(d)), context, { + // The file is a module: a sibling file is another one, reached only through a `use`. + sameDirectory: false, + accept: (n) => isRustNameInScope(n, ref, context), + }); } interface ModuleResolution { diff --git a/src/resolution/frameworks/swift.ts b/src/resolution/frameworks/swift.ts index 8c91bf95dc..5702aef4a0 100644 --- a/src/resolution/frameworks/swift.ts +++ b/src/resolution/frameworks/swift.ts @@ -7,6 +7,7 @@ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from '../types'; import { stripCommentsForRegex } from '../strip-comments'; +import { pickByNameAndKind } from './name-heuristic'; // No extract(): a SwiftUI view is its own struct node, and a UIKit controller // its class. A one-line `component`/`class` twin per `struct X: View` (and per @@ -41,7 +42,7 @@ export const swiftUIResolver: FrameworkResolver = { resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { // Pattern 1: View references (SwiftUI views are PascalCase ending in View) if (ref.referenceName.endsWith('View') && /^[A-Z]/.test(ref.referenceName)) { - const result = resolveByNameAndKind(ref.referenceName, VIEW_KINDS, VIEW_DIRS, context); + const result = resolveByNameAndKind(ref, VIEW_KINDS, VIEW_DIRS, context); if (result) { return { original: ref, @@ -54,7 +55,7 @@ export const swiftUIResolver: FrameworkResolver = { // Pattern 2: ViewModel/ObservableObject references if (ref.referenceName.endsWith('ViewModel') || ref.referenceName.endsWith('Store') || ref.referenceName.endsWith('Manager')) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, VIEWMODEL_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, VIEWMODEL_DIRS, context); if (result) { return { original: ref, @@ -67,7 +68,7 @@ export const swiftUIResolver: FrameworkResolver = { // Pattern 3: Model references if (/^[A-Z][a-zA-Z]+$/.test(ref.referenceName)) { - const result = resolveByNameAndKind(ref.referenceName, MODEL_KINDS, MODEL_DIRS, context); + const result = resolveByNameAndKind(ref, MODEL_KINDS, MODEL_DIRS, context); if (result) { return { original: ref, @@ -107,7 +108,7 @@ export const uikitResolver: FrameworkResolver = { resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { // Pattern 1: ViewController references if (ref.referenceName.endsWith('ViewController')) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, VC_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, VC_DIRS, context); if (result) { return { original: ref, @@ -120,7 +121,7 @@ export const uikitResolver: FrameworkResolver = { // Pattern 2: UIView subclass references if (ref.referenceName.endsWith('View') && !ref.referenceName.endsWith('ViewController')) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, UIVIEW_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, UIVIEW_DIRS, context); if (result) { return { original: ref, @@ -133,7 +134,7 @@ export const uikitResolver: FrameworkResolver = { // Pattern 3: Cell references if (ref.referenceName.endsWith('Cell')) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, CELL_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, CELL_DIRS, context); if (result) { return { original: ref, @@ -146,7 +147,7 @@ export const uikitResolver: FrameworkResolver = { // Pattern 4: Delegate/DataSource references if (ref.referenceName.endsWith('Delegate') || ref.referenceName.endsWith('DataSource')) { - const result = resolveByNameAndKind(ref.referenceName, PROTOCOL_KINDS, [], context); + const result = resolveByNameAndKind(ref, PROTOCOL_KINDS, [], context); if (result) { return { original: ref, @@ -206,7 +207,7 @@ export const vaporResolver: FrameworkResolver = { // Pattern 1: Controller references if (ref.referenceName.endsWith('Controller')) { - const result = resolveByNameAndKind(ref.referenceName, VAPOR_CONTROLLER_KINDS, VAPOR_CONTROLLER_DIRS, context); + const result = resolveByNameAndKind(ref, VAPOR_CONTROLLER_KINDS, VAPOR_CONTROLLER_DIRS, context); if (result) { return { original: ref, @@ -219,7 +220,7 @@ export const vaporResolver: FrameworkResolver = { // Pattern 2: Model references (Fluent) if (/^[A-Z][a-zA-Z]+$/.test(ref.referenceName)) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, FLUENT_MODEL_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, FLUENT_MODEL_DIRS, context); if (result) { return { original: ref, @@ -232,7 +233,7 @@ export const vaporResolver: FrameworkResolver = { // Pattern 3: Middleware references if (ref.referenceName.endsWith('Middleware')) { - const result = resolveByNameAndKind(ref.referenceName, VAPOR_CONTROLLER_KINDS, VAPOR_MIDDLEWARE_DIRS, context); + const result = resolveByNameAndKind(ref, VAPOR_CONTROLLER_KINDS, VAPOR_MIDDLEWARE_DIRS, context); if (result) { return { original: ref, @@ -495,29 +496,12 @@ const MODEL_KINDS = new Set(['struct', 'class']); const PROTOCOL_KINDS = new Set(['protocol']); const VAPOR_CONTROLLER_KINDS = new Set(['class', 'struct']); -/** - * Resolve a symbol by name using indexed queries instead of scanning all files. - */ +/** A framework name heuristic's pick (see name-heuristic.ts), preferring these folders. */ function resolveByNameAndKind( - name: string, + ref: UnresolvedRef, kinds: Set, preferredDirPatterns: string[], context: ResolutionContext, ): string | null { - const candidates = context.getNodesByName(name); - if (candidates.length === 0) return null; - - const kindFiltered = candidates.filter((n) => kinds.has(n.kind)); - if (kindFiltered.length === 0) return null; - - // Prefer candidates in framework-conventional directories - if (preferredDirPatterns.length > 0) { - const preferred = kindFiltered.filter((n) => - preferredDirPatterns.some((d) => n.filePath.includes(d)) - ); - if (preferred.length > 0) return preferred[0]!.id; - } - - // Fall back to any match - return kindFiltered[0]!.id; + return pickByNameAndKind(ref, kinds, (f) => preferredDirPatterns.some((d) => f.includes(d)), context); } From 83dac15cffad14e16c402efcd555ad9b84a7a912 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 1 Oct 2026 01:18:09 +0000 Subject: [PATCH 105/259] fix(js): a name the calling function binds is its local (#2226) A JS/TS parameter or plain var/let/const above the reference shadows a same-named function declared elsewhere in the file, whichever strategy found it; destructured call results, member calls and followed aliases are left alone. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/js-function-local-binding.test.ts | 81 +++++++++++++++++++++ src/resolution/name-matcher.ts | 63 ++++++++++++++++ 3 files changed, 145 insertions(+) create mode 100644 __tests__/js-function-local-binding.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 0374fc2041..fe4a555058 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- In JavaScript and TypeScript, a name the calling function binds itself, as a parameter or a plain `var`/`let`/`const` above the reference, now means that local instead of a same-named function elsewhere in the file. Before, every lodash helper's `object` parameter linked to a `function object() {}` declared inside `runInContext`, and zod's `new Class(…)` linked to an unrelated `Class` when `Class` was the function's parameter. Destructured hook results like `const { t } = useI18n()` still link to the function they come from. - Framework name conventions for Django, FastAPI, Flask, ASP.NET, Gin, SwiftUI and Vapor now only pick a class the reference can see. Examples of these conventions are "a `…View` is a view", "a `…Service` is a service" and "a capitalized name is a model". The pick starts from the reference's own file and then its package, and never takes a class declared inside some function or nested in another file's class. Before, Django REST framework's tests linked each file's own `MockView` and `Serializer` to another file's, and netbox's tests linked `TestForm()` to a form declared inside a different test function. - In Rust projects, a struct, handler or service name no longer resolves to an item declared inside some function body, and a file's own item comes before any other crate's. Before, axum's examples' `Uri` linked to a `struct Uri` declared inside a routing test, tokio's runtime `Handle` linked to `tokio-test`'s, and ripgrep's `Glob { … }` in globset linked to the CLI flags' `Glob`. - A method that calls a same-named method on another expression's result is no longer linked to itself in any language. Examples are Scala's `requestToArmeria(request).execute()` inside `execute()`, Rust's `self.0.into_route(state)` inside `into_route()`, and Kotlin's `this@Buffer.write(…)` from an inner object. A value whose initializer chain calls a same-named method, like sttp's `val response = basicRequest.get(…).response(…)`, no longer links to itself either. diff --git a/__tests__/js-function-local-binding.test.ts b/__tests__/js-function-local-binding.test.ts new file mode 100644 index 0000000000..a0360929bd --- /dev/null +++ b/__tests__/js-function-local-binding.test.ts @@ -0,0 +1,81 @@ +/** + * A JS/TS name the calling function binds itself — a parameter, or a `var` / + * `let` / `const` above the reference — is that local, never a same-named + * function declared elsewhere in the file. Every lodash helper lives inside + * `runInContext`, so `baseHas(object, key)`'s `object` and `mixin`'s + * `object(this.__wrapped__)` reached a `function object() {}` an IIFE declares + * there. A function that calls itself through its own `const` still does, and + * `if (handler) handler()` is not a parameter list. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-js-local-')); + const files: Record = { + 'lodash.js': `function runInContext(context) { + var baseCreate = (function() { + function object() {} + return function(proto) { + object.prototype = proto; + return new object; + }; + }()); + + function baseHas(object, key) { + return object != null && hasOwnProperty.call(object, key); + } + + function mixin(object, source) { + var result = object(this.__wrapped__); + return result; + } + + function handler() {} + + function run() { + if (handler) { + handler(); + } + const walk = (node) => (node ? walk(node.next) : null); + return walk(context); + } + + return { baseCreate, baseHas, mixin, run }; +} +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +const targetsFrom = (qualifiedName: string) => { + const ids = cg.getNodesInFile('lodash.js').filter((n) => n.qualifiedName === qualifiedName).map((n) => n.id); + return cg.getOutgoingEdgesFrom(ids).filter((e) => e.kind !== 'contains').map((e) => cg.getNode(e.target)!.qualifiedName); +}; + +describe('JS names the calling function binds', () => { + it('are its parameters, not a same-named function elsewhere in the file', () => { + expect(targetsFrom('runInContext::baseHas')).not.toContain('runInContext::object'); + expect(targetsFrom('runInContext::mixin')).not.toContain('runInContext::object'); + }); + + it('leave unbound names and a const’s own recursion alone', () => { + expect(targetsFrom('runInContext::run')).toContain('runInContext::handler'); + expect(targetsFrom('runInContext::run::walk')).toContain('runInContext::run::walk'); + }); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 9cb0459b6b..482690b328 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -546,6 +546,12 @@ export function matchFunctionRef( // test's parameter of its name receives. if (ref.language === 'python' && !candidates.some((n) => isFixtureInReach(n, ref.filePath, context)) && isPythonLocallyBound(ref.referenceName, ref, context)) return null; + // Likewise a JS/TS parameter or local: lodash's `baseHas(object, key)` passes its own `object`. + const jsLocal = jsFunctionLocalScope(ref.referenceName, ref, context); + if (jsLocal) { + candidates = candidates.filter((n) => n.filePath === ref.filePath && n.startLine >= jsLocal.start && n.startLine <= jsLocal.end); + if (candidates.length === 0) return null; + } // Swift implicit-self: a bare identifier can name a METHOD only of the // ENCLOSING type (`Button(action: handleTap)` written inside that type) — @@ -4230,6 +4236,51 @@ export function isRustNameInScope(candidate: Node, ref: UnresolvedRef, context: const TYPE_MEMBER_KINDS: ReadonlySet = new Set(['method', 'property', 'field', 'enum_member']); /** Per-context memo: `file\0name` → "the file binds this name locally". */ +/** Whether `n` lies outside the function that binds the reference's name itself (see jsFunctionLocalScope). */ +function isOutsideJsLocal(n: Node, ref: UnresolvedRef, context: ResolutionContext): boolean { + // `const indexName = this.dataSource.namingStrategy.indexName(…)`: a member, whatever the local's name. + if (ref.referenceKind === 'calls' && bareCallReceiver(ref, context) !== null) return false; + const scope = jsFunctionLocalScope(ref.referenceName, ref, context); + return scope !== null && !(n.filePath === ref.filePath && n.startLine >= scope.start && n.startLine <= scope.end); +} + +const JS_FN_LOCAL_MEMO = new WeakMap>(); + +/** + * The lines of the JS/TS function a reference sits in when that function binds + * the name itself — a parameter, or a `var`/`let`/`const` above the reference. + * Such a name is the local, never a same-named function declared elsewhere: + * every lodash helper lives inside `runInContext`, so `baseHas(object, key)`'s + * `object` and `mixin`'s `object(this.__wrapped__)` reached a `function + * object() {}` an IIFE declares there. Null when the function does not bind it. + */ +function jsFunctionLocalScope(name: string, ref: UnresolvedRef, context: ResolutionContext): { start: number; end: number } | null { + if (!JS_FAMILY.has(ref.language) || !/^[A-Za-z_$][\w$]*$/.test(name)) return null; + let memo = JS_FN_LOCAL_MEMO.get(context); + if (!memo) JS_FN_LOCAL_MEMO.set(context, (memo = new Map())); + const key = `${ref.fromNodeId}\0${name}\0${ref.line}`; + const hit = memo.get(key); + if (hit !== undefined) return hit; + let scope: { start: number; end: number } | null = null; + const fn = context.getNodeById?.(ref.fromNodeId); + if (fn && (fn.kind === 'function' || fn.kind === 'method') && fn.startLine <= ref.line && fn.endLine >= ref.line) { + const lines = context.getFileLines?.(ref.filePath) ?? context.readFile(ref.filePath)?.split(/\r?\n/) ?? []; + const text = stripCommentsForRegex(lines.slice(fn.startLine - 1, ref.line).join('\n'), 'javascript'); + const { param } = localBindingPatterns(name, 'g'); + const n = name.replace(/\$/g, '\\$'); + // A plain declaration. Destructuring re-binds what a call returns under + // the same name — `const { t } = useI18n()`, `const { getLabel } = + // useProps(props)` — which is the same-named function more often than not. + const declared = new RegExp(`\\b(?:const|let|var)\\s+${n}\\b(?!\\s*[,\\]}])`).test(text); + // A parameter list — never a control-flow head (`if (openMarkerClose) {`). + // A return type stays on its line, never a ternary's `: data.slice()` below `filter(canRowExpand)`. + const parameter = new RegExp(`(?>(); /** @@ -8387,6 +8438,15 @@ export function matchReference( // inside `execute()`: a member of what the receiver is, which is the calling // method only through a TS/JS field of the caller's own type. if (result && result.targetNodeId === ref.fromNodeId && isCollapsedNonRecursion(ref, context)) return null; + // A name the calling JS/TS function binds itself shadows the file's own: + // lodash's `mixin(object, …)` calling `object(this.__wrapped__)` is its + // parameter, whichever strategy (fuzzy included) found a `function object`. + if (result && JS_LOCAL_REF_KINDS.has(ref.referenceKind)) { + const target = context.getNodeById?.(result.targetNodeId); + // (A target of another name is what the local was followed to: `const + // selected = useStore(s => s.reset); selected()` is the store's `reset`.) + if (target && target.name === ref.referenceName && isOutsideJsLocal(target, ref, context)) return null; + } // Nor does a value's initializer call the value: sttp's `val response = // basicRequest.get(…).response(asStringAlways)` is a request's `response`. if (result && result.targetNodeId === ref.fromNodeId && ref.referenceKind === 'calls' && @@ -8394,6 +8454,9 @@ export function matchReference( return result ? retargetSelfOverload(result, ref, context) : result; } +/** Reference kinds a bare JS/TS local can be: a call, a value, a construction. */ +const JS_LOCAL_REF_KINDS: ReadonlySet = new Set(['calls', 'references', 'function_ref', 'instantiates']); + /** Node kinds that hold a value rather than run code. */ const VALUE_KINDS: ReadonlySet = new Set(['variable', 'constant', 'field', 'property']); From 3e6252f38bfbabaab31c48388361bb28517efae3 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 1 Oct 2026 01:25:35 +0000 Subject: [PATCH 106/259] fix(svelte,vue): a component's instance script declarations are its own (#2227) isVisibleAcrossFiles rejects a declaration in a .svelte instance + +
{items.length}
+`, + 'src/lib/examples/item-demo.svelte': ` + +{label} +`, + 'src/components/TabItem.vue': ` + + +`, + 'src/extensions/code-block.ts': `import { isActive } from "@/tiptap/core"; + +export function active(state: unknown) { + return isActive(state); +} +`, + 'src/components/CrudTable.vue': ` + + +`, + 'src/pages/GroupDataPage.vue': ` + + +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +const targetsFrom = (file: string) => { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg.getOutgoingEdgesFrom(ids).filter((e) => e.kind !== 'contains').map((e) => cg.getNode(e.target)!).map((t) => `${t.filePath}:${t.name}`); +}; + +describe('single-file component declarations', () => { + it('are private to the component', () => { + expect(targetsFrom('src/lib/examples/item-demo.svelte')).not.toContain('src/lib/examples/attachment-group.svelte:Item'); + expect(targetsFrom('src/extensions/code-block.ts')).not.toContain('src/components/TabItem.vue:isActive'); + }); + + it('except a type a Vue +`); + const file = r.nodes.find((n) => n.kind === 'file')!; + const component = componentOf(r); + expect(file).toMatchObject({ id: 'file:pages/index.vue', startLine: 1, endLine: 9 }); + const p = parents(r); + expect(p.get(component.id)).toEqual([file.id]); + expect(p.get(file.id)).toBeUndefined(); + const load = r.nodes.find((n) => n.name === 'load')!; + expect(p.get(load.id)).toEqual([component.id]); + for (const [, list] of p) expect(list).toHaveLength(1); + }); + + it('gives a +`); + expect(from(r, 'useFetch').filter((id) => id.startsWith('component:'))).toHaveLength(1); + expect(r.unresolvedReferences.filter((ref) => ref.referenceKind === 'imports').every((ref) => ref.fromNodeId === 'file:pages/index.vue')).toBe(true); + }); + + it('leaves a plain +`); + expect(from(r, 'registerWidget')).toEqual(['file:src/Widget.vue']); + }); + + it('gives a Svelte instance script to the component, and a module script to the file', () => { + const r = extractFromSource('src/routes/+page.svelte', ` + + +

items

+`); + const component = componentOf(r).id; + expect(from(r, 'onMount')).toEqual([component]); + expect(from(r, 'preload')).toEqual(['file:src/routes/+page.svelte']); + expect(from(r, 'register')).toEqual(['file:src/routes/+page.svelte']); + }); + + it('gives Astro frontmatter to the component, and makes a file node for a file without one', () => { + const r = extractFromSource('src/pages/blog.astro', `--- +const posts = await getCollection('blog'); +--- +
    {posts.map((p) =>
  • {p.title}
    • )}
+`); + expect(from(r, 'getCollection')).toEqual([componentOf(r).id]); + const bare = extractFromSource('src/pages/about.astro', `

about

\n`); + expect(bare.nodes.map((n) => n.kind).sort()).toEqual(['component', 'file']); + }); +}); diff --git a/src/extraction/astro-extractor.ts b/src/extraction/astro-extractor.ts index e38989375f..efdc33f795 100644 --- a/src/extraction/astro-extractor.ts +++ b/src/extraction/astro-extractor.ts @@ -2,6 +2,7 @@ import { Node, Edge, ExtractionResult, ExtractionError, UnresolvedReference } fr import { generateNodeId } from './tree-sitter-helpers'; import { TreeSitterExtractor } from './tree-sitter'; import { isLanguageSupported } from './grammars'; +import { foldScriptResult, sfcFileNode } from './sfc-script'; /** * Astro built-in components — compiler-provided (``) or shipped by @@ -45,8 +46,10 @@ export class AstroExtractor { const startTime = Date.now(); try { - // Create component node for the .astro file itself + // The file, holding the component the .astro file is + this.nodes.push(sfcFileNode(this.filePath, this.source, 'astro')); const componentNode = this.createComponentNode(); + this.edges.push({ source: `file:${this.filePath}`, target: componentNode.id, kind: 'contains' }); // Extract and process the frontmatter block (--- fenced, TypeScript) const frontmatter = this.extractFrontmatter(); @@ -199,45 +202,13 @@ export class AstroExtractor { const extractor = new TreeSitterExtractor(this.filePath, block.content, 'typescript'); const result = extractor.extract(); - // Offset line numbers from the block back to .astro file positions - for (const node of result.nodes) { - node.startLine += block.startLine; - node.endLine += block.startLine; - node.language = 'astro'; // Mark as astro, not TS - - this.nodes.push(node); - - // Add containment edge from component to this node - this.edges.push({ - source: componentNodeId, - target: node.id, - kind: 'contains', - }); - } - - // Offset edges (they reference line numbers) - for (const edge of result.edges) { - if (edge.line) { - edge.line += block.startLine; - } - this.edges.push(edge); - } - - // Offset unresolved references - for (const ref of result.unresolvedReferences) { - ref.line += block.startLine; - ref.filePath = this.filePath; - ref.language = 'astro'; - this.unresolvedReferences.push(ref); - } - - // Carry over errors - for (const error of result.errors) { - if (error.line) { - error.line += block.startLine; - } - this.errors.push(error); - } + // Frontmatter runs on every render of the component, and a +`, + 'pages/admin.vue': ` +`, + 'pages/orders/index.vue': ` +`, + 'components/index.vue': ` +`, + }, + astro: { + 'package.json': JSON.stringify({ name: 'site', private: true, dependencies: { astro: '^4.0.0' } }), + 'src/pages/index.astro': `--- +const title = 'Home'; +--- +

{title}

+`, + 'src/pages/about.astro': `

about

+`, + 'src/pages/blog/index.astro': `

blog

+`, + 'src/pages/api/hello.ts': `export const GET = async () => new Response('hi'); +export async function POST() { return new Response('ok'); } +`, + }, +}; + +const graphs: Record = {}; + +beforeAll(async () => { + for (const [name, files] of Object.entries(projects)) { + const root = fs.mkdtempSync(path.join(os.tmpdir(), `cg-page-${name}-`)); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + graphs[name] = { root, cg: await CodeGraph.init(root, { index: true }) }; + } +}); + +afterAll(() => { + for (const { root, cg } of Object.values(graphs)) { + cg.close(); + fs.rmSync(root, { recursive: true, force: true }); + } +}); + +/** route name → the files of what it calls or references. */ +function served(cg: CodeGraph): Record { + const out: Record = {}; + for (const route of cg.getNodesByKind('route')) { + out[route.name] = cg.getOutgoingEdgesFrom([route.id], ['calls', 'references']) + .map((e) => cg.getNode(e.target)!) + .map((n) => `${n.kind} ${n.filePath}`); + } + return out; +} + +describe('a Nuxt page', () => { + it('is served by its own file’s component', () => { + const map = served(graphs.nuxt!.cg); + expect(map['/']).toEqual(['component pages/index.vue']); + expect(map['/admin']).toEqual(['component pages/admin.vue']); + expect(map['/orders']).toEqual(['component pages/orders/index.vue']); + }); + + it('is listed with the component that serves it', () => { + const { entries } = buildRoutes(graphs.nuxt!.cg, new URLSearchParams()); + expect(entries.map((e) => `${e.url} ${e.handler} ${e.file}`).sort()).toEqual([ + '/ index pages/index.vue', + '/admin admin pages/admin.vue', + '/orders index pages/orders/index.vue', + ]); + }); +}); + +describe('an Astro page', () => { + it('is served by its own file’s component, and an endpoint by the verbs it exports', () => { + const map = served(graphs.astro!.cg); + expect(map['/']).toEqual(['component src/pages/index.astro']); + expect(map['/about']).toEqual(['component src/pages/about.astro']); + expect(map['/blog']).toEqual(['component src/pages/blog/index.astro']); + expect(map['/api/hello']).toEqual(['function src/pages/api/hello.ts', 'function src/pages/api/hello.ts']); + }); +}); diff --git a/docs/viewer-launch-changelog.md b/docs/viewer-launch-changelog.md index 7376e5517c..71b2a2e10d 100644 --- a/docs/viewer-launch-changelog.md +++ b/docs/viewer-launch-changelog.md @@ -125,6 +125,8 @@ These describe `codegraph ui` and its screens. They were taken out of `## [Unrel ## Fixes — Screens, links and navigation +- **Routes served by a component are listed.** Entry points and the Steps picker listed a route only when a function, method or class served it. A Vue Router, Nuxt, Svelte or Astro screen is served by a component, so those apps' routes weren't listed at all. They are now, each with the component that serves it. + - **Where the app goes after login is a fork, not two always-es.** A navigation whose destination comes back from a helper — `router.replace(await resolvePostLoginRoute())` over `return (await hasSeenWelcome(…)) ? '/home/' : '/welcome/'` — drew both screens with no condition, reading as if the welcome screen always shows. The two arms share a line, and only a column can tell them apart; each synthesized edge now carries its literal's own position, so the guard reader says which arm it is: `WHEN await hasSeenWelcome(…)` → home, and its negation → welcome. And the scan starts at the helper's body, so a literal-union return type — `Promise<'/welcome/' | '/home/'>`, whose routes are string literals too, written first — no longer stands in for the navigation itself. Re-index after upgrading to pick the positions up. - **A screen that talks to native code keeps its own navigations.** In a React Native or Expo app, a `router.push` written inside a listener for a native event was credited to whichever screen had *started* that round trip, not to the screen the push is written on. In one app that moved seven transitions off the capture screen and onto the review screen it opens — leaving the review screen looking as though nothing in the app could reach it, stranded in the "no transition reaches this" band at the bottom of the Screens tab, and printing Swift conditions like `Thread.isMainThread` on a JavaScript navigation. A navigation now belongs to the screen whose file it is written in; an event arriving from native code, from an HTTP call or off a queue is no longer read backwards as if it were a caller. diff --git a/src/db/queries.ts b/src/db/queries.ts index 443b851372..655f98a981 100644 --- a/src/db/queries.ts +++ b/src/db/queries.ts @@ -1204,7 +1204,8 @@ export class QueryBuilder { if (!this.stmts.getRoutingManifest) { // Edge kind varies across framework resolvers: Spring/Rails/ // Laravel/Drupal emit `references`, Express emits `calls`. Accept - // both — the semantic is the same (route → its handler). + // both — the semantic is the same (route → its handler). A screen in + // a Vue / Svelte / Astro app is served by a `component`. this.stmts.getRoutingManifest = this.db.prepare(` SELECT r.name AS url, @@ -1220,7 +1221,7 @@ export class QueryBuilder { JOIN nodes h ON e.target = h.id WHERE r.kind = 'route' AND e.kind IN ('references', 'calls') - AND h.kind IN ('function', 'method', 'class', 'constant', 'variable') + AND h.kind IN ('function', 'method', 'class', 'constant', 'variable', 'component') ORDER BY r.file_path, r.start_line LIMIT ? `); diff --git a/src/resolution/frameworks/astro.ts b/src/resolution/frameworks/astro.ts index 4645335bad..635d19e971 100644 --- a/src/resolution/frameworks/astro.ts +++ b/src/resolution/frameworks/astro.ts @@ -7,6 +7,7 @@ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from '../types'; +import { pageComponentRef, resolvePageComponent } from './page-component'; /** * Astro virtual module prefixes — framework-provided, not user code @@ -47,6 +48,10 @@ export const astroResolver: FrameworkResolver = { }, resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + // A page route names the component its file is. + const page = resolvePageComponent(ref, context); + if (page) return page; + // Pattern 1: the `Astro` global (Astro.props, Astro.url, Astro.params, …) // — runtime-provided in every component's frontmatter. Resolving it as // framework-provided keeps it from name-matching a user symbol named Astro. @@ -95,8 +100,9 @@ export const astroResolver: FrameworkResolver = { return null; }, - extract(filePath: string, _content: string) { + extract(filePath: string, content: string) { const nodes: Node[] = []; + const references: UnresolvedRef[] = []; const now = Date.now(); // Normalize to forward slashes @@ -118,8 +124,9 @@ export const astroResolver: FrameworkResolver = { !/\.config\.[a-z]+$/.test(base) ) { const routePath = filePathToAstroRoute(afterPages); - - nodes.push({ + const isPage = normalized.endsWith('.astro'); + const language = isPage ? 'astro' : 'typescript'; + const route: Node = { id: `route:${filePath}:${routePath}:1`, kind: 'route', name: routePath, @@ -129,16 +136,31 @@ export const astroResolver: FrameworkResolver = { endLine: 1, startColumn: 0, endColumn: 0, - language: normalized.endsWith('.astro') ? 'astro' : 'typescript', + language, updatedAt: now, - }); + }; + nodes.push(route); + if (isPage) { + references.push(pageComponentRef(route, '.astro', 'astro')); + } else { + // An endpoint is served by the verbs it exports: `export const GET: + // APIRoute = …`, `export async function POST(…)`. + for (const m of content.matchAll(ENDPOINT_EXPORT)) { + const verb = m[1] ?? m[2]!; + const line = content.slice(0, m.index).split('\n').length; + references.push({ fromNodeId: route.id, referenceName: verb, referenceKind: 'references', line, column: 0, filePath, language, candidates: [verb] }); + } + } } } - return { nodes, references: [] }; + return { nodes, references }; }, }; +/** An Astro endpoint's exported handler: `export const GET`, `export async function POST`. */ +const ENDPOINT_EXPORT = /^\s*export\s+(?:(?:async\s+)?function\s+(GET|POST|PUT|PATCH|DELETE|OPTIONS|HEAD|ALL)\b|const\s+(GET|POST|PUT|PATCH|DELETE|OPTIONS|HEAD|ALL)\b)/gm; + /** * Check if string is PascalCase */ diff --git a/src/resolution/frameworks/page-component.ts b/src/resolution/frameworks/page-component.ts new file mode 100644 index 0000000000..38d311b113 --- /dev/null +++ b/src/resolution/frameworks/page-component.ts @@ -0,0 +1,44 @@ +/** + * A file-routed page — Nuxt's `pages/admin.vue`, Astro's `src/pages/about.astro` + * — is served by the component its file IS: the single-file-component + * extractor's one component per file, named after the file. Without the link a + * page route stood alone, so the viewer's Steps tab drew nothing for a page and + * the routes list never named what serves it. + * + * The route says so with a `calls` reference by that name, the way a Next.js + * page names its default export, and {@link resolvePageComponent} lands it on + * the SAME file's component — never on a same-named one elsewhere, since every + * `index.vue` is a component named `index`. + */ + +import type { Language, Node } from '../../types'; +import type { ResolutionContext, ResolvedRef, UnresolvedRef } from '../types'; + +/** The component name a single-file-component extractor gives a file: its basename, extension off. */ +export function pageComponentName(filePath: string, extension: string): string { + const base = filePath.split(/[/\\]/).pop() ?? filePath; + return base.endsWith(extension) ? base.slice(0, -extension.length) : base; +} + +export function pageComponentRef(route: Node, extension: string, language: Language): UnresolvedRef { + const name = pageComponentName(route.filePath, extension); + return { + fromNodeId: route.id, + referenceName: name, + referenceKind: 'calls', + line: 1, + column: 0, + filePath: route.filePath, + language, + candidates: [name], + }; +} + +/** A page route's reference to its file's own component, or null when this is not one. */ +export function resolvePageComponent(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + if (ref.referenceKind !== 'calls' || !ref.fromNodeId.startsWith(`route:${ref.filePath}:`)) return null; + const component = context + .getNodesInFile(ref.filePath) + .find((n) => n.kind === 'component' && n.name === ref.referenceName); + return component ? { original: ref, targetNodeId: component.id, confidence: 0.95, resolvedBy: 'framework' } : null; +} diff --git a/src/resolution/frameworks/vue.ts b/src/resolution/frameworks/vue.ts index e894e9b468..6a5d9027c4 100644 --- a/src/resolution/frameworks/vue.ts +++ b/src/resolution/frameworks/vue.ts @@ -8,6 +8,7 @@ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from '../types'; import { dependsOn } from './package-deps'; +import { pageComponentRef, resolvePageComponent } from './page-component'; /** The languages a Vue app's scripts are written in. */ const VUE_SCRIPT_LANGUAGES: ReadonlySet = new Set(['vue', 'javascript', 'typescript', 'tsx', 'jsx']); @@ -212,12 +213,14 @@ export const nuxtResolver: FrameworkResolver = { return context.getAllFiles().some((f) => /(?:^|\/)nuxt\.config\.(?:[cm]?[jt]s)$/.test(f)); }, - resolve(): ResolvedRef | null { - return null; + resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + // A page route names the component its file is. + return resolvePageComponent(ref, context); }, extract(filePath: string, _content: string) { const nodes: Node[] = []; + const references: UnresolvedRef[] = []; const now = Date.now(); // Forward slashes, and a leading `/` so an app at the repository root @@ -229,7 +232,7 @@ export const nuxtResolver: FrameworkResolver = { if (pagesIndex !== -1 && normalized.endsWith('.vue')) { const routePath = filePathToNuxtRoute(normalized, pagesIndex + '/pages/'.length); if (routePath !== null) { - nodes.push({ + const route: Node = { id: `route:${filePath}:${routePath}:1`, kind: 'route', name: routePath, @@ -241,7 +244,9 @@ export const nuxtResolver: FrameworkResolver = { endColumn: 0, language: 'vue', updatedAt: now, - }); + }; + nodes.push(route); + references.push(pageComponentRef(route, '.vue', 'vue')); } } @@ -292,7 +297,7 @@ export const nuxtResolver: FrameworkResolver = { }); } - return { nodes, references: [] }; + return { nodes, references }; }, }; From 104b8de3e6325315456c861c6cb6648916b81b46 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 1 Oct 2026 21:40:38 +0000 Subject: [PATCH 137/259] fix(laravel): a string controller resolves without a Controller suffix (#2271) Only `XController@action` was claimed and resolved, and the written namespace was stripped, so akaunting's 'Common\\Uploads@inline' (and a Route::resource with a string controller plus options) bound 304 of 313 routes to nothing. Any `Class@action` is claimed; the namespace path under App\\Http\\Controllers is kept and picks between same-named controllers; a resource reads its first argument and resolves its controller class in the Laravel resolver (no `use` brings a string-named class into scope). Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/laravel-controller-strings.test.ts | 74 +++++++++++++ src/resolution/frameworks/laravel.ts | 104 ++++++++++++++----- 3 files changed, 152 insertions(+), 27 deletions(-) create mode 100644 __tests__/laravel-controller-strings.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index d71cb51f33..e165649c7a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- In Laravel apps, a route whose controller is named by string now links to its handler, without needing a `Controller` suffix and keeping the namespace it's written with: `'Common\Uploads@inline'`, and `Route::resource('companies', 'Common\Companies', [...])` with an options array. Two same-named controllers in different namespaces are told apart by that namespace. Before, akaunting linked 304 of its 313 routes to nothing. - In Nuxt and Astro projects, a file-routed page is now linked to the component its file defines. An Astro endpoint is linked to the HTTP-verb handlers it exports, like `export const GET`. Before, a page route like Nuxt's `pages/admin.vue` or Astro's `src/pages/about.astro` linked to nothing, so callers, impact and flows stopped at the route. - In Rails apps, a `resources` line's `only:` and `except:` are now read in every form Rails accepts: `%i[new create index]`, `%w(index show)`, a quoted name, and the older `:only => [...]`. Before, only a bracketed list was read, so maybe's `resources :family_exports, only: %i[new create index]` became all seven routes, four of them to actions the controller doesn't have. - In Vue, Svelte and Astro files, a component now owns its script, and what a `\n{$title}\n'), + ).toBe('php'); + // Prose that happens to put Pascal keywords on their own lines. + expect(detectLanguage('f.inc', '
  • \n Implementation\n
    • \n
  • \n Begin\n
    • \n')).toBe('php'); + }); + + it('keeps the path-only answer (no source) on PHP', () => { + expect(detectLanguage('defs.inc')).toBe('php'); + expect(detectLanguage('defs.inc', '')).toBe('php'); + }); + + it('lets an explicit codegraph.json mapping for .inc win', () => { + expect(detectLanguage('defs.inc', DIRECTIVES, { '.inc': 'php' })).toBe('php'); + expect(detectLanguage('defs.inc', '#define X 1\n', { '.inc': 'cpp' })).toBe('cpp'); + expect(detectLanguage('page.inc', ' { + it('loads the Pascal grammar when an .inc file is in the set', () => { + expect(preloadLanguagesForFiles(['defs.inc'])).toEqual(expect.arrayContaining(['php', 'pascal'])); + }); + + it('adds nothing for a PHP project without .inc files, or when .inc is mapped explicitly', () => { + expect(preloadLanguagesForFiles(['index.php', 'a.module'])).not.toContain('pascal'); + expect(preloadLanguagesForFiles(['defs.inc'], { '.inc': 'php' })).not.toContain('pascal'); + }); + }); + + describe('index and sync', () => { + let dir: string; + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-pascal-inc-')); + clearProjectConfigCache(); + }); + afterEach(() => { + clearProjectConfigCache(); + fs.rmSync(dir, { recursive: true, force: true }); + }); + const write = (rel: string, body: string) => { + const p = path.join(dir, rel); + fs.mkdirSync(path.dirname(p), { recursive: true }); + fs.writeFileSync(p, body); + }; + const languageOf = (cg: CodeGraph, rel: string) => cg.getFiles().find((f) => f.path === rel)?.language; + const symbolsIn = (cg: CodeGraph, rel: string) => + cg.getNodesInFile(rel).filter((n) => n.kind !== 'file').map((n) => `${n.kind}:${n.name}:${n.language}`).sort(); + + // Runs first, before anything in this file has loaded the Pascal grammar + // through a `.pas` file: an include-only tree must still get a parser. + it('indexes a tree of Pascal includes with no .pas file', async () => { + write('inc/defs.inc', DIRECTIVES); + write('inc/routines.inc', ROUTINES); + const cg = await CodeGraph.init(dir, { index: true }); + try { + expect(languageOf(cg, 'inc/defs.inc')).toBe('pascal'); + expect(languageOf(cg, 'inc/routines.inc')).toBe('pascal'); + expect(symbolsIn(cg, 'inc/routines.inc')).toEqual(['function:Bar:pascal', 'function:Foo:pascal']); + expect(cg.getFiles().find((f) => f.path === 'inc/routines.inc')?.errors ?? []).toEqual([]); + } finally { + cg.close(); + } + }); + + it('classifies each include by content, and sync keeps the language', async () => { + write('U.pas', 'unit U;\n\ninterface\n\n{$I defs.inc}\n\nprocedure Foo;\n\nimplementation\n\n{$I routines.inc}\n\nend.\n'); + write('defs.inc', DIRECTIVES); + write('routines.inc', ROUTINES); + write('drupal/mymodule.inc', ' f.path === 'defs.inc')?.errors ?? []).toEqual([]); + expect(symbolsIn(cg, 'routines.inc')).toEqual(['function:Bar:pascal', 'function:Foo:pascal']); + expect(symbolsIn(cg, 'drupal/mymodule.inc')).toEqual(['function:mymodule_helper:php']); + + write('defs.inc', DIRECTIVES + '{$DEFINE USE_FAST_MM}\n'); + write('routines.inc', ROUTINES + 'procedure Baz;\nbegin\nend;\n'); + await cg.sync(); + + expect(languageOf(cg, 'defs.inc')).toBe('pascal'); + expect(languageOf(cg, 'routines.inc')).toBe('pascal'); + expect(symbolsIn(cg, 'routines.inc')).toEqual(['function:Bar:pascal', 'function:Baz:pascal', 'function:Foo:pascal']); + expect(languageOf(cg, 'drupal/mymodule.inc')).toBe('php'); + } finally { + cg.close(); + } + }); + + it('honors a codegraph.json .inc mapping through index and sync', async () => { + write('codegraph.json', JSON.stringify({ extensions: { '.inc': 'php' } })); + write('defs.inc', DIRECTIVES); + const cg = await CodeGraph.init(dir, { index: true }); + try { + expect(languageOf(cg, 'defs.inc')).toBe('php'); + write('defs.inc', DIRECTIVES + '{$DEFINE MORE}\n'); + await cg.sync(); + expect(languageOf(cg, 'defs.inc')).toBe('php'); + } finally { + cg.close(); + } + }); + }); +}); diff --git a/scripts/kernel-parity.mjs b/scripts/kernel-parity.mjs index d02ab6d6c6..5209369d8c 100644 --- a/scripts/kernel-parity.mjs +++ b/scripts/kernel-parity.mjs @@ -198,8 +198,12 @@ process.env.CODEGRAPH_KERNEL_LANGS = 'all'; for (const { file, lang: extLang } of files) { const source = fs.readFileSync(file, 'utf8'); const rel = path.relative(ROOT, file); - // `.h` resolves C vs C++ by content — the same call the indexer makes. - const lang = extLang === 'detect' || extLang === 'javascript' || extLang === 'jsx' ? detectLanguage(rel, source) : extLang; + // `.h` resolves C vs C++ by content — the same call the indexer makes; so + // does `.inc` (PHP vs Pascal, #2279 — a Pascal include is not a kernel file). + const lang = + extLang === 'detect' || extLang === 'javascript' || extLang === 'jsx' || path.extname(file).toLowerCase() === '.inc' + ? detectLanguage(rel, source) + : extLang; if (!KERNEL_LANGS.has(lang)) continue; if (langFilter && !langFilter.has(lang)) continue; processed++; diff --git a/src/extraction/grammars.ts b/src/extraction/grammars.ts index 5913349f3b..6a05b96c0d 100644 --- a/src/extraction/grammars.ts +++ b/src/extraction/grammars.ts @@ -561,9 +561,101 @@ export function detectLanguage(filePath: string, source?: string, overrides?: Re if (looksLikeObjc(source)) return 'objc'; } + // `.inc` is PHP's include extension (Drupal) and Pascal/Delphi's too + // (`{$I defs.inc}`: directive blocks, declaration fragments), so it is + // decided per file by content (#2279). An explicit codegraph.json mapping + // for `.inc` is the user's answer and is not second-guessed. + if (lang === 'php' && ext === '.inc' && source && !(overrides && overrides[ext]) && looksLikePascalInclude(source)) { + return 'pascal'; + } + return lang; } +/** A PHP open tag: `\n]*>)?`; +const PAS_QNAME = String.raw`[a-z_][\w.]*(?:<[^>\n]*>)?`; +const PAS_COMMENT = String.raw`(?:\/\/[^\n]*|\{[^$}\n][^}\n]*\}[ \t]*)?`; +const PAS_SECTION_BREAK = String.raw`[ \t]*${PAS_COMMENT}(?:\r?\n[ \t]*${PAS_COMMENT})+`; +const pascalLine = (shape: string): RegExp => new RegExp(PAS_LINE + shape, 'im'); + +/** + * Line shapes only Pascal writes, any one of which makes an untagged `.inc` + * Pascal. Each leans on Pascal's own punctuation, so the dialects that share a + * keyword with it stay out: JavaScript `const x = 1;` / `function f() {`, + * SourcePawn `function void (int client);`, C++ `const T X::Y = …`, VBScript + * `Const X = 1` / `Function F(a)`, Smarty `{$var}`, Makefile `X := y`, prose + * with `Begin` on a line of its own. Every pattern stays linear on a long + * whitespace run: no run can be split two ways between neighbouring + * quantifiers. + */ +const PASCAL_INCLUDE_SIGNALS: readonly RegExp[] = [ + // A compiler directive: a name and an argument (`{$IFDEF X}`, `{$DEFINE X}`, + // `{$I file.inc}`, `{$WARN X OFF}`), a bare `{$ELSE}` / `{$ENDIF}` / + // `{$IFEND}`, or a switch (`{$R-}`, `{$A+,B-}`) — or the `{%MainUnit x.pp}` + // line Lazarus opens its include files with. + pascalLine(String.raw`\{(?:\$(?:[a-z]\w*[ \t]+[^\s}]|(?:else|endif|ifend)[ \t]*\}|[a-z][+-][,}])|%MainUnit\b)`), + // A routine header, closed by `;`: `procedure Foo;`, `procedure TForm1.Click(Sender: TObject);`, + // `class constructor Create;` — and a function's result type after its + // parameters (`function Bar(A: Integer): string;`; bare `function Bar;` is + // the implementation-section short form). The parameter list stops at any + // parenthesis, so a file of unclosed `procedure X(` lines stays linear. + pascalLine( + String.raw`(?:class[ \t]+)?(?:(?:procedure|constructor|destructor)[ \t]+${PAS_QNAME}[ \t]*(?:\([^()]*\)[ \t]*)?` + + String.raw`|function[ \t]+${PAS_QNAME}[ \t]*(?:(?:\([^()]*\)[ \t]*)?:[ \t]*[\w.]+(?:<[^>\n]*>)?[ \t]*)?);` + ), + // `unit Foo;` and a `uses A, B;` clause. + pascalLine(String.raw`unit[ \t]+[a-z_][\w.]*[ \t]*;`), + pascalLine(String.raw`uses\s+[a-z_][\w.]*(?:\s*,\s*[a-z_][\w.]*)*\s*;`), + // A `const` section, then `X = …` / `X: T = …`; or a typed constant on one + // line (`const Max: Integer = 10;`). The type never holds a `:`. + pascalLine( + String.raw`(?:const|resourcestring)(?:${PAS_SECTION_BREAK}[a-z_]\w*[ \t]*(?::[^=;:\n]+)?=|[ \t]+[a-z_]\w*[ \t]*:[^=;:\n]+=)` + ), + // A `var` section: `G, H: Integer;`, on the keyword's line or below it. + pascalLine( + String.raw`(?:var|threadvar)(?:${PAS_SECTION_BREAK}|[ \t]+)[a-z_]\w*(?:[ \t]*,[ \t]*[a-z_]\w*)*[ \t]*:(?!:)[^;\n]*;` + ), + // A `type` section, then `TFoo =`; or `type TFoo = class…` (record / + // interface / set of / array / procedure type) on one line. + pascalLine( + String.raw`type(?:${PAS_SECTION_BREAK}${PAS_NAME}[ \t]*=|[ \t]+${PAS_NAME}[ \t]*=[ \t]*(?:packed[ \t]+)?` + + String.raw`(?:class|record|object|interface|dispinterface|set[ \t]+of|array|reference[ \t]+to|procedure|function)\b)` + ), +]; + +/** A `begin` … `end;` block: both halves needed, so neither alone flips a file. */ +const PASCAL_BEGIN_RE = pascalLine(String.raw`begin\b`); +const PASCAL_END_RE = pascalLine(String.raw`end[ \t]*[;.][ \t]*$`); + +/** + * Whether an `.inc` file is a Pascal include rather than a PHP one (#2279). + * + * A PHP include always opens a PHP tag somewhere, so a tag anywhere keeps the + * file PHP. Without one, a Pascal-only line shape (`PASCAL_INCLUDE_SIGNALS`, + * or a `begin` … `end;` pair) makes it Pascal. Anything else keeps the PHP + * mapping — untagged text is inline HTML to PHP, so nothing is extracted — + * rather than handing a C / assembly / POV-Ray / ASP `.inc` to the Pascal + * grammar's error recovery. + * + * Deliberately per file, not "does this project have `.pas` files": the + * answer depends only on the file's own bytes, so a full index, a sync of one + * edited include, and a fresh re-index always agree — a project-level gate + * would flip an untouched include whenever the last `.pas` file came or went. + */ +function looksLikePascalInclude(source: string): boolean { + if (PHP_OPEN_TAG_RE.test(source)) return false; + if (PASCAL_INCLUDE_SIGNALS.some((re) => re.test(source))) return true; + return PASCAL_BEGIN_RE.test(source) && PASCAL_END_RE.test(source); +} + /** Whether a JavaScript file's leading comments carry Flow's `@flow` pragma (and not `@noflow`). */ export function hasFlowPragma(source: string): boolean { const head = source.slice(0, 4096).replace(/^#![^\n]*\n/, ''); diff --git a/src/extraction/index.ts b/src/extraction/index.ts index f87eeacd4c..0ca3145be2 100644 --- a/src/extraction/index.ts +++ b/src/extraction/index.ts @@ -808,6 +808,15 @@ export function preloadLanguagesForFiles( if (!languages.includes(ambiguous)) languages.push(ambiguous); } } + // An `.inc` path-detects as PHP but may read as Pascal (#2279) — unless + // codegraph.json maps `.inc` explicitly, which detectLanguage never overrides. + if ( + !languages.includes('pascal') && + !(overrides && overrides['.inc']) && + files.some((f) => f.toLowerCase().endsWith('.inc')) + ) { + languages.push('pascal'); + } return languages; } From a4bd890ec634f00c8e5e418ec8e4c0fe6babcaad Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 04:42:23 +0000 Subject: [PATCH 150/259] fix(python): a call on self is never resolved through a same-named import (#2290) The Python extractor records `self.get_ip(request)` by its bare name, so the import strategy matched it to a function the file imports as `get_ip` and the call never reached the class's own (or inherited) method. A bare-named Python call written on `self` now skips import resolution; the method strategies then find it. A plain `get_ip(request)` still binds to the import. Across allauth, DRF, netbox and mealie this retargets 16 calls, all from an imported helper to the class's own method; no edge is added or lost. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/python-self-call-not-import.test.ts | 94 +++++++++++++++++++ src/resolution/index.ts | 9 +- src/resolution/name-matcher.ts | 9 ++ 4 files changed, 111 insertions(+), 2 deletions(-) create mode 100644 __tests__/python-self-call-not-import.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 5a5b87f38b..cff9f381b9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -166,6 +166,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - When an agent asks CodeGraph about a separate git repository nested inside an indexed project, one the project's index leaves out (for example because the parent's `.gitignore` excludes it), it now gets the usual "isn't indexed" guidance instead of answers from the parent project's code. Nested repositories the parent does index, such as submodules, work as before. Thanks @wstczyw for the report. (#2110) - Editing a file that defines the same name more than once, like two classes that each have an `execute` method or a method's overloads, no longer moves every caller from other files onto one of them during `codegraph sync`. A sync interrupted partway through a file also no longer loses those callers until a full re-index. Thanks @ijbranch for the report. (#2276) - Delphi and Free Pascal include files (`.inc`) are now indexed as Pascal. Before, they were read as PHP and came up empty. A `.inc` file with a ` { + const dirs: string[] = []; + let cg: CodeGraph | undefined; + + afterEach(() => { + cg?.close(); + cg = undefined; + for (const dir of dirs.splice(0)) fs.rmSync(dir, { recursive: true, force: true }); + }); + + async function callsIn(files: Record): Promise { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-py-self-')); + dirs.push(dir); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(dir, rel)), { recursive: true }); + fs.writeFileSync(path.join(dir, rel), content); + } + cg = await CodeGraph.init(dir, { index: true }); + const rows = (cg as any).db.db + .prepare( + `SELECT s.qualified_name s, t.qualified_name t, t.file_path f FROM edges e + JOIN nodes s ON s.id = e.source JOIN nodes t ON t.id = e.target WHERE e.kind = 'calls'`, + ) + .all() as { s: string; t: string; f: string }[]; + return rows.map((r) => `${r.s} -> ${r.t} (${r.f})`).sort(); + } + + const httpkit = 'def get_ip(request):\n return "1.2.3.4"\n'; + + it('a method calling its own same-named method through self', async () => { + expect( + await callsIn({ + 'pkg/__init__.py': '', + 'pkg/httpkit.py': httpkit, + 'pkg/adapter.py': [ + 'from pkg.httpkit import get_ip', + '', + '', + 'class Adapter:', + ' def get_ip(self, request):', + ' return get_ip(request)', + '', + ' def notify(self, request):', + ' return self.get_ip(request)', + '', + ].join('\n'), + }), + ).toEqual([ + // The bare call is still the imported function. + 'Adapter::get_ip -> get_ip (pkg/httpkit.py)', + 'Adapter::notify -> Adapter::get_ip (pkg/adapter.py)', + ]); + }); + + it('a method a base class in another file declares', async () => { + expect( + await callsIn({ + 'pkg/__init__.py': '', + 'pkg/httpkit.py': httpkit, + 'pkg/base.py': 'class Base:\n def get_ip(self, request):\n return "base"\n', + 'pkg/adapter.py': [ + 'from pkg.base import Base', + 'from pkg.httpkit import get_ip', + '', + '', + 'class Adapter(Base):', + ' def notify(self, request):', + ' return self.get_ip(request)', + '', + ' def raw(self, request):', + ' return get_ip(request)', + '', + ].join('\n'), + }), + ).toEqual([ + 'Adapter::notify -> Base::get_ip (pkg/base.py)', + 'Adapter::raw -> get_ip (pkg/httpkit.py)', + ]); + }); +}); diff --git a/src/resolution/index.ts b/src/resolution/index.ts index b0c80e01ac..f4bb91a237 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -21,7 +21,7 @@ import { isImportableKind, CPP_DEFINE_SIGNATURE, } from './types'; -import { matchJsStoreBindingCall, isUnresolvedJsMemberCall, isVisibleAcrossFiles, matchReference, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesCodeBoundary, gateLanguageMatch, dumpNameMatcherProfile, clearNameMatcherMemos, isRustNameInScope, CASE_INSENSITIVE_LANGUAGES } from './name-matcher'; +import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, isVisibleAcrossFiles, matchReference, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesCodeBoundary, gateLanguageMatch, dumpNameMatcherProfile, clearNameMatcherMemos, isRustNameInScope, CASE_INSENSITIVE_LANGUAGES } from './name-matcher'; import { isVisibleCppMacro, clearCppMacroVisibility } from './cpp-macro-visibility'; import { isCppConstructorRef, matchCppConstructor } from './cpp-constructor'; import { gateSwiftTypeTarget, clearSwiftTypeVisibility, swiftExtendedConformances } from './swift-type-visibility'; @@ -1240,7 +1240,12 @@ export class ReferenceResolver { } const tImp = this.profileStages ? process.hrtime.bigint() : 0n; - const importResult = this.gateLanguage(resolveViaImport(ref, this.context), ref); + // `self.get_ip()` is a method call on the instance even when the file + // also imports a function named `get_ip`: the import never names it. + const selfCall = ref.language === 'python' && ref.referenceKind === 'calls' && + this.context.getImportMappings(ref.filePath, ref.language).some((m) => m.localName === ref.referenceName) && + isPythonSelfCall(ref, this.context); + const importResult = selfCall ? null : this.gateLanguage(resolveViaImport(ref, this.context), ref); if (this.profileStages) this.stageAdd('viaImport', ref, !!importResult, tImp); if (importResult) { if (importResult.confidence >= 0.9) return importResult; diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 16609dd72f..1f5d6f8a2f 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -1714,6 +1714,15 @@ function bareCallReceiver(ref: UnresolvedRef, context: ResolutionContext): { rec return { receiver: head.slice(-40), links }; } +/** + * Whether a Python call recorded by its bare name was written on the instance, + * `self.get_ip(request)`. The extractor drops the `self.`, so a same-named name + * the file imports would otherwise claim the method call (#2074 follow-up). + */ +export function isPythonSelfCall(ref: UnresolvedRef, context: ResolutionContext): boolean { + return ref.language === 'python' && bareCallReceiver(ref, context)?.receiver === 'self'; +} + /** Whether a call recorded by its bare name is written on something other than the caller's own object. */ function isCollapsedNonRecursion(ref: UnresolvedRef, context: ResolutionContext): boolean { const written = bareCallReceiver(ref, context); From f838c5a02ab05cbd938039ae89aef9542604eb57 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 04:47:36 +0000 Subject: [PATCH 151/259] fix(resolution): resolve method values on untyped Python module globals (#2074) (#2291) * fix(resolution): resolve method values on untyped Python module globals (#1820) A method value whose receiver is a module global without a static type (`thread_pool_exec(settings.conn.fetch, ...)`, where `conn = None` is rebound by `global conn; conn = Backend()`) resolved to a `variable` node and produced no edge. The global's type is now the set of classes its module assigns to it, at module scope or in a function that declares it `global`. One class resolves to its own method; several bind to the nearest declaration every candidate inherits, as a base-typed receiver does. No edge (a wrong callback edge is worse than none) when: - any binding of the global in its module is not a whole constructor call: a factory, a conditional, a tuple target, `for`/`with`/import, a star import, or an annotation the constructor contradicts; - the caller or an enclosing function binds the name itself: parameter, assignment (any target position), loop, comprehension, `as`, `case` pattern, lambda parameter; - the receiver is a deeper chain (`settings.conn.pool.fetch`); - the importing file rebinds the name, or imports it from two sources. Statements are read with their continuation lines, and only statement lines (bracket depth 0) can assign. Not seen: rebinding the global from another module (`settings.conn = X`) or through `globals()`. `pkg.mod.Cls` now resolves through `import pkg.mod`, unless two namespace imports share the last segment. Python import mappings also read parenthesized from-imports (`from x import (\n A,\n B,\n)`), which previously yielded the single name `(` and hid every class imported that way from receiver and inheritance resolution. Comments and docstrings are stripped first. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018CQubRS1bXvaoApgPyLf66 (cherry picked from commit 0aab32a18b59cb42d895e0fbc54d885f0b4ee38b) * fix(resolution): account for writes to a Python module global from other files (#2074) The module-global receiver type was read from the global's own module only, so `settings.conn = Decoy()` in another module, or `globals()["conn"] = X`, left a stale type and a wrong edge. - A production write `. = Cls(...)` from another file joins the type set (resolved in the writing file); any other production write (another value, a tuple target, `setattr`/`patch.object`) makes the type unknown. The module is matched through each file's imports, including aliases and relative imports. - Test files install doubles (`settings.conn = MagicMock()`, `monkeypatch.setattr(settings, "conn", ...)`): they do not change the production type, but a ref inside such a test resolves nothing. - `globals()[...] = ...` with the global's name or a computed key, and `globals().update(...)`, in the global's module make the type unknown. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018CQubRS1bXvaoApgPyLf66 (cherry picked from commit 205571122653ea42b8d00f68fbb4082585a1922c) * fix(resolution): match a module global's writers by file, not dotted suffix Review follow-up to the cross-module write scan. - A write now lands on the file its import names: relative imports resolve exactly, absolute ones by path suffix; when a spelling could name another file sharing the tail (`x/settings.py` and `y/settings.py` for `import settings`), a write through it makes the type unknown instead of feeding or clearing the wrong module. - `import a.b` binds `a`: the module is spelled `a.b`; the mapping's last-segment name is not treated as an alias of it. - Test doubles are recognised by the narrow test-suite set plus any `conftest.py`; examples, fixtures and benchmarks are production writers. - Writes inside `if __name__ == "__main__":` are script code, not module state, in the global's module and elsewhere. - Namespace-dict writes: `globals()`, `vars()` and `sys.modules[__name__]` are read per statement (continuations joined, strings ignored); any use other than a literal-key read or `.get`, or a literal-key write of the name, makes the type unknown. `.__dict__[...]` / `.update(` from another file does too. - `settings.conn: Store = Store()` and `settings.conn = (Store())` are plain constructor writes. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018CQubRS1bXvaoApgPyLf66 (cherry picked from commit 7aab29269a7fabf3a9b1b9dafaa544ada76f4a7a) * fix(resolution): explicit dotted aliases, one-line __main__ guards, scoped vars() Review follow-up. - `import pkg.settings as settings` binds `settings` even though it equals the last segment; the import mapping cannot tell it from a plain `import pkg.settings`, so the source line decides. - `if __name__ == "__main__": stmt` on one line is script code, and a `globals()` write inside a `__main__` block no longer counts. - `vars()` is the module dict only at module scope; inside a function it is the locals, so `return vars()` there no longer makes the type unknown. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018CQubRS1bXvaoApgPyLf66 (cherry picked from commit 0a5fb697970ab21edbf11f3baddfa072bbeae593) * fix(resolution): explicit-alias check matches the exact module, outside strings Review follow-up: `import other.pkg.settings as settings` or a string `"import pkg.settings as settings"` no longer marks a plain `import pkg.settings` as explicitly aliased, which attached a write to the wrong module. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018CQubRS1bXvaoApgPyLf66 (cherry picked from commit a5569a07ea2c09bdb73e1262007c8d501fccd8ec) * test(resolution): give the Python module-global adversarial table its own timeout It indexes one project per case, which takes about nine seconds on macOS, past vitest's five-second default. Co-Authored-By: Claude Opus 5.5 * docs(changelog): Python module globals and multi-line imports (#2074) Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Josh66 Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 2 + __tests__/function-ref.test.ts | 305 ++++++++++++++++++ __tests__/resolution.test.ts | 22 ++ src/resolution/import-resolver.ts | 10 +- src/resolution/name-matcher.ts | 499 ++++++++++++++++++++++++++++++ 5 files changed, 834 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cff9f381b9..cdb9a6a6f8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -167,6 +167,8 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - Editing a file that defines the same name more than once, like two classes that each have an `execute` method or a method's overloads, no longer moves every caller from other files onto one of them during `codegraph sync`. A sync interrupted partway through a file also no longer loses those callers until a full re-index. Thanks @ijbranch for the report. (#2276) - Delphi and Free Pascal include files (`.inc`) are now indexed as Pascal. Before, they were read as PHP and came up empty. A `.inc` file with a ` { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-fnref-global-')); + fs.writeFileSync(path.join(tmpDir, 'store.py'), 'class Store:\n def fetch(self, ids):\n return ids\n'); + fs.writeFileSync(path.join(tmpDir, 'decoy.py'), 'class Decoy:\n def fetch(self, ids):\n return []\n'); + fs.writeFileSync(path.join(tmpDir, 'settings.py'), `from decoy import Decoy +conn = None + +def init(): + global conn + from store import Store + conn = Store() + +def shadow(): + conn = Decoy() + return conn + +def same_file(pool): + pool.submit(conn.fetch, []) + +def param_shadow(pool, conn): + pool.submit(conn.fetch, []) + +def loop_shadow(pool, items): + for conn in items: + pool.submit(conn.fetch, []) +`); + fs.writeFileSync(path.join(tmpDir, 'consumer.py'), `import settings +from settings import conn + +def via_module(pool): + pool.submit(settings.conn.fetch, []) + +def via_name(pool): + pool.submit(conn.fetch, []) + +def import_shadow(pool, settings): + pool.submit(settings.conn.fetch, []) + +def local_import(pool): + from settings import conn + pool.submit(conn.fetch, []) +`); + const cg = CodeGraph.initSync(tmpDir); + try { + await cg.indexAll(); + const store = cg.getNodesByName('fetch').find(n => n.qualifiedName.startsWith('Store::'))!; + const into = cg.getIncomingEdges(store.id).filter(e => e.kind === 'references' && e.metadata?.fnRef === true); + // A parameter or loop variable of the same name is not the global; a local import of it is. + expect(sourceNames(cg, into)).toEqual(['local_import', 'same_file', 'via_module', 'via_name']); + const decoy = cg.getNodesByName('fetch').find(n => n.qualifiedName.startsWith('Decoy::'))!; + expect(cg.getIncomingEdges(decoy.id).filter(e => e.metadata?.fnRef === true)).toHaveLength(0); + } finally { cg.close(); } + }); + + it('#1820: a module global with several backends binds to the declaration they share', async () => { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-fnref-global-many-')); + fs.mkdirSync(path.join(tmpDir, 'backends')); + fs.writeFileSync(path.join(tmpDir, 'backends', '__init__.py'), ''); + fs.writeFileSync(path.join(tmpDir, 'backends', 'base.py'), + 'class Base:\n def fetch(self, ids):\n raise NotImplementedError("backend")\n' + + ' def search(self):\n raise NotImplementedError("backend")\n'); + fs.writeFileSync(path.join(tmpDir, 'backends', 'es.py'), + 'from backends.base import Base\nclass ES(Base):\n def search(self):\n return []\n'); + fs.writeFileSync(path.join(tmpDir, 'backends', 'vec.py'), + 'from backends.base import (\n Base,\n)\nclass Vec(Base):\n def fetch(self, ids):\n return ids\n' + + ' def search(self):\n return []\n'); + fs.writeFileSync(path.join(tmpDir, 'other.py'), 'class Other:\n def fetch(self, ids):\n return ids\n'); + fs.writeFileSync(path.join(tmpDir, 'settings.py'), `import backends.es +conn = None + +def init(engine): + global conn + if engine == "es": + conn = backends.es.ES() + else: + from backends import vec as vec_module + conn = vec_module.Vec() +`); + fs.writeFileSync(path.join(tmpDir, 'unknown.py'), `from other import Other +conn = None +mixed = None + +def init(flag): + global conn, mixed + conn = make_conn() + mixed = Other() if flag else None +`); + fs.writeFileSync(path.join(tmpDir, 'consumer.py'), `import settings +import unknown + +def gc(pool): + pool.submit(settings.conn.fetch, []) + +def find(pool): + pool.submit(settings.conn.search) + +def opaque(pool): + pool.submit(unknown.conn.fetch, []) + pool.submit(unknown.mixed.fetch, []) +`); + const cg = CodeGraph.initSync(tmpDir); + try { + await cg.indexAll(); + const byOwner = (owner: string, name = 'fetch') => cg.getNodesByName(name).find(n => n.qualifiedName.startsWith(`${owner}::`))!; + const fnRefs = (owner: string, name = 'fetch') => cg.getIncomingEdges(byOwner(owner, name).id) + .filter(e => e.kind === 'references' && e.metadata?.fnRef === true); + // ES inherits Base.fetch and Vec overrides it: the shared declaration is Base.fetch. + expect(sourceNames(cg, fnRefs('Base'))).toEqual(['gc']); + expect(fnRefs('Vec')).toHaveLength(0); + // Both backends override search: the declaration they inherit is still Base.search. + expect(sourceNames(cg, fnRefs('Base', 'search'))).toEqual(['find']); + expect([...fnRefs('ES', 'search'), ...fnRefs('Vec', 'search')]).toHaveLength(0); + // An opaque initializer (`make_conn()`, a conditional) leaves the global's type unknown. + expect(fnRefs('Other')).toHaveLength(0); + } finally { cg.close(); } + }); + + // One index per adversarial case: several seconds on a loaded machine. + it('#1820: a module global never binds through a shadow, an unknown value or a deeper chain', async () => { + const header = `from store import Store +from decoy import Decoy +conn = None + +def init(): + global conn + conn = Store() +`; + // Each case: its files, and the fnRef edges into any `fetch` it must produce. + const cases: Record, string[]]> = { + control: [{ 'm.py': header + 'def cb(pool):\n pool.submit(conn.fetch)\n' }, ['cb -> Store::fetch']], + tuple_first: [{ 'm.py': header + 'def cb(pool):\n conn, _ = (Decoy(), None)\n pool.submit(conn.fetch)\n' }, []], + loop_and_nested_global: [{ 'm.py': header + + 'def cb(pool, items):\n for conn in items:\n pass\n def nested():\n global conn\n pool.submit(conn.fetch)\n' }, []], + multiline_conditional: [{ 'm.py': header.replace('conn = Store()', 'conn = Store(\n x=1,\n ) if flag else Decoy()') + + 'def cb(pool):\n pool.submit(conn.fetch)\n' }, []], + multiline_constructor: [{ 'm.py': header.replace('conn = Store()', 'conn = Store(\n x=1,\n )') + + 'def cb(pool):\n pool.submit(conn.fetch)\n' }, ['cb -> Store::fetch']], + closure_param: [{ 'm.py': header + 'def outer(conn):\n def cb(pool):\n pool.submit(conn.fetch)\n return cb\n' }, []], + closure_local: [{ 'm.py': header + 'def outer(pool):\n conn = Decoy()\n def cb():\n pool.submit(conn.fetch)\n return cb\n' }, []], + lambda_param: [{ 'm.py': header + 'def cb(pool):\n pool.submit(lambda conn: pool.map(conn.fetch))\n' }, []], + comprehension: [{ 'm.py': header + 'def cb(pool, xs):\n [pool.submit(conn.fetch) for conn in xs]\n' }, []], + except_as: [{ 'm.py': header + 'def cb(pool):\n try:\n pass\n except Exception as conn:\n pool.submit(conn.fetch)\n' }, []], + match_as: [{ 'm.py': header + 'def cb(pool, x):\n match x:\n case Decoy() as conn:\n pool.submit(conn.fetch)\n' }, []], + global_in_nested_def: [{ 'm.py': 'from decoy import Decoy\nconn = None\n\ndef init():\n def helper():\n global conn\n conn = Decoy()\n\n' + + 'def cb(pool):\n pool.submit(conn.fetch)\n' }, []], + keyword_in_multiline_call: [{ 'm.py': 'from decoy import Decoy\nconn = None\napp = dict(\n conn=Decoy()\n)\n\n' + + 'def cb(pool):\n pool.submit(conn.fetch)\n' }, []], + tuple_rebind: [{ 'm.py': header + 'def reset():\n global conn\n conn, other = Decoy(), 1\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, []], + with_rebind: [{ 'm.py': header + 'def reset():\n global conn\n with open_decoy() as conn:\n pass\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, []], + docstring: [{ 'm.py': 'from store import Store\nconn = None\n"""\nconn = Store()\n"""\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, []], + deeper_chain: [{ 'settings.py': header, 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.pool.fetch)\n' }, []], + importer_rebinds: [{ 'settings.py': header, + 'c.py': 'from settings import conn\nfrom decoy import Decoy\n\ndef reset():\n global conn\n conn = Decoy()\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, []], + local_import_of_other_module: [{ 'settings.py': header, 'other_settings.py': 'from decoy import Decoy\nconn = Decoy()\n', + 'c.py': 'import settings\n\ndef cb(pool):\n import other_settings as settings\n pool.submit(settings.conn.fetch)\n' }, []], + local_from_import_of_other_module: [{ 'settings.py': header, 'other_settings.py': 'from decoy import Decoy\nconn = Decoy()\n', + 'c.py': 'from settings import conn\n\ndef cb(pool):\n from other_settings import conn\n pool.submit(conn.fetch)\n' }, []], + comment_in_parenthesized_import: [{ 'settings.py': header + 'other = 1\n', + 'c.py': 'from settings import (\n other, # see fetch()\n conn,\n)\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, ['cb -> Store::fetch']], + star_import_override: [{ 'local.py': 'from decoy import Decoy\nconn = Decoy()\n', 'settings.py': header + 'from local import *\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, []], + match_sequence_capture: [{ 'm.py': header + 'def cb(pool, x):\n match x:\n case [conn]:\n pool.submit(conn.fetch)\n' }, []], + match_keyword_capture: [{ 'm.py': header + 'def cb(pool, x):\n match x:\n case Decoy(inner=conn):\n pool.submit(conn.fetch)\n' }, []], + match_bare_capture: [{ 'm.py': header + 'def cb(pool, x):\n match x:\n case conn:\n pool.submit(conn.fetch)\n' }, []], + multiline_loop_target: [{ 'm.py': header + 'def cb(pool, xs):\n for (a,\n conn) in xs:\n pool.submit(conn.fetch)\n' }, []], + multiline_tuple_local: [{ 'm.py': header + 'def cb(pool):\n (a,\n conn) = 1, Decoy()\n pool.submit(conn.fetch)\n' }, []], + multiline_tuple_rebind: [{ 'm.py': header + 'def reset():\n global conn\n (a,\n conn) = 1, Decoy()\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, []], + annotation_contradicts_value: [{ 'm.py': 'from store import Store\nfrom decoy import Decoy\nconn: Store = Decoy()\n\n' + + 'def cb(pool):\n pool.submit(conn.fetch)\n' }, []], + annotation_over_factory: [{ 'm.py': 'from store import Store\nconn: Store = make_store()\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, + ['cb -> Store::fetch']], + external_production_write: [{ 'settings.py': header, + 'reset.py': 'import settings\nfrom decoy import Decoy\n\ndef reset():\n settings.conn = Decoy()\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, []], + external_aliased_write: [{ 'pkg/__init__.py': '', 'pkg/settings.py': header, + 'reset.py': 'import pkg.settings as cfg\n\ndef reset():\n cfg.conn = make_conn()\n', + 'c.py': 'from pkg import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, []], + external_relative_write: [{ 'pkg/__init__.py': '', 'pkg/settings.py': header, + 'pkg/reset.py': 'from . import settings\nfrom decoy import Decoy\n\ndef reset():\n settings.conn = Decoy()\n', + 'c.py': 'from pkg import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, []], + external_setattr: [{ 'settings.py': header, + 'reset.py': 'import settings\nfrom decoy import Decoy\n\ndef reset():\n setattr(settings, "conn", Decoy())\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, []], + external_subclass_write: [{ 'settings.py': header, + 'sub.py': 'from store import Store\nclass SubStore(Store):\n def fetch(self, ids):\n return ids\n', + 'reset.py': 'import settings\nfrom sub import SubStore\n\ndef reset():\n settings.conn = SubStore()\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb -> Store::fetch']], + test_double_write: [{ 'settings.py': header, + 'tests/test_c.py': 'import settings\nfrom unittest.mock import MagicMock\n\ndef test_cb(pool):\n settings.conn = MagicMock()\n pool.submit(settings.conn.fetch)\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb -> Store::fetch']], + test_monkeypatch: [{ 'settings.py': header, + 'tests/test_c.py': 'import settings\n\ndef test_cb(monkeypatch, pool):\n monkeypatch.setattr(settings, "conn", object())\n pool.submit(settings.conn.fetch)\n' }, []], + globals_literal_write: [{ 'm.py': header + 'def reset():\n globals()["conn"] = Decoy()\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, []], + globals_dynamic_write: [{ 'm.py': header + 'for name, obj in [("conn", Decoy())]:\n globals()[name] = obj\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, []], + unrelated_attribute_write: [{ 'settings.py': header, + 'other.py': 'from decoy import Decoy\n\nclass Holder:\n def __init__(self):\n self.conn = Decoy()\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb -> Store::fetch']], + // A write lands on the module its import names, never on another module sharing its tail. + write_path_is_exact: [{ 'x/__init__.py': '', 'y/__init__.py': '', 'x/settings.py': header, 'y/settings.py': header, + 'reset.py': 'from x import settings\nfrom decoy import Decoy\n\ndef reset():\n settings.conn = Decoy()\n', + 'cx.py': 'from x import settings\n\ndef cb_x(pool):\n pool.submit(settings.conn.fetch)\n', + 'cy.py': 'from y import settings\n\ndef cb_y(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb_y -> Store::fetch']], + ambiguous_write_target: [{ 'x/__init__.py': '', 'y/__init__.py': '', 'x/settings.py': header, 'y/settings.py': header, + 'reset.py': 'import settings\nfrom decoy import Decoy\n\ndef reset():\n settings.conn = Decoy()\n', + 'cx.py': 'from x import settings\n\ndef cb_x(pool):\n pool.submit(settings.conn.fetch)\n', + 'cy.py': 'from y import settings\n\ndef cb_y(pool):\n pool.submit(settings.conn.fetch)\n' }, []], + relative_write_is_exact: [{ 'pkg/__init__.py': '', 'pkg/sub/__init__.py': '', 'pkg/settings.py': header, + 'other/__init__.py': '', 'other/pkg/__init__.py': '', 'other/pkg/settings.py': header, + 'pkg/sub/reset.py': 'from .. import settings\n\ndef reset():\n settings.conn = make_conn()\n', + 'c2.py': 'from other.pkg import settings\n\ndef cb2(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb2 -> Store::fetch']], + namespace_import_binds_root: [{ 'pkg/__init__.py': '', 'pkg/settings.py': header, + 'c.py': 'import pkg.settings\nfrom decoy import Decoy\n\nsettings = Bag()\nsettings.conn = Decoy()\n', + 'd.py': 'from pkg import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb -> Store::fetch']], + // `other.pkg.settings as settings` is not an explicit alias of `pkg.settings`: the + // write goes to the other module, so pkg's global keeps its type (the consumer's + // relative import names pkg/settings.py exactly). + alias_of_longer_module_is_not_explicit: [{ 'pkg/__init__.py': '', 'pkg/settings.py': header, + 'other/__init__.py': '', 'other/pkg/__init__.py': '', 'other/pkg/settings.py': 'conn = None\n', + 'reset.py': 'import pkg.settings\nimport other.pkg.settings as settings\nfrom decoy import Decoy\n\ndef reset():\n settings.conn = Decoy()\n', + 'pkg/d.py': 'from . import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb -> Store::fetch']], + alias_in_string_is_not_explicit: [{ 'pkg/__init__.py': '', 'pkg/settings.py': 'conn = None\n', + 'reset.py': 'import pkg.settings\nfrom decoy import Decoy\nnote = "import pkg.settings as settings"\n\ndef reset():\n settings.conn = Decoy()\n', + 'd.py': 'from pkg import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, []], + explicit_alias_equal_to_leaf: [{ 'pkg/__init__.py': '', 'pkg/settings.py': header, + 'reset.py': 'import pkg.settings as settings\nfrom decoy import Decoy\n\ndef reset():\n settings.conn = Decoy()\n', + 'd.py': 'from pkg import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, []], + one_line_main_write: [{ 'settings.py': header, + 'run.py': 'import settings\nfrom decoy import Decoy\n\nif __name__ == "__main__": settings.conn = Decoy()\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb -> Store::fetch']], + globals_in_main_block: [{ 'm.py': header + 'if __name__ == "__main__":\n globals()["conn"] = Decoy()\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, + ['cb -> Store::fetch']], + vars_in_function_is_locals: [{ 'm.py': header + 'def dump():\n return vars()\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, + ['cb -> Store::fetch']], + example_write_is_production: [{ 'settings.py': header, + 'examples/wire.py': 'import settings\nfrom decoy import Decoy\n\ndef wire():\n settings.conn = Decoy()\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, []], + conftest_write_is_test: [{ 'settings.py': header, + 'pkg/conftest.py': 'import settings\nfrom decoy import Decoy\n\ndef fixture():\n settings.conn = Decoy()\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb -> Store::fetch']], + main_block_write: [{ 'settings.py': header, + 'run.py': 'import settings\nfrom decoy import Decoy\n\nif __name__ == "__main__":\n settings.conn = Decoy()\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb -> Store::fetch']], + own_main_block_write: [{ 'm.py': header + 'if __name__ == "__main__":\n conn = Decoy()\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, + ['cb -> Store::fetch']], + globals_continued_key: [{ 'm.py': header + 'def reset():\n globals()[\n "conn"] = Decoy()\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, []], + globals_alias: [{ 'm.py': header + 'def reset():\n g = globals()\n g["conn"] = Decoy()\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, []], + globals_in_string: [{ 'm.py': header + 'NOTE = "globals().update(mapping)"\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, + ['cb -> Store::fetch']], + external_dict_write: [{ 'settings.py': header, + 'reset.py': 'import settings\nfrom decoy import Decoy\n\ndef reset():\n settings.__dict__["conn"] = Decoy()\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, []], + external_annotated_write: [{ 'settings.py': header, + 'reset.py': 'import settings\nfrom store import Store\n\ndef reset():\n settings.conn: Store = Store()\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb -> Store::fetch']], + external_parenthesized_write: [{ 'settings.py': header, + 'reset.py': 'import settings\nfrom store import Store\n\ndef reset():\n settings.conn = (Store())\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb -> Store::fetch']], + dotted_import_collision: [{ 'alpha/__init__.py': '', 'beta/__init__.py': '', + 'alpha/foo.py': 'class Client:\n def fetch(self):\n return 1\n', 'beta/foo.py': 'class Client:\n def fetch(self):\n return 2\n', + 'm.py': 'import beta.foo\nimport alpha.foo\nconn = None\n\ndef init():\n global conn\n conn = alpha.foo.Client()\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, []], + dotted_base_collision: [{ 'alpha/__init__.py': '', 'beta/__init__.py': '', + 'alpha/foo.py': 'class Client:\n def fetch(self):\n return 1\n', 'beta/foo.py': 'class Client:\n def fetch(self):\n return 2\n', + 'm.py': 'import beta.foo\nimport alpha.foo\n\nclass Sub(alpha.foo.Client):\n pass\n\ndef go(pool, s: Sub):\n pool.submit(s.fetch)\n' }, []], + }; + const got: Record = {}; + for (const [name, [files, _]] of Object.entries(cases)) { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), `cg-fnref-global-${name}-`)); + const all = { 'store.py': 'class Store:\n def fetch(self, ids):\n return ids\n', + 'decoy.py': 'class Decoy:\n def fetch(self, ids):\n return []\n', ...files }; + for (const [file, content] of Object.entries(all)) { + fs.mkdirSync(path.dirname(path.join(tmpDir, file)), { recursive: true }); + fs.writeFileSync(path.join(tmpDir, file), content); + } + const cg = CodeGraph.initSync(tmpDir); + try { + await cg.indexAll(); + got[name] = cg.getNodesByName('fetch').flatMap(t => cg.getIncomingEdges(t.id) + .filter(e => e.kind === 'references' && e.metadata?.fnRef === true) + .map(e => `${cg.getNode(e.source)?.name} -> ${t.qualifiedName}`)).sort(); + } finally { cg.close(); fs.rmSync(tmpDir, { recursive: true, force: true }); tmpDir = undefined; } + } + expect(got).toEqual(Object.fromEntries(Object.entries(cases).map(([name, [, want]]) => [name, want]))); + }, 60_000); + + it('#1820: a module global re-resolves after its module changes (sync)', async () => { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-fnref-global-sync-')); + const settings = 'from store import Store\nfrom decoy import Decoy\nconn = None\n\ndef init():\n global conn\n conn = Store()\n'; + const consumer = 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n'; + fs.writeFileSync(path.join(tmpDir, 'store.py'), 'class Store:\n def fetch(self, ids):\n return ids\n'); + fs.writeFileSync(path.join(tmpDir, 'decoy.py'), 'class Decoy:\n def fetch(self, ids):\n return []\n'); + fs.writeFileSync(path.join(tmpDir, 'settings.py'), settings); + fs.writeFileSync(path.join(tmpDir, 'consumer.py'), consumer); + const cg = CodeGraph.initSync(tmpDir); + const edges = () => cg.getNodesByName('fetch').flatMap(t => cg.getIncomingEdges(t.id) + .filter(e => e.metadata?.fnRef === true).map(e => `${cg.getNode(e.source)?.name} -> ${t.qualifiedName}`)); + try { + await cg.indexAll(); + expect(edges()).toEqual(['cb -> Store::fetch']); + fs.writeFileSync(path.join(tmpDir, 'settings.py'), settings.replace('conn = Store()', 'conn = Decoy()')); + fs.writeFileSync(path.join(tmpDir, 'consumer.py'), consumer + '\n# touched\n'); + await cg.sync(); + expect(edges()).toEqual(['cb -> Decoy::fetch']); + } finally { cg.close(); } + }); + it('#1820: Go receiver types disambiguate method values and reject external fields', async () => { tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-fnref-go-scope-')); fs.writeFileSync(path.join(tmpDir, 'main.go'), `package demo diff --git a/__tests__/resolution.test.ts b/__tests__/resolution.test.ts index 9bdf768cd0..14b5198f47 100644 --- a/__tests__/resolution.test.ts +++ b/__tests__/resolution.test.ts @@ -648,6 +648,28 @@ from ..services import auth_service expect(mappings.some((m) => m.localName === 'helper')).toBe(true); expect(mappings.some((m) => m.localName === 'User')).toBe(true); }); + + it('should extract parenthesized Python from-imports', () => { + const content = ` +from store.base import ( + Base, + helper as h, # a comment, with a comma and a call() +) +""" +from docstring import NotAnImport +""" +from store.rows import (Row, Col) +`; + + const mappings = extractImportMappings('src/main.py', content, 'python'); + + expect(mappings.map((m) => [m.localName, m.exportedName, m.source])).toEqual([ + ['Base', 'Base', 'store.base'], + ['h', 'helper', 'store.base'], + ['Row', 'Row', 'store.rows'], + ['Col', 'Col', 'store.rows'], + ]); + }); }); describe('JVM FQN Import Resolution', () => { diff --git a/src/resolution/import-resolver.ts b/src/resolution/import-resolver.ts index dc3b583959..7fa9912e23 100644 --- a/src/resolution/import-resolver.ts +++ b/src/resolution/import-resolver.ts @@ -11,6 +11,7 @@ import { UnresolvedRef, ResolvedRef, ResolutionContext, ImportMapping, ReExport import { applyAliases } from './path-aliases'; import { extractLocalExportAliases } from './alias-binding'; import { resolveWorkspaceImport } from './workspace-packages'; +import { stripCommentsForRegex } from './strip-comments'; import { resolveMethodOnType, resolveObjectLiteralMember, @@ -937,7 +938,7 @@ export function extractImportMappings( // whole SFC (markup + styles included) is safe. mappings.push(...extractJSImports(content)); } else if (language === 'python') { - mappings.push(...extractPythonImports(content)); + mappings.push(...extractPythonImports(stripCommentsForRegex(content, 'python'))); } else if (language === 'go') { mappings.push(...extractGoImports(content)); } else if (language === 'java' || language === 'kotlin') { @@ -1066,13 +1067,14 @@ function extractJSImports(content: string): ImportMapping[] { function extractPythonImports(content: string): ImportMapping[] { const mappings: ImportMapping[] = []; - // from X import Y - const fromImportRegex = /from\s+([\w.]+)\s+import\s+([^#\n]+)/g; + // from X import Y, and the parenthesized form `from X import (\n Y,\n Z,\n)` + const fromImportRegex = /from\s+([\w.]+)\s+import\s+(\([^)]*\)|[^#\n]+)/g; let match; while ((match = fromImportRegex.exec(content)) !== null) { const [, source, imports] = match; - const names = imports!.split(',').map((s) => s.trim()); + const names = imports!.trim().replace(/^\(|\)$/g, '') + .split(',').map((s) => s.trim()); for (const name of names) { const aliasMatch = name.match(/(\w+)\s+as\s+(\w+)/); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 1f5d6f8a2f..701c0ea03c 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -281,6 +281,17 @@ function matchMemberFunctionRef(ref: UnresolvedRef, context: ResolutionContext): if (ref.language === 'python') { const cls = pythonRefClass(receiver, ref, context); if (cls) return result(pythonMembers(cls, member, ref, context)); + // `mod.global` / an imported `global` itself — never a deeper chain (`mod.global.field`). + const segments = receiver.split('.'); + const hit = segments.length <= 2 ? context.resolveImport?.({ ...ref, referenceName: receiver, referenceKind: 'references' }) : null; + const global = hit && context.getNodeById?.(hit.targetNodeId); + // A local import of the root binds what the file imports, unless the file imports it from two places. + if (global && global.name === segments[segments.length - 1] && isPythonModuleGlobal(global, context) && + pythonImportKeys(segments[0]!, ref.filePath, context).size === 1 && + !pythonGlobalBindings(segments[0]!, ref.filePath, context).some(b => b.kind !== 'import') && + !pythonBindsLocally(segments[0]!, ref, context, false)) { + return result(pythonGlobalMembers(global, member, ref, context)); + } } const imported = context.resolveImport?.(ref); const node = imported && context.getNodeById?.(imported.targetNodeId); @@ -305,6 +316,11 @@ function matchMemberFunctionRef(ref: UnresolvedRef, context: ResolutionContext): type = pythonFieldType(receiver, owner, ref, context); } else { type = pythonLocalType(receiver, ref, context); + const global = type === null + ? context.getNodesInFile(ref.filePath).find(n => n.name === receiver && isPythonModuleGlobal(n, context)) : undefined; + if (global && !pythonBindsLocally(receiver, ref, context, true)) { + return result(pythonGlobalMembers(global, member, ref, context)); + } } // A type name used directly (`Store.fetch`) is scoped just like an annotation. if (!type && /^[A-Z]\w*$/.test(receiver)) type = receiver; @@ -337,6 +353,14 @@ function matchMemberFunctionRef(ref: UnresolvedRef, context: ResolutionContext): function pythonRefClass(name: string, ref: UnresolvedRef, context: ResolutionContext): Node | null { const imports = context.getImportMappings(ref.filePath, 'python'); + // `import pkg.mod` then `pkg.mod.Cls`: the mapping keys the module by its last segment. + const module = imports.find(i => i.isNamespace && name.startsWith(`${i.source}.`) && + /^\w+$/.test(name.slice(i.source.length + 1))); + if (module) { + // Two modules with the same last segment (`import a.foo`, `import b.foo`) share the key: refuse. + if (imports.filter(i => i.localName === module.localName).length !== 1) return null; + name = `${module.localName}.${name.slice(module.source.length + 1)}`; + } if (imports.some(i => i.localName === name.split('.')[0])) { const hit = context.resolveImport?.({ ...ref, referenceName: name, referenceKind: 'references' }); const node = hit && context.getNodeById?.(hit.targetNodeId); @@ -411,6 +435,478 @@ function pythonLocalType(receiver: string, ref: UnresolvedRef, context: Resoluti return caller?.signature?.match(new RegExp(`\\b${receiver}\\s*:\\s*["']?([\\w.]+)`))?.[1] ?? null; } +/** A module-scope Python variable (not a class attribute or a function local). */ +function isPythonModuleGlobal(node: Node, context: ResolutionContext): boolean { + return node.language === 'python' && (node.kind === 'variable' || node.kind === 'constant') && + !context.getNodesInFile(node.filePath).some(n => + (n.kind === 'class' || n.kind === 'function' || n.kind === 'method') && + n.startLine <= node.startLine && n.endLine >= node.startLine); +} + +/** How one line binds a name: `global`, a plain `name = value` / `name: T`, an import, or any other binding. */ +type PythonBinding = + | { kind: 'global' } + | { kind: 'assign'; type: string | null; value: string; line: number } + | { kind: 'import'; key: string } + | { kind: 'other' }; + +const PYTHON_STATEMENT_STARTS = new WeakMap>(); +/** Per line: does it start a statement (bracket depth 0, no `\` continuation)? String contents are skipped. */ +function pythonStatementStarts(filePath: string, context: ResolutionContext): boolean[] { + let files = PYTHON_STATEMENT_STARTS.get(context); + if (!files) { files = new Map(); PYTHON_STATEMENT_STARTS.set(context, files); } + let starts = files.get(filePath); + if (starts) return starts; + starts = []; + let depth = 0; + let continued = false; + for (const line of pythonMemberLines(filePath, context)) { + starts.push(depth === 0 && !continued); + let quote = ''; + for (let i = 0; i < line.length; i++) { + const c = line[i]!; + if (quote) { if (c === '\\') i++; else if (c === quote) quote = ''; continue; } + if (c === '"' || c === "'") quote = c; + else if (c === '(' || c === '[' || c === '{') depth++; + else if (c === ')' || c === ']' || c === '}') depth = Math.max(0, depth - 1); + } + continued = /\\\s*$/.test(line); + } + files.set(filePath, starts); + return starts; +} + +/** Line indexes that belong to `scope` itself (null: the module), not to a def or class nested in it. */ +function pythonOwnLines(scope: Node | null, filePath: string, context: ResolutionContext): number[] { + const count = pythonMemberLines(filePath, context).length; + const from = scope ? scope.startLine : 1; + const to = scope ? Math.min(scope.endLine, count) : count; + const nested = new Uint8Array(to - from + 1); + for (const n of context.getNodesInFile(filePath)) { + if ((n.kind !== 'function' && n.kind !== 'method' && n.kind !== 'class') || n.id === scope?.id) continue; + if (n.startLine < from || n.endLine > to || (scope && n.startLine <= scope.startLine)) continue; + nested.fill(1, n.startLine - from, n.endLine - from + 1); + } + const own: number[] = []; + for (let l = from; l <= to; l++) if (!nested[l - from]) own.push(l - 1); + return own; +} + +/** Split `a = b = value` at its top-level assignment operators; null when the line assigns nothing. */ +function pythonAssignment(line: string): { targets: string[]; value: string; augmented: boolean } | null { + const targets: string[] = []; + let depth = 0; + let quote = ''; + let start = 0; + let augmented = false; + for (let i = 0; i < line.length; i++) { + const c = line[i]!; + if (quote) { if (c === '\\') i++; else if (c === quote) quote = ''; continue; } + if (c === '"' || c === "'") { quote = c; continue; } + if (c === '(' || c === '[' || c === '{') { depth++; continue; } + if (c === ')' || c === ']' || c === '}') { depth--; continue; } + if (c !== '=' || depth !== 0) continue; + const prev = line[i - 1] ?? ''; + if (line[i + 1] === '=') { i++; continue; } // == + if (prev === '!' || prev === ':') continue; // != and the walrus + if ((prev === '<' || prev === '>') && line[i - 2] !== prev) continue; // <= >= + const op = line.slice(start, i).match(/(?:\/\/|\*\*|>>|<<|[-+*/%&|^@])$/)?.[0]; + if (op) augmented = true; + targets.push(line.slice(start, i - (op?.length ?? 0))); + start = i + 1; + } + return targets.length ? { targets, value: line.slice(start), augmented } : null; +} + +/** + * Every way the statement starting at line `index` binds `name`, read across + * its continuation lines. Only statement lines can assign or import; any line + * can bind through `as`, a loop, a lambda or the walrus. + */ +function pythonLineBindings(lines: string[], index: number, statements: boolean[], name: string): PythonBinding[] { + let line = lines[index]!; + if (statements[index]) for (let j = index + 1; j < lines.length && !statements[j]; j++) line += '\n' + lines[j]; + const out: PythonBinding[] = []; + // An import statement can name `name` on a continuation line: `from x import (\n name,\n)`. + const imported = statements[index] ? line.match(/^\s*(?:from\s+([\w.]+)\s+)?import\s+([\s\S]*)$/) : null; + if (imported) { + const names = imported[2]!; + // `from x import *` can bind any name. + if (imported[1] && names.trim() === '*') return [{ kind: 'other' }]; + if (!names.includes(name)) return out; + for (const part of names.replace(/[()\\]/g, ' ').split(',')) { + const m = part.trim().match(/^([\w.]+)(?:\s+as\s+(\w+))?$/); + const local = m && (m[2] ?? (imported[1] ? m[1]! : m[1]!.split('.')[0]!)); + if (local !== name) continue; + out.push({ kind: 'import', key: imported[1] ? `${imported[1]}:${m![1]}` : `${m![2] ? m![1] : local}:*` }); + } + return out; + } + if (!line.includes(name)) return out; + const word = new RegExp(`(? s.trim() === name) + ? [declared[1] === 'global' ? { kind: 'global' } : { kind: 'other' }] : []; + } + if (statements[index]) { + // A `case` pattern binds its capture names (`case [name]:`, `case Cls(k=name):`, `case name:`). + if (/^\s*case\b/.test(line)) return [{ kind: 'other' }]; + const assignment = pythonAssignment(line); + if (assignment) { + const target = assignment.targets.length === 1 && !assignment.augmented ? assignment.targets[0]!.trim() : ''; + const annotated = target.match(new RegExp(`^${name}\\s*:\\s*["']?([\\w.]+)["']?$`)); + if (target === name || annotated) { + out.push({ kind: 'assign', type: annotated?.[1] ?? null, value: assignment.value.trim(), line: index }); + } else if (assignment.targets.some(t => word.test(t))) { + out.push({ kind: 'other' }); + } + } else { + const annotated = line.match(new RegExp(`^\\s*${name}\\s*:\\s*["']?([\\w.]+)["']?\\s*$`)); + if (annotated) out.push({ kind: 'assign', type: annotated[1]!, value: '', line: index }); + else if (new RegExp(`^\\s*del\\b`).test(line)) out.push({ kind: 'other' }); + } + } + if (new RegExp(`\\b${name}[ \\t]*:=|\\bas[ \\t]+${name}\\b`).test(line)) out.push({ kind: 'other' }); + for (const loop of line.matchAll(/\bfor\s+([^:]+?)\s+in\b/g)) if (word.test(loop[1]!)) out.push({ kind: 'other' }); + for (const lambda of line.matchAll(/\blambda\b([^:]*):/g)) if (word.test(lambda[1]!)) out.push({ kind: 'other' }); + return out; +} + +/** + * Whether `name`, read at the ref, is bound by the calling function or one that + * encloses it (parameter, assignment, loop, `as`, lambda, import) rather than + * being the module global. With `importsBind` false, an import of the name is + * not a shadow: it binds the same module the file imports. + */ +function pythonBindsLocally(name: string, ref: UnresolvedRef, context: ResolutionContext, importsBind: boolean): boolean { + const lines = pythonMemberLines(ref.filePath, context); + const statements = pythonStatementStarts(ref.filePath, context); + const param = new RegExp(`[(,]\\s*\\*{0,2}${name}\\s*[:=,)]`); + const scopes = context.getNodesInFile(ref.filePath).filter(n => + (n.kind === 'function' || n.kind === 'method') && n.startLine <= ref.line && n.endLine >= ref.line) + .sort((a, b) => b.startLine - a.startLine); + for (const scope of scopes) { + const own = pythonOwnLines(scope, ref.filePath, context); + const bindings = own.flatMap(i => pythonLineBindings(lines, i, statements, name)); + if (bindings.some(b => b.kind === 'global')) return false; + const def = own.find(i => /^\s*(?:async\s+)?def\b/.test(lines[i]!)); + let header = scope.signature ?? ''; + for (let j = def ?? lines.length; j < lines.length && (j === def || !statements[j]); j++) header += lines[j]; + if (param.test(header)) return true; + if (bindings.some(b => b.kind !== 'global' && (b.kind !== 'import' || importsBind))) return true; + } + return false; +} + +/** Distinct sources a file imports `name` from, at any scope (`from a import x` → `a:x`). */ +function pythonImportKeys(name: string, filePath: string, context: ResolutionContext): Set { + return pythonNameScan(context, `imports\0${filePath}\0${name}`, () => scanPythonImportKeys(name, filePath, context)); +} + +const PYTHON_NAME_SCANS = new WeakMap>(); +/** Per-file, per-name scans are shared by every ref in the file; cleared with the other memos on sync. */ +function pythonNameScan(context: ResolutionContext, key: string, scan: () => T): T { + let memo = PYTHON_NAME_SCANS.get(context); + if (!memo) { memo = new Map(); PYTHON_NAME_SCANS.set(context, memo); } + if (!memo.has(key)) memo.set(key, scan()); + return memo.get(key) as T; +} + +function scanPythonImportKeys(name: string, filePath: string, context: ResolutionContext): Set { + const lines = pythonMemberLines(filePath, context); + const statements = pythonStatementStarts(filePath, context); + const keys = new Set(); + for (let i = 0; i < lines.length; i++) { + if (!statements[i] || !/\bimport\b/.test(lines[i]!)) continue; + for (const b of pythonLineBindings(lines, i, statements, name)) if (b.kind === 'import') keys.add(b.key); + } + return keys; +} + +/** Every binding of module global `name` in `filePath`: at module scope, and in each function that declares it `global`. */ +function pythonGlobalBindings(name: string, filePath: string, context: ResolutionContext): PythonBinding[] { + return pythonNameScan(context, `globals\0${filePath}\0${name}`, () => scanPythonGlobalBindings(name, filePath, context)); +} + +function scanPythonGlobalBindings(name: string, filePath: string, context: ResolutionContext): PythonBinding[] { + const lines = pythonMemberLines(filePath, context); + const statements = pythonStatementStarts(filePath, context); + const scopes = context.getNodesInFile(filePath).filter(n => n.kind === 'class' || n.kind === 'function' || n.kind === 'method'); + const regions = [pythonOwnLines(null, filePath, context)]; + for (let i = 0; i < lines.length; i++) { + if (!/^\s*global\b/.test(lines[i]!) || !pythonLineBindings(lines, i, statements, name).length) continue; + const scope = scopes.filter(n => n.startLine <= i + 1 && n.endLine >= i + 1).sort((a, b) => b.startLine - a.startLine)[0]; + if (scope && scope.kind !== 'class') regions.push(pythonOwnLines(scope, filePath, context)); + } + const script = pythonMainBlockLines(filePath, context); + return regions.flatMap(region => region.filter(i => !script.has(i)).flatMap(i => pythonLineBindings(lines, i, statements, name))) + .filter(b => b.kind !== 'global'); +} + +const PYTHON_GLOBAL_CLASSES = new WeakMap>(); +/** + * The classes a module global can hold. It has no static type (`conn = None`, + * rebound by `global conn; conn = Backend()`), so its type is the set of + * classes its module assigns to it: at module scope, or in a function that + * declares it `global`. Any other binding of it there (an opaque value, a + * tuple target, `for`/`with`/import, a star import, a `globals()` write) + * makes the type unknown (null). Production writes from other modules + * (`settings.conn = X`) join the set; see pythonExternalWrites. + */ +function pythonGlobalClasses(global: Node, ref: UnresolvedRef, context: ResolutionContext): Node[] | null { + let memo = PYTHON_GLOBAL_CLASSES.get(context); + if (!memo) { memo = new Map(); PYTHON_GLOBAL_CLASSES.set(context, memo); } + if (memo.has(global.id)) return memo.get(global.id)!; + const file = global.filePath; + const external = pythonExternalWrites(global, context); + // Each type is resolved in the file that wrote it (its imports name the class). + const writes: Array<{ type: string; file: string }> = [...external.writes]; + let classes: Node[] | null = external.unknown || pythonDynamicGlobalWrite(global.name, file, context) ? null : []; + for (const b of classes ? pythonGlobalBindings(global.name, file, context) : []) { + if (b.kind !== 'assign') { classes = null; break; } + const constructor = b.value && b.value !== 'None' ? pythonConstructorCall(b.value) : null; + if (b.type) { + // `conn: Base = make()` trusts the annotation; `conn: A = B()` contradicts it. + if (constructor && constructor.split('.').pop() !== b.type.split('.').pop()) { classes = null; break; } + writes.push({ type: b.type, file }); + continue; + } + if (b.value === 'None') continue; + if (!constructor) { classes = null; break; } + writes.push({ type: constructor, file }); + } + const seen = new Set(); + for (const write of classes ? writes : []) { + const cls = pythonRefClass(write.type, { ...ref, filePath: write.file }, context); + if (!cls) { classes = null; break; } + if (!seen.has(cls.id)) { seen.add(cls.id); classes!.push(cls); } + } + memo.set(global.id, classes); + return classes; +} + +/** + * The repo files a Python module path can name from `fromFile`. Relative + * paths (`..settings`) resolve exactly; absolute ones match a file path + * suffix, so a source root (`src/`) still resolves — and two files sharing + * the tail (`x/settings.py`, `y/settings.py`) both come back. + */ +function pythonModuleFiles(dotted: string, fromFile: string, context: ResolutionContext): string[] { + const dots = dotted.match(/^\.+/)?.[0].length ?? 0; + const parts = dotted.slice(dots).split('.').filter(Boolean); + if (!parts.length) return []; + let dir = ''; + if (dots) { + dir = path.posix.dirname(fromFile.replace(/\\/g, '/')); + for (let i = 1; i < dots; i++) dir = path.posix.dirname(dir); + if (dir === '.') dir = ''; + } + const rel = [dir, ...parts].filter(Boolean).join('/'); + const matches = (file: string, want: string) => dots ? file === want : file === want || file.endsWith(`/${want}`); + const last = parts[parts.length - 1]!; + return [ + ...context.getNodesByName(`${last}.py`), ...context.getNodesByName(`${last}.pyi`), ...context.getNodesByName('__init__.py'), + ].filter(n => n.kind === 'file' && (matches(n.filePath, `${rel}.py`) || matches(n.filePath, `${rel}.pyi`) || matches(n.filePath, `${rel}/__init__.py`))) + .map(n => n.filePath); +} + +/** + * How `filePath` spells the module `moduleFile`: `aliases` import exactly that + * file; `ambiguous` could also be another file sharing its dotted tail. + * `import a.b` binds `a`, so that module is spelled `a.b` (the mapping's + * last-segment `localName` is not a binding); `import a.b as c` binds `c`. + */ +function pythonModuleAliases(filePath: string, moduleFile: string, context: ResolutionContext): { aliases: string[]; ambiguous: string[] } { + const aliases = new Set(); + const ambiguous = new Set(); + for (const m of context.getImportMappings(filePath, 'python')) { + const dotted = m.isNamespace ? m.source + : /^\.+$/.test(m.source) ? `${m.source}${m.exportedName}` : `${m.source}.${m.exportedName}`; + const files = pythonModuleFiles(dotted, filePath, context); + if (!files.includes(moduleFile)) continue; + // The mapping cannot tell `import a.b` from `import a.b as b`; the source line can. + // Exactly this module (not `other.a.b`), outside string literals. + const explicit = new RegExp(`\\bimport\\s[^\\n]*(? explicit.test(blankPythonStrings(line))); + (files.length === 1 ? aliases : ambiguous).add(plainDotted ? m.source : m.localName); + } + return { aliases: [...aliases], ambiguous: [...ambiguous] }; +} + +const PYTHON_MAIN_GUARD = /^if\s+(?:__name__\s*==\s*(['"])__main__\1|(['"])__main__\2\s*==\s*__name__)\s*:/; +/** Line indexes inside a top-level `if __name__ == "__main__":` block — script code, not module state. */ +function pythonMainBlockLines(filePath: string, context: ResolutionContext): Set { + return pythonNameScan(context, `main\0${filePath}`, () => { + const lines = pythonMemberLines(filePath, context); + const inside = new Set(); + for (let i = 0; i < lines.length; i++) { + const guard = lines[i]!.match(PYTHON_MAIN_GUARD); + if (!guard) continue; + if (lines[i]!.slice(guard[0].length).trim()) inside.add(i); // `if __name__ == "__main__": stmt` + for (let j = i + 1; j < lines.length && !/^\S/.test(lines[j]!); j++) inside.add(j); + } + return inside; + }); +} + +/** Blank single-line string contents (triple-quoted strings are already blanked by the comment stripper). */ +function blankPythonStrings(text: string): string { + return text.replace(/(['"])(?:\\.|(?!\1)[^\\\n])*\1/g, m => m[0] + ' '.repeat(m.length - 2) + m[0]); +} + +/** Test code installs doubles: the narrow test-suite set, plus pytest's `conftest.py` wherever it sits. */ +function isPythonTestFile(filePath: string): boolean { + return isTestPath(filePath) || /(?:^|\/)conftest\.py$/.test(filePath.replace(/\\/g, '/')); +} + +/** + * Writes to module global `global` from OTHER files. A production write + * `. = Cls(...)` adds a type; any other production write + * (another value, a tuple target, `setattr`) makes the type unknown. Test + * files install doubles (`settings.conn = MagicMock()`, `monkeypatch.setattr`) + * that do not define the production type; they are recorded as `writers`, and + * a ref inside a writer resolves nothing through the global. + */ +function pythonExternalWrites(global: Node, context: ResolutionContext): { writes: Array<{ type: string; file: string }>; unknown: boolean; writers: Set } { + return pythonNameScan(context, `external\0${global.id}`, () => { + const out = { writes: [] as Array<{ type: string; file: string }>, unknown: false, writers: new Set() }; + const name = global.name; + const escape = (s: string) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); + for (const file of context.getAllFiles()) { + if (file === global.filePath || !/\.pyi?$/.test(file) || !context.readFile(file)?.includes(name)) continue; + const test = isPythonTestFile(file); + const { aliases, ambiguous } = pythonModuleAliases(file, global.filePath, context); + if (!test && !aliases.length && !ambiguous.length) continue; + const spell = (names: string[]) => `(?:${names.map(escape).join('|')})`; + // Test files: any receiver (the double may sit on any spelling of the module). + const receiver = test ? '[\\w.]+' : spell([...aliases, ...ambiguous]); + const target = new RegExp(`(? target.test(t.trim())) ?? false; + if (!assigned && !dynamic.test(statement)) continue; + if (test) { out.writers.add(file); continue; } + // A write through an ambiguous spelling may land on another module: unknown. + const single = assignment && assignment.targets.length === 1 && !assignment.augmented && exact.test(assignment.targets[0]!.trim()); + const value = single ? assignment!.value.trim() : ''; + if (value === 'None') continue; + const constructor = value ? pythonConstructorCall(value) : null; + if (!constructor) { out.unknown = true; return out; } + out.writes.push({ type: constructor, file }); + } + } + return out; + }); +} + +/** + * Whether the global's own module can write it through its namespace dict: + * `globals()` / `vars()` / `sys.modules[__name__]` used as anything but a + * literal-key read or `.get`, or a literal-key write of this name. Read per + * statement (continuations joined), with string contents ignored. + */ +function pythonDynamicGlobalWrite(name: string, filePath: string, context: ResolutionContext): boolean { + const lines = pythonMemberLines(filePath, context); + const statements = pythonStatementStarts(filePath, context); + const script = pythonMainBlockLines(filePath, context); + // `vars()` is the module dict only at module scope; inside a function it is the locals. + const moduleLines = new Set(pythonOwnLines(null, filePath, context)); + for (let i = 0; i < lines.length; i++) { + if (!statements[i] || script.has(i)) continue; + let statement = lines[i]!; + for (let j = i + 1; j < lines.length && !statements[j]; j++) statement += '\n' + lines[j]; + const code = blankPythonStrings(statement); + const namespace = moduleLines.has(i) + ? /\bglobals\(\s*\)|\bvars\(\s*\)|\bsys\.modules\s*\[\s*__name__\s*\]/g + : /\bglobals\(\s*\)|\bsys\.modules\s*\[\s*__name__\s*\]/g; + const uses = code.match(namespace)?.length ?? 0; + if (!uses) continue; + const targets = pythonAssignment(statement)?.targets ?? []; + let safe = 0; + for (const m of statement.matchAll(/\bglobals\(\s*\)\s*(?:\[\s*(['"])(\w+)\1\s*\]|\.\s*get\s*\()/g)) { + const key = m[2]; + const written = key !== undefined && targets.some(t => t.includes(m[0])); + if (written && key === name) return true; + safe++; + } + if (uses > safe) return true; + } + return false; +} + +/** + * The method a module global's value can dispatch to: one class resolves to + * its own method; several bind to the nearest declaration they all inherit, + * as a base-typed receiver does. Otherwise, no edge. + */ +function pythonGlobalMembers(global: Node, member: string, ref: UnresolvedRef, context: ResolutionContext): Node[] { + // A test that installs its own double sees the double, not the production type. + if (pythonExternalWrites(global, context).writers.has(ref.filePath)) return []; + const classes = pythonGlobalClasses(global, ref, context); + if (!classes) return []; + const targets = [...new Map(classes.flatMap(cls => pythonMembers(cls, member, ref, context)).map(n => [n.id, n])).values()]; + if (targets.length <= 1) return targets; + // Several targets: the nearest declaration every candidate class inherits, + // whether or not a candidate overrides it. + const owner = (n: Node) => context.getNodesInFile(n.filePath).find(c => + c.kind === 'class' && n.qualifiedName === `${c.qualifiedName}::${member}`); + const declarations = new Map(); + const queue = [...targets]; + while (queue.length && declarations.size < 32) { + const decl = queue.shift()!; + const cls = decl && owner(decl); + if (!cls || declarations.has(decl.id)) continue; + declarations.set(decl.id, { decl, cls }); + queue.push(...pythonBases(cls, ref, context).flatMap(base => pythonMembers(base, member, ref, context))); + } + const inherits = (cls: Node, base: Node) => cls.id === base.id || pythonDerivesFrom(cls, base, ref, context); + const shared = [...declarations.values()].filter(d => classes.every(c => inherits(c, d.cls))); + const nearest = shared.filter(d => shared.every(o => inherits(d.cls, o.cls))); + return nearest.length === 1 ? [nearest[0]!.decl] : targets; +} + +/** `Cls(...)` / `pkg.mod.Cls(...)` as the WHOLE (possibly multi-line) value; else null (`Cls() if x else y`). */ +function pythonConstructorCall(text: string): string | null { + // `(Cls())` is the same value; peel parentheses that wrap the whole expression. + for (let wrapped = text.trim(); wrapped.startsWith('('); ) { + let depth = 0; + let close = -1; + for (let i = 0; i < wrapped.length && close < 0; i++) { + if (wrapped[i] === '(') depth++; + else if (wrapped[i] === ')' && --depth === 0) close = i; + } + if (close !== wrapped.length - 1) break; + text = wrapped = wrapped.slice(1, -1).trim(); + } + const callee = text.match(/^((?:[A-Za-z_]\w*\.)*[A-Z]\w*)\s*\(/); + if (!callee) return null; + let depth = 0; + let quote = ''; + for (let i = callee[0].length - 1; i < text.length; i++) { + const c = text[i]!; + if (quote) { if (c === '\\') i++; else if (c === quote) quote = ''; continue; } + if (c === '"' || c === "'") quote = c; + else if (c === '(' || c === '[' || c === '{') depth++; + else if (c === ')' || c === ']' || c === '}') { + if (--depth === 0) return /^[\s\\]*$/.test(text.slice(i + 1)) ? callee[1]! : null; + } + } + return null; +} + /** Read a field's own annotation/initializer, or a constructor parameter assigned to it. */ function pythonFieldType(receiver: string, owner: Node, ref: UnresolvedRef, context: ResolutionContext): string | null { const lines = pythonMemberLines(ref.filePath, context); @@ -6094,6 +6590,9 @@ function getInferScanStates(context: ResolutionContext): Map Date: Fri, 2 Oct 2026 04:49:00 +0000 Subject: [PATCH 152/259] fix(cpp): bound the macro-visibility include walk (#2127) (#2292) * fix(cpp): bound the macro-visibility include walk (#2127) isVisibleCppMacro walks the call site's translation unit to learn whether a function-like macro is visible there (#1838). A header whose guard the walk cannot decide -- `#define X_H 1` (the OpenSceneGraph / osgEarth idiom, which guardsItself does not read as a guard), or any header reached under an unknown `#if`, where neither its guard nor `#pragma once` settles -- was re-scanned on every inclusion path, and every re-scan pushed its define events onto the cached timeline again. On a layered include graph that is 2^depth scans and 2^depth timeline entries: "Resolving refs" died with "Ineffective mark-compacts near heap limit" at the first ref whose file reached such a tree, at the same ref whatever the heap size. The walk now keeps a version that every observable change bumps (definitions, `#pragma once`, the macro-name set). A re-entry with the same inherited truth, while the version still equals that of an earlier visit which itself changed nothing, is skipped: it would start from the same state, take the same branches and change nothing again, and the events it would push repeat each macro's current state, already its last event. Include cycles cut by the recursion stack are recorded per visit and must still be cut for a skip. A hard budget of 1M directive events per walk bounds what is left; past it the timeline is unknown and nothing at or after the cut line is suppressed, so the budget can only fall back to ordinary resolution, never invent a macro. Answers are unchanged: 20,000 random include graphs (guards bare / valued / pragma once / under flags, #if/#elif/#else, #undef, cycles) give identical isVisibleCppMacro results to the previous walker on 1.25M queries, with about one re-entry in eight skipped. Co-Authored-By: Claude Opus 5.5 (cherry picked from commit 73db32f81ab9ffe1b02b4443914e9fd74cc2c4e1) * docs(changelog): credit the C/C++ macro walk bound Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: danusha2345 Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/cpp-macro-include-walk.test.ts | 125 +++++++++++++++++++++++ src/resolution/cpp-macro-visibility.ts | 68 ++++++++++-- 3 files changed, 185 insertions(+), 9 deletions(-) create mode 100644 __tests__/cpp-macro-include-walk.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index cdb9a6a6f8..b83668890e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -169,6 +169,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Python, a call written on the instance, like `self.get_breadcrumbs(...)`, now links to the class's own method even when the file also imports a function of the same name. Before, the import claimed it: Django REST Framework's renderers, netbox's change logging and django-allauth's account adapter all linked their own method calls to an imported helper. - In Python, a method passed as a value through a module-level variable that a setup function assigns, like `settings.docStoreConn.delete_payload_fields` after `global docStoreConn; docStoreConn = QdrantConnection()`, now links to that class's method. When several backend classes can be assigned, it links to the declaration they all inherit. Thanks @JosefAschauer. (#2074) - In Python, an import split over several lines in parentheses, `from x import (A, B)` written one name per line, is now read in full. Before, its names were invisible to resolution, so Django REST Framework's `serializers.CharField()` and netbox's model mixins linked to the wrong class or to nothing. Thanks @JosefAschauer. +- Indexing a large C or C++ project no longer runs out of memory partway through "Resolving refs". Headers whose include guard is written as `#define X_H 1`, as in OpenSceneGraph and osgEarth, or that are included under a build flag codegraph cannot know, used to be re-read on every include path, which grows exponentially with the depth of the include tree. Each is now read once per include state, and a hard limit stops the check on any include tree that is still too large. Thanks @danusha2345, and @coolhitmanleon for the report. (#2127) ## [1.6.1] - 2026-09-29 diff --git a/__tests__/cpp-macro-include-walk.test.ts b/__tests__/cpp-macro-include-walk.test.ts new file mode 100644 index 0000000000..acaf9c0d0f --- /dev/null +++ b/__tests__/cpp-macro-include-walk.test.ts @@ -0,0 +1,125 @@ +/** + * #2127 — the C/C++ macro-visibility walk must stay bounded on deep include + * graphs. + * + * A header whose guard the walk cannot decide — `#define X_H 1` (the OSG / + * osgEarth idiom, which the guard reader does not take for a guard), or any + * header reached under an unknown `#if` — used to be re-scanned on every + * inclusion path: a layered include graph where each header includes the two + * of the next layer cost 2^depth header scans and grew the timeline with + * every one of them, which is how "Resolving refs" ran out of heap on + * osgEarth. The answers must not change: a macro definitely visible at the + * call site is still a macro, and an unknown one still is not. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import { isVisibleCppMacro } from '../src/resolution/cpp-macro-visibility'; +import type { ResolutionContext, UnresolvedRef } from '../src/resolution/types'; + +type Guard = 'value' | 'bare' | 'unknown-flag'; + +/** `depth` layers of two headers; each includes both headers of the next layer. */ +function layeredHeaders(depth: number, guard: Guard): Record { + const files: Record = {}; + for (let d = 0; d < depth; d++) { + for (const side of ['a', 'b']) { + const g = `H_${d}_${side}`; + const open = + guard === 'value' ? `#ifndef ${g}\n#define ${g} 1\n` : + guard === 'bare' ? `#ifndef ${g}\n#define ${g}\n` : + `#ifdef USE_${d}\n`; + const next = d + 1 < depth ? `#include "h${d + 1}_a.h"\n#include "h${d + 1}_b.h"\n` : ''; + files[`h${d}_${side}.h`] = `${open}${next}#define M_${d}_${side}(x) (x)\nint f_${d}_${side}(int);\n#endif\n`; + } + } + return files; +} + +/** A context over in-memory files where every name is both a macro and a function. */ +function contextOver(files: Record): ResolutionContext { + return { + getNodesByName: (name: string) => [ + { kind: 'constant', signature: `#define ${name}(v) (v)` }, + { kind: 'function' }, + ], + readFile: (f: string) => files[f] ?? null, + fileExists: (f: string) => f in files, + getAllFiles: () => Object.keys(files), + } as unknown as ResolutionContext; +} + +const call = (name: string, line: number): UnresolvedRef => + ({ language: 'cpp', referenceKind: 'calls', referenceName: name, filePath: 'unit.cpp', line }) as UnresolvedRef; + +describe('#2127 — the macro-visibility include walk is bounded', () => { + it.each(['value', 'bare', 'unknown-flag'] as const)( + 'a 40-layer include graph (2^40 inclusion paths) is walked in well under a second (%s guards)', + (guard) => { + const files = { + ...layeredHeaders(40, guard), + 'trace.h': '#define TRACE(v) ((void)(v))\n', + 'unit.cpp': '#include "h0_a.h"\n#include "h0_b.h"\n#include "trace.h"\nvoid unit() { TRACE(1); }\n', + }; + const context = contextOver(files); + const started = performance.now(); + // Before the fix this walk never finished (2^40 header scans). + expect(isVisibleCppMacro(call('TRACE', 4), context)).toBe(true); + expect(performance.now() - started).toBeLessThan(1000); + // A header macro under an undecidable guard is not definitely visible, + // and a call above the include that defines TRACE is still a call. + expect(isVisibleCppMacro(call('M_39_a', 4), context)).toBe(guard === 'bare'); + expect(isVisibleCppMacro(call('TRACE', 2), context)).toBe(false); + }, + ); + + it('a walk that cannot be shortened stops at its budget and suppresses nothing past it', () => { + // No guards and a known flag toggled on every inclusion: each visit + // changes the state, so no re-entry repeats an earlier one and the real + // preprocessor would expand all 2^22 paths too. The walk gives up instead: + // before the exploding include the macro is still known, after it nothing is. + const files: Record = { + 'toggle.h': '#ifdef T\n#undef T\n#else\n#define T\n#endif\n', + 'unit.cpp': '#define T\n#define TRACE(v) ((void)(v))\nvoid before() { TRACE(1); }\n#include "h0.h"\nvoid after() { TRACE(2); }\n', + }; + for (let d = 0; d < 22; d++) { + files[`h${d}.h`] = d + 1 < 22 ? `#include "toggle.h"\n#include "h${d + 1}.h"\n#include "h${d + 1}.h"\n` : '#include "toggle.h"\n'; + } + const context = contextOver(files); + const started = performance.now(); + expect(isVisibleCppMacro(call('TRACE', 3), context)).toBe(true); + expect(isVisibleCppMacro(call('TRACE', 5), context)).toBe(false); + expect(performance.now() - started).toBeLessThan(10_000); + }); +}); + +describe('#2127 — indexing a deep include graph keeps #1838 macro suppression', () => { + const roots: string[] = []; + afterEach(() => { + for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true }); + }); + + it('the macro reached through value-guarded headers still does not bind to the decoy function', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cpp-walk-')); + roots.push(root); + const files = { + ...layeredHeaders(24, 'value'), + 'trace.h': '#define TRACE(v) ((void)(v))\n', + 'unit.cpp': '#include "h0_a.h"\n#include "h0_b.h"\n#include "trace.h"\nvoid unit() { TRACE(1); }\n', + 'decoy.cpp': 'void TRACE(int v) {}\nvoid caller() { TRACE(2); }\n', + }; + for (const [rel, content] of Object.entries(files)) fs.writeFileSync(path.join(root, rel), content); + const cg = await CodeGraph.init(root, { index: true }); + try { + const fn = (name: string) => cg.getNodesByKind('function').find((n) => n.name === name)!; + const callees = (name: string) => + cg.getCallees(fn(name).id).filter((r) => r.edge.kind === 'calls').map((r) => r.node.filePath); + expect(callees('unit')).toEqual([]); + expect(callees('caller')).toEqual(['decoy.cpp']); + } finally { + cg.close(); + } + }, 60_000); +}); diff --git a/src/resolution/cpp-macro-visibility.ts b/src/resolution/cpp-macro-visibility.ts index e5c76e2f11..b0b3ed18a8 100644 --- a/src/resolution/cpp-macro-visibility.ts +++ b/src/resolution/cpp-macro-visibility.ts @@ -15,6 +15,12 @@ * flags remain possible, so only definite macro visibility suppresses a call. * Object-like definitions participate in conditions but never enter the cached * call-site timelines (vendor headers can contain tens of thousands of them). + * + * A guard the walk cannot decide (`#define X_H 1`, or any header reached under + * an unknown `#if`) re-enters its header on every inclusion path, which is + * exponential in the include graph's depth (#2127). A re-entry that provably + * repeats an earlier visit which changed nothing is skipped, and a hard budget + * bounds whatever is left: past it, nothing is known and nothing is suppressed. */ import * as path from 'path'; import { maskCppRawStrings } from '../extraction/languages/c-cpp'; @@ -29,17 +35,22 @@ type FileEvent = | { kind: 'branch'; op: string; expression: string; guard: boolean; line: number } | { kind: 'once'; line: number }; type Event = { line: number; defined: Truth }; +/** Macro name → events in root-file line order; from `cutoff` on, nothing is known. */ +type Timeline = { events: Map; cutoff: number }; type Cache = { summaries: Map; includes: Map; /** Indexed files by basename, for `#include "dir/name.h"` that no include root explains. */ byBasename: Map | null; /** Per root file: macro name → define/undef events in root-file line order. */ - roots: Map>; + roots: Map; }; const memo = new WeakMap(); const ROOT_TIMELINE_CAP = 32; +/** Directive events one translation-unit walk may evaluate (#2127). */ +const WALK_EVENT_BUDGET = 1_000_000; +const NO_CUTS: string[] = []; const and = (a: Truth, b: Truth): Truth => a === false || b === false ? false : a === true && b === true ? true : undefined; @@ -79,7 +90,8 @@ export function isVisibleCppMacro(ref: UnresolvedRef, context: ResolutionContext if (cache.roots.size >= ROOT_TIMELINE_CAP) cache.roots.delete(cache.roots.keys().next().value!); cache.roots.set(rootKey, timeline); } - const before = (timeline.get(ref.referenceName) ?? []).filter((e) => e.line <= ref.line); + if (ref.line >= timeline.cutoff) return false; + const before = (timeline.events.get(ref.referenceName) ?? []).filter((e) => e.line <= ref.line); return before.length > 0 && before[before.length - 1]!.defined === true; } @@ -250,12 +262,24 @@ function walkTranslationUnit( language: UnresolvedRef['language'], context: ResolutionContext, cache: Cache -): Map { +): Timeline { const timeline = new Map(); const definitions = new Map(); const scanning = new Set(); const macroNames = new Set(); const once = new Map(); + // Every change a later directive could observe (definitions, `#pragma once`, + // the macro-name set) bumps `version`. A visit that left it unchanged, entered + // again with the same inherited truth while it is still unchanged, starts from + // the same state, so it would take the same branches and change nothing again; + // the events it would push repeat each name's current state, which is already + // its last event. Include cycles cut by the recursion stack (`cuts`) must still + // be cut for the repeat to hold. + let version = 0; + const cuts: string[] = []; + const visits = new Map(); + let budget = WALK_EVENT_BUDGET; + let cutoff = Infinity; const condition = (expression: string): Truth => { const text = expression.trim(); if (/^(?:0x[\da-f]+|\d+)[uUlL]*$/i.test(text)) return Number(text.replace(/[uUlL]+$/, '')) !== 0; @@ -267,12 +291,28 @@ function walkTranslationUnit( return /^\w+$/.test(text) ? definitions.get(text)?.value : undefined; }; const scan = (file: string, inherited: Truth, includeLine?: number): void => { - if (inherited === false || scanning.has(file) || once.get(file) === true) return; + if (inherited === false || once.get(file) === true || cutoff !== Infinity) return; + if (scanning.has(file)) { + cuts.push(file); + return; + } + const seen = visits.get(file); + if ( + seen && seen.inherited === inherited && seen.start === seen.end && seen.end === version && + seen.cuts.every((c) => scanning.has(c)) + ) { + cuts.push(...seen.cuts); + return; + } + const start = version; + const firstCut = cuts.length; scanning.add(file); let active: Truth = inherited; const frames: Array<{ parent: Truth; taken: Truth }> = []; for (const ev of summarize(file, context, cache)) { const line = includeLine ?? ev.line; + if (cutoff === Infinity && --budget < 0) cutoff = line; + if (cutoff !== Infinity) break; if (ev.kind === 'branch') { if (ev.op === 'if' || ev.op === 'ifdef' || ev.op === 'ifndef') { const known = definitions.get(ev.expression.trim())?.defined; @@ -294,7 +334,9 @@ function walkTranslationUnit( } if (active === false) continue; if (ev.kind === 'once') { - once.set(file, or(once.get(file) ?? false, active)); + const next = or(once.get(file) ?? false, active); + if (next !== (once.get(file) ?? false)) version++; + once.set(file, next); continue; } if (ev.kind === 'include') { @@ -309,12 +351,17 @@ function walkTranslationUnit( // A name no directive has touched is unknown, not undefined: the build // can set it on the command line. So an `#undef` under an undecidable // `#if` leaves it unknown (#2069); only a certain one clears it. - definitions.set(ev.name, { + const next = { defined: defining ? or(prior?.defined, active) : and(prior?.defined, not(active)), value: defining && active === true ? condition(ev.value) : undefined, macro: now, - }); - if (ev.functionLike) macroNames.add(ev.name); + }; + if (next.defined !== prior?.defined || next.value !== prior?.value || next.macro !== prior?.macro) version++; + definitions.set(ev.name, next); + if (ev.functionLike && !macroNames.has(ev.name)) { + macroNames.add(ev.name); + version++; + } if (macroNames.has(ev.name)) { const events = timeline.get(ev.name) ?? []; events.push({ line, defined: now }); @@ -322,8 +369,11 @@ function walkTranslationUnit( } } scanning.delete(file); + const own = cuts.length > firstCut ? [...new Set(cuts.splice(firstCut))] : NO_CUTS; + cuts.push(...own); + visits.set(file, { inherited, start, end: version, cuts: own }); }; scan(rootFile, true); - return timeline; + return { events: timeline, cutoff }; } From 7c01910e545165cae56353044014ca03d479e646 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 05:00:47 +0000 Subject: [PATCH 153/259] fix(cfml): tag-based calls, wrapped functions and are indexed (#2091) (#2296) * fix(cfml): tag-based calls, wrapped functions and are indexed (#2091) Tag-based CFML lost almost all of its call edges. The tag walk sent only and bodies through call extraction, although the cfml grammar parses the expressions in , /, and #hash# (output, strings, attributes) with the same rules as cfscript. The walk now records each such expression with the scope that owns it (the enclosing , else the component, else the file) and reads them all through the existing cfscript extraction in one parse per file: the expressions are copied out on their original lines, each ended with a `;`, and every reference is handed back to its expression's scope with its column restored. is read the same way. A nested in a generic container tag at component or template scope ( in Application.cfc, ) was skipped because only direct children were examined; it is now extracted (never through an ERROR node, where a lost would misfile methods as top-level functions). , a generic tag to the grammar, now yields an interface node with its functions as methods and an `extends` ref per comma-separated parent, so `implements` resolves to it. The tag-based parse tree is now freed after the walk. Resolution: the CFML/Ruby/C# bare-call scope filters evaluated the receiver-on-line check once per same-named candidate though only the ref decides it. With tag calls extracted, that made Mach-II's resolution ~8x slower; the check is now memoized per ref (edges byte-identical). Co-Authored-By: Claude Opus 5.5 * docs(changelog): tag-based CFML calls (#2091) Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/cfml-tag-calls.test.ts | 85 ++++++++ __tests__/extraction.test.ts | 156 +++++++++++++ src/extraction/cfml-extractor.ts | 363 ++++++++++++++++++++++++++----- src/resolution/name-matcher.ts | 14 ++ 5 files changed, 560 insertions(+), 59 deletions(-) create mode 100644 __tests__/cfml-tag-calls.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index b83668890e..01a871b167 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -170,6 +170,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Python, a method passed as a value through a module-level variable that a setup function assigns, like `settings.docStoreConn.delete_payload_fields` after `global docStoreConn; docStoreConn = QdrantConnection()`, now links to that class's method. When several backend classes can be assigned, it links to the declaration they all inherit. Thanks @JosefAschauer. (#2074) - In Python, an import split over several lines in parentheses, `from x import (A, B)` written one name per line, is now read in full. Before, its names were invisible to resolution, so Django REST Framework's `serializers.CharField()` and netbox's model mixins linked to the wrong class or to nothing. Thanks @JosefAschauer. - Indexing a large C or C++ project no longer runs out of memory partway through "Resolving refs". Headers whose include guard is written as `#define X_H 1`, as in OpenSceneGraph and osgEarth, or that are included under a build flag codegraph cannot know, used to be re-read on every include path, which grows exponentially with the depth of the include tree. Each is now read once per include state, and a hard limit stops the check on any include tree that is still too large. Thanks @danusha2345, and @coolhitmanleon for the report. (#2127) +- In tag-based CFML, calls written in tags are now linked: in ``, ``/``, ``, `` and `#…#` expressions. Before, only `` and `` code was read, so callers and impact found almost nothing in tag-based components. Functions wrapped in tags like `` or `` are now indexed too, and a `` is indexed as an interface that `implements` links to. Re-index CFML projects after upgrading. Thanks @HarryMuc for the report. (#2091) ## [1.6.1] - 2026-09-29 diff --git a/__tests__/cfml-tag-calls.test.ts b/__tests__/cfml-tag-calls.test.ts new file mode 100644 index 0000000000..f2741ed2d7 --- /dev/null +++ b/__tests__/cfml-tag-calls.test.ts @@ -0,0 +1,85 @@ +/** + * Tag-based CFML end to end (#2091). Most calls in a tag-based component sit + * in ``/``/`` and `#hash#` expressions, not in + * ``; on a 6,000-file tag-based codebase only 6.4% of methods had a + * caller in the graph. Functions wrapped in `` (the + * Application.cfc shape) and `` components were missing as well, + * so their callers and implementers had nothing to land on. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-cfml-tag-calls-')); + const files: Record = { + 'model/Svc.cfc': ` +\t +\t\t +\t\t +\t\t +\t +\t +\t +\t + +`, + 'Application.cfc': ` +\t +\t\t +\t\t\t +\t\t +\t\t +\t + +`, + 'model/ISearchable.cfc': ` +\t + +`, + 'model/Catalog.cfc': ` +\t + +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +const edgesFrom = (file: string, kind: string): string[] => { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg + .getOutgoingEdgesFrom(ids) + .filter((e) => e.kind === kind) + .map((e) => `${cg.getNode(e.source)!.qualifiedName} -> ${cg.getNode(e.target)!.qualifiedName}`) + .sort(); +}; + +describe('tag-based CFML (#2091)', () => { + it('links calls written in , and to the methods they call', () => { + expect(edgesFrom('model/Svc.cfc', 'calls')).toEqual(['Svc::a -> Svc::b', 'Svc::a -> Svc::c', 'Svc::a -> Svc::d']); + }); + + it('indexes a method wrapped in and the calls it makes', () => { + expect(edgesFrom('Application.cfc', 'calls')).toEqual(['Application::onRequestStart -> Application::loadConfig']); + }); + + it('resolves to a component', () => { + const iface = cg.getNodesInFile('model/ISearchable.cfc').find((n) => n.kind === 'interface'); + expect(iface?.name).toBe('ISearchable'); + expect(edgesFrom('model/Catalog.cfc', 'implements')).toEqual(['model/Catalog.cfc::Catalog -> model/ISearchable.cfc::ISearchable']); + }); +}); diff --git a/__tests__/extraction.test.ts b/__tests__/extraction.test.ts index 7fad827adb..c6a522decf 100644 --- a/__tests__/extraction.test.ts +++ b/__tests__/extraction.test.ts @@ -11099,6 +11099,162 @@ import foo.cfm; expect(result.nodes.find((n) => n.name === 'innerHelper')?.kind).toBe('function'); }); }); + + describe('Calls in tag expressions outside / (#2091)', () => { + const callsFrom = (result: ReturnType, fromId: string | undefined) => + result.unresolvedReferences + .filter((r) => r.fromNodeId === fromId && (r.referenceKind === 'calls' || r.referenceKind === 'instantiates')) + .map((r) => `${r.referenceKind === 'instantiates' ? 'new ' : ''}${r.referenceName}@${r.line}`) + .sort(); + + it('should extract calls in , , and , attributed to the enclosing method', () => { + const code = ` +\t +\t\t +\t\t +\t\t +\t\t +\t\t +\t +\t + +`; + const result = extractFromSource('Svc.cfc', code); + const a = result.nodes.find((n) => n.kind === 'method' && n.name === 'a'); + expect(a).toBeDefined(); + expect(callsFrom(result, a?.id)).toEqual(['b@3', 'c@4', 'd@7', 'e@5']); + }); + + it('should extract #hash# expressions in output, strings and tag attributes — once each, alongside bodies', () => { + const code = ` + + + + #fmt(y)# and #variables.mailer.send()# + + SELECT #col()# FROM t + + + +`; + const result = extractFromSource('View.cfc', code); + const render = result.nodes.find((n) => n.kind === 'method' && n.name === 'render'); + expect(callsFrom(result, render?.id)).toEqual([ + 'col@7', + 'dsn@7', + 'fmt@5', + 'getItems@6', + 'new Widget@8', + 'svc.load@3', + 'userName@4', + 'variables.mailer.send@5', + ]); + // `` is a function local — no variable node for it. + expect(result.nodes.find((n) => n.name === 'y')).toBeUndefined(); + }); + + it('should attribute component-scope and template-scope calls to the component and the file', () => { + const component = `\n\n\n\n`; + const cfc = extractFromSource('Pseudo.cfc', component); + const cls = cfc.nodes.find((n) => n.kind === 'class'); + expect(callsFrom(cfc, cls?.id)).toEqual(['setup@2']); + + const template = `\n#renderList(items)#\n`; + const cfm = extractFromSource('index.cfm', template); + const file = cfm.nodes.find((n) => n.kind === 'file'); + expect(callsFrom(cfm, file?.id)).toEqual(['loadItems@1', 'renderList@2']); + }); + + it("should keep each call's own line and column (several per line, and across lines)", () => { + const code = ` + + #last()# + + +`; + const lines = code.split('\n'); + const result = extractFromSource('Pos.cfc', code); + const calls = result.unresolvedReferences.filter((r) => r.referenceKind === 'calls'); + expect(calls.map((r) => r.referenceName).sort()).toEqual(['a', 'b.c', 'count', 'd', 'e', 'last', 'other.fetch', 'sum']); + for (const r of calls) { + expect(lines[r.line - 1]!.slice(r.column).startsWith(r.referenceName)).toBe(true); + } + }); + + it('should read a attribute as an expression', () => { + const code = `\n\n\n\n\n\n`; + const result = extractFromSource('Queue.cfc', code); + const drain = result.nodes.find((n) => n.kind === 'method' && n.name === 'drain'); + expect(callsFrom(result, drain?.id)).toEqual(['hasNext@3']); + }); + + it('should not read HTML \n

    also(not)

    \n    \n\n`; + const result = extractFromSource('Page.cfc', code); + expect(result.unresolvedReferences.filter((r) => r.referenceKind === 'calls')).toEqual([]); + }); + }); + + describe(' inside a generic container tag (#2091)', () => { + it('should extract a method nested in (the Application.cfc shape)', () => { + const code = ` +\t +\t\t +\t\t\t +\t\t +\t + +`; + const result = extractFromSource('Application.cfc', code); + const cls = result.nodes.find((n) => n.kind === 'class'); + const method = result.nodes.find((n) => n.kind === 'method' && n.name === 'onRequestStart'); + expect(method).toBeDefined(); + expect(method?.qualifiedName).toBe('Application::onRequestStart'); + expect(result.edges.some((e) => e.kind === 'contains' && e.source === cls?.id && e.target === method?.id)).toBe(true); + const call = result.unresolvedReferences.find((r) => r.referenceName === 'loadConfig'); + expect(call?.fromNodeId).toBe(method?.id); + }); + + it('should extract a top-level function nested in in a template', () => { + const code = `\n\n\n`; + const result = extractFromSource('lib.cfm', code); + expect(result.nodes.find((n) => n.name === 'helper')?.kind).toBe('function'); + }); + }); + + describe(' (#2091)', () => { + const code = ` +\t +\t\t +\t +\t + +`; + + it('should extract an interface node named from the file, with its functions as methods', () => { + const result = extractFromSource('ISearchable.cfc', code); + const iface = result.nodes.find((n) => n.kind === 'interface'); + expect(iface?.name).toBe('ISearchable'); + expect(iface?.startLine).toBe(1); + expect(iface?.endLine).toBe(6); + expect(result.nodes.filter((n) => n.kind === 'class')).toHaveLength(0); + const methods = result.nodes.filter((n) => n.kind === 'method'); + expect(methods.map((m) => m.qualifiedName).sort()).toEqual(['ISearchable::count', 'ISearchable::search']); + for (const m of methods) { + expect(result.edges.some((e) => e.kind === 'contains' && e.source === iface?.id && e.target === m.id)).toBe(true); + } + expect(methods.find((m) => m.name === 'search')?.returnType).toBe('any'); + }); + + it('should extract each interface it extends', () => { + const result = extractFromSource('ISearchable.cfc', code); + const iface = result.nodes.find((n) => n.kind === 'interface'); + const ext = result.unresolvedReferences.filter((r) => r.referenceKind === 'extends'); + expect(ext.map((r) => r.referenceName)).toEqual(['IBase', 'IOther']); + expect(ext.every((r) => r.fromNodeId === iface?.id)).toBe(true); + }); + }); }); describe('COBOL Extraction', () => { diff --git a/src/extraction/cfml-extractor.ts b/src/extraction/cfml-extractor.ts index 8395940c10..efd2ea6ea6 100644 --- a/src/extraction/cfml-extractor.ts +++ b/src/extraction/cfml-extractor.ts @@ -4,6 +4,55 @@ import { generateNodeId, NodeIdAllocator } from './tree-sitter-helpers'; import { TreeSitterExtractor } from './tree-sitter'; import { getParser } from './grammars'; +/** Tags whose own children include an expression: ``, ``, ``, ``, and `#…#`. */ +const TAG_EXPRESSION_PARENTS: ReadonlySet = new Set([ + 'cf_set_tag', + 'cf_if_tag', + 'cf_elseif_tag', + 'cf_return_tag', + 'hash_expression', +]); + +/** + * The expression node types (shared with the cfscript grammar) that can + * contain a call. A positive list, so a tag's markup — body content, its + * `var` keyword, an ERROR — is never mistaken for an expression; a bare + * identifier or literal can't call anything and is skipped. + */ +const TAG_EXPRESSION_TYPES: ReadonlySet = new Set([ + 'call_expression', + 'member_expression', + 'subscript_expression', + 'new_expression', + 'assignment_expression', + 'augmented_assignment_expression', + 'binary_expression', + 'unary_expression', + 'update_expression', + 'ternary_expression', + 'elvis_expression', + 'parenthesized_expression', + 'sequence_expression', + 'string', + 'array', + 'object', + 'ordered_struct', + 'function_expression', + 'arrow_function', +]); + +/** An expression in tag markup and the scope (function, component or file node) that owns its calls. */ +interface TagExpression { + startIndex: number; + endIndex: number; + /** Where it starts in the file — the same line in the source extractTagExpressions synthesizes. */ + row: number; + column: number; + /** Its start column in the synthesized source. */ + textColumn: number; + scopeId: string; +} + /** * CfmlExtractor - Extracts code relationships from CFML source (.cfc/.cfm). * @@ -15,7 +64,9 @@ import { getParser } from './grammars'; * raw AST, so this extractor replicates it: a file whose first real token * isn't `<` is delegated wholesale to the cfscript grammar (the dominant * modern style); otherwise the file is walked tag-by-tag with the cfml - * grammar, delegating any `` tag bodies the same way. + * grammar, delegating any `` tag bodies the same way — and the + * expressions written in tags themselves (``, ``, `#hash#`, …) + * through the same cfscript extraction (see extractTagExpressions). */ export class CfmlExtractor { private filePath: string; @@ -26,6 +77,7 @@ export class CfmlExtractor { private edges: Edge[] = []; private unresolvedReferences: UnresolvedReference[] = []; private errors: ExtractionError[] = []; + private tagExpressions: TagExpression[] = []; /** `language` is the file's detected language — `'cfml'` for `.cfc`/`.cfm`, `'cfscript'` for `.cfs`. Both dialect-switch internally; this only controls the language tag stamped onto emitted nodes/refs. */ constructor(filePath: string, source: string, language: Language = 'cfml') { @@ -113,8 +165,14 @@ export class CfmlExtractor { return; } - const fileNode = this.createFileNode(); - this.walkProgram(tree.rootNode, fileNode.id); + try { + const fileNode = this.createFileNode(); + this.walkProgram(tree.rootNode, fileNode.id); + } finally { + // Trees hold wasm heap memory V8's GC never sees — free it per file. + tree.delete(); + } + this.extractTagExpressions(); } /** Build the file's own `kind:'file'` node, spanning the whole source. Tag-based files need this explicitly — unlike `extractBareScript` (which delegates the whole file to `TreeSitterExtractor` and inherits its file node), `extractTagBased` walks the tree itself and has no other source of one. */ @@ -151,17 +209,11 @@ export class CfmlExtractor { if (child.type === 'cf_component_open_tag') { child = this.extractComponent(child, fileNodeId).nextSibling; continue; - } else if (child.type === 'cf_function_tag') { - // A cffunction outside any cfcomponent wrapper (rare, but legal in a - // .cfm template) — extract as a top-level function, contained by the file. - this.extractFunctionTag(child, undefined, fileNodeId); - } else if (child.type === 'cf_script_tag') { - this.delegateScriptTag(child, fileNodeId); - } else if (child.type === 'cf_query_tag') { - this.delegateQueryTag(child, fileNodeId); - } else { - this.delegateNestedTags(child, fileNodeId); } + // Template scope: a cffunction outside any cfcomponent wrapper (rare, + // but legal in a .cfm template) is a top-level function contained by + // the file, and template code's calls are the file's. + this.visitTag(child, root.type, fileNodeId, undefined, true); child = child.nextSibling; } } @@ -208,20 +260,7 @@ export class CfmlExtractor { language: this.language, }); } - const implementsAttr = this.tagAttr(openTag, 'implements'); - if (implementsAttr) { - for (const iface of implementsAttr.split(',').map((s) => s.trim()).filter(Boolean)) { - this.unresolvedReferences.push({ - fromNodeId: classNode.id, - referenceName: iface, - referenceKind: 'implements', - filePath: this.filePath, - line: openTag.startPosition.row + 1, - column: openTag.startPosition.column, - language: this.language, - }); - } - } + this.pushInheritanceRefs(classNode.id, this.tagAttr(openTag, 'implements'), 'implements', openTag); // Walk siblings between the open tag and its close tag. let sibling = openTag.nextSibling; @@ -231,15 +270,9 @@ export class CfmlExtractor { lastNode = sibling; break; } - if (sibling.type === 'cf_function_tag') { - this.extractFunctionTag(sibling, classNode.id, classNode.id, classNode.name); - } else if (sibling.type === 'cf_script_tag') { - this.delegateScriptTag(sibling, classNode.id, classNode.name); - } else if (sibling.type === 'cf_query_tag') { - this.delegateQueryTag(sibling, classNode.id); - } else { - this.delegateNestedTags(sibling, classNode.id, classNode.name); - } + // Component scope: cffunctions are its methods, pseudo-constructor + // code's calls (a component-level ``) are the component's. + this.visitTag(sibling, openTag.parent?.type ?? '', classNode.id, classNode.name, true); lastNode = sibling; sibling = sibling.nextSibling; } @@ -247,6 +280,54 @@ export class CfmlExtractor { return lastNode; } + /** + * `...` (#2091). Unlike + * `` (see the implicit-end-tag note on `extractComponent`) the + * grammar has no dedicated rule for it: it is a generic `cf_tag` whose body + * — the `` signatures — is ordinary nested children. Named like + * a component (the file name, unless a `name` attribute says otherwise), so + * `` resolves to it. An interface may + * extend several interfaces, comma-separated. + */ + private extractInterfaceTag(tag: SyntaxNode, containerId: string): void { + const startTag = this.cfStartTag(tag) ?? tag; + const name = this.tagAttr(startTag, 'name') ?? this.componentNameFromPath(); + const id = this.nodeIds.generate(this.filePath, 'interface', name, tag.startPosition.row + 1, tag.startPosition.column); + this.nodes.push({ + id, + kind: 'interface', + name, + qualifiedName: `${this.filePath}::${name}`, + filePath: this.filePath, + language: this.language, + startLine: tag.startPosition.row + 1, + endLine: tag.endPosition.row + 1, + startColumn: tag.startPosition.column, + endColumn: tag.endPosition.column, + isExported: true, + updatedAt: Date.now(), + }); + this.edges.push({ source: containerId, target: id, kind: 'contains' }); + this.pushInheritanceRefs(id, this.tagAttr(startTag, 'extends'), 'extends', startTag); + this.delegateNestedTags(tag, id, name, true); + } + + /** One unresolved `extends`/`implements` ref per name in a comma-separated tag attribute. */ + private pushInheritanceRefs(fromNodeId: string, list: string | undefined, kind: 'extends' | 'implements', tag: SyntaxNode): void { + if (!list) return; + for (const name of list.split(',').map((s) => s.trim()).filter(Boolean)) { + this.unresolvedReferences.push({ + fromNodeId, + referenceName: name, + referenceKind: kind, + filePath: this.filePath, + line: tag.startPosition.row + 1, + column: tag.startPosition.column, + language: this.language, + }); + } + } + /** * `...`. * `parentClassId` decides `method` vs top-level `function`; `containerId` is @@ -290,36 +371,190 @@ export class CfmlExtractor { this.edges.push({ source: containerId, target: fnNode.id, kind: 'contains' }); } - // Delegate any / bodies nested inside this function, at - // any depth (e.g. inside // control-flow tags). + // Walk the body, at any depth (e.g. inside // + // control-flow tags): its / bodies and tag + // expressions are this function's. this.delegateNestedTags(tag, fnNode.id); } /** - * Recursively delegates any `cf_script_tag`/`cf_query_tag` found within - * `node`'s subtree — e.g. a ``/`` nested inside - * ``/``/`` control-flow tags, which (unlike - * ``'s body — see the implicit-end-tag note on `extractComponent`) - * ARE normal children, just possibly several levels deep, so a direct-children - * check misses them. Does not descend into a nested `cf_function_tag` — that - * has its own scope and is walked separately. `parentClassName` rides along - * so a `` at component scope classifies its functions as methods - * scoped under the component. + * Visit one node of tag markup. `containerId` is the scope the node sits in + * — the function whose body it is, else the component, else the file — and + * owns whatever calls it makes. `declScope` is true outside function bodies + * (component/interface/template scope), where a `` declares a + * function of that scope; `parentClassName` is set at component/interface + * scope, where those functions are methods. + * + * - ``/`` bodies go to their own grammars (see + * delegateScriptTag / delegateQueryTag). + * - A `` is extracted when reached at declaration scope, at any + * depth: `` around Application.cfc's handlers + * (#2091) or `` around a template's helpers doesn't make them + * any less the scope's functions. Never through an ERROR node — at file + * scope that is typically a `` the parser lost (behind an + * unclosed ``), whose methods must not be misfiled as + * top-level functions. Inside a function body a `` would be + * its own scope; CFML rejects that, so it is skipped. + * - An expression in tag position — the value of ``, the condition + * of ``/``, the operand of ``, a `#hash#` + * anywhere in markup, attributes and strings included — is recorded for + * extractTagExpressions. The grammar parses these structurally (the same + * expression rules as cfscript), so `parentType` (the node's parent) is + * enough to tell the expression from the tag around it. + */ + private visitTag(node: SyntaxNode, parentType: string, containerId: string, parentClassName: string | undefined, declScope: boolean): void { + switch (node.type) { + case 'cf_script_tag': + this.delegateScriptTag(node, containerId, parentClassName); + return; + case 'cf_query_tag': + // The SQL body is opaque to this grammar; the walk below only reaches + // the tag's attributes (`datasource="#dsn()#"`), never the body twice. + this.delegateQueryTag(node, containerId); + break; + case 'cf_function_tag': + if (declScope) this.extractFunctionTag(node, parentClassName ? containerId : undefined, containerId, parentClassName); + return; + case 'cf_tag': { + const name = this.cfTagName(node); + if (name === 'interface' && declScope && !parentClassName) { + this.extractInterfaceTag(node, containerId); + return; + } + // `` — the one common tag attribute whose + // plain-text value is a CFML expression rather than a literal. + if (name === 'loop') { + const condition = this.tagAttrValueNode(this.cfStartTag(node) ?? node, 'condition'); + if (condition) this.addTagExpression(condition, containerId); + } + break; + } + default: + if (TAG_EXPRESSION_PARENTS.has(parentType) && TAG_EXPRESSION_TYPES.has(node.type)) { + this.addTagExpression(node, containerId); + return; + } + } + this.delegateNestedTags(node, containerId, parentClassName, declScope && node.type !== 'ERROR'); + } + + /** + * Visit `node`'s children (see visitTag) — e.g. a ``/`` + * nested inside ``/``/`` control-flow tags, which + * (unlike ``'s body — see the implicit-end-tag note on + * `extractComponent`) ARE normal children, just possibly several levels + * deep, so a direct-children check misses them. `parentClassName` rides + * along so a `` at component scope classifies its functions as + * methods scoped under the component. */ - private delegateNestedTags(node: SyntaxNode, containerId: string | undefined, parentClassName?: string): void { + private delegateNestedTags(node: SyntaxNode, containerId: string, parentClassName?: string, declScope = false): void { for (let i = 0; i < node.namedChildCount; i++) { const child = node.namedChild(i); - if (!child) continue; - if (child.type === 'cf_script_tag') { - this.delegateScriptTag(child, containerId, parentClassName); - } else if (child.type === 'cf_query_tag') { - this.delegateQueryTag(child, containerId); - } else if (child.type === 'cf_function_tag') { - continue; + if (child) this.visitTag(child, node.type, containerId, parentClassName, declScope); + } + } + + /** Record an expression in tag markup, owned by `scopeId`, for extractTagExpressions. */ + private addTagExpression(node: SyntaxNode, scopeId: string): void { + if (node.endIndex <= node.startIndex) return; + this.tagExpressions.push({ + startIndex: node.startIndex, + endIndex: node.endIndex, + row: node.startPosition.row, + column: node.startPosition.column, + textColumn: 0, + scopeId, + }); + } + + /** + * Extract calls from the expressions visitTag recorded, through the same + * cfscript extraction a `` body gets — one parse per file, not + * one per expression: the expressions are copied out in order, each on its + * own line (the line breaks between them are kept, so every line number is + * the file's) and ended with a `;`, and the result is read with the + * cfscript grammar. Each reference is handed back to the scope whose + * expression contains it, its column shifted back to the file's. Only + * references are kept: an expression declares nothing (`` + * is a function local, and its `var` isn't copied), so a closure written in + * one is part of its scope's code. + */ + private extractTagExpressions(): void { + const exprs = this.tagExpressions; + if (exprs.length === 0) return; + exprs.sort((a, b) => a.startIndex - b.startIndex); + + const parts: string[] = []; + let pos = 0; + let column = 0; // column in the synthesized source + for (const expr of exprs) { + if (pos > 0) { + parts.push(';'); + column++; + } + let breaks = 0; + for (let at = this.source.indexOf('\n', pos); at !== -1 && at < expr.startIndex; at = this.source.indexOf('\n', at + 1)) breaks++; + if (breaks > 0) { + parts.push('\n'.repeat(breaks)); + column = 0; + } + expr.textColumn = column; + const text = this.source.slice(expr.startIndex, expr.endIndex); + parts.push(text); + const lastBreak = text.lastIndexOf('\n'); + column = lastBreak === -1 ? column + text.length : text.length - lastBreak - 1; + pos = expr.endIndex; + } + + const result = new TreeSitterExtractor(this.filePath, parts.join(''), 'cfscript').extract(); + for (const ref of result.unresolvedReferences) { + const expr = this.tagExpressionAt(ref.line - 1, ref.column); + // Only an expression's first line moved; its later lines are verbatim. + if (ref.line - 1 === expr.row) ref.column += expr.column - expr.textColumn; + ref.fromNodeId = expr.scopeId; + ref.filePath = this.filePath; + ref.language = this.language; + this.unresolvedReferences.push(ref); + } + // A broken expression is routine in hand-written markup and the file's + // symbols came from the tag walk, so only a genuine failure is reported — + // not the extractor's "no symbols" warning, which an expressions-only + // source always earns. + for (const error of result.errors) { + if (error.severity === 'error') this.errors.push(error); + } + } + + /** The recorded expression containing a synthesized-source position: the last one starting at or before it. */ + private tagExpressionAt(row: number, column: number): TagExpression { + const exprs = this.tagExpressions; + let lo = 0; + let hi = exprs.length - 1; + let found = 0; + while (lo <= hi) { + const mid = (lo + hi) >> 1; + const e = exprs[mid]!; + if (e.row < row || (e.row === row && e.textColumn <= column)) { + found = mid; + lo = mid + 1; } else { - this.delegateNestedTags(child, containerId, parentClassName); + hi = mid - 1; } } + return exprs[found]!; + } + + /** A generic `cf_tag`'s start tag — where its name and attributes live. */ + private cfStartTag(tag: SyntaxNode): SyntaxNode | undefined { + return tag.namedChildren.find( + (c: SyntaxNode) => c.type === 'cf_start_tag' || c.type === 'cf_start_tag_with_selfclose' + ); + } + + /** A generic `cf_tag`'s name, lowercased, without the `cf` prefix (`` → `loop`). */ + private cfTagName(tag: SyntaxNode): string | undefined { + const nameNode = this.cfStartTag(tag)?.namedChildren.find((c: SyntaxNode) => c.type === 'cf_tag_name'); + return nameNode ? this.source.substring(nameNode.startIndex, nameNode.endIndex).toLowerCase() : undefined; } /** @@ -445,6 +680,18 @@ export class CfmlExtractor { /** Read a `cf_attribute`'s value by name from a tag node's direct `cf_attribute`/`cf_tag_attributes` children. */ private tagAttr(tag: SyntaxNode, attrName: string): string | undefined { + const valueNode = this.tagAttrValueNode(tag, attrName); + if (valueNode === undefined) return undefined; + if (!valueNode) return ''; + return this.source.substring(valueNode.startIndex, valueNode.endIndex); + } + + /** + * The `attribute_value` node of a tag attribute — `undefined` when the tag + * has no such attribute, `null` when its value isn't plain text (empty, or + * a `#hash#` expression). + */ + private tagAttrValueNode(tag: SyntaxNode, attrName: string): SyntaxNode | null | undefined { const attrs: SyntaxNode[] = []; for (let i = 0; i < tag.namedChildCount; i++) { const child = tag.namedChild(i); @@ -467,9 +714,7 @@ export class CfmlExtractor { const valueWrapper = attr.namedChildren.find( (c: SyntaxNode) => c.type === 'quoted_cf_attribute_value' || c.type === 'cf_attribute_value' ); - const valueNode = valueWrapper?.namedChildren.find((c: SyntaxNode) => c.type === 'attribute_value'); - if (!valueNode) return ''; - return this.source.substring(valueNode.startIndex, valueNode.endIndex); + return valueWrapper?.namedChildren.find((c: SyntaxNode) => c.type === 'attribute_value') ?? null; } return undefined; } diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 701c0ea03c..e0b4ad28b8 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -4644,6 +4644,8 @@ function cfmlComponentFiles(dotted: string, from: string, context: ResolutionCon return files; } +const NO_RECEIVER_LINES = new WeakMap>(); + /** * Whether a bare reference is receiver-less at its call site, name case * aside: the name is not preceded by a `.` on its line (true when the line @@ -4651,6 +4653,17 @@ function cfmlComponentFiles(dotted: string, from: string, context: ResolutionCon * links of a chain (`newFuture(f).then(g)`) over bare. */ function hasNoReceiverOnLine(ref: UnresolvedRef, context: ResolutionContext): boolean { + // Asked once per CANDIDATE by the per-language scope filters, though only + // the ref decides it — a bare `init()` in a CFML codebase has hundreds of + // same-named methods, each re-reading the line (#2091). + let memo = NO_RECEIVER_LINES.get(context); + if (!memo) NO_RECEIVER_LINES.set(context, (memo = new WeakMap())); + let answer = memo.get(ref); + if (answer === undefined) memo.set(ref, (answer = readNoReceiverOnLine(ref, context))); + return answer; +} + +function readNoReceiverOnLine(ref: UnresolvedRef, context: ResolutionContext): boolean { const line = context.getFileLines?.(ref.filePath)?.[ref.line - 1] ?? context.readFile(ref.filePath)?.split('\n')[ref.line - 1]; if (line === undefined) return true; const lower = line.toLowerCase(); @@ -6629,6 +6642,7 @@ export function clearNameMatcherMemos(context: ResolutionContext): void { KOTLIN_FILE_SCOPES.delete(context); RUBY_ANCESTRY.delete(context); CFML_CHAINS.delete(context); + NO_RECEIVER_LINES.delete(context); OBJC_SUPERS.delete(context); CSHARP_SUPERS.delete(context); CSHARP_STATIC_USINGS.delete(context); From 45a680df07df2297dd7997a9376501351088eee9 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 06:16:25 +0000 Subject: [PATCH 154/259] fix(mcp): one daemon per project however its root is cased on Windows (#2278) (#2285) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(mcp): canonicalize the daemon rendezvous root on Windows A project spelled d:\work\codegraph or D:\work\codegraph is one directory, but the daemon socket/pipe name and the registry record name hashed a raw path.resolve, which preserves whichever casing the caller passed. A proxy that computed the other casing probed a socket nobody bound, spawned a redundant daemon, and that daemon exited on the lock the running one held, so the session silently served the graph in-process: no shared watcher, no auto-sync, one extra copy of the index per open window. Both names now come from one shared canonicalProjectRoot (native realpath, lowercased on win32), so every spelling meets the same daemon and codegraph list / stop --all see one record per project. (cherry picked from commit 181178801827351d713e1d552886c0482e6db67b) * fix(mcp): share one in-process project however its root is cased (#2278) The engine's project lifecycle and `codegraph stop`'s registry fallback keyed a project by its realpath, which keeps the caller's casing, so two spellings of one root opened two graphs and two watchers in one process. Both now use the same identity key as the daemon socket. Co-Authored-By: Claude Opus 5.5 (cherry picked from commit 0a3239e603d5796b9f2cf636b4c09635239e32dd) * docs(changelog): file the Windows rendezvous fix under Unreleased Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: 养心堂主 Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/daemon-rendezvous-case.test.ts | 123 +++++++++++++++++++++++ src/directory.ts | 34 +++++++ src/mcp/daemon-paths.ts | 12 ++- src/mcp/daemon-registry.ts | 11 +- src/mcp/project-lifecycle.ts | 13 ++- 6 files changed, 184 insertions(+), 10 deletions(-) create mode 100644 __tests__/daemon-rendezvous-case.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 01a871b167..62d005a12f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -171,6 +171,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Python, an import split over several lines in parentheses, `from x import (A, B)` written one name per line, is now read in full. Before, its names were invisible to resolution, so Django REST Framework's `serializers.CharField()` and netbox's model mixins linked to the wrong class or to nothing. Thanks @JosefAschauer. - Indexing a large C or C++ project no longer runs out of memory partway through "Resolving refs". Headers whose include guard is written as `#define X_H 1`, as in OpenSceneGraph and osgEarth, or that are included under a build flag codegraph cannot know, used to be re-read on every include path, which grows exponentially with the depth of the include tree. Each is now read once per include state, and a hard limit stops the check on any include tree that is still too large. Thanks @danusha2345, and @coolhitmanleon for the report. (#2127) - In tag-based CFML, calls written in tags are now linked: in ``, ``/``, ``, `` and `#…#` expressions. Before, only `` and `` code was read, so callers and impact found almost nothing in tag-based components. Functions wrapped in tags like `` or `` are now indexed too, and a `` is indexed as an interface that `implements` links to. Re-index CFML projects after upgrading. Thanks @HarryMuc for the report. (#2091) +- On Windows, a project opened with a different drive-letter or folder-name case, like `d:\work\app` and `D:\Work\App`, now joins the CodeGraph background server that is already running for it. Before, each spelling started a server of its own that exited at once, so the session quietly served the graph by itself with no shared file watching or auto-sync, and `codegraph list` showed the project twice. Thanks @greatjackzhou. (#2278) ## [1.6.1] - 2026-09-29 diff --git a/__tests__/daemon-rendezvous-case.test.ts b/__tests__/daemon-rendezvous-case.test.ts new file mode 100644 index 0000000000..6c8a1a1a3e --- /dev/null +++ b/__tests__/daemon-rendezvous-case.test.ts @@ -0,0 +1,123 @@ +/** + * One project must map to ONE daemon rendezvous key, however its root is + * spelled — the regression behind "the MCP server starts but never shares the + * daemon". + * + * The daemon's named pipe (Windows) / tmpdir socket (POSIX) is + * `…codegraph-.slice(0,16)`, and the lockfile is shared + * while the SOCKET NAME is derived independently by each process. So the moment + * two processes hash the same directory differently they stop meeting: the + * proxy's probe finds nothing, it spawns a redundant daemon, and that daemon + * exits on the lock the first one holds + * (`Another daemon (pid N) already holds the lock; exiting.`) — the session then + * serves in-process, without the shared watcher or auto-sync. + * + * That is reachable on Windows because the root arrives two ways: a cwd-derived + * root is the on-disk casing (`D:\work\codegraph`), while a client-supplied + * `rootUri`/`workspaceFolders` path arrives as `file:///d%3A/…` → `d:\…`, and + * `path.resolve` preserves whichever it got (NTFS is case-insensitive, so both + * name one directory). Pinned here at the level that actually broke: the key, + * not the filename. + * + * The real-filesystem assertions are Windows-gated — on POSIX the two spellings + * are genuinely different directories, so only the Windows case has a + * case-insensitive filesystem to converge on. + */ + +import { afterEach, describe, expect, it } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { canonicalProjectRoot } from '../src/directory'; +import { getDaemonSocketPath } from '../src/mcp/daemon-paths'; +import { acquireProject } from '../src/mcp/project-lifecycle'; +import type CodeGraph from '../src/index'; + +const tmpDirs: string[] = []; +function makeDir(): string { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-rendezvous-')); + tmpDirs.push(dir); + return dir; +} +afterEach(() => { + while (tmpDirs.length) { + try { + fs.rmSync(tmpDirs.pop()!, { recursive: true, force: true }); + } catch { + /* best-effort */ + } + } +}); + +/** The same drive letter, forced to one case. No-op on a path without a drive. */ +function withDriveCase(p: string, upper: boolean): string { + return p.replace(/^[a-z]:/i, (drive) => (upper ? drive.toUpperCase() : drive.toLowerCase())); +} + +describe('daemon rendezvous key', () => { + it('collapses redundant path segments to one socket name', () => { + const root = makeDir(); + const indirect = path.join(root, 'a', '..', 'b', '.'); + const direct = path.join(root, 'b'); + // `.`, `..` and a trailing separator all name the same directory, so both + // spellings must resolve to the same socket — otherwise `--path` with a + // stray separator is enough to miss a running daemon. + expect(getDaemonSocketPath(indirect)).toBe(getDaemonSocketPath(direct)); + }); + + it.runIf(process.platform === 'win32')( + 'maps both drive-letter cases of one directory onto one pipe', + () => { + const root = makeDir(); + const upper = withDriveCase(root, true); + const lower = withDriveCase(root, false); + + expect(canonicalProjectRoot(upper)).toBe(canonicalProjectRoot(lower)); + expect(getDaemonSocketPath(upper)).toBe(getDaemonSocketPath(lower)); + // …and the converged key is the on-disk casing, lowercased, so it does not + // depend on which of the two the caller happened to hold. + expect(canonicalProjectRoot(upper)).toBe(fs.realpathSync.native(root).toLowerCase()); + }, + ); + + it.runIf(process.platform === 'win32')( + 'converges case variants even when the root cannot be realpath’d', + () => { + // The `.codegraph/` root normally exists, so this is the fallback arm — + // but a divergence there would fail identically, so pin it. + const missing = path.join(os.tmpdir(), 'cg-rendezvous-absent', 'nested'); + expect(canonicalProjectRoot(missing)).toBe(canonicalProjectRoot(missing.toUpperCase())); + }, + ); +}); + +/** True when the temp filesystem ignores case (default macOS APFS, NTFS). */ +function caseInsensitiveTmp(): boolean { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-CaseProbe-')); + try { + return fs.existsSync(dir.replace('cg-CaseProbe-', 'cg-caseprobe-')); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +} + +describe('in-process project sharing (#2278)', () => { + it.runIf(caseInsensitiveTmp())('opens one graph for two casings of one root', async () => { + const root = path.join(makeDir(), 'Repo'); + fs.mkdirSync(root); + let opened = 0; + const open = () => { + opened++; + return { close() {}, isIndexing: () => false } as unknown as CodeGraph; + }; + const a = acquireProject(root, open, {}); + const b = acquireProject(path.join(path.dirname(root), 'REPO'), open, {}); + try { + expect(opened).toBe(1); + expect(b.cg).toBe(a.cg); + } finally { + await a.release(); + await b.release(); + } + }); +}); diff --git a/src/directory.ts b/src/directory.ts index d99860f04e..c458457e4e 100644 --- a/src/directory.ts +++ b/src/directory.ts @@ -303,6 +303,40 @@ export function statInode(p: string): string | null { } } +/** + * Canonicalize a project root for IDENTITY: hashing it into a rendezvous key + * (the daemon's named pipe / tmpdir socket, a registry record). Not for display + * — the caller-visible root keeps its own spelling. + * + * The contract is that two spellings of ONE directory yield ONE string, because + * a client that derives a different key silently fails to meet the daemon that + * is already running: it probes a socket nobody bound, spawns a redundant + * daemon, and that daemon dies on the lock the first one holds, so the session + * degrades to a single-process engine. `path.resolve` alone does NOT satisfy + * that contract — NTFS is case-insensitive, so `d:\work\codegraph` and + * `D:\work\codegraph` name one directory but two strings, and non-native + * `fs.realpathSync` keeps whichever casing the caller passed (only `.native` + * asks the filesystem for the on-disk name — the same reason + * {@link isSameIndexRoot} uses it). Both spellings really do occur: a + * cwd-derived root is the on-disk case, while a client-supplied + * `rootUri`/`workspaceFolders` path arrives as `file:///d%3A/…`. + * + * The Windows lowercase is belt-and-braces on top of the native realpath — NTFS + * can't hold two directories differing only by case, and it also absorbs a + * `\\?\`-prefixed native result diverging from a plain one. + */ +export function canonicalProjectRoot(projectRoot: string): string { + const resolved = path.resolve(projectRoot); + let canonical = resolved; + try { + canonical = fs.realpathSync.native(resolved); + } catch { + // ENOENT/EACCES/ELOOP — the root is normally there (`.codegraph/` lives in + // it), so this is a fallback rather than a path we expect to take. + } + return process.platform === 'win32' ? canonical.toLowerCase() : canonical; +} + /** * Whether two resolved index roots are one index spelled two ways — a symlinked * checkout, or a case-variant on a case-insensitive mount (macOS, NTFS, WSL diff --git a/src/mcp/daemon-paths.ts b/src/mcp/daemon-paths.ts index 34751814f3..8f78d5063e 100644 --- a/src/mcp/daemon-paths.ts +++ b/src/mcp/daemon-paths.ts @@ -32,14 +32,20 @@ import * as crypto from 'crypto'; import * as net from 'net'; import * as os from 'os'; import * as path from 'path'; -import { getCodeGraphDir } from '../directory'; +import { canonicalProjectRoot, getCodeGraphDir } from '../directory'; /** Soft upper bound for in-project socket paths. */ const POSIX_SOCKET_PATH_LIMIT = 100; -/** Short stable identifier for a project root — used in tmpdir/pipe names. */ +/** + * Short stable identifier for a project root — used in tmpdir/pipe names. + * + * Hashed over {@link canonicalProjectRoot}, never a raw `path.resolve`: the key + * is a rendezvous, so every spelling of one directory must land on one name or + * a proxy probes a pipe the running daemon never bound (see that function). + */ function projectHash(projectRoot: string): string { - return crypto.createHash('sha256').update(path.resolve(projectRoot)).digest('hex').slice(0, 16); + return crypto.createHash('sha256').update(canonicalProjectRoot(projectRoot)).digest('hex').slice(0, 16); } /** diff --git a/src/mcp/daemon-registry.ts b/src/mcp/daemon-registry.ts index b4a01c307f..c6c4043b60 100644 --- a/src/mcp/daemon-registry.ts +++ b/src/mcp/daemon-registry.ts @@ -22,6 +22,7 @@ import * as fs from 'fs'; import * as os from 'os'; import * as path from 'path'; import * as crypto from 'crypto'; +import { canonicalProjectRoot } from '../directory'; import { getDaemonPidPath, getDaemonSocketCandidates, @@ -50,8 +51,14 @@ export function getRegistryDir(): string { return path.join(os.homedir(), '.codegraph', 'daemons'); } +/** + * One record per project, so it is keyed the same way the daemon socket is: + * over {@link canonicalProjectRoot}, not a raw `path.resolve` — otherwise the + * same project spelled with another drive-letter case files two records, and + * `list` over-lists while `stop --all` misses one. + */ function recordPath(root: string): string { - const hash = crypto.createHash('sha256').update(path.resolve(root)).digest('hex').slice(0, 16); + const hash = crypto.createHash('sha256').update(canonicalProjectRoot(root)).digest('hex').slice(0, 16); return path.join(getRegistryDir(), `${hash}.json`); } @@ -242,7 +249,7 @@ export async function stopDaemonAt(root: string, options: { preserveUnverified?: } if (pid == null) { const rec = listDaemons({ prune: false }).find( - (r) => path.resolve(r.root) === path.resolve(root) + (r) => canonicalProjectRoot(r.root) === canonicalProjectRoot(root) ); pid = rec?.pid ?? null; if (rec) identity = rec; diff --git a/src/mcp/project-lifecycle.ts b/src/mcp/project-lifecycle.ts index ba405f57d1..b7a52c1a5b 100644 --- a/src/mcp/project-lifecycle.ts +++ b/src/mcp/project-lifecycle.ts @@ -2,13 +2,15 @@ import { realpathSync } from 'fs'; import type { Socket } from 'net'; import type CodeGraph from '../index'; -import { isInitialized } from '../directory'; +import { canonicalProjectRoot, isInitialized } from '../directory'; import { LockUnavailableError, watchDisabledReason } from '../sync'; import { getDaemonSocketCandidates } from './daemon-paths'; import { connectWithHello } from './proxy'; import { markWriterReady, readWriterLock, releaseWriterLock, tryAcquireWriterLock } from './writer-lock'; interface Project { + /** Identity key: one entry however the root is spelled (#2278). */ + key: string; cg: CodeGraph; refs: number; owner: boolean; @@ -35,13 +37,14 @@ export function acquireProject( options: Parameters[0], ): ProjectLease { root = realpathSync(root); - let project = projects.get(root); + const key = canonicalProjectRoot(root); + let project = projects.get(key); if (!project) { const cg = open(); - project = { cg, refs: 0, owner: false, caughtUp: false, retirement: null, gate: null, socket: null, options, + project = { key, cg, refs: 0, owner: false, caughtUp: false, retirement: null, gate: null, socket: null, options, timer: setInterval(() => { void ready(root, project!); }, 1000) }; project.timer.unref(); - projects.set(root, project); + projects.set(key, project); } const entry = project; entry.refs++; @@ -80,7 +83,7 @@ function retire(root: string, project: Project): Promise { setTimeout(finish, 25); return; } - projects.delete(root); + projects.delete(project.key); project.cg.close(); if (project.owner) releaseWriterLock(root); resolve(); From e7389b22fb18381eae9639feb6e09ae2af5d128d Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 06:18:02 +0000 Subject: [PATCH 155/259] fix(mcp): a session serving in-process hands the project back to the shared daemon (#2277) (#2293) * fix(mcp): a session serving in-process hands the project back to the shared daemon (#2277) When a proxy lost its daemon (or never reached one) it served the rest of the session through an in-process engine. That engine takes the project's writer lock in fallback mode and nothing ever released it or tried the daemon again, so every later daemon start for the project failed on the held lock and every other session ran read-only without auto-sync until that one session ended. Sessions that fell back read-only stayed read-only for life too. The degraded proxy now retries the shared daemon with backoff (5 s, doubling to a 5 min cap; CODEGRAPH_DAEMON_RETRY_MS / _MAX_MS, 0 disables) through the existing probe/spawn/poll path. A writing engine is stopped first, after the calls it is serving have answered, so its lock is free before a daemon is spawned and two writers never overlap; calls arriving meanwhile are buffered and go to the daemon, or are served in-process again if none comes up. A read-only engine keeps serving until a daemon answers and is closed once its calls finish. The client's initialize is replayed to the new daemon with its reply suppressed, as on first attach. Co-Authored-By: Claude Opus 5.5 * docs(changelog): a degraded session hands the project back (#2277) Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/mcp-daemon.test.ts | 138 ++++++++++++++++++++++++++++++ src/mcp/engine.ts | 5 ++ src/mcp/proxy.ts | 159 +++++++++++++++++++++++++++++------ 4 files changed, 279 insertions(+), 24 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 62d005a12f..cf27df4914 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -172,6 +172,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - Indexing a large C or C++ project no longer runs out of memory partway through "Resolving refs". Headers whose include guard is written as `#define X_H 1`, as in OpenSceneGraph and osgEarth, or that are included under a build flag codegraph cannot know, used to be re-read on every include path, which grows exponentially with the depth of the include tree. Each is now read once per include state, and a hard limit stops the check on any include tree that is still too large. Thanks @danusha2345, and @coolhitmanleon for the report. (#2127) - In tag-based CFML, calls written in tags are now linked: in ``, ``/``, ``, `` and `#…#` expressions. Before, only `` and `` code was read, so callers and impact found almost nothing in tag-based components. Functions wrapped in tags like `` or `` are now indexed too, and a `` is indexed as an interface that `implements` links to. Re-index CFML projects after upgrading. Thanks @HarryMuc for the report. (#2091) - On Windows, a project opened with a different drive-letter or folder-name case, like `d:\work\app` and `D:\Work\App`, now joins the CodeGraph background server that is already running for it. Before, each spelling started a server of its own that exited at once, so the session quietly served the graph by itself with no shared file watching or auto-sync, and `codegraph list` showed the project twice. Thanks @greatjackzhou. (#2278) +- An agent session that lost its connection to CodeGraph's shared background server, or couldn't reach it at startup, no longer stops that server from running for the rest of the session. It reconnects on its own once the server can start, so other sessions on the same project get auto-sync back instead of running read-only. Thanks @ijbranch for the report. (#2277) ## [1.6.1] - 2026-09-29 diff --git a/__tests__/mcp-daemon.test.ts b/__tests__/mcp-daemon.test.ts index 71cc7b71ff..5b8b305002 100644 --- a/__tests__/mcp-daemon.test.ts +++ b/__tests__/mcp-daemon.test.ts @@ -817,4 +817,142 @@ describe('Shared MCP daemon (issue #411)', () => { expect(resp.result !== undefined || resp.error !== undefined).toBe(true); expect(isAlive(server.child.pid!)).toBe(true); }, 45000); + + it('a proxy serving in-process hands the writer lock back to a fresh daemon (#2277)', async () => { + // After its daemon dies, the proxy's in-process engine takes writer.pid in + // fallback mode. Every later daemon start used to fail on that lock for the + // rest of the session, so every other session on the project ran read-only + // without auto-sync. The degraded proxy now retries the daemon: it stops + // its engine (releasing the lock), lets a daemon start, and proxies again. + const env = { + CODEGRAPH_DAEMON_IDLE_TIMEOUT_MS: '30000', + CODEGRAPH_PPID_POLL_MS: '5000', + // The default, spelled out: long enough to observe the lockout first. + CODEGRAPH_DAEMON_RETRY_MS: '5000', + }; + const a = spawnServer(tempDir, env); + servers.push(a); + const proxyPid = a.child.pid!; + sendInitialize(a.child, `file://${tempDir}`, 1); + await waitFor(() => findResponse(a.stdout, 1), 20000, 25, 'initialize response'); + await waitFor(() => a.stderr.some((l) => l.includes('Attached to shared daemon')), 8000, 25, 'first daemon attach'); + sendMessage(a.child, { jsonrpc: '2.0', id: 2, method: 'tools/call', params: { name: 'codegraph_status', arguments: {} } }); + await waitFor(() => findResponse(a.stdout, 2), 30000, 25, 'warm tools/call via daemon'); + const firstDaemon = readLockPid(realRoot)!; + + process.kill(firstDaemon, 'SIGTERM'); + expect(await waitProcessExit(firstDaemon, 8000)).toBe(true); + await waitFor(() => a.stderr.some((l) => l.includes('serving this session in-process')), 8000, 25, 'in-process failover'); + sendMessage(a.child, { jsonrpc: '2.0', id: 3, method: 'tools/call', params: { name: 'codegraph_status', arguments: {} } }); + await waitFor(() => findResponse(a.stdout, 3), 15000, 25, 'in-process tools/call'); + + // The lockout: the in-process engine owns the project's writer slot. + expect(readWriterInfo(realRoot)).toMatchObject({ pid: proxyPid, mode: 'fallback' }); + + // Another session starting now cannot get a daemon while that lock is held. + const b = spawnServer(tempDir, env); + servers.push(b); + sendInitialize(b.child, `file://${tempDir}`, 1); + await waitFor(() => findResponse(b.stdout, 1), 20000, 25, 'second session initialize'); + + // Keep calling through the handover: every request gets exactly one reply. + let nextId = 4; + const send = (): void => { + sendMessage(a.child, { jsonrpc: '2.0', id: nextId++, method: 'tools/call', params: { name: 'codegraph_status', arguments: {} } }); + }; + const ticker = setInterval(send, 250); + let daemonWriter: { pid: number; mode: string } | null = null; + try { + daemonWriter = await waitFor(() => { + const w = readWriterInfo(realRoot); + return w && w.mode === 'daemon' && w.pid !== proxyPid && isAlive(w.pid) ? w : null; + }, 30000, 25, 'a daemon to own the writer lock'); + await waitFor( + () => a.stderr.some((l) => l.includes(`Attached to shared daemon`) && l.includes(`(pid ${daemonWriter!.pid},`)), + 15000, 25, 'the proxy to reattach to the new daemon', + ); + } finally { + clearInterval(ticker); + } + send(); + const lastId = nextId - 1; + await waitFor(() => findResponse(a.stdout, lastId), 15000, 25, 'a tools/call through the new daemon'); + for (let id = 2; id <= lastId; id++) { + const replies = a.stdout.filter((line) => { + try { const m = JSON.parse(line); return m.id === id && ('result' in m || 'error' in m); } catch { return false; } + }); + expect(replies, `replies to request ${id}`).toHaveLength(1); + expect(JSON.parse(replies[0]).error, `request ${id}`).toBeUndefined(); + } + expect(readLockPid(realRoot)).toBe(daemonWriter!.pid); + + // The session that started during the lockout ends up on the shared daemon too. + await waitFor( + () => b.stderr.some((l) => l.includes('Attached to shared daemon') && l.includes(`(pid ${daemonWriter!.pid},`)), + 30000, 25, 'the second session to attach', + ); + sendMessage(b.child, { jsonrpc: '2.0', id: 2, method: 'tools/call', params: { name: 'codegraph_status', arguments: {} } }); + const bReply = await waitFor(() => findResponse(b.stdout, 2), 15000, 25, 'second session tools/call'); + expect(bReply.error).toBeUndefined(); + }, 90000); + + it('a read-only in-process session moves to the shared daemon once one can start (#2277)', async () => { + // A session that fell back read-only (another daemon held the project) + // used to stay that way for life. Once the blocker is gone it now starts + // and attaches to a daemon of its own version. + const sockPath = getDaemonSocketPath(realRoot); + const pidPath = path.join(realRoot, '.codegraph', 'daemon.pid'); + fs.writeFileSync(pidPath, JSON.stringify({ pid: process.pid, version: '0.0.0-mismatch', socketPath: sockPath, startedAt: Date.now() })); + const miniServer = net.createServer((sock) => { + sock.write(JSON.stringify({ codegraph: '0.0.0-mismatch', pid: process.pid, socketPath: sockPath, protocol: 1 }) + '\n'); + }); + await new Promise((resolve) => miniServer.listen(sockPath, () => resolve())); + let miniServerOpen = true; + try { + const server = spawnServer(tempDir, { + CODEGRAPH_DAEMON_IDLE_TIMEOUT_MS: '30000', + CODEGRAPH_DAEMON_RETRY_MS: '300', + CODEGRAPH_DAEMON_RETRY_MAX_MS: '1000', + }); + servers.push(server); + sendInitialize(server.child, `file://${tempDir}`, 1); + await waitFor(() => server.stderr.some((l) => l.includes('serving this session in-process')), 10000, 25, 'in-process fallback'); + sendMessage(server.child, { jsonrpc: '2.0', id: 2, method: 'tools/call', params: { name: 'codegraph_status', arguments: {} } }); + const local = await waitFor(() => findResponse(server.stdout, 2), 10000, 25, 'read-only tools/call'); + expect(local.error).toBeUndefined(); + expect(server.stderr.some((l) => l.includes('Serving reads in-process without auto-sync'))).toBe(true); + // Retries while the other version still answers start no daemon. + await waitFor( + () => server.stderr.filter((l) => l.includes('differs from ours')).length >= 3, + 10000, 25, 'repeated retries against the other version', + ); + expect(countListeningLines(realRoot)).toBe(0); + + // The other-version daemon goes away. + await new Promise((resolve) => miniServer.close(() => resolve())); + miniServerOpen = false; + fs.rmSync(pidPath, { force: true }); + + const attached = await waitFor( + () => server.stderr.find((l) => l.includes('Attached to shared daemon')), + 30000, 25, 'the session to attach to a daemon', + ); + const daemonPid = readLockPid(realRoot)!; + expect(attached).toContain(`(pid ${daemonPid},`); + expect(readWriterInfo(realRoot)).toMatchObject({ pid: daemonPid, mode: 'daemon' }); + sendMessage(server.child, { jsonrpc: '2.0', id: 3, method: 'tools/call', params: { name: 'codegraph_status', arguments: {} } }); + const viaDaemon = await waitFor(() => findResponse(server.stdout, 3), 15000, 25, 'tools/call through the daemon'); + expect(viaDaemon.error).toBeUndefined(); + expect(JSON.stringify(viaDaemon.result)).toContain('CodeGraph Status'); + } finally { + if (miniServerOpen) await new Promise((resolve) => miniServer.close(() => resolve())); + } + }, 60000); }); + +function readWriterInfo(root: string): { pid: number; mode: string } | null { + try { + const info = JSON.parse(fs.readFileSync(path.join(root, '.codegraph', 'writer.pid'), 'utf8')); + return typeof info.pid === 'number' && typeof info.mode === 'string' ? info : null; + } catch { return null; } +} diff --git a/src/mcp/engine.ts b/src/mcp/engine.ts index 2898f203b1..5409c36d48 100644 --- a/src/mcp/engine.ts +++ b/src/mcp/engine.ts @@ -167,6 +167,11 @@ export class MCPEngine { this.toolHandler.setDefaultProjectHint(projectPath); } + /** Whether this engine only reads: no watcher, no sync, no writer slot. */ + isReadOnly(): boolean { + return this.opts.readOnly; + } + /** Project root that the engine resolved on first init (null if none). */ getProjectPath(): string | null { return this.projectPath; diff --git a/src/mcp/proxy.ts b/src/mcp/proxy.ts index 17c5a59cb1..a983c2ebdc 100644 --- a/src/mcp/proxy.ts +++ b/src/mcp/proxy.ts @@ -39,6 +39,15 @@ import type { MCPEngine } from './engine'; /** Default poll cadence for the PPID watchdog (same as the direct server). */ const DEFAULT_PPID_POLL_MS = 5000; +/** + * How long a proxy serving in-process waits before trying the shared daemon + * again, doubling after each miss up to the cap (#2277). Tunable through + * `CODEGRAPH_DAEMON_RETRY_MS` (`0` turns retrying off) and + * `CODEGRAPH_DAEMON_RETRY_MAX_MS`. + */ +const DEFAULT_DAEMON_RETRY_MS = 5_000; +const DEFAULT_DAEMON_RETRY_MAX_MS = 300_000; + /** * Env var that opts INTO the "attached to shared daemon" log line. Off by * default: the line is benign INFO, but MCP hosts render any server stderr at @@ -196,8 +205,9 @@ export interface LocalHandshakeDeps { /** Probe → spawn → retry → hello-verify; resolves a connected daemon socket, * or null when the daemon path is genuinely unavailable (→ in-process fallback). */ getDaemonSocket(): Promise; - /** Lazily create an in-process engine — used ONLY if the daemon never comes up, - * preserving the "a broken daemon never wedges a session" guarantee. */ + /** Lazily create an in-process engine — used only while the daemon is + * unreachable, preserving the "a broken daemon never wedges a session" + * guarantee. Called again whenever the previous engine was retired (#2277). */ makeEngine(): MCPEngine; /** Project root for the fallback engine's lazy init. */ root: string; @@ -215,6 +225,12 @@ export interface LocalHandshakeDeps { * the local one). If the daemon never comes up (version mismatch / spawn fail), * a lazily-created in-process engine serves the calls — so the handshake speedup * never costs the old fall-back-to-direct robustness. + * + * Serving in-process is not for the rest of the session (#2277): that engine + * may own the project's writer lock, and no daemon can start while it does, so + * every other session on the project would be stuck without auto-sync. The + * proxy retries the daemon with backoff, handing the writer lock back first, + * and proxies again once one answers. */ export async function runLocalHandshakeProxy(deps: LocalHandshakeDeps): Promise { // The proxy is long-lived and can serve fallback tool calls in-process. Match @@ -224,6 +240,7 @@ export async function runLocalHandshakeProxy(deps: LocalHandshakeDeps): Promise< let daemonStatus: 'connecting' | 'ready' | 'failed' = 'connecting'; let daemonSocket: net.Socket | null = null; let clientInitId: unknown = undefined; // suppress the daemon's reply to the forwarded initialize + let clientInitLine: string | undefined; // replayed to a daemon that comes up mid-session (#2277) // Telemetry attribution for the in-process fallback only — calls routed to // the daemon are counted by the daemon's own session (which receives the // forwarded initialize, clientInfo included), never double-counted here. @@ -231,7 +248,14 @@ export async function runLocalHandshakeProxy(deps: LocalHandshakeDeps): Promise< const pending: string[] = []; // client lines buffered until the daemon resolves let engine: MCPEngine | null = null; let engineReady: Promise | null = null; + // Calls being served in-process; the engine is only ever stopped after they finish. + const localCalls = new Set>(); let shuttingDown = false; + const retryBaseMs = parseDelayMs(process.env.CODEGRAPH_DAEMON_RETRY_MS, DEFAULT_DAEMON_RETRY_MS); + const retryMaxMs = Math.max(retryBaseMs, parseDelayMs(process.env.CODEGRAPH_DAEMON_RETRY_MAX_MS, DEFAULT_DAEMON_RETRY_MAX_MS)); + let retryDelayMs = retryBaseMs; + let retryTimer: NodeJS.Timeout | null = null; + let attachedAt = 0; // Requests forwarded to the daemon and not yet answered, keyed by JSON-RPC id. // If the daemon dies mid-session (#662 — e.g. an MCP host SIGTERM's it when a // new session starts), these would otherwise hang forever; we re-serve them @@ -256,24 +280,47 @@ export async function runLocalHandshakeProxy(deps: LocalHandshakeDeps): Promise< const shutdown = (): void => { if (shuttingDown) return; shuttingDown = true; try { livenessWatchdog?.stop(); } catch { /* ignore */ } + if (retryTimer) clearTimeout(retryTimer); try { daemonSocket?.destroy(); } catch { /* ignore */ } try { engine?.stop(); } catch { /* ignore */ } process.exit(0); }; - const ensureEngine = (): Promise => { - if (!engine) engine = deps.makeEngine(); - if (!engineReady) engineReady = engine.ensureInitialized(deps.root).catch(() => { /* degraded */ }); - return engineReady; + // Resolves the engine a call was started on, so a call in flight while the + // engine is being retired still finishes on it. + const ensureEngine = async (): Promise => { + if (!engine) { + engine = deps.makeEngine(); + engineReady = engine.ensureInitialized(deps.root).catch(() => { /* degraded */ }); + } + const current = engine; + await engineReady; + return current; + }; + // Stop the in-process engine once the calls it is serving have answered. + // Stopping it releases the writer lock if it held one. + const retireEngine = async (): Promise => { + const retired = engine; + engine = null; + engineReady = null; + await Promise.allSettled([...localCalls]); + try { await retired?.stop(); } catch { /* best-effort */ } }; // Daemon-unavailable fallback: serve a client message in-process. - const handleLocally = async (line: string): Promise => { + const handleLocally = (line: string): Promise => { + const call = serveLocally(line); + localCalls.add(call); + const done = (): void => { localCalls.delete(call); }; + call.then(done, done); + return call; + }; + const serveLocally = async (line: string): Promise => { let msg: JsonRpc; try { msg = JSON.parse(line) as JsonRpc; } catch { return; } const id = msg.id; if (msg.method === 'tools/call' && id !== undefined) { try { - await ensureEngine(); + const local = await ensureEngine(); const params = (msg.params || {}) as { name: string; arguments?: Record }; - const result = await engine!.getToolHandler().execute(params.name, params.arguments || {}, exploreSession); + const result = await local.getToolHandler().execute(params.name, params.arguments || {}, exploreSession); writeClient({ jsonrpc: '2.0', id, result }); getTelemetry().recordUsage('mcp_tool', params.name, !result.isError, telemetryClient); } catch (err) { @@ -314,6 +361,7 @@ export async function runLocalHandshakeProxy(deps: LocalHandshakeDeps): Promise< let msg: JsonRpc; try { msg = JSON.parse(line) as JsonRpc; } catch { routeToDaemon(line); continue; } if (msg.method === 'initialize') { clientInitId = msg.id; + clientInitLine = line; const initParams = (msg.params ?? {}) as { clientInfo?: { name?: unknown; version?: unknown } }; if (initParams.clientInfo) { telemetryClient = { @@ -364,15 +412,58 @@ export async function runLocalHandshakeProxy(deps: LocalHandshakeDeps): Promise< }); // ---- daemon connection (background) ---- - let socket: net.Socket | null = null; - try { socket = await deps.getDaemonSocket(); } catch { socket = null; } + const connectDaemon = async (): Promise => { + let socket: net.Socket | null = null; + try { socket = await deps.getDaemonSocket(); } catch { socket = null; } + // `socket.destroyed`: the connect-window error guard above can absorb an + // 'error' that already destroyed the socket before we got here (#974) — treat + // a dead socket as "no daemon" so we cleanly fall back to the in-process engine. + if (socket && (socket.destroyed || shuttingDown)) { + try { socket.destroy(); } catch { /* ignore */ } + socket = null; + } + return socket; + }; + // Every buffered call binds to the engine now, so a later retirement waits for all of them. + const serveBufferedLocally = (): void => { + for (const line of pending.splice(0)) void handleLocally(line); + }; + const scheduleDaemonRetry = (): void => { + if (shuttingDown || retryTimer || retryBaseMs <= 0) return; + const delay = retryDelayMs; + retryDelayMs = Math.min(retryDelayMs * 2, retryMaxMs); + retryTimer = setTimeout(() => { retryTimer = null; void retryDaemon(); }, delay); + retryTimer.unref?.(); + }; + const retryDaemon = async (): Promise => { + if (shuttingDown || daemonStatus !== 'failed') return; + // While a writing engine owns writer.pid no daemon can start (#2277). Hand + // the lock back first, once the calls it is serving have answered, and + // buffer new calls until a daemon answers or we resume in-process. One + // writer at a time: the engine is fully stopped before a daemon is spawned. + const handover = engine !== null && !engine.isReadOnly(); + if (handover) { + daemonStatus = 'connecting'; + await retireEngine(); + } + const socket = await connectDaemon(); + if (shuttingDown) return; + if (socket) { + process.stderr.write('[CodeGraph MCP] Shared daemon reachable; proxying this session to it instead of serving in-process.\n'); + attachDaemon(socket, true); + // A read-only engine holds no lock: let its calls finish, then close it. + if (!handover) void retireEngine(); + return; + } + daemonStatus = 'failed'; + serveBufferedLocally(); + scheduleDaemonRetry(); + }; - // `!socket.destroyed`: the connect-window error guard above can absorb an - // 'error' that already destroyed the socket before we got here (#974) — treat - // a dead socket as "no daemon" so we cleanly fall back to the in-process engine. - if (socket && !socket.destroyed && !shuttingDown) { + const attachDaemon = (socket: net.Socket, replayInitialize: boolean): void => { daemonSocket = socket; daemonStatus = 'ready'; + attachedAt = Date.now(); let sockBuf = ''; socket.setEncoding('utf8'); socket.on('data', (chunk: string) => { @@ -397,33 +488,48 @@ export async function runLocalHandshakeProxy(deps: LocalHandshakeDeps): Promise< // The daemon going away does NOT end the session (#662). An MCP host can // SIGTERM the shared daemon when another session starts; if we exited here, // this host would silently lose CodeGraph and any in-flight request would - // hang. Instead, fall back to the in-process engine for the rest of the - // session and re-serve whatever the dead daemon never answered. + // hang. Instead, fall back to the in-process engine until a daemon answers + // again, and re-serve whatever the dead daemon never answered. const onDaemonLost = (): void => { - if (shuttingDown || daemonStatus !== 'ready') return; // host teardown, or already handled + if (shuttingDown || daemonSocket !== socket) return; // host teardown, or already handled daemonStatus = 'failed'; - try { daemonSocket?.destroy(); } catch { /* ignore */ } daemonSocket = null; + try { socket.destroy(); } catch { /* ignore */ } process.stderr.write( `[CodeGraph MCP] Shared daemon connection lost; serving this session in-process (degraded), re-serving ${inflight.size} in-flight request(s).\n` ); const orphaned = [...inflight.values()]; inflight.clear(); for (const line of orphaned) void handleLocally(line); + // A link that stayed up past the longest wait starts the backoff over; one + // that keeps dropping right after it attaches keeps backing off. + if (Date.now() - attachedAt >= retryMaxMs) retryDelayMs = retryBaseMs; + scheduleDaemonRetry(); }; socket.on('close', onDaemonLost); socket.on('error', onDaemonLost); + // A daemon that came up mid-session has never seen this client: prime its + // session with the client's own initialize, whose reply is suppressed + // above like the first one. + if (replayInitialize && clientInitLine !== undefined) { + try { socket.write(clientInitLine + '\n'); } catch { /* close path */ } + } for (const line of pending) { trackInflight(line); if (process.env.CODEGRAPH_MCP_DEBUG) process.stderr.write(`[mcp-debug] proxy-flush ${line.slice(0, 80)}\n`); try { socket.write(line + '\n'); } catch { /* ignore */ } } pending.length = 0; + }; + + const socket = await connectDaemon(); + if (socket) { + attachDaemon(socket, false); } else if (!shuttingDown) { daemonStatus = 'failed'; process.stderr.write('[CodeGraph MCP] Shared daemon unavailable; serving this session in-process (degraded).\n'); - const buffered = pending.splice(0); - for (const line of buffered) await handleLocally(line); + serveBufferedLocally(); + scheduleDaemonRetry(); } await new Promise(() => { /* stdin keeps the loop alive; exit via shutdown() */ }); @@ -588,10 +694,15 @@ function startPpidWatchdog(socket: net.Socket): void { } function parsePollMs(raw: string | undefined): number { - if (raw === undefined || raw === '') return DEFAULT_PPID_POLL_MS; + return parseDelayMs(raw, DEFAULT_PPID_POLL_MS); +} + +/** A non-negative millisecond env value; unset or malformed gives `fallback`. */ +function parseDelayMs(raw: string | undefined, fallback: number): number { + if (raw === undefined || raw === '') return fallback; const parsed = Number(raw); - if (!Number.isFinite(parsed)) return DEFAULT_PPID_POLL_MS; - if (parsed < 0) return DEFAULT_PPID_POLL_MS; + if (!Number.isFinite(parsed)) return fallback; + if (parsed < 0) return fallback; return Math.floor(parsed); } From 7bbc1fb7f6bc4627a6e7930f7229b34879a1cf57 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 06:18:46 +0000 Subject: [PATCH 156/259] fix(tests): the test suite no longer writes to the developer's real home dir (#2275) (#2295) Running the suite on a machine with Claude Code configured could wire the CodeGraph prompt hook into the developer's own ~/.claude/settings.json: every upgrade test whose fake upgrade succeeded ran the real post-upgrade prompt-hook self-heal, which detected the real global Claude profile and wrote it. Daemon suites likewise left records in the real ~/.codegraph/daemons. On Windows a HOME-only redirect would not have helped, since os.homedir() reads USERPROFILE there. - The prompt-hook writer is now injected through UpgradeDeps (wirePromptHook); the CLI passes the real defaultWirePromptHook and the upgrade tests a recorder. New tests pin that a successful upgrade calls the recorder and that the real writer never touches a configured profile. - A vitest setupFiles entry gives every engine test file a throwaway home: HOME/USERPROFILE (plus HOMEDRIVE/HOMEPATH/APPDATA/LOCALAPPDATA on Windows), XDG_CONFIG_HOME, and GIT_CONFIG_GLOBAL (seeded with a dummy identity) point into it, and CLAUDE_CONFIG_DIR/CODEX_HOME/COPILOT_HOME/ HERMES_HOME are cleared. Spawned CLI/MCP children inherit it. A sandbox test asserts os.homedir(), the Claude profile, the daemon registry, a child process and git's global config all resolve inside it. Co-authored-by: Claude Opus 5.5 --- AGENTS.md | 2 + __tests__/setup-home-sandbox.ts | 66 +++++++++++++++++ __tests__/test-home-sandbox.test.ts | 65 +++++++++++++++++ __tests__/upgrade.test.ts | 109 +++++++++++++++++++++++++++- src/bin/codegraph.ts | 1 + src/upgrade/index.ts | 26 ++++++- vitest.config.mts | 3 + vitest.workspace.mts | 6 +- 8 files changed, 270 insertions(+), 8 deletions(-) create mode 100644 __tests__/setup-home-sandbox.ts create mode 100644 __tests__/test-home-sandbox.test.ts diff --git a/AGENTS.md b/AGENTS.md index 2abe06c4ad..0f5e9a882e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -181,6 +181,8 @@ Tests live in `__tests__/` and mirror the module they cover. Notable ones beyond Tests create temp dirs with `fs.mkdtempSync` and clean up in `afterEach`. They write real files and exercise real SQLite — there is no DB mocking. +Every engine test file runs in a throwaway home dir (`__tests__/setup-home-sandbox.ts`, a `setupFiles` entry): `HOME`/`USERPROFILE` (+ Windows vars), `XDG_CONFIG_HOME` and `GIT_CONFIG_GLOBAL` point into it, and `CLAUDE_CONFIG_DIR`/`CODEX_HOME`/… are cleared; spawned children inherit it. It's a backstop — still inject writes to global state (e.g. `UpgradeDeps.wirePromptHook`, #2275). + ### Windows-gated tests Behavior that differs by platform (path resolution, drive letters, `SENSITIVE_PATHS`, `%APPDATA%` config dirs, CRLF) must be gated, not assumed. Use `it.runIf(process.platform === 'win32')(...)` for Windows-only assertions and `it.runIf(process.platform !== 'win32')(...)` for POSIX-only ones — e.g. `/etc` is sensitive on POSIX but resolves to `C:\etc` (non-existent) on Windows, so an ungated `/etc` assertion fails on Windows. Validate the Windows side for real (see below); don't merge a Windows-gated test you haven't seen run. diff --git a/__tests__/setup-home-sandbox.ts b/__tests__/setup-home-sandbox.ts new file mode 100644 index 0000000000..5acb473944 --- /dev/null +++ b/__tests__/setup-home-sandbox.ts @@ -0,0 +1,66 @@ +/** + * Give every engine test file a throwaway home directory (#2275). + * + * The suite exercises code that writes to the user's GLOBAL state: the Claude + * prompt hook in `~/.claude/settings.json`, the daemon registry under + * `~/.codegraph/daemons`, every agent target's global config. A test that + * forgot to redirect the home dir wrote to the developer's real one — a + * successful fake `codegraph upgrade` wired the prompt hook into their own + * Claude profile, and the daemon suites left thousands of registry records + * behind. CI never noticed: its runners have no Claude profile to edit. + * + * So the home dir is redirected here, before any test module loads, rather + * than file by file. Everything that resolves a home dir is covered: + * `os.homedir()` reads HOME on POSIX but USERPROFILE on Windows; git on + * Windows falls back to HOMEDRIVE+HOMEPATH; the per-agent overrides + * (CLAUDE_CONFIG_DIR, CODEX_HOME, ...) win over either, so they are cleared; + * and git's global config is pinned to a seeded file, so a test's `git + * commit` neither reads the developer's identity, signing or hooks nor + * depends on them. Spawned CLI / MCP processes inherit all of it. + * + * Tests that redirect the home dir themselves (spying on `os.homedir`, or + * setting HOME and restoring it) keep working: they restore to this sandbox. + */ +import { afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; + +// Resolved, so a comparison against a realpath'd cwd (macOS `/var` → +// `/private/var`) sees the same home everything else does. +const home = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-test-home-'))); + +process.env.HOME = home; +process.env.USERPROFILE = home; +if (process.platform === 'win32') { + const drive = /^([A-Za-z]:)(\\.*)$/.exec(home); + if (drive) { + process.env.HOMEDRIVE = drive[1]; + process.env.HOMEPATH = drive[2]; + } + process.env.APPDATA = path.join(home, 'AppData', 'Roaming'); + process.env.LOCALAPPDATA = path.join(home, 'AppData', 'Local'); +} else { + // Only ever set on Windows in real life; on POSIX a stray one would point + // the opencode/Copilot legacy-path logic somewhere real. + delete process.env.APPDATA; + delete process.env.LOCALAPPDATA; +} +process.env.XDG_CONFIG_HOME = path.join(home, '.config'); +for (const override of ['CLAUDE_CONFIG_DIR', 'CODEX_HOME', 'COPILOT_HOME', 'HERMES_HOME']) { + delete process.env[override]; +} + +// Both spellings: GIT_CONFIG_GLOBAL for git >= 2.32, and `$HOME/.gitconfig` +// (the same file) for older ones. +const gitConfig = path.join(home, '.gitconfig'); +fs.writeFileSync(gitConfig, '[user]\n\tname = CodeGraph Test\n\temail = test@codegraph.invalid\n'); +process.env.GIT_CONFIG_GLOBAL = gitConfig; + +afterAll(() => { + try { + fs.rmSync(home, { recursive: true, force: true, maxRetries: 3 }); + } catch { + /* a straggling child still holding a file there; it's in the temp dir */ + } +}); diff --git a/__tests__/test-home-sandbox.test.ts b/__tests__/test-home-sandbox.test.ts new file mode 100644 index 0000000000..f5ebb7a718 --- /dev/null +++ b/__tests__/test-home-sandbox.test.ts @@ -0,0 +1,65 @@ +/** + * The suite's home-dir sandbox (`setup-home-sandbox.ts`, #2275) is in force: + * nothing a test runs — in-process or spawned — resolves the developer's real + * home, Claude profile, daemon registry or git identity. + */ +import { describe, it, expect } from 'vitest'; +import { execFileSync } from 'child_process'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { claudeTarget } from '../src/installer/targets/claude'; +import { getRegistryDir } from '../src/mcp/daemon-registry'; + +const tmpRoot = fs.realpathSync(os.tmpdir()); +const under = (p: string, dir: string): boolean => path.resolve(p).startsWith(dir + path.sep); +/** The sandbox home — asserted to be one, so the checks below can't pass on a real home. */ +function sandboxHome(): string { + const home = os.homedir(); + expect(under(home, tmpRoot), home).toBe(true); + return home; +} + +describe('test home sandbox', () => { + it('os.homedir() is a throwaway dir under the temp root, not the real home', () => { + const home = sandboxHome(); + expect(path.basename(home)).toMatch(/^codegraph-test-home-/); + // The account's profile dir, which ignores HOME / USERPROFILE. + expect(home).not.toBe(os.userInfo().homedir); + }); + + it('the global Claude profile and the daemon registry resolve inside it', () => { + const home = sandboxHome(); + expect(process.env.CLAUDE_CONFIG_DIR).toBeUndefined(); + expect(under(claudeTarget.detect('global').configPath, home)).toBe(true); + expect(under(getRegistryDir(), home)).toBe(true); + }); + + it('a spawned child sees the same home', () => { + const childHome = execFileSync(process.execPath, ['-p', 'require("os").homedir()'], { + encoding: 'utf-8', + windowsHide: true, + }).trim(); + expect(childHome).toBe(sandboxHome()); + }); + + it("git's global config is the seeded sandbox file", () => { + const gitConfig = process.env.GIT_CONFIG_GLOBAL ?? ''; + expect(under(gitConfig, sandboxHome())).toBe(true); + const name = execFileSync('git', ['config', '--global', '--get', 'user.name'], { + encoding: 'utf-8', + windowsHide: true, + }).trim(); + expect(name).toBe('CodeGraph Test'); + }); + + // os.homedir() reads USERPROFILE there, and git falls back to + // HOMEDRIVE+HOMEPATH — a HOME-only sandbox would not hold. + it.runIf(process.platform === 'win32')('covers the Windows home variables', () => { + const home = sandboxHome(); + expect(process.env.USERPROFILE).toBe(home); + expect(`${process.env.HOMEDRIVE}${process.env.HOMEPATH}`).toBe(home); + expect(under(process.env.APPDATA ?? '', home)).toBe(true); + expect(under(process.env.LOCALAPPDATA ?? '', home)).toBe(true); + }); +}); diff --git a/__tests__/upgrade.test.ts b/__tests__/upgrade.test.ts index 244957af4e..e1b05e4d40 100644 --- a/__tests__/upgrade.test.ts +++ b/__tests__/upgrade.test.ts @@ -14,6 +14,7 @@ import { reindexAdvisory, runUpgrade, verifyResolvedVersion, + defaultWirePromptHook, buildWindowsUpgradeScript, NPM_PACKAGE, type InstallMethod, @@ -230,13 +231,15 @@ interface Calls { captures: Array<{ cmd: string; args: string[] }>; logs: string[]; errors: string[]; + /** How many times the upgrade asked deps to wire the Claude prompt hook. */ + promptHookWires: number; } function makeDeps( overrides: Partial & { method: InstallMethod; currentVersion: string }, runExit = 0 ): { deps: UpgradeDeps; calls: Calls } { - const calls: Calls = { runs: [], captures: [], logs: [], errors: [] }; + const calls: Calls = { runs: [], captures: [], logs: [], errors: [], promptHookWires: 0 }; const deps: UpgradeDeps = { currentVersion: overrides.currentVersion, method: overrides.method, @@ -252,6 +255,12 @@ function makeDeps( return overrides.capture ? overrides.capture(cmd, args) : null; }, hasCommand: overrides.hasCommand ?? ((c) => c === 'curl'), + // A recorder, never the real writer: that one edits the GLOBAL Claude + // profile, which here would be the developer's own (#2275). + wirePromptHook: async () => { + calls.promptHookWires += 1; + return overrides.wirePromptHook ? overrides.wirePromptHook() : false; + }, log: (m) => calls.logs.push(m), warn: (m) => calls.logs.push(m), error: (m) => calls.errors.push(m), @@ -473,6 +482,104 @@ describe('runUpgrade beta signup offer', () => { }); }); +// --------------------------------------------------------------------------- +// Post-upgrade prompt-hook self-heal — only ever through deps (#2275). The +// real writer edits the GLOBAL Claude profile; while the upgrade called it +// directly, every successful fake upgrade in this file could wire the hook +// into the developer's own ~/.claude/settings.json. +// --------------------------------------------------------------------------- + +describe('post-upgrade prompt-hook self-heal', () => { + // A configured global Claude profile the REAL writer would act on, so an + // upgrade that bypasses deps shows up as a settings.json written here. + const KEYS = ['CLAUDE_CONFIG_DIR', 'CODEGRAPH_NO_PROMPT_HOOK', 'CODEGRAPH_PROMPT_HOOK'] as const; + const saved: Partial> = {}; + let profile: string; + + function configureProfile(): void { + fs.writeFileSync( + path.join(profile, '.claude.json'), + JSON.stringify({ mcpServers: { codegraph: { command: 'codegraph', args: ['serve', '--mcp'] } } }), + ); + } + + beforeEach(() => { + for (const k of KEYS) saved[k] = process.env[k]; + delete process.env.CODEGRAPH_NO_PROMPT_HOOK; + delete process.env.CODEGRAPH_PROMPT_HOOK; + profile = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-upgrade-claude-')); + process.env.CLAUDE_CONFIG_DIR = profile; + }); + afterEach(() => { + for (const k of KEYS) { + if (saved[k] === undefined) delete process.env[k]; + else process.env[k] = saved[k]; + } + fs.rmSync(profile, { recursive: true, force: true }); + }); + + it('a successful upgrade wires the hook through deps, never the real writer', async () => { + configureProfile(); + const { deps, calls } = makeDeps({ method: { kind: 'npm', scope: 'global' }, currentVersion: '0.9.8' }); + expect(await runUpgrade({}, deps)).toBe(0); + expect(fs.existsSync(path.join(profile, 'settings.json')), 'the real writer ran').toBe(false); + expect(calls.promptHookWires).toBe(1); + }); + + it('notes the hook only when the writer changed something', async () => { + const quiet = makeDeps({ method: { kind: 'npm', scope: 'global' }, currentVersion: '0.9.8' }); + expect(await runUpgrade({}, quiet.deps)).toBe(0); + expect(quiet.calls.logs.join('\n')).not.toMatch(/front-load hook/); + + const wired = makeDeps({ + method: { kind: 'npm', scope: 'global' }, + currentVersion: '0.9.8', + wirePromptHook: async () => true, + }); + expect(await runUpgrade({}, wired.deps)).toBe(0); + expect(wired.calls.logs.join('\n')).toMatch(/Enabled the CodeGraph front-load hook/); + }); + + it('the kill-switch skips the writer entirely', async () => { + process.env.CODEGRAPH_NO_PROMPT_HOOK = '1'; + const { deps, calls } = makeDeps({ method: { kind: 'npm', scope: 'global' }, currentVersion: '0.9.8' }); + expect(await runUpgrade({}, deps)).toBe(0); + expect(calls.promptHookWires).toBe(0); + }); + + it('does not wire on --check, when up to date, or when the upgrade fails', async () => { + const check = makeDeps({ method: { kind: 'npm', scope: 'global' }, currentVersion: '0.9.8' }); + expect(await runUpgrade({ check: true }, check.deps)).toBe(0); + const current = makeDeps({ method: { kind: 'npm', scope: 'global' }, currentVersion: '0.9.9' }); + expect(await runUpgrade({}, current.deps)).toBe(0); + const failed = makeDeps({ method: { kind: 'npm', scope: 'global' }, currentVersion: '0.9.8' }, 1); + expect(await runUpgrade({}, failed.deps)).toBe(1); + for (const { calls } of [check, current, failed]) expect(calls.promptHookWires).toBe(0); + }); + + it('a throwing writer never fails the upgrade', async () => { + const { deps } = makeDeps({ + method: { kind: 'npm', scope: 'global' }, + currentVersion: '0.9.8', + wirePromptHook: async () => { throw new Error('EACCES'); }, + }); + expect(await runUpgrade({}, deps)).toBe(0); + }); + + // The production writer itself, pointed at the temp profile above. + it('the real writer wires a configured profile once, and leaves an unconfigured one alone', async () => { + const settings = path.join(profile, 'settings.json'); + expect(await defaultWirePromptHook()).toBe(false); + expect(fs.existsSync(settings)).toBe(false); + + configureProfile(); + expect(await defaultWirePromptHook()).toBe(true); + const written = JSON.parse(fs.readFileSync(settings, 'utf-8')); + expect(JSON.stringify(written.hooks.UserPromptSubmit)).toMatch(/prompt-hook/); + expect(await defaultWirePromptHook()).toBe(false); // idempotent + }); +}); + // --------------------------------------------------------------------------- // Post-upgrade self-heal of installed agent surfaces // --------------------------------------------------------------------------- diff --git a/src/bin/codegraph.ts b/src/bin/codegraph.ts index 64078dfab6..e5f8eacfc6 100644 --- a/src/bin/codegraph.ts +++ b/src/bin/codegraph.ts @@ -2859,6 +2859,7 @@ program run: up.defaultRun, capture: up.defaultCapture, hasCommand: up.hasCommand, + wirePromptHook: up.defaultWirePromptHook, log: (m: string) => console.log(m), warn: (m: string) => warn(m), error: (m: string) => error(m), diff --git a/src/upgrade/index.ts b/src/upgrade/index.ts index 625e45b831..76354378a8 100644 --- a/src/upgrade/index.ts +++ b/src/upgrade/index.ts @@ -297,6 +297,15 @@ export interface UpgradeDeps { warn: (msg: string) => void; error: (msg: string) => void; platform: NodeJS.Platform; + /** + * Wire Claude Code's front-load prompt hook into the GLOBAL Claude profile + * when that profile already has CodeGraph configured; resolves true when it + * changed the settings file. That file is the user's real + * `~/.claude/settings.json`, so the writer is injected like every other side + * effect here — the CLI passes {@link defaultWirePromptHook}, unit tests a + * recorder (#2275: tests that reached the real one rewrote the developer's). + */ + wirePromptHook: () => Promise; /** * Offer the one-time CodeGraph Pro beta opt-in after a successful update * (see installer/beta-signup — self-gating: TTY only, and silent forever @@ -528,10 +537,7 @@ function selfHealInstalledSurfaces(deps: UpgradeDeps): void { */ async function selfHealPromptHook(deps: UpgradeDeps): Promise { if (process.env.CODEGRAPH_NO_PROMPT_HOOK === '1' || process.env.CODEGRAPH_PROMPT_HOOK === '0') return; - const { claudeTarget, writePromptHookEntry } = await import('../installer/targets/claude'); - if (!claudeTarget.detect('global').alreadyConfigured) return; - const res = writePromptHookEntry('global'); - if (res.action === 'created' || res.action === 'updated') { + if (await deps.wirePromptHook()) { deps.log( c.dim('Enabled the CodeGraph front-load hook for Claude Code (structural prompts). Disable any time: CODEGRAPH_NO_PROMPT_HOOK=1'), ); @@ -714,3 +720,15 @@ export function defaultCapture(cmd: string, args: string[]): { code: number; std if (r.error) return null; return { code: r.status ?? -1, stdout: r.stdout ?? '' }; } + +/** + * The production `UpgradeDeps.wirePromptHook`: writes the hook only when the + * global Claude profile already carries CodeGraph's MCP entry, and leaves the + * file byte-for-byte alone once the hook is there. + */ +export async function defaultWirePromptHook(): Promise { + const { claudeTarget, writePromptHookEntry } = await import('../installer/targets/claude'); + if (!claudeTarget.detect('global').alreadyConfigured) return false; + const res = writePromptHookEntry('global'); + return res.action === 'created' || res.action === 'updated'; +} diff --git a/vitest.config.mts b/vitest.config.mts index ae02ba84c6..96fb2573a6 100644 --- a/vitest.config.mts +++ b/vitest.config.mts @@ -14,6 +14,9 @@ export default defineConfig({ include: ['__tests__/**/*.test.ts'], // Suites that spawn the built CLI need a current dist/ (#1879). globalSetup: ['./__tests__/global-setup-dist.ts'], + // A throwaway home dir (and git global config) per test file, so nothing + // the suite runs can write to the developer's real one (#2275). + setupFiles: ['./__tests__/setup-home-sandbox.ts'], /** * Several MCP integration tests (mcp-daemon, mcp-initialize, mcp-ppid-watchdog, * mcp-roots) spawn `dist/bin/codegraph.js serve --mcp` with `process.execPath` diff --git a/vitest.workspace.mts b/vitest.workspace.mts index c37fae14ba..9abdc721c7 100644 --- a/vitest.workspace.mts +++ b/vitest.workspace.mts @@ -20,9 +20,9 @@ import { defineWorkspace } from 'vitest/config'; * browser builds of `web-tree-sitter` and friends, and the failures that * causes look nothing like their cause. * - * The engine project `extends` the shared base, so the env vars and Node guard - * in `vitest.config.mts` still apply to every engine test. The ui project does - * not — see the note on it. + * The engine project `extends` the shared base, so the env vars, Node guard and + * home-dir sandbox in `vitest.config.mts` still apply to every engine test. The + * ui project does not — see the note on it. */ export default defineWorkspace([ { From 273e5cdc4c2ce60a5e99b134c2d05681a8de6cd7 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 06:19:41 +0000 Subject: [PATCH 157/259] fix(upgrade): a Windows upgrade no longer breaks the install while CodeGraph is running (#2185) (#2294) * fix(upgrade): a Windows upgrade no longer breaks the install while CodeGraph is running (#2185) On Windows, `codegraph upgrade` renamed only node.exe aside and then Copy-Item'd the new bundle over current\. Any open agent session runs a CodeGraph MCP server from current\, which keeps the native kernel loaded, so the copy failed on lib\kernel\codegraph-kernel.node halfway through, leaving no node.exe and a mix of versions that no codegraph command could repair. Re-running install.ps1 failed the same way, because it Remove-Items current\. Both now share one PowerShell function (a test pins the two copies equal). The bundle is unpacked next to current\ (same volume), then swapped in file by file: every file being replaced or dropped is first renamed aside (Windows allows renaming a running exe or a loaded DLL, just not overwriting or deleting it), then the staged file is moved in. Any failure or interruption undoes every step. Renamed-aside files, including the node.exe.old-* left by earlier upgrades, are deleted on the next successful run once nothing holds them. The upgrade's exit code now tells whether the install was left unchanged or needs a reinstall, and the message says which. Co-Authored-By: Claude Opus 5.5 * docs(changelog): Windows upgrade with CodeGraph running (#2185) Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/upgrade.test.ts | 154 ++++++++++++++++++++++++++++++++-- install.ps1 | 105 ++++++++++++++++++++--- src/upgrade/index.ts | 172 +++++++++++++++++++++++++++++++------- 4 files changed, 382 insertions(+), 50 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cf27df4914..348a804c87 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -173,6 +173,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In tag-based CFML, calls written in tags are now linked: in ``, ``/``, ``, `` and `#…#` expressions. Before, only `` and `` code was read, so callers and impact found almost nothing in tag-based components. Functions wrapped in tags like `` or `` are now indexed too, and a `` is indexed as an interface that `implements` links to. Re-index CFML projects after upgrading. Thanks @HarryMuc for the report. (#2091) - On Windows, a project opened with a different drive-letter or folder-name case, like `d:\work\app` and `D:\Work\App`, now joins the CodeGraph background server that is already running for it. Before, each spelling started a server of its own that exited at once, so the session quietly served the graph by itself with no shared file watching or auto-sync, and `codegraph list` showed the project twice. Thanks @greatjackzhou. (#2278) - An agent session that lost its connection to CodeGraph's shared background server, or couldn't reach it at startup, no longer stops that server from running for the rest of the session. It reconnects on its own once the server can start, so other sessions on the same project get auto-sync back instead of running read-only. Thanks @ijbranch for the report. (#2277) +- On Windows, `codegraph upgrade` no longer breaks the install while an agent session is using CodeGraph. It used to stop partway and leave CodeGraph unable to start; it now swaps the new files in with CodeGraph running and puts the previous version back if anything fails, and re-running the PowerShell installer works the same way. If an earlier upgrade already broke your install, re-run `irm https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.ps1 | iex` to repair it. Thanks @tippmar-nr for the report. (#2185) ## [1.6.1] - 2026-09-29 diff --git a/__tests__/upgrade.test.ts b/__tests__/upgrade.test.ts index e1b05e4d40..ad46a17ad4 100644 --- a/__tests__/upgrade.test.ts +++ b/__tests__/upgrade.test.ts @@ -16,6 +16,8 @@ import { verifyResolvedVersion, defaultWirePromptHook, buildWindowsUpgradeScript, + WINDOWS_SWAP_FUNCTION, + WINDOWS_UPGRADE_DAMAGED, NPM_PACKAGE, type InstallMethod, type UpgradeDeps, @@ -211,17 +213,124 @@ describe('version helpers', () => { expect(a).toContain('codegraph index -f'); }); - it('buildWindowsUpgradeScript targets the right asset per arch and renames-not-deletes the exe', () => { + it('buildWindowsUpgradeScript targets the right asset per arch', () => { const arm = buildWindowsUpgradeScript('C:\\cg\\current', 'v1.2.3', 'arm64'); expect(arm).toContain('releases/download/v1.2.3/codegraph-win32-arm64.zip'); expect(arm).toContain("$dest='C:\\cg\\current'"); - expect(arm).toContain('Rename-Item'); // never Remove-Item on the locked exe - expect(arm).not.toMatch(/Remove-Item[^;]*\$dest'?\s*;/); // doesn't delete current\ + expect(arm).toContain("Join-Path $stage 'codegraph-win32-arm64'"); const x64 = buildWindowsUpgradeScript('C:\\cg\\current', 'v1.2.3', 'x64'); expect(x64).toContain('codegraph-win32-x64.zip'); }); }); +// --------------------------------------------------------------------------- +// Windows file swap (#2185) — a running CodeGraph process (an agent session's +// MCP server) keeps node.exe and the native kernel locked: they can be renamed +// but not overwritten or deleted. The upgrade used to rename only node.exe and +// then Copy-Item over the rest, so the locked kernel failed the copy halfway +// and left the install with no node.exe. PowerShell can't run on the macOS +// test host, so these pin the script's shape; the behavior is validated on a +// real Windows machine. +// --------------------------------------------------------------------------- + +describe('windows bundle swap script (#2185)', () => { + const INSTALL_PS1 = path.join(__dirname, '..', 'install.ps1'); + const installPs1 = () => fs.readFileSync(INSTALL_PS1, 'utf-8').replace(/\r\n/g, '\n'); + const script = () => buildWindowsUpgradeScript('C:\\Users\\me\\AppData\\Local\\codegraph\\current', 'v1.6.2', 'x64'); + /** The swap function's own statements, comments dropped. */ + const swapCode = () => WINDOWS_SWAP_FUNCTION.split('\n').filter((l) => !l.trim().startsWith('#')).join('\n'); + + it('install.ps1 carries the same swap function, verbatim', () => { + const m = /# >>> Install-CodeGraphFiles[^\n]*\n([\s\S]*?)# <<< Install-CodeGraphFiles/.exec(installPs1()); + expect(m, 'install.ps1 lost its Install-CodeGraphFiles markers').not.toBeNull(); + expect(m![1]).toBe(WINDOWS_SWAP_FUNCTION); + }); + + it('never copies over or deletes current\\ in place', () => { + for (const text of [script(), installPs1()]) { + expect(text).not.toMatch(/Copy-Item/); + expect(text).not.toMatch(/Remove-Item[^\n]*\$dest\b/); + expect(text).not.toMatch(/Expand-Archive[^\n]*\$dest\b/); + } + }); + + it('unpacks next to current\\ (same volume), not into %TEMP%, then swaps', () => { + const s = script(); + expect(s).toContain(`$stage=Join-Path (Split-Path -Parent $dest) ('.staging-'`); + expect(s).toMatch(/Expand-Archive -Path \$zip -DestinationPath \$stage/); + expect(s.indexOf('Expand-Archive')).toBeLessThan(s.indexOf('Install-CodeGraphFiles $(')); + // install.ps1 stages next to current\ too. + expect(installPs1()).toContain(`$stage = Join-Path $installDir ('.staging-'`); + expect(installPs1()).toMatch(/Install-CodeGraphFiles \$\(.*\) \$dest$/m); + }); + + it('renames every replaced file aside before moving the staged file in', () => { + const code = swapCode(); + // Replaced files and files the new version drops are both renamed aside… + expect(code.match(/Move-Logged \$at "\$at\.old-\$token"/g)).toHaveLength(2); + // …and the aside-rename of a target comes before the staged file moves in. + const aside = code.indexOf('if ([IO.File]::Exists($at)) { Move-Logged $at "$at.old-$token" }'); + const moveIn = code.indexOf('Move-Logged ($stageDir + $rel) $at'); + expect(aside).toBeGreaterThan(0); + expect(moveIn).toBeGreaterThan(aside); + // No in-place overwrite or delete of a live file during the swap. + expect(code).not.toMatch(/\[IO\.File\]::(Copy|Replace)\(/); + expect(code).not.toMatch(/Move-Item|Rename-Item/); + }); + + it('rolls back on failure and on interruption, and says whether the install still works', () => { + const code = swapCode(); + expect(code).toMatch(/catch \{[\s\S]*\$lost = Undo-Logged[\s\S]*\} finally \{\s*if \(-not \$done\) \{ \[void\]\(Undo-Logged\) \}/); + // Undo walks the log backwards, moving each file back to where it was. + expect(code).toContain('for ($n = $undo.Count - 1; $n -ge 0; $n--)'); + expect(code).toContain('[IO.File]::Move($u[1], $u[0])'); + expect(code).toContain('Nothing was changed: the existing install still works.'); + expect(code).toContain("$e.Data['codegraphDamaged'] = [bool]$lost"); + // The upgrade script maps that flag to the exit code runUpgrade reads. + expect(script()).toContain(`$code=if($_.Exception.Data['codegraphDamaged']){${WINDOWS_UPGRADE_DAMAGED}}else{1}`); + expect(script().trimEnd().endsWith('exit $code')).toBe(true); + }); + + it('refuses a download that is not a bundle before touching current\\', () => { + const code = swapCode(); + expect(code).toContain("foreach ($need in 'node.exe', 'bin\\codegraph.cmd')"); + expect(code.indexOf("foreach ($need in")).toBeLessThan(code.indexOf('try {')); + }); + + it('cleans up renamed-aside leftovers, including ones from earlier upgrades', () => { + const m = /\$asideName = '([^']+)'/.exec(WINDOWS_SWAP_FUNCTION); + expect(m).not.toBeNull(); + const aside = new RegExp(m![1]!, 'i'); // PowerShell -match is case-insensitive + // The pre-fix upgrade left node.exe.old-<32-hex guid>; this one uses 8 hex. + expect(aside.test('node.exe.old-0123456789abcdef0123456789ABCDEF')).toBe(true); + expect(aside.test('codegraph-kernel.node.old-deadbeef')).toBe(true); + for (const shipped of ['node.exe', 'codegraph-kernel.node', 'old-deadbeef.js', 'x.old-1234567', 'a.old-deadbeef.js']) { + expect(aside.test(shipped), shipped).toBe(false); + } + // No file a bundle actually ships looks like a leftover. + const walk = (dir: string): string[] => + fs.readdirSync(dir, { withFileTypes: true }).flatMap((e) => (e.isDirectory() ? walk(path.join(dir, e.name)) : [e.name])); + const distDir = path.join(__dirname, '..', 'dist'); + if (fs.existsSync(distDir)) expect(walk(distDir).filter((n) => aside.test(n))).toEqual([]); + // Deleted only after a successful swap, and a still-locked one is skipped. + const code = swapCode(); + expect(code.indexOf('[IO.File]::Delete($f.FullName)')).toBeGreaterThan(code.indexOf('$done = $true')); + expect(code).toContain('try { [IO.File]::Delete($f.FullName) } catch {}'); + }); + + it('quotes the install path for PowerShell', () => { + const s = buildWindowsUpgradeScript("C:\\Users\\o'brien\\codegraph\\current", 'v1.6.2', 'x64'); + expect(s).toContain("$dest='C:\\Users\\o''brien\\codegraph\\current'"); + }); + + it('fits a Windows command line even for a long install path', () => { + const root = `C:\\${'very-long-directory-name\\'.repeat(8)}codegraph\\current`; + const encoded = Buffer.from(buildWindowsUpgradeScript(root, 'v10.20.30', 'arm64'), 'utf16le').toString('base64'); + // CreateProcess caps the whole command line at 32,767 characters. + expect(encoded.length).toBeLessThan(24_000); + }); +}); + // --------------------------------------------------------------------------- // runUpgrade orchestration — mocked side-effects // --------------------------------------------------------------------------- @@ -335,11 +444,10 @@ describe('runUpgrade', () => { expect(calls.runs).toHaveLength(1); expect(calls.runs[0].cmd).toBe('powershell.exe'); const decoded = decodeEncodedCommand(calls.runs[0].args); - // Downloads the right asset, renames the locked exe aside, copies over current\. + // Downloads the right asset and swaps it in with the shared rename-aside function. expect(decoded).toContain('releases/download/v0.9.9/codegraph-win32-'); - expect(decoded).toContain('Rename-Item'); - expect(decoded).toContain('node.exe.old-'); - expect(decoded).toContain('Copy-Item'); + expect(decoded).toContain(WINDOWS_SWAP_FUNCTION); + expect(decoded).toMatch(/^\s*Install-CodeGraphFiles .* \$dest$/m); }); it('windows bundle: a non-zero installer exit is a failure', async () => { @@ -353,7 +461,37 @@ describe('runUpgrade', () => { ); const code = await runUpgrade({}, deps); expect(code).toBe(1); - expect(calls.errors.join('\n')).toMatch(/exited with code/i); + // Exit 1 is the script's "failed, install unchanged" code (#2185). + expect(calls.errors.join('\n')).toMatch(/did not complete; your existing install was left as it was/i); + }); + + it('windows bundle: an incomplete rollback points at the reinstall command', async () => { + const { deps, calls } = makeDeps( + { + method: { kind: 'bundle', os: 'windows', bundleRoot: 'C:/x/codegraph/current', installDir: 'C:/x/codegraph' }, + currentVersion: '0.9.8', + platform: 'win32', + }, + WINDOWS_UPGRADE_DAMAGED + ); + const code = await runUpgrade({}, deps); + expect(code).toBe(1); + expect(calls.errors.join('\n')).toMatch(/could not be put back/i); + expect(calls.logs.join('\n')).toContain('install.ps1 | iex'); + expect(calls.runs).toHaveLength(1); // no post-upgrade refresh/probe after a failure + }); + + it('windows bundle: any other exit code is reported as-is', async () => { + const { deps, calls } = makeDeps( + { + method: { kind: 'bundle', os: 'windows', bundleRoot: 'C:/x/codegraph/current', installDir: 'C:/x/codegraph' }, + currentVersion: '0.9.8', + platform: 'win32', + }, + -1 + ); + expect(await runUpgrade({}, deps)).toBe(1); + expect(calls.errors.join('\n')).toMatch(/exited with code -1/i); }); it('npm global: shells out to npm install -g @pkg@latest', async () => { diff --git a/install.ps1 b/install.ps1 index 7c0b523caf..a079205b46 100644 --- a/install.ps1 +++ b/install.ps1 @@ -5,7 +5,8 @@ # # irm https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.ps1 | iex # -# Upgrade with `codegraph upgrade` (or just re-run this). To uninstall: remove +# Upgrade with `codegraph upgrade` (or just re-run this -- safe even while agent +# sessions are running CodeGraph). To uninstall: remove # $env:LOCALAPPDATA\codegraph and drop its \current\bin entry from your user PATH. # # Environment: @@ -13,6 +14,81 @@ # CODEGRAPH_INSTALL_DIR install location (default: %LOCALAPPDATA%\codegraph) $ErrorActionPreference = 'Stop' + +# >>> Install-CodeGraphFiles -- keep identical to WINDOWS_SWAP_FUNCTION in src/upgrade/index.ts +function Install-CodeGraphFiles([string]$Stage, [string]$Dest) { + # Move an unpacked bundle into $Dest. Windows can't overwrite or delete a + # running node.exe or a loaded .node addon, but it can rename one, so every + # file being replaced (or dropped by the new version) is first renamed aside + # to .old-. Any failure puts every file back, so the install is + # never left half-replaced or without its node.exe. + $ErrorActionPreference = 'Stop' + $stageDir = (Resolve-Path -LiteralPath $Stage).ProviderPath.TrimEnd('\') + foreach ($need in 'node.exe', 'bin\codegraph.cmd') { + if (-not (Test-Path -LiteralPath (Join-Path $stageDir $need))) { throw "The CodeGraph download is incomplete (no $need); nothing was changed." } + } + $token = [guid]::NewGuid().ToString('N').Substring(0, 8) + $asideName = '\.old-[0-9a-f]{8,32}$' + $files = @{}; $dirs = @{} + foreach ($i in @(Get-ChildItem -LiteralPath $stageDir -Recurse -Force)) { + $rel = $i.FullName.Substring($stageDir.Length) + if ($i.PSIsContainer) { $dirs[$rel] = $true } else { $files[$rel] = $true } + } + $undo = New-Object System.Collections.ArrayList + function Move-Logged([string]$From, [string]$To) { [IO.File]::Move($From, $To); [void]$undo.Add(@($From, $To)) } + function Undo-Logged { + $lost = 0 + for ($n = $undo.Count - 1; $n -ge 0; $n--) { + $u = $undo[$n] + try { if ($u.Count -eq 2) { [IO.File]::Move($u[1], $u[0]) } else { [IO.Directory]::Delete($u[0]) } } catch { if ($u.Count -eq 2) { $lost++ } } + } + $undo.Clear() + $lost + } + $done = $false; $at = $Dest + try { + if (-not (Test-Path -LiteralPath $Dest)) { [void][IO.Directory]::CreateDirectory($Dest); [void]$undo.Add(@($Dest)) } + $destDir = (Resolve-Path -LiteralPath $Dest).ProviderPath.TrimEnd('\') + foreach ($f in @(Get-ChildItem -LiteralPath $destDir -Recurse -Force -File)) { + if (-not $files.ContainsKey($f.FullName.Substring($destDir.Length)) -and $f.Name -notmatch $asideName) { + $at = $f.FullName; Move-Logged $at "$at.old-$token" + } + } + foreach ($rel in @($dirs.Keys | Sort-Object Length)) { + $at = $destDir + $rel + if (-not [IO.Directory]::Exists($at)) { [void][IO.Directory]::CreateDirectory($at); [void]$undo.Add(@($at)) } + } + foreach ($rel in @($files.Keys)) { + $at = $destDir + $rel + if ([IO.File]::Exists($at)) { Move-Logged $at "$at.old-$token" } + Move-Logged ($stageDir + $rel) $at + } + $done = $true + } catch { + $x = $_.Exception; while ($x.InnerException) { $x = $x.InnerException } + $lost = Undo-Logged + $msg = "Could not replace $at ($($x.Message))." + if ($lost) { + $msg += " $lost file(s) could not be put back, so the install may not start. Close your agent sessions and any running codegraph commands, then reinstall: irm https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.ps1 | iex" + } else { + $msg += " Nothing was changed: the existing install still works. If another program has CodeGraph's files open, close your agent sessions (they run the CodeGraph MCP server) and any running codegraph commands, then try again." + } + $e = New-Object System.Exception($msg); $e.Data['codegraphDamaged'] = [bool]$lost; throw $e + } finally { + # Interrupted (Ctrl+C) without reaching catch: still put everything back. + if (-not $done) { [void](Undo-Logged) } + } + # Delete what this run and earlier ones renamed aside. A file a running + # process still holds can't be deleted yet; the next install retries it. + foreach ($f in @(Get-ChildItem -LiteralPath $destDir -Recurse -Force -File -ErrorAction SilentlyContinue)) { + if ($f.Name -match $asideName) { try { [IO.File]::Delete($f.FullName) } catch {} } + } + foreach ($d in @(Get-ChildItem -LiteralPath $destDir -Recurse -Force -Directory -ErrorAction SilentlyContinue | Sort-Object { $_.FullName.Length } -Descending)) { + if (-not $dirs.ContainsKey($d.FullName.Substring($destDir.Length))) { try { [IO.Directory]::Delete($d.FullName) } catch {} } + } +} +# <<< Install-CodeGraphFiles + $repo = 'colbymchenry/codegraph' $installDir = if ($env:CODEGRAPH_INSTALL_DIR) { $env:CODEGRAPH_INSTALL_DIR } else { Join-Path $env:LOCALAPPDATA 'codegraph' } @@ -27,25 +103,28 @@ if (-not $version) { } if (-not $version) { throw "codegraph: could not resolve latest version; set CODEGRAPH_VERSION." } -# 3. Download + extract the bundle into a stable 'current' dir (overwritten on upgrade). +# 3. Download the bundle and unpack it next to the stable 'current' dir (same +# volume, so moving it in is renames, not copies), then move it into 'current'. +# 'current' is never deleted wholesale: a CodeGraph process that is still +# running (an agent session's MCP server) holds node.exe and the native kernel, +# which can't be deleted -- Install-CodeGraphFiles renames them aside instead. $url = "https://github.com/$repo/releases/download/$version/codegraph-$target.zip" Write-Host "Installing CodeGraph $version ($target)..." $tmp = Join-Path $env:TEMP ("cg-" + [guid]::NewGuid().ToString()) New-Item -ItemType Directory -Force -Path $tmp | Out-Null $zip = Join-Path $tmp 'cg.zip' -Invoke-WebRequest -Uri $url -OutFile $zip - $dest = Join-Path $installDir 'current' -if (Test-Path $dest) { Remove-Item -Recurse -Force $dest } -New-Item -ItemType Directory -Force -Path $dest | Out-Null -Expand-Archive -Path $zip -DestinationPath $dest -Force -# Archives contain a top-level codegraph-\ dir; flatten it. -$inner = Join-Path $dest "codegraph-$target" -if (Test-Path $inner) { - Get-ChildItem -Force $inner | Move-Item -Destination $dest -Force - Remove-Item -Recurse -Force $inner +$stage = Join-Path $installDir ('.staging-' + [guid]::NewGuid().ToString('N').Substring(0, 8)) +try { + Invoke-WebRequest -Uri $url -OutFile $zip + New-Item -ItemType Directory -Force -Path $stage | Out-Null + Expand-Archive -Path $zip -DestinationPath $stage -Force + # Archives contain a top-level codegraph-\ dir. + $inner = Join-Path $stage "codegraph-$target" + Install-CodeGraphFiles $(if (Test-Path $inner) { $inner } else { $stage }) $dest +} finally { + Remove-Item -LiteralPath $stage, $tmp -Recurse -Force -ErrorAction SilentlyContinue } -Remove-Item -Recurse -Force $tmp # 4. Put the launcher dir on the user's PATH. $binDir = Join-Path $dest 'bin' diff --git a/src/upgrade/index.ts b/src/upgrade/index.ts index 76354378a8..0e088e9982 100644 --- a/src/upgrade/index.ts +++ b/src/upgrade/index.ts @@ -17,11 +17,12 @@ * vendored `node` binary and a `bin/codegraph` launcher next to its `lib/`, so * we can recognize it from the running file's path without a marker file. * - * Windows wrinkle: a running `node.exe` is locked and can't be deleted, so the - * bundle's `current\` dir can't be overwritten in place by the process doing - * the upgrade. We therefore spawn a DETACHED helper that waits for this - * process to exit (releasing the lock), then runs `install.ps1`. This is the - * conventional Windows self-update dance (rustup/nvm-windows do the same). + * Windows wrinkle: a running `node.exe` and a loaded `.node` addon are locked — + * they can't be overwritten or deleted, by this process or by the MCP servers + * of open agent sessions. They CAN be renamed, so the Windows upgrade unpacks + * the new bundle next to `current\` and swaps it in file by file, renaming each + * replaced file aside and rolling every step back on failure (see + * `WINDOWS_SWAP_FUNCTION`, shared verbatim with `install.ps1`). */ import * as fs from 'fs'; @@ -34,6 +35,7 @@ export const REPO = 'colbymchenry/codegraph'; export const NPM_PACKAGE = '@colbymchenry/codegraph'; const RAW_BASE = `https://raw.githubusercontent.com/${REPO}/main`; export const INSTALL_SH_URL = `${RAW_BASE}/install.sh`; +export const INSTALL_PS1_URL = `${RAW_BASE}/install.ps1`; // --------------------------------------------------------------------------- // Install-method detection (pure — fully unit-testable via injected probes) @@ -578,36 +580,140 @@ function upgradeUnixBundle( return 0; } +/** + * The PowerShell function that moves an unpacked Windows bundle into the + * install's `current\` dir. Shared VERBATIM with `install.ps1` (a test pins the + * two copies equal), so a first install, a re-run of the installer, and + * `codegraph upgrade` all replace files the same way. + * + * Why file-by-file renames (#2185): every open agent session runs a CodeGraph + * MCP server from `current\`, which keeps `node.exe` and the native kernel + * (`lib\kernel\codegraph-kernel.node`) locked. Windows refuses to overwrite or + * delete a running exe or a loaded DLL but does let it be renamed. The old + * upgrade renamed only `node.exe` and then `Copy-Item`ed over the rest, so the + * locked kernel failed the copy halfway — leaving no `node.exe` and a mix of + * versions, which no `codegraph` command could repair. Now each file being + * replaced (or dropped by the new version) is renamed aside to + * `.old-` before the staged file is moved in; any failure undoes + * every step, and the renamed-aside files are deleted on the next successful + * run once nothing holds them. + */ +export const WINDOWS_SWAP_FUNCTION = String.raw`function Install-CodeGraphFiles([string]$Stage, [string]$Dest) { + # Move an unpacked bundle into $Dest. Windows can't overwrite or delete a + # running node.exe or a loaded .node addon, but it can rename one, so every + # file being replaced (or dropped by the new version) is first renamed aside + # to .old-. Any failure puts every file back, so the install is + # never left half-replaced or without its node.exe. + $ErrorActionPreference = 'Stop' + $stageDir = (Resolve-Path -LiteralPath $Stage).ProviderPath.TrimEnd('\') + foreach ($need in 'node.exe', 'bin\codegraph.cmd') { + if (-not (Test-Path -LiteralPath (Join-Path $stageDir $need))) { throw "The CodeGraph download is incomplete (no $need); nothing was changed." } + } + $token = [guid]::NewGuid().ToString('N').Substring(0, 8) + $asideName = '\.old-[0-9a-f]{8,32}$' + $files = @{}; $dirs = @{} + foreach ($i in @(Get-ChildItem -LiteralPath $stageDir -Recurse -Force)) { + $rel = $i.FullName.Substring($stageDir.Length) + if ($i.PSIsContainer) { $dirs[$rel] = $true } else { $files[$rel] = $true } + } + $undo = New-Object System.Collections.ArrayList + function Move-Logged([string]$From, [string]$To) { [IO.File]::Move($From, $To); [void]$undo.Add(@($From, $To)) } + function Undo-Logged { + $lost = 0 + for ($n = $undo.Count - 1; $n -ge 0; $n--) { + $u = $undo[$n] + try { if ($u.Count -eq 2) { [IO.File]::Move($u[1], $u[0]) } else { [IO.Directory]::Delete($u[0]) } } catch { if ($u.Count -eq 2) { $lost++ } } + } + $undo.Clear() + $lost + } + $done = $false; $at = $Dest + try { + if (-not (Test-Path -LiteralPath $Dest)) { [void][IO.Directory]::CreateDirectory($Dest); [void]$undo.Add(@($Dest)) } + $destDir = (Resolve-Path -LiteralPath $Dest).ProviderPath.TrimEnd('\') + foreach ($f in @(Get-ChildItem -LiteralPath $destDir -Recurse -Force -File)) { + if (-not $files.ContainsKey($f.FullName.Substring($destDir.Length)) -and $f.Name -notmatch $asideName) { + $at = $f.FullName; Move-Logged $at "$at.old-$token" + } + } + foreach ($rel in @($dirs.Keys | Sort-Object Length)) { + $at = $destDir + $rel + if (-not [IO.Directory]::Exists($at)) { [void][IO.Directory]::CreateDirectory($at); [void]$undo.Add(@($at)) } + } + foreach ($rel in @($files.Keys)) { + $at = $destDir + $rel + if ([IO.File]::Exists($at)) { Move-Logged $at "$at.old-$token" } + Move-Logged ($stageDir + $rel) $at + } + $done = $true + } catch { + $x = $_.Exception; while ($x.InnerException) { $x = $x.InnerException } + $lost = Undo-Logged + $msg = "Could not replace $at ($($x.Message))." + if ($lost) { + $msg += " $lost file(s) could not be put back, so the install may not start. Close your agent sessions and any running codegraph commands, then reinstall: irm https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.ps1 | iex" + } else { + $msg += " Nothing was changed: the existing install still works. If another program has CodeGraph's files open, close your agent sessions (they run the CodeGraph MCP server) and any running codegraph commands, then try again." + } + $e = New-Object System.Exception($msg); $e.Data['codegraphDamaged'] = [bool]$lost; throw $e + } finally { + # Interrupted (Ctrl+C) without reaching catch: still put everything back. + if (-not $done) { [void](Undo-Logged) } + } + # Delete what this run and earlier ones renamed aside. A file a running + # process still holds can't be deleted yet; the next install retries it. + foreach ($f in @(Get-ChildItem -LiteralPath $destDir -Recurse -Force -File -ErrorAction SilentlyContinue)) { + if ($f.Name -match $asideName) { try { [IO.File]::Delete($f.FullName) } catch {} } + } + foreach ($d in @(Get-ChildItem -LiteralPath $destDir -Recurse -Force -Directory -ErrorAction SilentlyContinue | Sort-Object { $_.FullName.Length } -Descending)) { + if (-not $dirs.ContainsKey($d.FullName.Substring($destDir.Length))) { try { [IO.Directory]::Delete($d.FullName) } catch {} } + } +} +`; + +/** Single-quote a value for PowerShell (a `'` inside is doubled). */ +function psQuote(value: string): string { + return `'${value.replace(/'/g, "''")}'`; +} + +/** Exit code of the upgrade script when files could not all be put back. */ +export const WINDOWS_UPGRADE_DAMAGED = 2; + /** Build the in-place Windows upgrade script (exported for unit-testing). */ export function buildWindowsUpgradeScript(bundleRoot: string, version: string, arch: string): string { const target = `win32-${arch}`; const url = `https://github.com/${REPO}/releases/download/${version}/codegraph-${target}.zip`; - // Windows can't DELETE a running exe but CAN rename it, so we upgrade IN - // PLACE: download → rename the locked node.exe aside → extract the new bundle - // over current\. Synchronous, no detached helper (which dies under SSH/job - // objects and has worse UX). The running process keeps its renamed node.exe - // mapped; the NEXT `codegraph` invocation uses the new one. We can't reuse - // install.ps1 here — it `Remove-Item`s current\, which fails on the locked exe. + // Synchronous, no detached helper (which dies under SSH/job objects and has + // worse UX). The bundle is unpacked into a sibling of current\ — the same + // volume, so the swap is renames, never a half-finished copy — and nothing + // in current\ changes until it is fully unpacked. The running process keeps + // its renamed node.exe mapped; the NEXT `codegraph` invocation uses the new + // one. Exit codes: 0 installed, 1 failed with the install unchanged, + // WINDOWS_UPGRADE_DAMAGED when the rollback could not restore every file. return [ `$ErrorActionPreference='Stop'`, - `$dest='${bundleRoot}'`, - `$url='${url}'`, - `Write-Host "Downloading $url"`, + WINDOWS_SWAP_FUNCTION, + `$dest=${psQuote(bundleRoot)}`, + `$url=${psQuote(url)}`, `$tmp=Join-Path $env:TEMP ('cg-up-'+[guid]::NewGuid().ToString('N'))`, - `New-Item -ItemType Directory -Force -Path $tmp | Out-Null`, - `$zip=Join-Path $tmp 'cg.zip'`, - `Invoke-WebRequest -Uri $url -OutFile $zip`, - `$stage=Join-Path $tmp 'stage'`, - `Expand-Archive -Path $zip -DestinationPath $stage -Force`, - `$inner=Join-Path $stage 'codegraph-${target}'`, - `$src=if(Test-Path $inner){$inner}else{$stage}`, - `$node=Join-Path $dest 'node.exe'`, - `if(Test-Path $node){Rename-Item -Path $node -NewName ('node.exe.old-'+[guid]::NewGuid().ToString('N')) -Force}`, - `Copy-Item -Path (Join-Path $src '*') -Destination $dest -Recurse -Force`, - `Get-ChildItem -Path $dest -Filter 'node.exe.old-*' -ErrorAction SilentlyContinue | ForEach-Object { try { Remove-Item $_.FullName -Force -ErrorAction Stop } catch {} }`, - `Remove-Item -Recurse -Force $tmp -ErrorAction SilentlyContinue`, - `Write-Host "Installed CodeGraph ${version} to $dest"`, - ].join(';'); + `$stage=Join-Path (Split-Path -Parent $dest) ('.staging-'+[guid]::NewGuid().ToString('N').Substring(0,8))`, + `$code=0`, + `try {`, + ` Write-Host "Downloading $url"`, + ` New-Item -ItemType Directory -Force -Path $tmp | Out-Null`, + ` $zip=Join-Path $tmp 'cg.zip'`, + ` Invoke-WebRequest -Uri $url -OutFile $zip`, + ` Expand-Archive -Path $zip -DestinationPath $stage -Force`, + ` $inner=Join-Path $stage 'codegraph-${target}'`, + ` Install-CodeGraphFiles $(if(Test-Path $inner){$inner}else{$stage}) $dest`, + ` Write-Host "Installed CodeGraph ${version} to $dest"`, + `} catch {`, + ` [Console]::Error.WriteLine($_.Exception.Message)`, + ` $code=if($_.Exception.Data['codegraphDamaged']){${WINDOWS_UPGRADE_DAMAGED}}else{1}`, + `}`, + `Remove-Item -LiteralPath $stage,$tmp -Recurse -Force -ErrorAction SilentlyContinue`, + `exit $code`, + ].join('\n'); } function upgradeWindowsBundle( @@ -624,7 +730,15 @@ function upgradeWindowsBundle( deps.log(c.dim(`Downloading and installing ${latest}…`)); const code = deps.run('powershell.exe', ['-NoProfile', '-ExecutionPolicy', 'Bypass', '-EncodedCommand', encoded]); if (code !== 0) { - deps.error(`Installer exited with code ${code}.`); + // The script has already printed what went wrong and what to do. + if (code === WINDOWS_UPGRADE_DAMAGED) { + deps.error('The upgrade failed and some files could not be put back.'); + deps.log(c.dim(`Close your agent sessions and any running codegraph commands, then reinstall: irm ${INSTALL_PS1_URL} | iex`)); + } else if (code === 1) { + deps.error(`The upgrade to ${latest} did not complete; your existing install was left as it was.`); + } else { + deps.error(`Installer exited with code ${code}.`); + } return 1; } deps.log(''); From 8abebda02cbe893a393d60ecc4f231f18d194f4a Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 07:07:34 +0000 Subject: [PATCH 158/259] test(windows): two #1910 tests no longer time out on Windows (#2298) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit oversize-file-not-read's drift test wrote a 1.4 MB `.js` made of one statement repeated 110k times. Windows Defender's script scan of that freshly written file takes about 54 s on its first read (the viewer's read in the test), so the test always timed out on Windows and its cleanup then hit EPERM on the still open file. A varied 2.4 MB `.js` and a real 2 MB bundle scan in 35 ms and 144 ms, so this is the fixture, not something users hit; the fixture now has a different statement on every line. The mpeg-ts git-path cases build a git repository and index it, the last one twice. At ~100 ms per git process on Windows that is 5–6 s, past vitest's 5 s default; the describe now allows 30 s. Co-authored-by: Claude Opus 5.5 --- __tests__/mpeg-ts-not-typescript.test.ts | 5 ++++- __tests__/oversize-file-not-read.test.ts | 13 +++++++++++-- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/__tests__/mpeg-ts-not-typescript.test.ts b/__tests__/mpeg-ts-not-typescript.test.ts index 5d44003d2e..dfc23a1950 100644 --- a/__tests__/mpeg-ts-not-typescript.test.ts +++ b/__tests__/mpeg-ts-not-typescript.test.ts @@ -215,7 +215,10 @@ describe('MPEG-TS video named .ts is skipped, real TypeScript is indexed (#1910) }); }); -describe('a video .ts never stays pending (#1910)', () => { +// Each case builds a git repository and indexes it, the last one twice over. On +// Windows every git process costs ~100 ms, which puts these cases at 5–6 s there, +// past vitest's 5 s default. +describe('a video .ts never stays pending (#1910)', { timeout: 30_000 }, () => { const dirs: string[] = []; afterEach(() => { for (const d of dirs.splice(0)) fs.rmSync(d, { recursive: true, force: true }); }); const gitProject = (): string => { diff --git a/__tests__/oversize-file-not-read.test.ts b/__tests__/oversize-file-not-read.test.ts index f32267f73f..fad014c375 100644 --- a/__tests__/oversize-file-not-read.test.ts +++ b/__tests__/oversize-file-not-read.test.ts @@ -89,8 +89,17 @@ describe('an unchanged file over the size limit is not reported as drifted (#191 dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-oversize-drift-')); fs.writeFileSync(path.join(dir, 'app.ts'), 'export function alpha() { return 1; }\n'); // 1.4 MB of ordinary text: over the index limit, under the viewer's 8 MB read cap. - const line = 'const x = 1;\n'; - fs.writeFileSync(path.join(dir, 'big.js'), line.repeat(Math.ceil((1.4 * 1024 * 1024) / line.length))); + // Each line differs: Windows Defender's script scan of a freshly written `.js` + // made of one statement repeated 110k times takes close to a minute on the + // first read, so the viewer's read below blew the test timeout there. Real + // code of the same size is scanned in milliseconds. + const line = 'const last = 0;\n'; + const lines: string[] = []; + for (let size = 0, i = 0; size < 1.4 * 1024 * 1024; i++) { + lines.push(`const x${i} = ${i};\n`); + size += lines[i]!.length; + } + fs.writeFileSync(path.join(dir, 'big.js'), lines.join('')); const cg = await CodeGraph.init(dir, { index: true }); try { const record = cg.getFiles().find((f) => f.path === 'big.js')!; From 91b9bba1336cb6fe33113b0df25c37ae07b078ca Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 08:48:01 +0000 Subject: [PATCH 159/259] fix(extraction): never terminate a parse worker while it is loading grammars (#2301) On a loaded Windows machine `codegraph init` / `index` sometimes died right after the parse phase with 0xC0000005 (STATUS_ACCESS_VIOLATION), no WER report and no output past the last parsed file. The parse pool's destroy() terminated every worker at once, and a short index often ends while a late-spawned worker is still compiling its grammar WASM; terminating a worker with that compile in flight crashes the process. Reproduced on a Windows 11 ARM64 VM without CodeGraph's pipeline: starting three parse workers with preloaded grammar bytes and terminating them mid-load crashed 5 of 720 child processes, terminating them only after grammars-loaded crashed 0 of 360. Under the real load (24 parallel `codegraph init` runs on 4 cores) main crashed after parsing in 8 of 1,200 runs; with this change 0 of 1,080 did. destroy() now waits for a worker that is still loading to report grammars-loaded or exit before terminating it, capped at 15 s so a wedged load can't hold teardown forever. Recycling only ever touches idle workers, and a hung parse is still killed by the hard timeout. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/parse-pool.test.ts | 79 ++++++++++++++++++++++++++++++++++++ src/extraction/parse-pool.ts | 54 ++++++++++++++++++++++-- 3 files changed, 131 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 348a804c87..f7b195d0d2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -174,6 +174,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - On Windows, a project opened with a different drive-letter or folder-name case, like `d:\work\app` and `D:\Work\App`, now joins the CodeGraph background server that is already running for it. Before, each spelling started a server of its own that exited at once, so the session quietly served the graph by itself with no shared file watching or auto-sync, and `codegraph list` showed the project twice. Thanks @greatjackzhou. (#2278) - An agent session that lost its connection to CodeGraph's shared background server, or couldn't reach it at startup, no longer stops that server from running for the rest of the session. It reconnects on its own once the server can start, so other sessions on the same project get auto-sync back instead of running read-only. Thanks @ijbranch for the report. (#2277) - On Windows, `codegraph upgrade` no longer breaks the install while an agent session is using CodeGraph. It used to stop partway and leave CodeGraph unable to start; it now swaps the new files in with CodeGraph running and puts the previous version back if anything fails, and re-running the PowerShell installer works the same way. If an earlier upgrade already broke your install, re-run `irm https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.ps1 | iex` to repair it. Thanks @tippmar-nr for the report. (#2185) +- On Windows, `codegraph init`, `codegraph index` and the background server's syncs no longer sometimes crash right after parsing on a busy machine (exit code 3221225477, an access violation). Shutting down the parsing threads could stop one while it was still loading its language grammars, which took the whole process down; they now finish loading first. ## [1.6.1] - 2026-09-29 diff --git a/__tests__/parse-pool.test.ts b/__tests__/parse-pool.test.ts index 6211481a44..ac51b984b5 100644 --- a/__tests__/parse-pool.test.ts +++ b/__tests__/parse-pool.test.ts @@ -248,3 +248,82 @@ describe('ParseWorkerPool', () => { await expect(pool.requestParse(task('y.ts'))).rejects.toThrow(/destroyed/); }); }); + +/** + * A fake whose grammar load takes `loadMs` (or never finishes, with `null`), and + * which records when it finished loading and when it was terminated. + */ +class SlowLoadWorker implements ParsePoolWorker { + private msgCb?: (m: unknown) => void; + private exitCb?: (code: number) => void; + loadedAt: number | null = null; + terminatedAt: number | null = null; + constructor(private loadMs: number | null) {} + on(event: string, cb: (...args: any[]) => void): void { + if (event === 'message') this.msgCb = cb; + else if (event === 'exit') this.exitCb = cb; + } + postMessage(msg: unknown): void { + if ((msg as { type: string }).type !== 'load-grammars' || this.loadMs === null) return; + setTimeout(() => { + if (this.terminatedAt !== null) return; + this.loadedAt = performance.now(); + this.msgCb?.({ type: 'grammars-loaded' }); + }, this.loadMs); + } + exit(code: number): void { this.exitCb?.(code); } + terminate(): Promise { this.terminatedAt = performance.now(); return Promise.resolve(0); } +} + +describe('teardown never terminates a worker mid-load', () => { + // Terminating a worker whose grammar WASM is still compiling can crash the + // process with an access violation (0xC0000005) on Windows; a pool torn down + // right after a short index often has a late-spawned worker in that state. + const poolOf = (worker: SlowLoadWorker, loadSettleMs?: number) => + new ParseWorkerPool({ languages: ['typescript'] as Language[], size: 1, createWorker: () => worker, loadSettleMs }); + + it('waits for a loading worker to finish before terminating it', async () => { + const w = new SlowLoadWorker(120); + const pool = poolOf(w); + await pool.destroy(); + expect(w.loadedAt).not.toBeNull(); + expect(w.terminatedAt).not.toBeNull(); + expect(w.terminatedAt!).toBeGreaterThanOrEqual(w.loadedAt!); + }); + + it('terminates a worker that is already loaded right away', async () => { + const w = new SlowLoadWorker(0); + const pool = poolOf(w); + await sleep(20); + expect(w.loadedAt).not.toBeNull(); + const t0 = performance.now(); + await pool.destroy(); + expect(w.terminatedAt! - t0).toBeLessThan(50); + }); + + it('does not wait past the cap on a load that never finishes', async () => { + const w = new SlowLoadWorker(null); + const pool = poolOf(w, 80); + const t0 = performance.now(); + await pool.destroy(); + expect(w.terminatedAt).not.toBeNull(); + expect(w.terminatedAt! - t0).toBeGreaterThanOrEqual(70); + expect(w.terminatedAt! - t0).toBeLessThan(1000); + }); + + it('does not wait on a worker that died while loading', async () => { + const w = new SlowLoadWorker(null); + // The pool respawns a crashed worker; the replacement loads normally. The + // default 15s cap stays, so waiting on the dead one would time the test out. + const workers = [w]; + const pool = new ParseWorkerPool({ + languages: ['typescript'] as Language[], size: 1, + createWorker: () => workers.shift() ?? new SlowLoadWorker(0), + }); + w.exit(1); + await sleep(20); + const t0 = performance.now(); + await pool.destroy(); + expect(performance.now() - t0).toBeLessThan(200); + }); +}); diff --git a/src/extraction/parse-pool.ts b/src/extraction/parse-pool.ts index 9b2b100e4e..62967a3dfb 100644 --- a/src/extraction/parse-pool.ts +++ b/src/extraction/parse-pool.ts @@ -75,6 +75,16 @@ const MAX_SCALED_PARSE_TIMEOUT_MS = 20_000; * only a worker that stays silent the whole window is treated as hung. */ const HARD_KILL_MULTIPLIER = 3; +/** + * How long `destroy()` waits for a worker that is still loading its grammars + * before terminating it anyway. Terminating a worker whose WebAssembly + * compiles are still in flight can take the whole process down with an access + * violation (0xC0000005, seen on Windows): a pool torn down right after a short + * index often has a late-spawned worker in exactly that state. A load finishes + * in well under a second normally and a few seconds under heavy load; the cap + * only bounds a load that is wedged. + */ +const GRAMMAR_LOAD_SETTLE_MS = 15_000; /** * Max workers cold-starting at once. A worker's cold start is heavy (module load * + grammar WASM compile); starting the whole pool simultaneously thrashes CPU. @@ -169,6 +179,8 @@ export interface ParseWorkerPoolOptions { parseTimeoutMs?: number; /** Worker factory (tests inject a fake). Defaults to a real `worker_threads` Worker. */ createWorker?: () => ParsePoolWorker; + /** How long destroy() waits on a worker still loading grammars (tests shorten it). Default 15s. */ + loadSettleMs?: number; /** Optional verbose logger (the orchestrator's `[worker] …` logger). */ log?: (msg: string) => void; /** @@ -190,6 +202,10 @@ export class ParseWorkerPool { // Spawned but not yet 'grammars-loaded'. Growth counts these so a single first // parse doesn't spawn the whole pool before the eager worker reports ready. private pending = new Set(); + // Each worker's grammar load, settled when it reports 'grammars-loaded' or + // goes away — what destroy() waits on before terminating it (see + // GRAMMAR_LOAD_SETTLE_MS). + private loads = new Map; settle: () => void }>(); private parseCounts = new Map(); private nextId = 1; private totalCrashes = 0; @@ -200,6 +216,7 @@ export class ParseWorkerPool { private readonly recycleInterval: number; private readonly parseTimeoutMs: number; private readonly createWorker: () => ParsePoolWorker; + private readonly loadSettleMs: number; private readonly log: (msg: string) => void; private readonly grammarBuffers?: Record; @@ -209,6 +226,7 @@ export class ParseWorkerPool { this.maxSize = Math.max(1, Math.min(opts.size, MAX_PARSE_POOL_SIZE)); this.recycleInterval = opts.recycleInterval ?? DEFAULT_RECYCLE_INTERVAL; this.parseTimeoutMs = opts.parseTimeoutMs ?? DEFAULT_PARSE_TIMEOUT_MS; + this.loadSettleMs = opts.loadSettleMs ?? GRAMMAR_LOAD_SETTLE_MS; this.log = opts.log ?? (() => {}); if (opts.createWorker) { this.createWorker = opts.createWorker; @@ -278,9 +296,12 @@ export class ParseWorkerPool { this.workers.add(w); this.pending.add(w); this.parseCounts.set(w, 0); + let settle!: () => void; + const settled = new Promise((resolve) => { settle = resolve; }); + this.loads.set(w, { settled, settle }); w.on('message', (m) => this.onMessage(w, (m ?? {}) as ParseWorkerMessage)); - w.on('error', (e) => this.onWorkerGone(w, `Worker error: ${e?.message ?? 'unknown'}`)); - w.on('exit', (code) => { if (code !== 0) this.onWorkerGone(w, `Worker exited with code ${code}`); }); + w.on('error', (e) => { this.loadSettled(w); this.onWorkerGone(w, `Worker error: ${e?.message ?? 'unknown'}`); }); + w.on('exit', (code) => { this.loadSettled(w); if (code !== 0) this.onWorkerGone(w, `Worker exited with code ${code}`); }); // Load grammars; the worker replies 'grammars-loaded' and only then is idle. // Pre-read WASM bytes (when the orchestrator provided them) make this a // memory load instead of a per-spawn disk read. @@ -289,6 +310,7 @@ export class ParseWorkerPool { private onMessage(w: ParsePoolWorker, m: ParseWorkerMessage): void { if (m.type === 'grammars-loaded') { + this.loadSettled(w); if (!this.workers.has(w)) return; // recycled/destroyed before ready this.pending.delete(w); this.idle.push(w); @@ -351,6 +373,32 @@ export class ParseWorkerPool { if (this.healthy && !this.destroyed) this.spawnOne(); } + /** The worker's grammar load is over: it reported ready, errored or exited. */ + private loadSettled(w: ParsePoolWorker): void { + const load = this.loads.get(w); + if (!load) return; + this.loads.delete(w); + load.settle(); + } + + /** + * Terminate a worker, but not while it is still loading grammars: wait for + * the load to finish (or the worker to go away), up to `loadSettleMs` + * (GRAMMAR_LOAD_SETTLE_MS — see that constant for the crash this avoids). + */ + private async terminateSettled(w: ParsePoolWorker): Promise { + const load = this.loads.get(w); + if (load) { + let cap: NodeJS.Timeout | undefined; + await Promise.race([ + load.settled, + new Promise((resolve) => { cap = setTimeout(resolve, this.loadSettleMs); cap.unref?.(); }), + ]); + clearTimeout(cap); + } + try { await w.terminate(); } catch { /* already gone */ } + } + private removeWorker(w: ParsePoolWorker): void { this.workers.delete(w); this.pending.delete(w); @@ -471,6 +519,6 @@ export class ParseWorkerPool { } this.inflight.clear(); this.queue = []; - await Promise.all(ws.map((w) => Promise.resolve(w.terminate()).catch(() => { /* already gone */ }))); + await Promise.all(ws.map((w) => this.terminateSettled(w))); } } From e0f9c95e2f7e5b8752b5b1bbd1362842c44b1ff3 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 10:20:23 +0000 Subject: [PATCH 160/259] fix(resolution): a busy machine doesn't boot the resolver pool for a small project (#2303) The resolve loop boots the worker pool mid-run when the remaining refs' projected sequential settle (observed per-ref rate x refs left) clears 400 ms. Load inflates the observed rate, so on a loaded 4-core Windows VM a ten-file project (19 refs) projected 506 ms for its last 11 and booted three workers in 122 of 240 runs under 24 parallel inits (0 of 80 at 8). The pool only fans out batches of 1,000 refs, so it resolved nothing; synthesis then waited out its boot (dedupe-merge 6.3 s vs a 6.5 s boot) and its three extra database connections pushed the VM's free commit to 11 MB. Adaptive engagement now also needs 20,000 refs left. tokio-class Rust repos (~55k refs at ~36 us) still engage; a smaller remainder never repays a seconds-long boot. With the floor, the same load booted the pool in 0 of 240 runs. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/resolver-pool-sizing.test.ts | 16 +++++++++++++++- src/resolution/index.ts | 4 ++-- src/resolution/resolver-pool.ts | 17 +++++++++++++++++ 4 files changed, 35 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f7b195d0d2..0669709589 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -175,6 +175,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - An agent session that lost its connection to CodeGraph's shared background server, or couldn't reach it at startup, no longer stops that server from running for the rest of the session. It reconnects on its own once the server can start, so other sessions on the same project get auto-sync back instead of running read-only. Thanks @ijbranch for the report. (#2277) - On Windows, `codegraph upgrade` no longer breaks the install while an agent session is using CodeGraph. It used to stop partway and leave CodeGraph unable to start; it now swaps the new files in with CodeGraph running and puts the previous version back if anything fails, and re-running the PowerShell installer works the same way. If an earlier upgrade already broke your install, re-run `irm https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.ps1 | iex` to repair it. Thanks @tippmar-nr for the report. (#2185) - On Windows, `codegraph init`, `codegraph index` and the background server's syncs no longer sometimes crash right after parsing on a busy machine (exit code 3221225477, an access violation). Shutting down the parsing threads could stop one while it was still loading its language grammars, which took the whole process down; they now finish loading first. +- On a busy machine, indexing or syncing a small project no longer starts extra background threads for resolving references that it can't use. Under heavy load the slower pace made a ten-file project look big enough to need them, which added seconds to the run and held extra memory. ## [1.6.1] - 2026-09-29 diff --git a/__tests__/resolver-pool-sizing.test.ts b/__tests__/resolver-pool-sizing.test.ts index cd949a4f0e..e70b7a6911 100644 --- a/__tests__/resolver-pool-sizing.test.ts +++ b/__tests__/resolver-pool-sizing.test.ts @@ -8,7 +8,7 @@ */ import { describe, it, expect } from 'vitest'; import * as os from 'os'; -import { ResolverPool } from '../src/resolution/resolver-pool'; +import { ResolverPool, ADAPTIVE_ENGAGE_MIN_REFS, shouldEngageAdaptively } from '../src/resolution/resolver-pool'; import { cgroupMemoryAvailable, darwinMemoryAvailable, @@ -106,3 +106,17 @@ describe('memory budget helpers', () => { } ); }); + +describe('adaptive pool engagement needs a real amount of work left', () => { + it('does not boot the pool for a small remainder, however slow the observed rate', () => { + // A busy machine: 11 refs left projected at 506 ms (seen on a loaded 4-core VM). + expect(shouldEngageAdaptively(506, 11, 400)).toBe(false); + expect(shouldEngageAdaptively(60_000, ADAPTIVE_ENGAGE_MIN_REFS - 1, 400)).toBe(false); + }); + + it('boots it for a large remainder whose projected settle clears the bar', () => { + // tokio-class: ~55k Rust refs at ~36 µs each. + expect(shouldEngageAdaptively(1_980, 55_000, 400)).toBe(true); + expect(shouldEngageAdaptively(399, 55_000, 400)).toBe(false); + }); +}); diff --git a/src/resolution/index.ts b/src/resolution/index.ts index f4bb91a237..1eaf14b145 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -27,7 +27,7 @@ import { isCppConstructorRef, matchCppConstructor } from './cpp-constructor'; import { gateSwiftTypeTarget, clearSwiftTypeVisibility, swiftExtendedConformances } from './swift-type-visibility'; import { gateTypeParameter, clearTypeParameterMemos } from './type-parameters'; import { resolveViaImport, resolvePhpImportedStaticCall, resolvePhpQualifiedClassRef, resolveJvmImport, extractImportMappings, extractReExports, loadCppIncludeDirs, isPhpIncludePathRef, isCobolCopybookRef, isNixPathImportRef, isJsPathImportRef, isBoundToOutOfRepoImport, clearImportResolverMemos, resolveImportPath, isExternalImport } from './import-resolver'; -import { ResolverPool, minRefsForPool } from './resolver-pool'; +import { ResolverPool, minRefsForPool, shouldEngageAdaptively } from './resolver-pool'; import { resolveAliasBinding } from './alias-binding'; import { detectFrameworks } from './frameworks'; import { synthesizeCallbackEdges } from './callback-synthesizer'; @@ -2109,7 +2109,7 @@ export class ReferenceResolver { adaptiveSeqRefs += batch.length; const remaining = total - processed - batch.length; const projectedMs = (adaptiveSeqMs / Math.max(1, adaptiveSeqRefs)) * Math.max(0, remaining); - if (projectedMs >= ADAPTIVE_ENGAGE_SETTLE_MS) { + if (shouldEngageAdaptively(projectedMs, remaining, ADAPTIVE_ENGAGE_SETTLE_MS)) { if (process.env.CODEGRAPH_SYNTH_TIMINGS) { console.error(`[pool-timing] adaptive engage: projected ${Math.round(projectedMs)}ms sequential settle over ${remaining} remaining refs`); } diff --git a/src/resolution/resolver-pool.ts b/src/resolution/resolver-pool.ts index 56b9785a66..96f4340d8d 100644 --- a/src/resolution/resolver-pool.ts +++ b/src/resolution/resolver-pool.ts @@ -58,6 +58,23 @@ export function minRefsForPool(): number { return 150_000; } +/** + * Fewest refs still to settle for the batch loop to boot the pool mid-run + * from its observed per-ref rate. Booting takes seconds (each worker opens the + * database and warms a resolver), only batches of MIN_PARALLEL_BATCH refs fan + * out, and synthesis waits for a booting pool. A rate measured on a busy + * machine inflates every projection, so with no floor a ten-file project booted + * the pool in half its runs under heavy load and paid seconds for nothing. + * Repos the adaptive bar is for (Rust-class per-ref cost) have tens of + * thousands of refs and still engage. + */ +export const ADAPTIVE_ENGAGE_MIN_REFS = 20_000; + +/** Whether the resolve loop should boot the pool mid-run (see ADAPTIVE_ENGAGE_MIN_REFS). */ +export function shouldEngageAdaptively(projectedMs: number, remainingRefs: number, barMs: number): boolean { + return remainingRefs >= ADAPTIVE_ENGAGE_MIN_REFS && projectedMs >= barMs; +} + export class ResolverPool { private workers: PoolWorker[] = []; private nextId = 0; From 9d4e805bec36d62332d7fba0a735329ba7621777 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 10:36:42 +0000 Subject: [PATCH 161/259] test(windows): daemon-pid-reuse cleanup retries a still-held temp dir (#2304) afterEach SIGKILLs the servers and the detached daemon, waits 50 ms and removes the temp dir. A killed process releases its handles asynchronously on Windows, and under full-suite load the removal hit EPERM on the VM (the test itself passed; it passes 3/3 alone). Retry the removal like the other process-spawning suites do. Co-authored-by: Claude Opus 5.5 --- __tests__/daemon-pid-reuse.test.ts | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/__tests__/daemon-pid-reuse.test.ts b/__tests__/daemon-pid-reuse.test.ts index 48650ccdf5..8bb116a0ce 100644 --- a/__tests__/daemon-pid-reuse.test.ts +++ b/__tests__/daemon-pid-reuse.test.ts @@ -197,7 +197,9 @@ describe('Shared MCP daemon (issue #411)', () => { } await new Promise((r) => setTimeout(r, 50)); servers.length = 0; - fs.rmSync(tempDir, { recursive: true, force: true }); + // A killed process releases its handles asynchronously on Windows; under + // full-suite load 50 ms is not always enough, so retry EPERM/EBUSY. + fs.rmSync(tempDir, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); }); it('takes over after SIGKILL even when the stale PID has been reused (#1553)', async () => { From 3d86bcc0371079070d25901830fb2102100434c3 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 18:03:28 +0000 Subject: [PATCH 162/259] fix(windows): the query pool never terminates a worker still starting up (#2307) Terminating a worker thread while it is still loading its modules can crash the whole process on Windows (exit 3221225477 / 0xC0000005, no error, no dump). QueryPool.destroy() terminated every worker at once, and the daemon destroys its pool whenever it stops - a session that ends within a few seconds of starting had its eager worker in exactly that state. MCPEngine.stop() didn't even wait for the pool before the daemon called process.exit(). - QueryPool tracks each worker's start (settled on 'ready', error or exit); destroy() answers outstanding calls first, then terminates each worker once it has started, capped at 15 s - the parse pool's #2301 mechanism. - MCPEngine.stop() awaits the pool. - #2301's explanation is corrected: on the Windows VM, terminating workers mid grammar compile never crashed (0 in 2,161), but terminating them while they load CodeGraph's extraction modules did. The parse pool's wait already covers that window, so only its comment and changelog wording change. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 3 +- __tests__/query-pool.test.ts | 101 +++++++++++++++++++++++++++++++++++ src/extraction/parse-pool.ts | 14 ++--- src/mcp/engine.ts | 10 ++-- src/mcp/query-pool.ts | 62 +++++++++++++++++++-- 5 files changed, 173 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0669709589..86258f27b6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -174,8 +174,9 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - On Windows, a project opened with a different drive-letter or folder-name case, like `d:\work\app` and `D:\Work\App`, now joins the CodeGraph background server that is already running for it. Before, each spelling started a server of its own that exited at once, so the session quietly served the graph by itself with no shared file watching or auto-sync, and `codegraph list` showed the project twice. Thanks @greatjackzhou. (#2278) - An agent session that lost its connection to CodeGraph's shared background server, or couldn't reach it at startup, no longer stops that server from running for the rest of the session. It reconnects on its own once the server can start, so other sessions on the same project get auto-sync back instead of running read-only. Thanks @ijbranch for the report. (#2277) - On Windows, `codegraph upgrade` no longer breaks the install while an agent session is using CodeGraph. It used to stop partway and leave CodeGraph unable to start; it now swaps the new files in with CodeGraph running and puts the previous version back if anything fails, and re-running the PowerShell installer works the same way. If an earlier upgrade already broke your install, re-run `irm https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.ps1 | iex` to repair it. Thanks @tippmar-nr for the report. (#2185) -- On Windows, `codegraph init`, `codegraph index` and the background server's syncs no longer sometimes crash right after parsing on a busy machine (exit code 3221225477, an access violation). Shutting down the parsing threads could stop one while it was still loading its language grammars, which took the whole process down; they now finish loading first. +- On Windows, `codegraph init`, `codegraph index` and the background server's syncs no longer sometimes crash right after parsing on a busy machine (exit code 3221225477, an access violation). Shutting down the parsing threads could stop one while it was still starting up, which took the whole process down; they now finish starting first. - On a busy machine, indexing or syncing a small project no longer starts extra background threads for resolving references that it can't use. Under heavy load the slower pace made a ten-file project look big enough to need them, which added seconds to the run and held extra memory. +- On Windows, the CodeGraph background server no longer sometimes crashes as it shuts down (exit code 3221225477, an access violation) when it stops shortly after starting. Shutting down could stop one of its background threads while that thread was still starting up; it now lets the thread finish starting first. ## [1.6.1] - 2026-09-29 diff --git a/__tests__/query-pool.test.ts b/__tests__/query-pool.test.ts index 2087e6af47..7c28f48f04 100644 --- a/__tests__/query-pool.test.ts +++ b/__tests__/query-pool.test.ts @@ -182,6 +182,63 @@ describe('QueryPool', () => { expect(pool.healthy).toBe(false); }); + describe('destroy never terminates a worker still starting up', () => { + // Terminating a worker while it is still loading its modules can crash the + // whole process on Windows (0xC0000005), and the daemon destroys its pool + // whenever it stops. FakeWorker(…, null) never posts 'ready' by itself. + it('answers callers at once, then waits for the worker to start', async () => { + let worker!: FakeWorker; + const pool = new QueryPool({ + root: '/x', size: 1, softTimeoutMs: 10_000, + createWorker: () => (worker = new FakeWorker(() => ({ hang: true }), null)), + }); + const pending = pool.run('codegraph_explore', { query: 'q' }); + let destroyed = false; + const down = pool.destroy().then(() => { destroyed = true; }); + expect((await pending).isError).toBe(true); // not held behind the start + await sleep(50); + expect(destroyed).toBe(false); + expect(worker.alive).toBe(true); + worker.emitMessage({ type: 'ready', ok: true }); + await down; + expect(worker.alive).toBe(false); + }); + + it('terminates a started worker at once', async () => { + let worker!: FakeWorker; + const pool = new QueryPool({ root: '/x', size: 1, startSettleMs: 10_000, createWorker: () => (worker = new FakeWorker(() => ({ result: ok('r') }))) }); + await sleep(5); + const started = Date.now(); + await pool.destroy(); + expect(worker.alive).toBe(false); + expect(Date.now() - started).toBeLessThan(1_000); + }); + + it('gives up waiting on a start that never finishes', async () => { + let worker!: FakeWorker; + const pool = new QueryPool({ root: '/x', size: 1, startSettleMs: 60, createWorker: () => (worker = new FakeWorker(() => ({ hang: true }), null)) }); + const started = Date.now(); + await pool.destroy(); + expect(worker.alive).toBe(false); + expect(Date.now() - started).toBeGreaterThanOrEqual(55); + }); + + it('does not wait on a worker that died while starting', async () => { + const workers: FakeWorker[] = []; + const pool = new QueryPool({ + root: '/x', size: 1, startSettleMs: 10_000, + // The first never starts; its replacement does. + createWorker: () => { const w = new FakeWorker(() => ({ result: ok('r') }), workers.length ? true : null); workers.push(w); return w; }, + }); + workers[0].emitExit(); + await sleep(5); + const started = Date.now(); + await pool.destroy(); + expect(Date.now() - started).toBeLessThan(1_000); + expect(workers.every((w) => !w.alive)).toBe(true); + }); + }); + it('is not `ready` until a worker completes its cold start (#662 first-call stall)', async () => { // A worker cold start is seconds (tens under load); a call queued behind it // waits for the 45s busy backstop with nothing served. The ToolHandler must @@ -408,6 +465,50 @@ describe('MCP query pool with real projects (#1465)', () => { } }, 30000); + it('never terminates a real worker before it has started', async () => { + const alpha = await indexProject('alpha', 'alphaSymbol'); + const startedWhenEnded: boolean[] = []; + pool = new QueryPool({ + root: alpha, size: 1, + createWorker: () => { + const worker = new Worker(path.resolve(__dirname, '../dist/mcp/query-worker.js'), { workerData: { root: alpha } }); + let started = false; + worker.on('message', (m: { type?: string }) => { if (m?.type === 'ready') started = true; }); + const terminate = worker.terminate.bind(worker); + worker.terminate = () => { + startedWhenEnded.push(started); + return terminate(); + }; + return worker; + }, + }); + await pool.destroy(); // its eager worker is still loading + expect(startedWhenEnded).toEqual([true]); + }, 30000); + + it('engine stop waits for the query pool to shut down', async () => { + const alpha = await indexProject('alpha', 'alphaSymbol'); + const activeEngine = await start(alpha); + expect(pool).not.toBeNull(); + const realPool = pool!; + let release!: () => void; + const destroy = vi.spyOn(realPool, 'destroy').mockReturnValue(new Promise((r) => { release = r; })); + try { + let stopped = false; + const stopping = activeEngine.stop().then(() => { stopped = true; }); + // Everything else stop() closes is done well inside this; only the pool is held. + await Promise.race([stopping, sleep(1_500)]); + expect(stopped).toBe(false); + release(); + await stopping; + expect(stopped).toBe(true); + } finally { + release(); + destroy.mockRestore(); + await realPool.destroy(); + } + }, 30000); + it('honors size=0 for projectPath-only sessions', async () => { const alpha = await indexProject('alpha', 'alphaSymbol'); const workspace = path.join(tempDir, 'workspace'); diff --git a/src/extraction/parse-pool.ts b/src/extraction/parse-pool.ts index 62967a3dfb..b406a0916d 100644 --- a/src/extraction/parse-pool.ts +++ b/src/extraction/parse-pool.ts @@ -76,13 +76,13 @@ const MAX_SCALED_PARSE_TIMEOUT_MS = 20_000; */ const HARD_KILL_MULTIPLIER = 3; /** - * How long `destroy()` waits for a worker that is still loading its grammars - * before terminating it anyway. Terminating a worker whose WebAssembly - * compiles are still in flight can take the whole process down with an access - * violation (0xC0000005, seen on Windows): a pool torn down right after a short - * index often has a late-spawned worker in exactly that state. A load finishes - * in well under a second normally and a few seconds under heavy load; the cap - * only bounds a load that is wedged. + * How long `destroy()` waits for a worker that is still starting up — loading + * its modules, then its grammars — before terminating it anyway. Terminating a + * worker while it is still loading its modules can take the whole process down + * with an access violation (0xC0000005, seen on Windows): a pool torn down + * right after a short index often has a late-spawned worker in exactly that + * state. A start finishes in well under a second normally and a few seconds + * under heavy load; the cap only bounds a start that is wedged. */ const GRAMMAR_LOAD_SETTLE_MS = 15_000; /** diff --git a/src/mcp/engine.ts b/src/mcp/engine.ts index 5409c36d48..62a4c8041c 100644 --- a/src/mcp/engine.ts +++ b/src/mcp/engine.ts @@ -271,13 +271,13 @@ export class MCPEngine { // Detach + terminate the worker pool first so no tool call routes to a // worker mid-teardown; outstanding pool calls resolve with graceful guidance. + // Stopping waits for the workers to end: the daemon exits right after, and + // exiting while a worker is still starting up can crash the process. this.toolHandler.setQueryPool(null); - if (this.queryPool) { - void this.queryPool.destroy(); - this.queryPool = null; - } + const poolDown = this.queryPool ? this.queryPool.destroy() : Promise.resolve(); + this.queryPool = null; const drained = this.toolHandler.closeAll(); - this.stopPromise = drained.then(async () => { + this.stopPromise = Promise.all([drained, poolDown]).then(async () => { if (this.initPromise) await this.initPromise; if (this.defaultLease) { await this.defaultLease.release(); diff --git a/src/mcp/query-pool.ts b/src/mcp/query-pool.ts index b3a0f5322e..6b1f473991 100644 --- a/src/mcp/query-pool.ts +++ b/src/mcp/query-pool.ts @@ -69,6 +69,17 @@ const CRASH_BUDGET = 12; */ const MAX_CONCURRENT_SPAWN = 2; +/** + * How long `destroy()` waits for a worker that is still starting up before + * terminating it anyway. Terminating a worker while it is still loading its + * modules can take the whole process down with an access violation + * (0xC0000005, seen on Windows), and the daemon shuts its pool down whenever + * it stops — a session that ends within a few seconds of starting has a + * worker in exactly that state. A start takes about a second normally and a + * few under heavy load; the cap only bounds a start that is wedged. + */ +const WORKER_START_SETTLE_MS = 15_000; + /** Shape of a message a worker posts back (ready handshake or a tool result). */ interface WorkerMessage { type?: string; @@ -99,6 +110,8 @@ export interface QueryPoolOptions { maxRetries?: number; /** Worker factory (tests inject a fake). Defaults to a real `worker_threads` Worker. */ createWorker?: () => PoolWorker; + /** How long destroy() waits on a worker still starting up (tests shorten it). Default 15s. */ + startSettleMs?: number; } /** @@ -150,6 +163,10 @@ export class QueryPool { // a large DB open) saturate the box and starve the main loop. Grow only when // the queue outstrips idle + pending. private pendingWorkers = new Set(); + // Each worker's start, settled when it posts 'ready' (either way) or goes + // away — what destroy() waits on before terminating it (see + // WORKER_START_SETTLE_MS). + private starts = new Map; settle: () => void }>(); private nextId = 1; private totalCrashes = 0; private destroyed = false; @@ -158,6 +175,7 @@ export class QueryPool { private readonly softTimeoutMs: number; private readonly maxRetries: number; private readonly createWorker: () => PoolWorker; + private readonly startSettleMs: number; constructor(opts: QueryPoolOptions) { this.root = opts.root; @@ -165,6 +183,7 @@ export class QueryPool { this.softTimeoutMs = opts.softTimeoutMs ?? resolveBusyTimeoutMs(); this.maxRetries = opts.maxRetries ?? 1; this.createWorker = opts.createWorker ?? (() => new Worker(WORKER_FILE, { workerData: { root: this.root } })); + this.startSettleMs = opts.startSettleMs ?? WORKER_START_SETTLE_MS; this.spawnOne(); // one eager warm worker, ready for the first call } @@ -211,12 +230,18 @@ export class QueryPool { } this.workers.add(w); this.pendingWorkers.add(w); + let settle!: () => void; + const settled = new Promise((resolve) => { settle = resolve; }); + this.starts.set(w, { settled, settle }); w.on('message', (m) => this.onMessage(w, (m ?? {}) as WorkerMessage)); - w.on('error', () => this.onWorkerGone(w)); - w.on('exit', (code) => { if (code !== 0) this.onWorkerGone(w); }); + w.on('error', () => { this.startSettled(w); this.onWorkerGone(w); }); + w.on('exit', (code) => { this.startSettled(w); if (code !== 0) this.onWorkerGone(w); }); } private onMessage(w: PoolWorker, m: WorkerMessage): void { + // Before the retired-worker check: destroy() waits on this for a worker + // it has already let go of. + if (m?.type === 'ready') this.startSettled(w); if (!m || !this.workers.has(w)) return; // ignore late messages from retired workers if (m.type === 'ready') { if (!this.pendingWorkers.delete(w)) return; // already handled this handshake @@ -311,7 +336,36 @@ export class QueryPool { }); } - /** Terminate all workers and answer any outstanding calls gracefully. */ + private startSettled(w: PoolWorker): void { + const start = this.starts.get(w); + if (!start) return; + this.starts.delete(w); + start.settle(); + } + + /** + * Terminate a worker, but not while it is still starting up: wait for it to + * post 'ready' (or go away), up to `startSettleMs` (WORKER_START_SETTLE_MS — + * see that constant for the crash this avoids). + */ + private async terminateStarted(w: PoolWorker): Promise { + const start = this.starts.get(w); + if (start) { + let cap: NodeJS.Timeout | undefined; + await Promise.race([ + start.settled, + new Promise((resolve) => { cap = setTimeout(resolve, this.startSettleMs); cap.unref?.(); }), + ]); + clearTimeout(cap); + } + try { await w.terminate(); } catch { /* already gone */ } + } + + /** + * Terminate all workers and answer any outstanding calls gracefully. A worker + * still starting up is terminated once it has started (see + * WORKER_START_SETTLE_MS); outstanding calls are answered first. + */ async destroy(): Promise { if (this.destroyed) return; this.destroyed = true; @@ -324,6 +378,6 @@ export class QueryPool { } this.inflight.clear(); this.queue = []; - await Promise.all(ws.map((w) => Promise.resolve(w.terminate()).catch(() => { /* already gone */ }))); + await Promise.all(ws.map((w) => this.terminateStarted(w))); } } From 70fe0ecc52751c5a3f8a0912388411bd61e5de13 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 21:58:28 +0000 Subject: [PATCH 163/259] fix(windows): helper threads never end mid-start or with GC marking in flight (#2309) On Windows a worker thread that ends - terminate(), its own process.exit(), or the process exiting - while V8's concurrent marker is marking its heap crashes the whole process (exit 3221225477 / 0xC0000005, no error, no dump). Bisected on the Windows 11 ARM64 VM: --no-concurrent-marking alone stops it (18/480 -> 0/480; no other concurrency flag does), but it costs about a third of indexing speed, so CodeGraph avoids the dangerous moments instead: - src/worker-teardown.ts: owners terminate a worker only once it has started (its first message, posted after its modules load - mid-load is by far the worst moment), and a worker about to exit runs a full collection first, which finishes any marking in flight (gc reached at runtime, never global). - Resolver pool / store writer: the worker collects before exiting on 'close' (the normal path - main 21/480 crashed on the VM, now 2/1200), and the 5 s close fallback never terminates a worker still starting. - codegraph_status change counts: the worker posts 'loaded' after its heavy require and collects before answering; a deadline answers unknown at once but terminates only once loaded (3/720 -> 0/1440), and MCPEngine.stop() waits for measurements in flight. Co-authored-by: Claude Opus 5.5 --- AGENTS.md | 1 + CHANGELOG.md | 1 + __tests__/mcp-freshness-bounds.test.ts | 39 +++++++- __tests__/mcp-status-freshness.test.ts | 29 +++++- __tests__/worker-teardown.test.ts | 131 +++++++++++++++++++++++++ src/extraction/store-worker.ts | 3 + src/extraction/store-writer.ts | 16 ++- src/mcp/engine.ts | 10 +- src/mcp/index-freshness-worker.ts | 8 +- src/mcp/index-freshness.ts | 38 +++++-- src/resolution/resolver-pool.ts | 21 +++- src/resolution/resolver-worker.ts | 3 + src/worker-teardown.ts | 100 +++++++++++++++++++ 13 files changed, 376 insertions(+), 24 deletions(-) create mode 100644 __tests__/worker-teardown.test.ts create mode 100644 src/worker-teardown.ts diff --git a/AGENTS.md b/AGENTS.md index 0f5e9a882e..fd5daee8ef 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -214,6 +214,7 @@ For any Windows-specific PR, bug, or implementation, validate it on the real Win - Guest toolchain (winget): Node LTS, Git, and the **VC++ ARM64 redistributable** (required by `@rollup/rollup-win32-arm64-msvc`, which vitest pulls in). - Fetch a contributor PR head straight from their fork to dodge `pull//head` lag: `git fetch ` then `git checkout -f FETCH_HEAD`. - Windows baseline: as of #2053 the full suite passes on the Windows 11 (ARM64) VM. The only expected exception is `security.test.ts > Session marker symlink resistance > does not follow a pre-planted symlink`, which needs symlink privileges (Developer Mode) — confirm any other failure against `origin/main` before blaming your PR. The former `mcp-initialize.test.ts` / `mcp-roots.test.ts` `EPERM` teardown failures came from tests spawning `serve --mcp` without the runtime flags (its `--liftoff-only` re-exec grandchild kept the cwd / SQLite file open); spawn it with `WASM_RUNTIME_FLAGS` and await the child's exit before removing the temp dir. Windows checkouts may be CRLF — split source lines on `/\r?\n/` in tests. +- Windows worker crashes: a worker thread that ends (`terminate()`, its own `process.exit()`, or the process exiting) while V8's concurrent marker is marking its heap kills the process with exit 3221225477 (0xC0000005) and leaves no error or dump. It's worst while the worker is still loading its modules. Owners must not terminate a worker before its first message (`workerStarted` / `terminateOnceStarted`, or the pool's own handshake), and a worker that exits itself calls `collectBeforeExit()` first — both in `src/worker-teardown.ts`. `--no-concurrent-marking` also stops it, but measured about a third slower indexing on the Windows VM, so it is not used. ## Releases diff --git a/CHANGELOG.md b/CHANGELOG.md index 86258f27b6..9b76be6319 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -177,6 +177,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - On Windows, `codegraph init`, `codegraph index` and the background server's syncs no longer sometimes crash right after parsing on a busy machine (exit code 3221225477, an access violation). Shutting down the parsing threads could stop one while it was still starting up, which took the whole process down; they now finish starting first. - On a busy machine, indexing or syncing a small project no longer starts extra background threads for resolving references that it can't use. Under heavy load the slower pace made a ten-file project look big enough to need them, which added seconds to the run and held extra memory. - On Windows, the CodeGraph background server no longer sometimes crashes as it shuts down (exit code 3221225477, an access violation) when it stops shortly after starting. Shutting down could stop one of its background threads while that thread was still starting up; it now lets the thread finish starting first. +- On Windows, indexing, syncing and the CodeGraph background server no longer sometimes crash (exit code 3221225477, an access violation) when they stop one of their helper threads: the ones that resolve references, write the index, and count changes for `codegraph_status`. A thread is no longer stopped while it is still starting up, and each one finishes its memory cleanup before it exits. ## [1.6.1] - 2026-09-29 diff --git a/__tests__/mcp-freshness-bounds.test.ts b/__tests__/mcp-freshness-bounds.test.ts index 5710b4e137..a9dae57aff 100644 --- a/__tests__/mcp-freshness-bounds.test.ts +++ b/__tests__/mcp-freshness-bounds.test.ts @@ -5,7 +5,7 @@ import * as path from 'path'; import { createHash } from 'crypto'; import { Readable } from 'stream'; import { validateAnswerFiles } from '../src/mcp/answer-freshness'; -import { measurePendingChanges } from '../src/mcp/index-freshness'; +import { endFreshnessMeasurements, measurePendingChanges } from '../src/mcp/index-freshness'; vi.mock('fs', async importOriginal => { const actual = await importOriginal(); @@ -46,16 +46,47 @@ describe('bounded freshness validation (#1959)', () => { const pending = measurePendingChanges('/freshness-timeout'); expect(measurePendingChanges('/freshness-timeout')).toBe(pending); expect(workers).toHaveLength(1); + workers[0].emit('message', { type: 'loaded' }); // stuck past loading, in the scan await vi.advanceTimersByTimeAsync(8000); expect(await pending).toBeNull(); expect(workers[0].terminate).toHaveBeenCalledOnce(); }); + it('answers unknown on deadline but never terminates a worker still loading its modules', async () => { + // Terminating a worker while it loads its modules can crash the process + // on Windows (0xC0000005) — see worker-start.ts. + vi.useFakeTimers(); + const pending = measurePendingChanges('/freshness-still-loading'); + await vi.advanceTimersByTimeAsync(8000); + expect(await pending).toBeNull(); + await vi.advanceTimersByTimeAsync(1000); + expect(workers[0].terminate).not.toHaveBeenCalled(); + workers[0].emit('message', { type: 'loaded' }); + await vi.advanceTimersByTimeAsync(0); + expect(workers[0].terminate).toHaveBeenCalledOnce(); + }); + + it('a server shutting down ends measurements once they have loaded', async () => { + vi.useFakeTimers(); + const pending = measurePendingChanges('/freshness-shutdown'); + let ended = false; + const ending = endFreshnessMeasurements().then(() => { ended = true; }); + await vi.advanceTimersByTimeAsync(1000); + expect(ended).toBe(false); + expect(workers[0].terminate).not.toHaveBeenCalled(); + workers[0].emit('message', { type: 'loaded' }); + await ending; + expect(workers[0].terminate).toHaveBeenCalled(); + workers[0].emit('exit', 1); // what a terminated worker does + expect(await pending).toBeNull(); + }); + it('returns unknown on deadline even if worker termination is delayed', async () => { vi.useFakeTimers(); const pending = measurePendingChanges('/freshness-delayed-exit'); let release!: () => void; workers[0].terminate.mockImplementation(() => new Promise(resolve => { release = resolve; })); + workers[0].emit('message', { type: 'loaded' }); await vi.advanceTimersByTimeAsync(8000); expect(await pending).toBeNull(); release(); @@ -81,11 +112,13 @@ describe('bounded freshness validation (#1959)', () => { expect(await measurePendingChanges('/freshness-three')).toBeNull(); expect(workers).toHaveLength(2); workers[0].emit('error', new Error('worker failed')); - workers[1].emit('message', { added: -1, modified: 0, removed: 0 }); + workers[1].emit('message', { type: 'counts', counts: { added: -1, modified: 0, removed: 0 } }); expect(await first).toBeNull(); expect(await second).toBeNull(); + await new Promise((r) => setTimeout(r, 0)); // the slots free once each worker is terminated const retry = measurePendingChanges('/freshness-three'); - workers[2].emit('message', { added: 0, modified: 1, removed: 0 }); + workers[2].emit('message', { type: 'loaded' }); + workers[2].emit('message', { type: 'counts', counts: { added: 0, modified: 1, removed: 0 } }); expect(await retry).toEqual({ added: 0, modified: 1, removed: 0 }); }); }); diff --git a/__tests__/mcp-status-freshness.test.ts b/__tests__/mcp-status-freshness.test.ts index 923f77ae3f..c78844db53 100644 --- a/__tests__/mcp-status-freshness.test.ts +++ b/__tests__/mcp-status-freshness.test.ts @@ -1,8 +1,9 @@ -import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import * as fs from 'fs'; import * as os from 'os'; import * as path from 'path'; import { execFileSync } from 'child_process'; +import { Worker } from 'worker_threads'; import CodeGraph from '../src/index'; import { measurePendingChanges } from '../src/mcp/index-freshness'; import { ToolHandler } from '../src/mcp/tools'; @@ -52,6 +53,32 @@ describe('MCP status freshness (#1959)', () => { expect(await measurePendingChanges(path.join(root, 'missing'))).toBeNull(); }); + it('past its deadline, never terminates the worker before it has loaded its modules', async () => { + // Terminating a worker while it loads its modules can crash the whole + // process on Windows (0xC0000005). A 1 ms deadline passes while the real + // worker is still loading: the answer is unknown at once, and the worker + // is terminated only after it has posted 'loaded'. + const posted = new WeakSet(); + const loadedWhenEnded: boolean[] = []; + const emit = Worker.prototype.emit; + vi.spyOn(Worker.prototype, 'emit').mockImplementation(function (this: Worker, event: string | symbol, ...args: unknown[]) { + if (event === 'message') posted.add(this); + return emit.call(this, event, ...args); + }); + const terminate = Worker.prototype.terminate; + vi.spyOn(Worker.prototype, 'terminate').mockImplementation(function (this: Worker) { + loadedWhenEnded.push(posted.has(this)); + return terminate.call(this); + }); + try { + expect(await measurePendingChanges(root, 1)).toBeNull(); + await vi.waitFor(() => expect(loadedWhenEnded).toHaveLength(1), { timeout: 20_000 }); + expect(loadedWhenEnded).toEqual([true]); + } finally { + vi.restoreAllMocks(); + } + }, 30_000); + it('counts edits committed after the index even when the working tree is clean', async () => { fs.writeFileSync(path.join(root, 'modify.ts'), 'export const modify = 99;\n'); execFileSync('git', ['add', 'modify.ts'], { cwd: root, stdio: 'pipe' }); diff --git a/__tests__/worker-teardown.test.ts b/__tests__/worker-teardown.test.ts new file mode 100644 index 0000000000..22d63bc830 --- /dev/null +++ b/__tests__/worker-teardown.test.ts @@ -0,0 +1,131 @@ +/** + * On Windows, a worker thread that ends while V8's concurrent marker is + * marking its heap can crash the whole process (0xC0000005, no error, no + * dump) — most of all while it is still loading its modules. Owners wait for a + * worker's first message before terminating it, and workers collect garbage + * before exiting (worker-teardown.ts). These pin both halves, then end real + * resolver and store workers right after starting them and record whether + * each had started when it was terminated. + */ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { EventEmitter } from 'events'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { Worker } from 'worker_threads'; +import CodeGraph from '../src/index'; +import { terminateOnceStarted, workerStarted } from '../src/worker-teardown'; + +const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms)); + +class FakeWorker extends EventEmitter { + terminate = vi.fn(async () => 0); +} + +describe('terminating a worker once it has started', () => { + it('waits for the first message', async () => { + const w = new FakeWorker(); + const started = workerStarted(w as unknown as Worker); + const ending = terminateOnceStarted(w, started); + await sleep(30); + expect(w.terminate).not.toHaveBeenCalled(); + w.emit('message', { type: 'ready' }); + await ending; + expect(w.terminate).toHaveBeenCalledOnce(); + }); + + it('counts an error or an exit as started', async () => { + for (const event of ['error', 'exit'] as const) { + const w = new FakeWorker(); + w.on('error', () => {}); + const started = workerStarted(w as unknown as Worker); + w.emit(event, event === 'error' ? new Error('boom') : 1); + await terminateOnceStarted(w, started); + expect(w.terminate).toHaveBeenCalledOnce(); + } + }); + + it('stops waiting after the cap, and never rejects', async () => { + const w = new FakeWorker(); + w.terminate.mockRejectedValue(new Error('already gone')); + const t0 = Date.now(); + await terminateOnceStarted(w, new Promise(() => {}), 60); + expect(w.terminate).toHaveBeenCalledOnce(); + expect(Date.now() - t0).toBeGreaterThanOrEqual(55); + }); +}); + +describe('collecting garbage before a worker exits', () => { + it('runs a full collection in a real worker, and leaves no global gc behind', async () => { + const teardown = path.resolve(__dirname, '../dist/worker-teardown.js'); + const run = (code: string) => new Promise((resolve, reject) => { + const w = new Worker(code, { eval: true }); + w.once('message', resolve); + w.once('error', reject); + }); + const ran = await run(` + const { collectBeforeExit } = require(${JSON.stringify(teardown)}); + let junk = Array.from({ length: 200000 }, (_, i) => ({ i, s: 'x'.repeat(32) })); + junk = null; + const before = process.memoryUsage().heapUsed; + const collected = collectBeforeExit(); + const after = process.memoryUsage().heapUsed; + require('worker_threads').parentPort.postMessage({ collected, freed: before - after, gc: typeof globalThis.gc }); + `) as { collected: boolean; freed: number; gc: string }; + expect(ran.collected).toBe(true); + expect(ran.freed).toBeGreaterThan(0); + expect(ran.gc).toBe('undefined'); + // A worker created afterwards doesn't get a global gc either. + expect(await run(`require('worker_threads').parentPort.postMessage(typeof globalThis.gc)`)).toBe('undefined'); + }, 30_000); +}); + +describe('real workers are never terminated before they have started', () => { + let root: string; + let dbPath: string; + const seen = new WeakSet(); + const startedWhenEnded: boolean[] = []; + + beforeEach(async () => { + root = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-worker-start-'))); + fs.writeFileSync(path.join(root, 'app.ts'), 'export function alpha() { return beta(); }\nexport function beta() { return 1; }\n'); + const cg = await CodeGraph.init(root); + try { await cg.indexAll(); } finally { cg.close(); } + dbPath = path.join(root, '.codegraph', 'codegraph.db'); + startedWhenEnded.length = 0; + // A worker re-emits each message it posts; note which have, and what each + // had done by the time it was terminated. + const emit = Worker.prototype.emit; + vi.spyOn(Worker.prototype, 'emit').mockImplementation(function (this: Worker, event: string | symbol, ...args: unknown[]) { + if (event === 'message') seen.add(this); + return emit.call(this, event, ...args); + }); + const terminate = Worker.prototype.terminate; + vi.spyOn(Worker.prototype, 'terminate').mockImplementation(function (this: Worker) { + startedWhenEnded.push(seen.has(this)); + return terminate.call(this); + }); + }); + + afterEach(() => { + vi.restoreAllMocks(); + vi.unstubAllEnvs(); + fs.rmSync(root, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); + }); + + it('the resolver pool, torn down while its workers boot', async () => { + vi.stubEnv('CODEGRAPH_RESOLVE_WORKERS', '2'); + const { ResolverPool } = require('../dist/resolution/resolver-pool') as typeof import('../src/resolution/resolver-pool'); + const pool = ResolverPool.tryCreate(dbPath, root); + expect(pool).not.toBeNull(); + await pool!.destroy(0); // the close fallback fires at once, mid-boot + expect(startedWhenEnded).toEqual([true, true]); + }, 30_000); + + it('the store writer, closed while its worker boots', async () => { + const { StoreWriter } = require('../dist/extraction/store-writer') as typeof import('../src/extraction/store-writer'); + const writer = new StoreWriter(path.resolve(__dirname, '../dist/extraction/store-worker.js'), dbPath, false); + await writer.close(0); + expect(startedWhenEnded.every(Boolean)).toBe(true); + }, 30_000); +}); diff --git a/src/extraction/store-worker.ts b/src/extraction/store-worker.ts index a562f557af..65b1f20972 100644 --- a/src/extraction/store-worker.ts +++ b/src/extraction/store-worker.ts @@ -32,6 +32,7 @@ import { QueryBuilder } from '../db/queries'; import { createDatabase, SqliteDatabase } from '../db/sqlite-adapter'; import { finalizeStoreBundle, type KernelStoreBundle, type StoreBundle } from './store-writer'; import { decodeExtractBuffers } from './kernel/decode'; +import { collectBeforeExit } from '../worker-teardown'; if (!parentPort) { throw new Error('store-worker must be run as a worker thread'); @@ -135,6 +136,8 @@ port.on('message', (msg: InMessage) => { } catch { /* already closed */ } + // No GC marking in flight when the thread ends (worker-teardown.ts). + collectBeforeExit(); process.exit(0); break; } diff --git a/src/extraction/store-writer.ts b/src/extraction/store-writer.ts index 683f09117e..4accc3d3c4 100644 --- a/src/extraction/store-writer.ts +++ b/src/extraction/store-writer.ts @@ -9,6 +9,7 @@ import { Worker } from 'worker_threads'; import { ExtractionResult, Language, Node, Edge, UnresolvedReference, FileRecord } from '../types'; +import { terminateOnceStarted, workerStarted } from '../worker-teardown'; /** One file's complete store payload (pre-filtered — see storeFileBundle). */ export interface StoreBundle { @@ -66,6 +67,8 @@ export function finalizeStoreBundle( export class StoreWriter { private worker: Worker; + /** Settles on the worker's first message or its end — see worker-teardown.ts. */ + private started: Promise; private readyPromise: Promise; private firstError: Error | null = null; private drainWaiters = new Map void; reject: (e: Error) => void }>(); @@ -77,6 +80,7 @@ export class StoreWriter { constructor(workerScriptPath: string, dbPath: string, fastInit: boolean) { this.worker = new Worker(workerScriptPath); + this.started = workerStarted(this.worker); let readyResolve!: () => void; let readyReject!: (e: Error) => void; this.readyPromise = new Promise((resolve, reject) => { @@ -175,14 +179,18 @@ export class StoreWriter { return p; } - /** Close the worker's DB connection and join the thread. */ - async close(): Promise { + /** + * Close the worker's DB connection and join the thread; the worker collects + * garbage and exits by itself (worker-teardown.ts). One that hasn't by + * `timeoutMs` is terminated — but never while it is still starting up. + */ + async close(timeoutMs = 5000): Promise { if (this.exited) return; this.worker.postMessage({ type: 'close' }); await new Promise((resolve) => { const t = setTimeout(() => { - void this.worker.terminate().then(() => resolve()); - }, 5000); + void terminateOnceStarted(this.worker, this.started).then(() => resolve()); + }, timeoutMs); this.worker.once('exit', () => { clearTimeout(t); resolve(); diff --git a/src/mcp/engine.ts b/src/mcp/engine.ts index 62a4c8041c..cc0b43ba1d 100644 --- a/src/mcp/engine.ts +++ b/src/mcp/engine.ts @@ -18,6 +18,7 @@ import { ToolHandler } from './tools'; import { WslSharedIndexError } from '../db/wsl-shared-index'; import { assertNoRebuild, releaseWriterLock, tryAcquireWriterLock, writerLockHeldMessage } from './writer-lock'; import { QueryPool, resolvePoolSize } from './query-pool'; +import { endFreshnessMeasurements } from './index-freshness'; import { acquireProject, ProjectLease } from './project-lifecycle'; // Lazy-load the heavy CodeGraph chain (sqlite + query/graph/context layers) OFF @@ -271,13 +272,16 @@ export class MCPEngine { // Detach + terminate the worker pool first so no tool call routes to a // worker mid-teardown; outstanding pool calls resolve with graceful guidance. - // Stopping waits for the workers to end: the daemon exits right after, and - // exiting while a worker is still starting up can crash the process. + // Stopping waits for the workers to end — the pool's, and any + // `codegraph_status` change count still measuring: the daemon exits right + // after, and exiting while a worker is still starting up can crash the + // process. this.toolHandler.setQueryPool(null); const poolDown = this.queryPool ? this.queryPool.destroy() : Promise.resolve(); this.queryPool = null; + const measurementsDown = endFreshnessMeasurements(); const drained = this.toolHandler.closeAll(); - this.stopPromise = Promise.all([drained, poolDown]).then(async () => { + this.stopPromise = Promise.all([drained, poolDown, measurementsDown]).then(async () => { if (this.initPromise) await this.initPromise; if (this.defaultLease) { await this.defaultLease.release(); diff --git a/src/mcp/index-freshness-worker.ts b/src/mcp/index-freshness-worker.ts index 06d9787372..b678f1d20a 100644 --- a/src/mcp/index-freshness-worker.ts +++ b/src/mcp/index-freshness-worker.ts @@ -1,5 +1,6 @@ /** Exact CLI-parity change count, isolated from the MCP transport event loop. */ import { parentPort, workerData } from 'worker_threads'; +import { collectBeforeExit } from '../worker-teardown'; if (parentPort) { const port = parentPort; @@ -7,6 +8,9 @@ if (parentPort) { let counts: { added: number; modified: number; removed: number } | null = null; try { const CodeGraph = (require('../index') as typeof import('../index')).default; + // Loaded: from here on the owner may terminate this worker. Ending it while + // it loads those modules can crash the process on Windows (worker-teardown.ts). + port.postMessage({ type: 'loaded' }); cg = CodeGraph.openSync((workerData as { root: string }).root); const changes = cg.getChangedFiles(); counts = { @@ -19,5 +23,7 @@ if (parentPort) { } finally { try { cg?.close(); } catch { /* the worker is exiting */ } } - port.postMessage(counts); + // The owner terminates this worker on the answer: no GC marking in flight then. + collectBeforeExit(); + port.postMessage({ type: 'counts', counts }); } diff --git a/src/mcp/index-freshness.ts b/src/mcp/index-freshness.ts index ed6c66b02f..504216f425 100644 --- a/src/mcp/index-freshness.ts +++ b/src/mcp/index-freshness.ts @@ -1,6 +1,7 @@ import { existsSync } from 'fs'; import * as path from 'path'; import { Worker } from 'worker_threads'; +import { terminateOnceStarted, workerStarted } from '../worker-teardown'; export interface PendingChangeCounts { added: number; @@ -12,20 +13,31 @@ export interface PendingChangeCounts { const MEASURE_TIMEOUT_MS = 8_000; let liveMeasurements = 0; const active = new Map>(); +/** Every measurement worker not yet terminated, with its load — for {@link endFreshnessMeasurements}. */ +const live = new Map>(); -export function measurePendingChanges(root: string): Promise { +export function measurePendingChanges(root: string, timeoutMs = MEASURE_TIMEOUT_MS): Promise { const key = path.resolve(root); const existing = active.get(key); if (existing) return existing; // Status probes for many projects must not exhaust the shared daemon. if (liveMeasurements >= 2) return Promise.resolve(null); - const pending = runMeasurement(key).finally(() => active.delete(key)); + const pending = runMeasurement(key, timeoutMs).finally(() => active.delete(key)); active.set(key, pending); return pending; } -function runMeasurement(root: string): Promise { +/** + * Terminate every measurement still running, each once it has loaded its + * modules (worker-teardown.ts) — for a server that is about to exit, since + * exiting tears the workers down the same way terminating them does. + */ +export async function endFreshnessMeasurements(): Promise { + await Promise.all([...live].map(([worker, loaded]) => terminateOnceStarted(worker, loaded))); +} + +function runMeasurement(root: string, timeoutMs: number): Promise { // The compiled sibling is beside us in production; Vitest loads src/ but // builds dist/ before the tests, so use that copy for the worker there. const sibling = path.join(__dirname, 'index-freshness-worker.js'); @@ -41,6 +53,10 @@ function runMeasurement(root: string): Promise { return Promise.resolve(null); } + // Its first message is 'loaded', posted once its modules are in. Terminating + // it before then can crash the whole process on Windows (worker-teardown.ts). + const loaded = workerStarted(worker); + live.set(worker, loaded); liveMeasurements++; return new Promise(resolve => { let settled = false; @@ -48,13 +64,19 @@ function runMeasurement(root: string): Promise { if (settled) return; settled = true; clearTimeout(timer); - // A worker inside a synchronous Git call may take time to terminate. - // Return unknown on deadline, but keep its concurrency slot until exit. - void worker.terminate().catch(() => {}).finally(() => { liveMeasurements--; }); + // A worker inside a synchronous Git call may take time to terminate, and + // one still loading its modules must not be terminated yet. Return + // unknown on deadline, but keep its concurrency slot until it is gone. + void terminateOnceStarted(worker, loaded).finally(() => { + live.delete(worker); + liveMeasurements--; + }); resolve(counts); }; - const timer = setTimeout(() => finish(null), MEASURE_TIMEOUT_MS); - worker.once('message', (value: PendingChangeCounts | null) => { + const timer = setTimeout(() => finish(null), timeoutMs); + worker.on('message', (msg: { type?: string; counts?: PendingChangeCounts | null }) => { + if (msg?.type !== 'counts') return; + const value = msg.counts; const valid = value && typeof value === 'object' && ['added', 'modified', 'removed'].every( key => Number.isSafeInteger(value[key as keyof PendingChangeCounts]) && value[key as keyof PendingChangeCounts] >= 0, ); diff --git a/src/resolution/resolver-pool.ts b/src/resolution/resolver-pool.ts index 96f4340d8d..457e1d0c0f 100644 --- a/src/resolution/resolver-pool.ts +++ b/src/resolution/resolver-pool.ts @@ -16,6 +16,7 @@ import * as os from 'os'; import type { Edge, UnresolvedReference } from '../types'; import type { ResolvedRef, UnresolvedRef } from './types'; import { memoryBudgetBytes } from './memory-budget'; +import { terminateOnceStarted, workerStarted } from '../worker-teardown'; /** One synthesis pass's output: its edge list + worker-measured wall clock. */ export interface SynthPassResult { @@ -34,9 +35,14 @@ export interface ChunkResult { interface PoolWorker { worker: Worker; ready: Promise; + /** Settles on the worker's first message or its end — see worker-teardown.ts. */ + started: Promise; busy: number; } +/** How long destroy() waits for a worker to exit on 'close' before terminating it. */ +const CLOSE_TIMEOUT_MS = 5000; + const MIN_PARALLEL_BATCH = 1000; const CHUNK_SIZE = 500; @@ -166,13 +172,14 @@ export class ResolverPool { private constructor(workerScript: string, dbPath: string, projectRoot: string, size: number) { for (let i = 0; i < size; i++) { const worker = new Worker(workerScript); + const started = workerStarted(worker); let readyResolve!: () => void; let readyReject!: (e: Error) => void; const ready = new Promise((resolve, reject) => { readyResolve = resolve; readyReject = reject; }); - const pw: PoolWorker = { worker, ready, busy: 0 }; + const pw: PoolWorker = { worker, ready, started, busy: 0 }; worker.on('message', (msg: { type: string; id?: number; message?: string; edges?: Edge[]; ms?: number } & Partial) => { if (msg.type === 'ready') { readyResolve(); @@ -341,14 +348,20 @@ export class ResolverPool { ); } - async destroy(): Promise { + /** + * Ask every worker to close; each collects garbage and exits by itself (see + * worker-teardown.ts). One that hasn't by `closeTimeoutMs` is terminated — + * but never while it is still starting up: a pool torn down soon after it + * booted, on a busy machine, can have a worker still loading its modules. + */ + async destroy(closeTimeoutMs = CLOSE_TIMEOUT_MS): Promise { await Promise.all( this.workers.map( (pw) => new Promise((resolve) => { const t = setTimeout(() => { - void pw.worker.terminate().then(() => resolve()); - }, 5000); + void terminateOnceStarted(pw.worker, pw.started).then(() => resolve()); + }, closeTimeoutMs); pw.worker.once('exit', () => { clearTimeout(t); resolve(); diff --git a/src/resolution/resolver-worker.ts b/src/resolution/resolver-worker.ts index f1d3020128..6cc3ec71f7 100644 --- a/src/resolution/resolver-worker.ts +++ b/src/resolution/resolver-worker.ts @@ -26,6 +26,7 @@ import { QueryBuilder } from '../db/queries'; import { ReferenceResolver } from './index'; import { SYNTH_PASSES } from './callback-synthesizer'; import { createYielder } from './cooperative-yield'; +import { collectBeforeExit } from '../worker-teardown'; import type { UnresolvedReference } from '../types'; if (!parentPort) { @@ -126,6 +127,8 @@ port.on('message', (msg: InMessage) => { } catch { /* already closed */ } + // No GC marking in flight when the thread ends (worker-teardown.ts). + collectBeforeExit(); process.exit(0); break; } diff --git a/src/worker-teardown.ts b/src/worker-teardown.ts new file mode 100644 index 0000000000..fa5d08e34a --- /dev/null +++ b/src/worker-teardown.ts @@ -0,0 +1,100 @@ +/** + * Ending a worker thread without taking the process down. + * + * On Windows, a worker thread that ends — terminated, by its own + * `process.exit()`, or with the process — while V8's concurrent marker is + * still marking its heap can crash the whole process with an access violation + * (exit 3221225477 / 0xC0000005), with no error and no dump. Bisected on a + * Windows 11 ARM64 VM (Node 24): `--no-concurrent-marking` alone stops it, and + * no other concurrency flag does — but that flag makes indexing about a third + * slower, so CodeGraph avoids the dangerous moments instead. + * + * Measured there, 480 child processes per row, four workers each: + * + * ended while still loading CodeGraph's modules 18 crashed + * loaded and allocating, terminated by the owner 4 crashed + * loaded and allocating, exiting by itself 1 crashed + * loaded and allocating, full collection, then exit 0 crashed + * + * So an owner never ends a worker that is still starting (it waits for the + * worker's first message, which is posted only after its modules load), and a + * worker that is about to exit collects garbage first — a full collection + * finishes any marking in flight. + */ + +import type { Worker } from 'worker_threads'; + +/** + * Longest an owner waits for a worker to finish starting before terminating + * it anyway. A start takes about a second normally and a few under heavy + * load; the cap only bounds a start that is wedged. + */ +export const WORKER_START_SETTLE_MS = 15_000; + +/** + * Settles once `worker` has started — its first message — or has gone away. + * Call it right after creating the worker, before it can post anything. + */ +export function workerStarted(worker: Worker): Promise { + return new Promise((resolve) => { + worker.once('message', () => resolve()); + worker.once('error', () => resolve()); + worker.once('exit', () => resolve()); + }); +} + +/** + * Terminate `worker`, but not while it is still starting: wait for `started` + * (from {@link workerStarted}) first, up to `capMs`. Never rejects. + */ +export async function terminateOnceStarted( + worker: Pick, + started: Promise, + capMs = WORKER_START_SETTLE_MS +): Promise { + let cap: NodeJS.Timeout | undefined; + await Promise.race([ + started, + new Promise((resolve) => { + cap = setTimeout(resolve, capMs); + cap.unref?.(); + }), + ]); + clearTimeout(cap); + try { + await worker.terminate(); + } catch { + // already gone + } +} + +let collect: (() => void) | null | undefined; + +/** + * In a worker that is about to exit: run a full garbage collection, so no + * marking is in flight when the thread ends. Returns whether one ran. The + * collector is reached at runtime (no `--expose-gc` on the command line) and + * is never exposed as a global. + */ +export function collectBeforeExit(): boolean { + if (collect === undefined) { + try { + // eslint-disable-next-line @typescript-eslint/no-require-imports + (require('v8') as typeof import('v8')).setFlagsFromString('--expose-gc'); + // eslint-disable-next-line @typescript-eslint/no-require-imports + collect = (require('vm') as typeof import('vm')).runInNewContext('gc') as () => void; + // Only the context made above needed it; workers created later don't get a global gc. + // eslint-disable-next-line @typescript-eslint/no-require-imports + (require('v8') as typeof import('v8')).setFlagsFromString('--no-expose-gc'); + } catch { + collect = null; + } + } + if (!collect) return false; + try { + collect(); + return true; + } catch { + return false; + } +} From 2965b7a349706fd650013bc0d738741b3539c228 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Sat, 3 Oct 2026 04:46:58 +0000 Subject: [PATCH 164/259] docs(changelog): ready [Unreleased] for 1.6.2; release verify waits up to 5 min (#2314) - Adds the Highlights block and groups the 164 entries the way 1.6.1's notes are: New Features for the newly supported routes and newly indexed code, Fixes under sub-headings (Windows; the background server and CLI; indexing; across languages; then one per language family) so a skimmer finds their area. Duplicates left by merges are folded together (the Windows console window entry twice; two overlapping Vapor closure-route entries; three Windows crash entries into one). No entry's wording is otherwise changed. - The release workflow's "Verify every package is actually on the registry" gave each package 60 s. 1.6.1 published all seven packages fine, but the first took longer than that to appear and the run went red. It now waits up to 5 minutes per package. No viewer entries: those stay in docs/viewer-launch-changelog.md, and `codegraph ui` stays behind CODEGRAPH_UI=1 (checked on a fresh build: absent from --help, refused with the "not in this release yet" message). Co-authored-by: Claude Opus 5.5 --- .github/workflows/release.yml | 4 +- CHANGELOG.md | 300 +++++++++++++++++++--------------- 2 files changed, 174 insertions(+), 130 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index d48973f50f..38b097e17a 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -275,10 +275,12 @@ jobs: V="${{ steps.ver.outputs.version }}" # npm publish can print success without persisting; confirm against the # registry (with retries for propagation) so green means really shipped. + # Up to 5 minutes per package: 1.6.1's packages all published fine but + # took longer than the old 60 s to appear, which failed the run. for dir in release/npm/codegraph-* release/npm/main; do name=$(node -p "require('./$dir/package.json').name") ok= - for i in 1 2 3 4 5 6; do + for i in $(seq 1 30); do if npm view "$name@$V" version >/dev/null 2>&1; then ok=1; break; fi echo "waiting for $name@$V to appear ($i)…"; sleep 10 done diff --git a/CHANGELOG.md b/CHANGELOG.md index 9b76be6319..a333b1e2d1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,172 +12,214 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] +### Highlights + +- **Far fewer wrong links, in every language.** A call, import or type now resolves the way its language scopes names — through imports, packages, namespaces and the class it is written in — so it stops landing on an unrelated symbol that only shares its name. This covers TypeScript and JavaScript, Python, Java, Kotlin, Scala, C#, VB.NET, Swift, Objective-C, Go, Rust, C and C++, PHP, Ruby, Dart, Lua, R and more, and makes callers, impact and `codegraph_explore` answers more trustworthy. +- **Overloads, `super` and recursion resolve correctly.** A call reaches the overload its arguments fit, a `super` call reaches the parent's method, and a method that calls another overload, or another object's method of the same name, no longer looks like it calls itself. +- **Routes for more frameworks.** Angular (including Nx libraries), Vue Router admin apps, Rails nested routes, Flask's `add_url_rule`, ASP.NET FastEndpoints, Play subprojects, Vapor closure routes and Expo Router API routes are found. NestJS and Laravel routes are named by the path a request actually takes, and a Spring or Laravel test that calls a route by URL counts as covering it. +- **More of your code in the graph.** Vue Options API components, Java and Kotlin enum constants with bodies, Flow-typed JavaScript, Kotlin infix calls, React Native native views and events, Expo modules, tag-based CFML and Delphi `.inc` files are all indexed now. +- **Windows: no flashing windows, no random crashes.** The background server no longer pops up console windows while it works, sporadic access-violation crashes (exit code 3221225477) while indexing or shutting down are fixed, and `codegraph upgrade` works while an agent session is using CodeGraph. +- **A steadier background server.** Newer MCP clients such as Antigravity connect again, Claude Code sees the full text of `codegraph_status`, a session that lost the shared server reconnects by itself, and a project queried through `projectPath` is released after 10 idle minutes. +- **Indexing that finishes.** Large C and C++ projects no longer run out of memory while resolving references, a COBOL file can no longer stall indexing, and `codegraph sync` keeps every caller of a name that is defined more than once. +- **Upgrading:** re-index your projects after this release — most of these fixes are written while indexing. + +### New Features + +- In Nuxt and Astro projects, a file-routed page is now linked to the component its file defines. An Astro endpoint is linked to the HTTP-verb handlers it exports, like `export const GET`. Before, a page route like Nuxt's `pages/admin.vue` or Astro's `src/pages/about.astro` linked to nothing, so callers, impact and flows stopped at the route. +- React Router links and `navigate(…)` calls written through a route-config object, like bulletproof-react's `paths.app.discussion.getHref(id)`, now reach the route they open. A `` with an arrow-function attribute before its `to`, such as `onMouseEnter={() => …}`, is read too. +- React Router data-router routes are named by their full path: nested `children` paths are joined to their parent's, `path: paths.app.root.path` constants are read, and `lazy: () => import('./routes/x')` routes link to the module's component. A guard wrapper like `` no longer stands in for the page. On bulletproof-react, every route now has its real path and page. +- Vapor routes whose handler is a trailing closure, like `app.get("hello") { req in … }`, WebSocket routes like `app.webSocket("chat") { req, ws in … }`, and `routes.on(.POST, "x", use: handler)` registrations are now found, and a closure route links to what the closure calls, the way an Express inline handler does. Before, only routes with a `use:` handler were read, and a closure route linked to nothing, so callers, impact and flows stopped at it. +- Flow-typed JavaScript files (those with `@flow` in their header comment) are now parsed like TypeScript, so type annotations like `render(): React.Node` no longer cut classes short. This mostly affects React Native code: on React Native's own libraries the graph gained over 8,000 links, and component classes keep their methods. Files without the pragma are unchanged. Re-index after upgrading. +- A link made by a framework or bridge resolver now records which one made it (`metadata.framework`, such as `swift-objc-bridge` or `react-native-bridge`), as the README describes. Before, only synthesized event and view channels were named. +- ASP.NET apps built with FastEndpoints now have their routes: each endpoint class's `Get(…)` / `Post(…)` in `Configure()` becomes a route linked to its own `HandleAsync` / `ExecuteAsync`, including paths kept in a request class's `Route` constant. ardalis/CleanArchitecture went from no routes to 24. Minimal API routes written without a leading slash (`app.MapGet("api/todos", …)`) are now named `/api/todos`. +- Play projects kept in subdirectories are recognized, so each one's `conf/routes` is read. A repository with no Play build at its root, like playframework's samples, went from no routes to over 130. +- Rails routes are now read with their nesting: `namespace` and `scope` add their path and controller module, nested `resources` sit under their parent's `:id`, and `member` / `collection` blocks add their actions. A namespaced route now links to its own module's controller. Routes in a Rails engine's `config/routes.rb` are found too: solidus went from no routes to over 600, and mastodon's route-to-action links nearly tripled. +- Flask routes registered with `add_url_rule(…)` are now found and linked to their view function or class-based view. So are routes registered through a project's own helper that passes a list of paths and a `view_func=`. flaskbb, which registers every view that way, went from no routes to over 100. +- Vue components written with the Options API (`export default { methods: {…}, computed: {…}, watch: {…}, mounted() {…} }`), as every Vue 2 app is, now have their methods, computed properties, watchers and lifecycle hooks in the graph. Each one has its own callers and calls, and a template's `@click="handleLogin"` links to the method it runs. On vue-element-admin that's over 800 new links, where a component's calls used to belong to the whole file. +- React Native events whose name is held in a constant now link native code to the JavaScript that listens for them, like NetInfo's `addListener(PrivateTypes.DEVICE_CONNECTIVITY_EVENT, …)` or a Java `.emit(PROGRESS_EVENT, …)`. The constant is read the way the language scopes it, from the same file, an import or its owning class, so a parameter that happens to share a constant's name is never mistaken for it. +- React Native views declared with `requireNativeComponent('X')`, the older Paper style that react-native-maps and segmented-control use, now link from JavaScript to their native iOS and Android implementations, as Codegen specs already did, including in plain `.js` files. A JSX tag written under a different name than its component, like `` for `import Autocomplete from './autocomplete.vue'`, now links to the component it renders. +- Expo Router API routes (`app/hello+api.ts`) are now endpoints, one per exported method, like `GET /hello`, bound to the function that handles them. They used to show up as a screen named `/hello+api`. In a repository whose root declares Expo Router for an example app, a Next.js app in its own folder, like react-native-true-sheet's `docs/`, no longer gets Expo screens made from its files. +- Calls into an Expo module now reach the module's native functions even when the JavaScript names the module something else, like expo-camera's `CameraManager` for `requireNativeModule('ExpoCamera')`. Both the iOS and the Android implementations are linked. expo-camera's calls used to link back to the same-named method making the call, so a method looked like it called itself. +- Angular apps split across Nx libraries, like angular-spotify, now have their routes and navigation. A lazily loaded route written `async () => (await import('@app/home')).HomeModule` is followed through the library's `index.ts` to the module it re-exports, and a `routerLink` in one library now reaches a screen declared in another. Route paths built from a class constant (`RouterUtil.Configuration.Lyrics`, `` `issue/:${ProjectConst.IssueId}` ``) or an enum are now read. A redirect in a routes file that only lazy-loads others now counts, and `router.navigate(['project', 'issue', id])` without `relativeTo` is read from the root, as Angular does. Re-index Angular projects after upgrading. +- React Router apps now show navigation written as React Router v5's `` and through a styled link, like react-boilerplate's `HeaderLink = styled(Link)` used as ``. Apps built this way had routes but no links between their screens. +- SvelteKit routes are now named by the address a browser asks for. A `(group)` folder like `(app)` or `(marketing)` is no longer part of a route's path, and a parameter with a matcher, like `[id=integer]`, is just `:id`. So `goto('/blocks')` and `` now reach a page that lives in `src/routes/(app)/blocks/`, where before they reached nothing and the app's screens showed no navigation. Re-index SvelteKit projects after upgrading. +- Vue Router apps built like vue-element-admin, vue-admin-template or vben now have their routes and screens. Routes declared in a named table (`export const constantRoutes = [...]`, `const routes: RouteRecordRaw[] = [...]`), in per-module route files, or through `new Router(...)` are now read. Child routes are joined onto their parent's path, and the parent's component is the layout around them. A lazily loaded view (`() => import('@/views/dashboard/index')`) is bound to the file it names, and `this.$router.push(...)` counts as navigation. Nuxt's file-based routes now come only from a Nuxt app, so a plain Vue app's `pages/` folder no longer turns into made-up screens, and a Nuxt app at the root of its repository now gets its routes. Re-index Vue projects after upgrading. +- NestJS routes are now named by the path a request takes: the global prefix from `app.setGlobalPrefix('api')` is applied, along with its `exclude` list, and so is URI versioning from `app.enableVersioning(...)`, with `@Version`, `VERSION_NEUTRAL` and a controller's own `version` taken into account. A route that used to appear as `GET /user` is now `GET /api/v1/user`, so a front end's `this.http.get('/api/v1/user')` or `fetch` call connects to the controller method that serves it. Re-index NestJS projects after upgrading. +- Laravel routes are now named by the path a request takes: a leading `/` is added where the routes file leaves it out, `Route::prefix()` and `Route::group(['prefix' => …])` groups are applied, and routes in `routes/api.php` carry the `/api` prefix Laravel serves them under, read from your `RouteServiceProvider`, `bootstrap/app.php` or any file that mounts a routes file. A front-end `fetch('/api/…')` now connects to the Laravel route that serves it. Re-index Laravel projects after upgrading. +- Controllers that Spring or Laravel tests exercise by URL now count as tested. That covers MockMvc's `perform(post("/owners/new"))`, WebTestClient, TestRestTemplate, RestAssured, Laravel's `$this->postJson('api/me')`, Pest's `get('/about')`, and a project's own request helpers built on them. `codegraph_explore` now names the test suite that reaches such an endpoint, where before every one of them looked untested. Re-index Spring and Laravel projects after upgrading. +- In an Angular app, a method a template calls, like `(click)="toggleFavorite()"` or `(ngSubmit)="submitForm()"`, is now linked from its component. These methods used to show no callers at all, so impact and callers questions stopped at them, and dead-code checks could list them as unused. Re-index Angular projects after upgrading. +- Angular apps now have routes and navigation in the graph. Each screen in a `Routes` array is a route named by its full path, including children, lazily loaded `loadChildren` files and NgModules, paths written as route constants or `$localize` strings, and redirects. Each route is linked to its component. `router.navigate([...])`, `navigateByUrl`, a guard's `createUrlTree` and every `routerLink` in a component's template link to the screen they open, and a template's child components (``) are linked to the component that renders them. Questions like "where does this button go" and "what renders this component" now have answers in `codegraph_explore`. Re-index Angular projects after upgrading. +- Delphi and Free Pascal include files (`.inc`) are now indexed as Pascal. Before, they were read as PHP and came up empty. A `.inc` file with a ``, ``/``, ``, `` and `#…#` expressions. Before, only `` and `` code was read, so callers and impact found almost nothing in tag-based components. Functions wrapped in tags like `` or `` are now indexed too, and a `` is indexed as an interface that `implements` links to. Re-index CFML projects after upgrading. Thanks @HarryMuc for the report. (#2091) +- Java enum constants and Kotlin enum entries with a body of their own now have their methods in the graph. Examples are `PLUS { int apply(…) { … } }` and `NewBuffer { override fun pipe() … }`. Before, those methods were missing and every call inside them was dropped. jsoup's HTML tokenizer and tree builder are written this way, so over 2,000 of their calls were invisible. Re-index to pick this up. +- Kotlin infix calls, like `Users.id eq id1` or koin's `single { … } bind MyApi::class`, are now recorded as calls to the infix function. Before, they had no callers at all. On Exposed, `eq` now shows over a thousand callers. Bit operations on numbers (`x and 0xff`, `h shr 8`) are left to Kotlin's own operators, even where the project defines a same-named extension. Re-index after upgrading. + ### Fixes +#### Windows + - On Windows, terminal windows no longer flash open and closed while CodeGraph runs in the background. Since 1.6.1, a background MCP server popped up a console window (a full Windows Terminal window when that is the default terminal) several times when it started and again every time it re-synced a changed file. All of CodeGraph's git calls now run hidden. Thanks @Suharaz, @23q3, @A-Van-Gestel and @HarryMuc. (#2094, #2096) -- In Vapor apps, a route whose handler is a trailing closure, like `app.get("hello") { req in … }` or `app.webSocket("chat") { req, ws in … }`, now links to what the closure calls, the way an Express inline handler does. Before, the route linked to nothing, so callers, impact and flows stopped at it. +- On Windows, a project opened with a different drive-letter or folder-name case, like `d:\work\app` and `D:\Work\App`, now joins the CodeGraph background server that is already running for it. Before, each spelling started a server of its own that exited at once, so the session quietly served the graph by itself with no shared file watching or auto-sync, and `codegraph list` showed the project twice. Thanks @greatjackzhou. (#2278) +- On Windows, `codegraph upgrade` no longer breaks the install while an agent session is using CodeGraph. It used to stop partway and leave CodeGraph unable to start; it now swaps the new files in with CodeGraph running and puts the previous version back if anything fails, and re-running the PowerShell installer works the same way. If an earlier upgrade already broke your install, re-run `irm https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.ps1 | iex` to repair it. Thanks @tippmar-nr for the report. (#2185) +- On Windows, CodeGraph no longer sometimes crashes with exit code 3221225477 (an access violation) on a busy machine: right after parsing in `codegraph init`, `codegraph index` and background syncs, or as the background server shuts down. Stopping one of its helper threads (the ones that parse files, resolve references, write the index and count changes for `codegraph_status`) while it was still starting up, or before it had finished cleaning up its memory, took the whole process down. Threads now finish starting and clean up before they exit. + +#### The background server, sessions and the CLI + +- The Claude Code prompt hook no longer runs on the messages Claude Code uses to hand a subagent's report back to the main session. Before, such a long report could keep the hook busy past Claude Code's 30-second hook timeout and inject context unrelated to what you asked. Thanks @danusha2345, and @tippmar-nr for the report. (#2184) +- Newer MCP clients such as Antigravity 2.5 connect again: the server now answers their `server/discover` probe right away with "method not found", so they go straight on to the regular handshake instead of waiting on a reply that could take seconds or never come. Thanks @danusha2345, and @samem26 for the report. (#2084) +- A CodeGraph session that queried another project through `projectPath` no longer keeps that project locked for as long as it runs: after 10 minutes without a query it lets the project go, so the project's own session and `codegraph index` can take over again (tune with `CODEGRAPH_PROJECT_IDLE_TIMEOUT_MS`, `0` keeps it open). Thanks @danusha2345, and @bompus for the report. (#2087) +- Claude Code now shows the model the full text of `codegraph_status` and of explore's stale-index refusal. Both results also carried `structuredContent`, and Claude Code shows a result's `structuredContent` in place of its text, so the model saw only a freshness JSON object: no refusal message, no retry guidance and no status report. Both results are now text only. Thanks @bompus. (#2088) +- When an agent asks CodeGraph about a separate git repository nested inside an indexed project, one the project's index leaves out (for example because the parent's `.gitignore` excludes it), it now gets the usual "isn't indexed" guidance instead of answers from the parent project's code. Nested repositories the parent does index, such as submodules, work as before. Thanks @wstczyw for the report. (#2110) +- An agent session that lost its connection to CodeGraph's shared background server, or couldn't reach it at startup, no longer stops that server from running for the rest of the session. It reconnects on its own once the server can start, so other sessions on the same project get auto-sync back instead of running read-only. Thanks @ijbranch for the report. (#2277) + +#### Indexing and sync + - A COBOL file whose last line ends inside the sequence area (columns 1-6), such as a closing period indented only four spaces, no longer stalls indexing for over a minute and then ends up in the index with no symbols. -- In Laravel apps, a route whose controller is named by string now links to its handler, without needing a `Controller` suffix and keeping the namespace it's written with: `'Common\Uploads@inline'`, and `Route::resource('companies', 'Common\Companies', [...])` with an options array. Two same-named controllers in different namespaces are told apart by that namespace. Before, akaunting linked 304 of its 313 routes to nothing. -- In Nuxt and Astro projects, a file-routed page is now linked to the component its file defines. An Astro endpoint is linked to the HTTP-verb handlers it exports, like `export const GET`. Before, a page route like Nuxt's `pages/admin.vue` or Astro's `src/pages/about.astro` linked to nothing, so callers, impact and flows stopped at the route. -- In Rails apps, a `resources` line's `only:` and `except:` are now read in every form Rails accepts: `%i[new create index]`, `%w(index show)`, a quoted name, and the older `:only => [...]`. Before, only a bracketed list was read, so maybe's `resources :family_exports, only: %i[new create index]` became all seven routes, four of them to actions the controller doesn't have. +- Editing a file that defines the same name more than once, like two classes that each have an `execute` method or a method's overloads, no longer moves every caller from other files onto one of them during `codegraph sync`. A sync interrupted partway through a file also no longer loses those callers until a full re-index. Thanks @ijbranch for the report. (#2276) +- Indexing a large C or C++ project no longer runs out of memory partway through "Resolving refs". Headers whose include guard is written as `#define X_H 1`, as in OpenSceneGraph and osgEarth, or that are included under a build flag codegraph cannot know, used to be re-read on every include path, which grows exponentially with the depth of the include tree. Each is now read once per include state, and a hard limit stops the check on any include tree that is still too large. Thanks @danusha2345, and @coolhitmanleon for the report. (#2127) +- On a busy machine, indexing or syncing a small project no longer starts extra background threads for resolving references that it can't use. Under heavy load the slower pace made a ten-file project look big enough to need them, which added seconds to the run and held extra memory. + +#### Across languages + +- A type that extends another type of the same name now links to that other type instead of to itself. When the source spells out the qualifier, that qualifier picks between same-named candidates. Before, cats' `trait BigDecimalInstances extends cats.kernel.instances.BigDecimalInstances` and every `trait AllOps … with Functor.AllOps[…]` pointed back at the declaring trait, and Newtonsoft.Json's `JsonConverter : JsonConverter` extended itself. +- Framework name conventions for Django, FastAPI, Flask, ASP.NET, Gin, SwiftUI and Vapor now only pick a class the reference can see. Examples of these conventions are "a `…View` is a view", "a `…Service` is a service" and "a capitalized name is a model". The pick starts from the reference's own file and then its package, and never takes a class declared inside some function or nested in another file's class. Before, Django REST framework's tests linked each file's own `MockView` and `Serializer` to another file's, and netbox's tests linked `TestForm()` to a form declared inside a different test function. +- A method that calls a same-named method on another expression's result is no longer linked to itself in any language. Examples are Scala's `requestToArmeria(request).execute()` inside `execute()`, Rust's `self.0.into_route(state)` inside `into_route()`, and Kotlin's `this@Buffer.write(…)` from an inner object. A value whose initializer chain calls a same-named method, like sttp's `val response = basicRequest.get(…).response(…)`, no longer links to itself either. +- A method that hands its call on to another object is no longer linked to itself. In TypeScript and JavaScript, a call through `this.` or `window.` counts as recursion only when the field is declared as the method's own class. A guess from a receiver's name alone never lands on the calling method. Before, BookStack's `toggle()` doing `this.container.classList.toggle('open')`, `listen()` doing `window.$events.listen(…)`, and `FileStorage::delete` doing `$storage->delete($path)` each pointed at themselves. +- A function or name passed as a value now resolves to what is in scope where it's written. A function nested inside another function is only reachable from inside it. In Python, a parameter or local of the same name is that local. A pytest fixture is a test's parameter only in its own module or under its `conftest.py`. Before, httpx's `self._build_auth(auth)` linked to an `auth` a test defines inside another function, and `auth_flow(self, request)` handing `request` on linked to the package's `request()` function. +- In Java, C#, Kotlin, Swift, Scala, Dart and VB.NET, a call now reaches the overload whose parameters fit its arguments, not whichever same-named overload was indexed first. For example, `HashCodeBuilder.reflectionHashCode(this)` now reaches the one-argument overload with varargs, not a three-parameter one. Each overload's callers are now its own. +- A method that calls another overload of itself now links to that overload rather than to itself. An example is `toInstant(instant)` returning `toInstant(instant, Instant.EPOCH)`. The fuller overload now lists its convenience overloads among its callers, so its impact includes them. This applies to Java, C#, Kotlin, Swift, C++, Scala, Dart and VB.NET. Real recursion is unchanged. +- Production code no longer links to a same-named symbol inside a test suite, since tests aren't built into the program. Before, typeorm's `Record` types reached a test entity called `Record`, and okhttp's sample `@Override` annotations reached a test's nested `Override` class. Test-support code a project ships, such as a `testing/` folder or a `*-test` module, stays in reach. +- A method call on a variable is no longer guessed to be a test double's method just because the double has the only method of that name. That covered classes named `Mocked…`, `Fake…`, `Stub…` or `Dummy…`: django-allauth's `resp.json()` calls had all gone to its `MockedResponse`. A test that names the double still calls it. +- A call through a class name, like `THorse.Get(…)`, `MyView.as_view()` or `ColorChoices.values()`, now reaches the class method that class inherits. The inheritance is read from the class declarations themselves, in Pascal, Python, Ruby, PHP, TypeScript/JavaScript and the Java family. Before, such a call was resolved by guessing from the name. On Horse, `THorse.Get('/ping', …)` used to land on a route-group interface's `Get` 120 times. Re-index after upgrading. +- A method guessed from a receiver's name no longer lands on a request handler the web framework dispatches to, such as a Django or DRF view's `post` / `get` or a controller's `update` / `destroy`. It also no longer lands on a test double the calling file never mentions. On django-allauth, tests' `client.post(…)` used to land on a `ClientRegistrationView` 425 times. On mealie, `response.json()` used to land on a test's `_FakeHTTPResponse` 424 times. In Python, `site = Site.objects.create(…)` now makes `site` a `Site`, so `site.save()` reaches the model's own `save`. Re-index after upgrading. +- In C#, Java and Kotlin, a call through a field, property or parameter now lands on the method of the type it is declared with. That includes a type inherited from a base class with its type argument filled in, a generic or `for`-each declaration, a C# `using` alias, and a type parameter's bound. A call no longer falls back to a guess at some project class's same-named method, and a type from outside the project gives no link. On Newtonsoft.Json, `_innerWriter.WriteValue(…)` inside a writer wrapper used to land on the wrapper's own `WriteValue`, and a `TextWriter`'s `Write(…)` on a test writer's. On commons-lang, a `DateFormat` field's `parse(…)` used to land on the project's `DateParser` 284 times. +- A call on a variable declared with a type from outside the project (`List list`, `var sb = new StringBuilder()`, `signer = TimestampSigner()`) no longer lands on a project type's method with the same name. On gson, `list.add(…)` used to land on a project list wrapper; on commons-lang, `s.length()` and `buffer.append(…)` on project classes. In Python, a keyword argument inside a call (`prefix=IPNetwork(…)`) is no longer taken for an assignment to that name. +- A call on a type from outside the project, like `Integer.valueOf(…)`, `Arrays.asList(…)`, `Object.assign(…)` or Delphi's `Exception.Create(…)`, no longer lands on the project's own method with that name. On commons-lang, `Integer.valueOf` used to land on `StringUtils.valueOf` over 800 times. Go's exported variables and Pascal's capitalized parameters and locals are not mistaken for type names. +- In PHP, CFML, Pascal, COBOL and VB.NET, a call written in a different case than its definition now resolves, like `formatprice()` for `FormatPrice` or CFML's `ToMatchWithCase()` for `toMatchWithCase`. These languages ignore case in names, but such calls used to be dropped before resolution. +- In Java, Kotlin, Swift, C#, C and C++, Go, Scala, Ruby, Dart and the other case-sensitive languages, a name no longer links to a symbol whose name differs only in case. On jsoup, JUnit's `@Test` had linked every test to a `test` method; `new CookieManager()` had linked to a `cookieManager()` getter, and a `Method` type to a `method()`. PHP, Pascal, CFML, COBOL and VB.NET, whose names really do ignore case, are unchanged. +- A `super` call in an overriding method, like `super.viewDidLoad()`, `[super init]`, `base.Handle()`, `parent::setUp()` or `super().dispatch(…)`, no longer makes the method look like it calls itself. The call goes to the parent's version, so an override stops showing up as recursive and no longer lists itself among its own callers. +- A generic type parameter, like `A` in `def zipWith[A, B](…)`, `T` in ` T max(…)` or `class Foo`, no longer links to a project symbol that happens to share its name. On cats, one `implicit def A` had thousands of made-up dependents from every `A` in the library. The same goes for Rust, Dart, TypeScript, Java, Kotlin, Swift, Go, C# and C++. A class or other type declared inside a function is only linked from inside that function. In Scala, a `def`'s own parameter, like `f` in `(f: A => B)`, no longer links to a same-named field elsewhere. A bare name no longer reaches a type nested inside another type unless the code is inside that type, extends it or imports its members; the type in scope is linked instead, like cats' own `FlatMap` in place of `Eval`'s nested `FlatMap`. Re-index projects in these languages after upgrading. + +#### JavaScript and TypeScript, with Vue, Svelte, Astro, React, Express, NestJS and Hono + - In Vue, Svelte and Astro files, a component now owns its script, and what a ` diff --git a/telemetry-dashboard/public/styles.css b/telemetry-dashboard/public/styles.css index 73ec9751c1..e2dc7f88c7 100644 --- a/telemetry-dashboard/public/styles.css +++ b/telemetry-dashboard/public/styles.css @@ -107,6 +107,28 @@ button:focus-visible { padding: 0 4px; } +/* --- stale-data warning ------------------------------------------------- */ + +.data-warning { + margin-top: 16px; + padding: 12px 16px; + background: var(--surface); + border: 1px solid var(--oxblood); +} + +.data-warning p { + margin: 0; +} + +.data-warning p + p { + margin-top: 6px; +} + +.data-warning strong { + color: var(--oxblood); + font-weight: 600; +} + /* --- buttons ------------------------------------------------------------ */ button { diff --git a/telemetry-dashboard/scripts/fixture.sql b/telemetry-dashboard/scripts/fixture.sql index 108df44365..9e2ddef020 100644 --- a/telemetry-dashboard/scripts/fixture.sql +++ b/telemetry-dashboard/scripts/fixture.sql @@ -117,6 +117,12 @@ SELECT machine_id, min(day) FROM events GROUP BY machine_id; INSERT INTO daily_machines (day, machines, prod_machines) SELECT day, count(*), coalesce(sum(prod), 0) FROM machine_days GROUP BY day; +-- The activation funnel's input: each machine's earliest index day (FIRST_INDEX_DAY). +-- m04 and m06 never index, so theirs stays NULL. +UPDATE machine_first_seen + SET first_index_day = (SELECT min(day) FROM events e + WHERE e.machine_id = machine_first_seen.machine_id AND e.event = 'index'); + INSERT INTO daily_event_counts (day, event, count, machines) SELECT day, event, CASE WHEN event = 'usage_rollup' diff --git a/telemetry-dashboard/scripts/render-check.mjs b/telemetry-dashboard/scripts/render-check.mjs index 9411bec541..fd1a740fbd 100644 --- a/telemetry-dashboard/scripts/render-check.mjs +++ b/telemetry-dashboard/scripts/render-check.mjs @@ -176,8 +176,12 @@ const PROBE = `(() => { tableHidden: section.querySelector('[data-role="table"]').hidden, }; }); + const warning = document.getElementById('data-warning'); return { ready: document.body.dataset.ready === 'true', + warning: warning.hidden ? null : warning.textContent, + customFrom: document.querySelector('[data-role="custom-from"]').value, + customTo: document.querySelector('[data-role="custom-to"]').value, range: document.getElementById('range-summary').textContent, dataThrough: document.getElementById('data-through').textContent, refreshed: document.getElementById('refreshed-at').textContent, @@ -299,17 +303,32 @@ async function main() { // The very same registry the page just rendered from, imported here so the // expectations cannot drift from the panels under test. const { PANELS } = await import(pathToFileURL(join(root, 'public', 'panels.js')).href); + const { shortDay } = await import(pathToFileURL(join(root, 'public', 'theme.js')).href); + const utcDay = (ms) => new Date(ms).toISOString().slice(0, 10); + const today = utcDay(Date.now()); check(`all ${PANELS.length} panels are on the page`, view.panels.length === PANELS.length, `got ${view.panels.length}`); - const broken = view.panels.filter((p) => p.state !== 'ready'); + // The fixture lives in July, so a range ending today has nothing in it: every + // panel should say so plainly, and none should error. + const broken = view.panels.filter((p) => p.state !== 'ready' && p.state !== 'empty'); check( - 'every panel reached its ready state', + 'every panel settled without an error', broken.length === 0, broken.map((p) => `${p.id}: ${p.state} ${p.message}`).join(' | '), ); check('the default range is the 30-day preset', view.selectedPreset === 'Last 30 days', view.selectedPreset); - check('the range is stated in the filter row', /Jun|Jul/.test(view.range), view.range); - check('the data horizon is stated', view.dataThrough.includes('Jul 10'), view.dataThrough); + same('…ending today, not on the last rolled-up day', today, view.customTo); + same('…and starting 29 days before', utcDay(Date.now() - 29 * 864e5), view.customFrom); + check('the range is stated in the filter row', view.range.includes(shortDay(today)), view.range); + check('the rollup horizon is stated', view.dataThrough.includes('Event counts through Jul 10'), view.dataThrough); + check('…with yesterday\'s activity beside it', view.dataThrough.includes('0 machines active yesterday'), view.dataThrough); check('the refresh time is stated', view.refreshed.startsWith('Last refreshed'), view.refreshed); + // Nothing has arrived since the fixture's last day: the page must say so rather + // than present July as current. + check( + 'a stalled ingest is called out above the panels', + view.warning?.includes('No new events since Jul 10.') === true, + String(view.warning), + ); // A CSP violation surfaces here as a `security` log entry, which is the point // of the check: the page must work under `script-src 'self'` with no inline @@ -339,10 +358,10 @@ async function main() { (await cdp.evaluate(sessionId, 'document.body.dataset.ready === "true"')) === true, ); view = await cdp.evaluate(sessionId, PROBE); - const weekly = view.panels.find((p) => p.id === 'daily-production-users'); - check('a daily line now holds 7 points', weekly.chart?.labels.length === 7, `${weekly.chart?.labels.length}`); + same('the range now starts 6 days back', utcDay(Date.now() - 6 * 864e5), view.customFrom); + check('…and still ends today', view.customTo === today, view.customTo); check('the 7-day preset is marked selected', view.selectedPreset === 'Last 7 days', view.selectedPreset); - check('every panel re-rendered cleanly', view.panels.every((p) => p.state === 'ready')); + check('every panel re-rendered cleanly', view.panels.every((p) => p.state === 'ready' || p.state === 'empty')); console.log('\nA custom range works the same way'); await cdp.evaluate( @@ -358,6 +377,12 @@ async function main() { view = await cdp.evaluate(sessionId, PROBE); check('the fixture window is 10 days', view.panels.find((p) => p.id === 'daily-production-users').chart?.labels.length === 10); check('no preset stays highlighted', view.selectedPreset === null, view.selectedPreset); + const notReady = view.panels.filter((p) => p.state !== 'ready'); + check( + 'every panel reached its ready state', + notReady.length === 0, + notReady.map((p) => `${p.id}: ${p.state} ${p.message}`).join(' | '), + ); // -- every panel plots what the API returned ------------------------------ console.log('\nEvery panel plots the API’s own numbers'); @@ -428,6 +453,29 @@ async function main() { same('languages lead with typescript', 'typescript', byId.languages.chart.labels[0]); check('retention starts at 100%', byId.retention.chart.datasets[0].data[0] === 100); + console.log('\nPast the rollup\'s last day, lines stop instead of dropping to zero'); + await cdp.evaluate( + sessionId, + `document.body.dataset.ready = ""; + document.querySelector('[data-role="custom-from"]').value = "2026-07-05"; + document.querySelector('[data-role="custom-to"]').value = "2026-07-14"; + document.querySelector('[data-role="custom-apply"]').click();`, + ); + await waitFor('the over-the-edge render', async () => + (await cdp.evaluate(sessionId, 'document.body.dataset.ready === "true"')) === true, + ); + const edge = Object.fromEntries((await cdp.evaluate(sessionId, PROBE)).panels.map((p) => [p.id, p])); + same( + 'daily production users ends in gaps after Jul 10', + [[2, 3, 2, 1, 1, 1, null, null, null, null]], + edge['daily-production-users'].chart?.datasets.map((d) => d.data), + ); + same( + 'new installs are live, so they stay zero rather than gap', + [[2, 0, 0, 1, 2, 0, 0, 0, 0, 0]], + edge['new-installs'].chart?.datasets.map((d) => d.data), + ); + // Colour, spacing and label collisions are not things an assertion catches. // RENDER_SHOT=/tmp/dash.png npm run smoke:render → look at it. if (process.env.RENDER_SHOT) { diff --git a/telemetry-dashboard/scripts/seed-fixture.sh b/telemetry-dashboard/scripts/seed-fixture.sh index 88c8b1e271..3f4eb66085 100755 --- a/telemetry-dashboard/scripts/seed-fixture.sh +++ b/telemetry-dashboard/scripts/seed-fixture.sh @@ -11,17 +11,20 @@ set -uo pipefail cd "$(dirname "$0")/.." DB=codegraph-telemetry -MIGRATION=../telemetry-worker/migrations/0001_init.sql +MIGRATIONS=../telemetry-worker/migrations -if [[ ! -f "$MIGRATION" ]]; then - echo "seed: cannot find $MIGRATION — run this from a full checkout" >&2 +if [[ ! -f "$MIGRATIONS/0001_init.sql" ]]; then + echo "seed: cannot find $MIGRATIONS — run this from a full checkout" >&2 exit 1 fi -# The migration is plain CREATE TABLE, so a second run fails on "table already -# exists". That is the expected steady state here, hence the swallowed output — -# the fixture load below is the step whose failure actually matters. -npx wrangler d1 execute "$DB" --local --file="$MIGRATION" >/dev/null 2>&1 +# Applied in order, each as a plain file. A second run fails every one of them +# ("table already exists", "duplicate column") — the expected steady state here, +# hence the swallowed output. The fixture load below is the step whose failure +# actually matters, and it fails loudly if a migration is missing. +for migration in "$MIGRATIONS"/*.sql; do + npx wrangler d1 execute "$DB" --local --file="$migration" >/dev/null 2>&1 +done if ! npx wrangler d1 execute "$DB" --local --file=scripts/fixture.sql >/dev/null; then echo "seed: loading scripts/fixture.sql failed" >&2 diff --git a/telemetry-dashboard/scripts/smoke-api.sh b/telemetry-dashboard/scripts/smoke-api.sh index 964c1b6819..67008c47b4 100755 --- a/telemetry-dashboard/scripts/smoke-api.sh +++ b/telemetry-dashboard/scripts/smoke-api.sh @@ -92,12 +92,21 @@ check "health stays uncached" "no-store" \ "$(curl -sD - -o /dev/null -b "$JAR" "$BASE/api/health" | grep -i '^cache-control:' | cut -d' ' -f2- | tr -d '\r')" echo -echo "/api/meta — what the range picker anchors on" +echo "/api/meta — how current each kind of number is" +TODAY="$(node -e 'console.log(new Date().toISOString().slice(0, 10))')" +YESTERDAY="$(node -e 'console.log(new Date(Date.now() - 864e5).toISOString().slice(0, 10))')" META="$(get "/api/meta")" -field "latest day" latest_day 2026-07-10 "$META" -field "earliest day" earliest_day 2026-07-01 "$META" -field "raw events start" earliest_raw_day 2026-07-01 "$META" -field "retention window" retention_days 14 "$META" +field "today, as a UTC day" today "$TODAY" "$META" +field "latest rollup day" latest_rollup_day 2026-07-10 "$META" +field "…kept as latest_day" latest_day 2026-07-10 "$META" +field "earliest day" earliest_day 2026-07-01 "$META" +field "raw events start" earliest_raw_day 2026-07-01 "$META" +field "retention window" retention_days 14 "$META" +# The fixture stops on 07-10, months ago: nothing has arrived since, but the +# rollup did cover every day that saw activity, so it is not behind. +field "ingest stalled (no events since 07-10)" ingest_stalled true "$META" +field "rollup not behind (it covered every active day)" rollup_behind false "$META" +field "nobody active yesterday" machines_yesterday 0 "$META" echo echo "/api/summary — the big numbers (12 machines, one of them CI)" @@ -134,6 +143,7 @@ field "calls per day" datasets.0.data '[0,40,28,0,0,12,0,0,0,5]' "$TS" field "machines per day" datasets.1.data '[0,1,2,0,0,1,0,0,0,1]' "$TS" TS="$(get "/api/timeseries?metric=duration_buckets&$RANGE")" +field "coverage reported" covered_through 2026-07-10 "$TS" field "bucket order is the scale" datasets.0.label '<10s' "$TS" field "…and ends at the longest" datasets.3.label '5m+' "$TS" field "<10s over time" datasets.0.data '[2,0,1,1,0,0,0,1,0,0]' "$TS" @@ -141,6 +151,22 @@ field "10-60s over time" datasets.1.data '[0,2,0,0,0,0,1,0,0,1]' "$TS" field "1-5m over time" datasets.2.data '[0,0,0,0,1,1,0,0,0,0]' "$TS" field "5m+ over time" datasets.3.data '[0,0,1,0,0,0,0,0,1,0]' "$TS" +echo +echo "Past the rollup's last day: not counted yet, so null — never a zero" +# The fixture is rolled up through 07-10; 07-11 and 07-12 have not been. +LATE="from=2026-07-09&to=2026-07-12" +TS="$(get "/api/timeseries?metric=installs_uninstalls&$LATE")" +field "installs stop at the rollup" datasets.0.data '[2,0,null,null]' "$TS" +field "…and say where" covered_through 2026-07-10 "$TS" +field "the table twin keeps the gap" rows.2.Installs null "$TS" +TS="$(get "/api/timeseries?metric=production_users&$LATE")" +field "daily production users too" datasets.0.data '[1,1,null,null]' "$TS" +TS="$(get "/api/timeseries?metric=duration_buckets&$LATE")" +field "run length over time too" datasets.0.data '[0,0,null,null]' "$TS" +TS="$(get "/api/timeseries?metric=new_installs&$LATE")" +field "new installs are live: zeros, not gaps" datasets.0.data '[2,0,0,0]' "$TS" +field "…and claim no rollup coverage" covered_through null "$TS" + echo echo "/api/breakdown — bars and pies" # machine-days, taking the largest per-event count per day so one machine's @@ -209,6 +235,7 @@ field "…so two dropped" dropped 2 "$ACT" field "window" window_days 7 "$ACT" field "daily rate, null where no cohort" datasets.0.data '[75,50,100,null,100,null,null,100,100,null]' "$ACT" field "recent cohorts flagged incomplete" incomplete_from 2026-07-04 "$ACT" +field "cohorts counted through the rollup" covered_through 2026-07-10 "$ACT" field "…and the completed ones are not" rows.2.complete true "$ACT" field "…while the last week is" rows.8.complete false "$ACT" @@ -217,6 +244,14 @@ field "…while the last week is" rows.8.complete false "$ACT" ACT="$(get "/api/activation?window=1&$RANGE")" field "a 1-day window converts fewer" activated 9 "$ACT" +# m11 (indexed 07-10) and m12 (indexed 07-09) arrived on 07-09. 07-11 onward has +# not been rolled up, so those cohorts are gaps and stay out of the totals. +ACT="$(get "/api/activation?from=2026-07-09&to=2026-07-12")" +field "uncounted cohorts stay out of the totals" installs 2 "$ACT" +field "…both counted ones converted" activated 2 "$ACT" +field "…and draw as gaps, not 0%" datasets.0.data '[100,null,null,null]' "$ACT" +field "…with no conversions claimed" rows.2.activated null "$ACT" + echo echo "/api/retention — day 0–14, denominator per day" RET="$(get "/api/retention?$RANGE")" @@ -259,6 +294,19 @@ field "no bars" labels '[]' "$EMPTY" EMPTY="$(get "/api/timeseries?metric=production_users&from=2025-01-01&to=2025-01-03")" field "still a dense axis" datasets.0.data '[0,0,0]' "$EMPTY" +echo +echo "A rollup that stops is called out" +# One machine active yesterday that no rollup has covered — what a failed nightly +# run (or a database refusing writes) leaves behind. Removed again straight after. +STALE_ID=00000000-0000-4000-8000-0000000000ff +npx wrangler d1 execute codegraph-telemetry --local \ + --command "INSERT INTO machine_days (machine_id, day, prod) VALUES ('$STALE_ID', '$YESTERDAY', 1)" >/dev/null 2>&1 +META="$(get "/api/meta")" +field "rollup behind once a day goes un-rolled" rollup_behind true "$META" +field "…and yesterday's machine is counted" machines_yesterday 1 "$META" +npx wrangler d1 execute codegraph-telemetry --local \ + --command "DELETE FROM machine_days WHERE machine_id = '$STALE_ID'" >/dev/null 2>&1 + echo printf '%d passed, %d failed\n' "$PASS" "$FAIL" [[ "$FAIL" -eq 0 ]] diff --git a/telemetry-dashboard/scripts/smoke-auth.sh b/telemetry-dashboard/scripts/smoke-auth.sh index 3fe357baec..73777cadb4 100755 --- a/telemetry-dashboard/scripts/smoke-auth.sh +++ b/telemetry-dashboard/scripts/smoke-auth.sh @@ -64,9 +64,10 @@ lacks() { # lacks fi } -echo "Seeding local D1 from the ingest worker's migration…" -npx wrangler d1 execute codegraph-telemetry --local \ - --file=../telemetry-worker/migrations/0001_init.sql >/dev/null 2>&1 +echo "Seeding local D1 from the ingest worker's migrations…" +for migration in ../telemetry-worker/migrations/*.sql; do + npx wrangler d1 execute codegraph-telemetry --local --file="$migration" >/dev/null 2>&1 +done echo "Starting wrangler dev on :${DASH_PORT}…" npx wrangler dev --port "$DASH_PORT" --ip 127.0.0.1 >"$LOG" 2>&1 & diff --git a/telemetry-dashboard/src/api.ts b/telemetry-dashboard/src/api.ts index 5b5fac3760..63b77556c6 100644 --- a/telemetry-dashboard/src/api.ts +++ b/telemetry-dashboard/src/api.ts @@ -3,10 +3,14 @@ * scoped by the same `?from=&to=` range the picker drives. * * Rules this file keeps: - * - **Rollups first.** Every panel is answered from `daily_*` / `machine_days`, - * which are kept forever. Only the activation funnel touches raw `events`, - * because "did this machine ever run an index" is not a daily aggregate — and - * that is also the only endpoint with a horizon (the retention window). + * - **Rollups only.** Every panel is answered from `daily_*`, `machine_days` and + * `machine_first_seen`, which are kept forever. No panel reads raw `events`: + * D1 runs one query at a time per database, so one slow scan there fails every + * panel queued behind it. (/api/meta reads the table's first and last day, one + * indexed lookup each.) + * - **Today is in range; uncounted days are not zeros.** Rolled-up numbers stop + * at the nightly rollup's last day and come back null after it, so a chart + * ending today draws a gap where the count has not happened yet, not a cliff. * - **Parameterized, always.** No value from the query string is ever * concatenated into SQL. Dimensions and metrics are looked up in the tables * below and rejected with a 400 if they are not there, so even the column @@ -88,9 +92,7 @@ export interface Range { /** * The range every endpoint shares. Absent params default to the last 30 days - * ending today so a bare `curl /api/summary` still answers something sensible; - * the dashboard itself always sends both, anchored on /api/meta's latest day so - * no chart ends on a day the nightly rollup has not written yet. + * ending today, which is also what the dashboard's presets send. */ function parseRange(url: URL): Range | ApiResult { const rawTo = url.searchParams.get('to'); @@ -212,43 +214,92 @@ interface MetaRow { earliest_active_day: string | null; earliest_raw_day: string | null; latest_raw_day: string | null; + machines_yesterday: number | null; } /** - * What the picker anchors on. The dashboard asks for this first and ends every - * default range on `latest_day`, because the nightly cron has not rolled up - * today yet — anchoring on the wall clock would put a phantom zero on the right - * edge of every line chart. + * How fresh the data is, and whether either writer has stopped. + * + * The page's ranges end on today; this is what tells it how much of that range + * each kind of number actually covers. Machine counts (`machine_days`, + * `machine_first_seen`) are written by the ingest worker as events arrive, so they + * run through today. Everything rolled up by the nightly cron runs through + * `latest_rollup_day` — normally yesterday. + * + * Every lookup here is a single min() or max() per subquery, on purpose: SQLite only + * answers those from an index when the aggregate stands alone, and + * `SELECT min(day), max(day) FROM events` is a full scan of the biggest table. + * + * The two flags exist because both writers have failed silently before: from + * 2026-08-11 almost no event was stored and no rollup ran, nothing errored where + * anyone would see it, and the dashboard kept presenting Aug 9 as "the latest day" + * for seven weeks. Each flag is a fact the page states, not a tuned threshold. */ async function meta(env: Env): Promise { + const now = Date.now(); + const today = utcDay(now); + const yesterday = utcDay(now - DAY_MS); const row = await env.DB.prepare( `SELECT (SELECT max(day) FROM daily_event_counts) AS latest_rollup_day, (SELECT min(day) FROM daily_event_counts) AS earliest_rollup_day, (SELECT max(day) FROM machine_days) AS latest_active_day, (SELECT min(day) FROM machine_days) AS earliest_active_day, (SELECT min(day) FROM events) AS earliest_raw_day, - (SELECT max(day) FROM events) AS latest_raw_day`, - ).first(); + (SELECT max(day) FROM events) AS latest_raw_day, + (SELECT count(*) FROM machine_days WHERE day = ?) AS machines_yesterday`, + ) + .bind(yesterday) + .first(); + + const latestRollup = row?.latest_rollup_day ?? null; + const latestRaw = row?.latest_raw_day ?? null; + const latestActive = row?.latest_active_day ?? null; + + // Nothing at all since before yesterday. (Client clocks may run a few minutes + // ahead, so the latest day can be tomorrow — that is fresh, not stale.) + const ingestStalled = latestRaw !== null && latestRaw < yesterday; + // The 00:30 UTC run rolls up yesterday, so the day before that must always be in + // by now. Only "behind" if there was activity after the last rolled-up day — a day + // nobody used codegraph would be a silent rollup, not a missed one. + const rollupBehind = + latestActive !== null && + (latestRollup === null || (latestRollup < addDays(today, -2) && latestActive > latestRollup)); - const latest = row?.latest_rollup_day ?? row?.latest_active_day ?? null; - const earliest = row?.earliest_rollup_day ?? row?.earliest_active_day ?? null; return { body: { - latest_day: latest, - earliest_day: earliest, - latest_rollup_day: row?.latest_rollup_day ?? null, - latest_active_day: row?.latest_active_day ?? null, - /** Below this day the activation funnel is blind — raw events are purged. */ + today, + /** The last day the nightly rollup covers. Kept under its old name for callers. */ + latest_day: latestRollup ?? latestActive, + earliest_day: row?.earliest_rollup_day ?? row?.earliest_active_day ?? null, + latest_rollup_day: latestRollup, + latest_active_day: latestActive, earliest_raw_day: row?.earliest_raw_day ?? null, - latest_raw_day: row?.latest_raw_day ?? null, + latest_raw_day: latestRaw, + machines_yesterday: row?.machines_yesterday ?? 0, + rollup_behind: rollupBehind, + ingest_stalled: ingestStalled, max_range_days: MAX_RANGE_DAYS, retention_days: RETENTION_DAYS, - generated_at: new Date().toISOString(), + generated_at: new Date(now).toISOString(), }, - cacheControl: CACHE_CONTROL, + // Short: this is the staleness check, and it is one indexed lookup per field. + cacheControl: 'private, max-age=60', }; } +/** + * The last day the nightly rollup has written. Rolled-up series stop here and go + * null after it — "not counted yet", which a chart draws as a gap, rather than a zero + * it would draw as a cliff. `daily_event_counts` is keyed (day, event), so this is + * one step down its primary key. + */ +const COVERAGE_SQL = 'SELECT max(day) AS day FROM daily_event_counts'; + +/** Like densify(), but days past `coveredThrough` are null: not counted yet, not zero. */ +function densifyCovered(labels: string[], byDay: Map, coveredThrough: string | null): (number | null)[] { + return labels.map((day) => (coveredThrough === null || day > coveredThrough ? null : (byDay.get(day) ?? 0))); +} + // --------------------------------------------------------------------------- // /api/summary — the big numbers // --------------------------------------------------------------------------- @@ -314,12 +365,19 @@ interface SeriesSpec { labels: [string] | [string, string]; sql: string; binds: (range: Range) => (string | number)[]; + /** + * Written by the ingest worker as events arrive rather than by the nightly + * rollup, so it runs through today instead of stopping at the rollup's last day. + */ + live?: boolean; } /** - * Every metric here reads a rollup table, so a line stays correct for days whose - * raw events are long gone. Each query returns (day, a[, b]) and is densified - * against the full day list, because a day with no rows means zero, not a gap. + * Every metric here reads a rollup table (or, if `live`, a table the ingest path + * keeps), so a line stays correct for days whose raw events are long gone. Each + * query returns (day, a[, b]) and is densified against the full day list, because + * a covered day with no rows means zero, not a gap. Days the rollup has not + * reached yet are the gap. */ const SERIES: Record = { installs_uninstalls: { @@ -341,6 +399,7 @@ const SERIES: Record = { WHERE first_day BETWEEN ? AND ? GROUP BY first_day`, binds: (r) => [r.from, r.to], + live: true, }, production_users: { title: 'Daily production users', @@ -379,9 +438,12 @@ async function timeseries(env: Env, url: URL, range: Range): Promise return fail(`unknown metric — one of: ${[...Object.keys(SERIES), 'duration_buckets'].join(', ')}`); } - const { results } = await env.DB.prepare(spec.sql) - .bind(...spec.binds(range)) - .all(); + const batch = await env.DB.batch([ + env.DB.prepare(spec.sql).bind(...spec.binds(range)), + env.DB.prepare(COVERAGE_SQL), + ]); + const results = rowsOf(batch[0]); + const coveredThrough = spec.live ? null : (firstOf<{ day: string | null }>(batch[1])?.day ?? null); const labels = dayList(range); const a = new Map(); @@ -391,19 +453,23 @@ async function timeseries(env: Env, url: URL, range: Range): Promise b.set(row.day, row.b ?? 0); } - const datasets = [{ label: spec.labels[0], data: densify(labels, a) }]; - if (spec.labels.length === 2) datasets.push({ label: spec.labels[1], data: densify(labels, b) }); + const fill = (byDay: Map): (number | null)[] => + spec.live ? densify(labels, byDay) : densifyCovered(labels, byDay, coveredThrough); + const datasets = [{ label: spec.labels[0], data: fill(a) }]; + if (spec.labels.length === 2) datasets.push({ label: spec.labels[1], data: fill(b) }); return { body: { range, metric, title: spec.title, + /** Last day with real numbers; null for a live series, which runs through today. */ + covered_through: spec.live ? null : coveredThrough, labels, datasets, rows: labels.map((day, i) => ({ day, - ...Object.fromEntries(datasets.map((d) => [d.label, d.data[i] ?? 0])), + ...Object.fromEntries(datasets.map((d) => [d.label, d.data[i] ?? null])), })), }, cacheControl: CACHE_CONTROL, @@ -412,14 +478,17 @@ async function timeseries(env: Env, url: URL, range: Range): Promise /** "Session run length over time": one series per duration bucket, bucket-ordered. */ async function durationBucketSeries(env: Env, range: Range): Promise { - const { results } = await env.DB.prepare( - `SELECT day, value, sum(count) AS n - FROM daily_dim_counts - WHERE dim = 'duration_bucket' AND event = 'index' AND day BETWEEN ? AND ? - GROUP BY day, value`, - ) - .bind(range.from, range.to) - .all<{ day: string; value: string; n: number }>(); + const batch = await env.DB.batch([ + env.DB.prepare( + `SELECT day, value, sum(count) AS n + FROM daily_dim_counts + WHERE dim = 'duration_bucket' AND event = 'index' AND day BETWEEN ? AND ? + GROUP BY day, value`, + ).bind(range.from, range.to), + env.DB.prepare(COVERAGE_SQL), + ]); + const results = rowsOf<{ day: string; value: string; n: number }>(batch[0]); + const coveredThrough = firstOf<{ day: string | null }>(batch[1])?.day ?? null; const labels = dayList(range); const perBucket = new Map>(); @@ -437,7 +506,7 @@ async function durationBucketSeries(env: Env, range: Range): Promise const datasets = order.map((bucket) => ({ label: bucket, - data: densify(labels, perBucket.get(bucket) ?? new Map()), + data: densifyCovered(labels, perBucket.get(bucket) ?? new Map(), coveredThrough), })); return { @@ -445,11 +514,12 @@ async function durationBucketSeries(env: Env, range: Range): Promise range, metric: 'duration_buckets', title: 'Indexing run length over time', + covered_through: coveredThrough, labels, datasets, rows: labels.map((day, i) => ({ day, - ...Object.fromEntries(datasets.map((d) => [d.label, d.data[i] ?? 0])), + ...Object.fromEntries(datasets.map((d) => [d.label, d.data[i] ?? null])), })), }, cacheControl: CACHE_CONTROL, @@ -600,11 +670,17 @@ interface ActivationRow { * reinstalls does not re-enter the funnel, which is what makes this a * conversion rate rather than an install-event ratio. * - * The LEFT JOIN rides events_machine_day (machine_id, day) and `count(DISTINCT)` - * absorbs the fan-out from a machine that indexed many times. This is the one - * endpoint that reads raw `events`, so it is bounded by the retention window — - * `raw_events_from` tells the caller where the data actually starts, and the UI - * says so rather than drawing a cliff and calling it a drop in conversion. + * "Ran an index" is `machine_first_seen.first_index_day`, which the nightly rollup + * keeps at the earliest day each machine indexed. That makes this a range read over + * one small table. It used to be a join against raw `events` — on production volume + * ~55 s per week of cohorts, which held D1's single query lane long enough to fail + * every other panel waiting behind it. A first index day is never before the first + * day (the ingest path keeps first_day at the machine's earliest event), so "within + * the window" is just `first_index_day <= first_day + window`. + * + * Because first_index_day is rolled up, cohorts after the rollup's last day have no + * conversions counted yet. They are left out of the totals and drawn as gaps — + * counting them would show a drop in conversion that is only a lag. */ async function activation(env: Env, url: URL, range: Range): Promise { const rawWindow = url.searchParams.get('window'); @@ -615,41 +691,40 @@ async function activation(env: Env, url: URL, range: Range): Promise const batch = await env.DB.batch([ env.DB.prepare( - `SELECT f.first_day AS day, - count(DISTINCT f.machine_id) AS installs, - count(DISTINCT CASE WHEN e.machine_id IS NOT NULL THEN f.machine_id END) AS activated - FROM machine_first_seen f - LEFT JOIN events e - ON e.machine_id = f.machine_id - AND e.event = 'index' - AND e.day >= f.first_day - AND e.day <= date(f.first_day, ?) - WHERE f.first_day BETWEEN ? AND ? - GROUP BY f.first_day`, + `SELECT first_day AS day, + count(*) AS installs, + count(CASE WHEN first_index_day <= date(first_day, ?) THEN 1 END) AS activated + FROM machine_first_seen + WHERE first_day BETWEEN ? AND ? + GROUP BY first_day`, // A bound modifier string, built from an integer this function validated — // date() takes the modifier as data, so nothing is concatenated into SQL. ).bind(`+${window} days`, range.from, range.to), - env.DB.prepare(`SELECT min(day) AS raw_from, max(day) AS raw_to FROM events`), + env.DB.prepare(COVERAGE_SQL), ]); const rows = rowsOf(batch[0]); const byDay = new Map(rows.map((r) => [r.day, r])); const labels = dayList(range); - const installs = rows.reduce((n, r) => n + (r.installs ?? 0), 0); - const activated = rows.reduce((n, r) => n + (r.activated ?? 0), 0); - // Cohorts younger than the window have not finished converting yet, so their // rate is a floor, not a result. Marked rather than dropped: hiding the last // week of a conversion chart is its own kind of lie. - const boundsRow = firstOf<{ raw_from: string | null; raw_to: string | null }>(batch[1]); - const latestRaw = boundsRow?.raw_to ?? utcDay(Date.now()); - const incompleteFrom = addDays(latestRaw, -(window - 1)); + const coveredThrough = firstOf<{ day: string | null }>(batch[1])?.day ?? null; + const incompleteFrom = addDays(coveredThrough ?? utcDay(Date.now()), -(window - 1)); + const counted = (day: string): boolean => coveredThrough !== null && day <= coveredThrough; + let installs = 0; + let activated = 0; const detail = labels.map((day) => { const row = byDay.get(day); const dayInstalls = row?.installs ?? 0; + if (!counted(day)) { + return { day, installs: dayInstalls, activated: null, rate: null, complete: false }; + } const dayActivated = row?.activated ?? 0; + installs += dayInstalls; + activated += dayActivated; return { day, installs: dayInstalls, @@ -669,8 +744,8 @@ async function activation(env: Env, url: URL, range: Range): Promise rate: installs > 0 ? activated / installs : null, /** Cohorts from this day on have not had the full window to convert. */ incomplete_from: incompleteFrom, - /** Raw events start here; a range reaching further back under-counts. */ - raw_events_from: boundsRow?.raw_from ?? null, + /** The last cohort day in the totals: later cohorts have no conversions counted yet. */ + covered_through: coveredThrough, labels, datasets: [ { diff --git a/telemetry-worker/README.md b/telemetry-worker/README.md index 204074c914..296ec2251d 100644 --- a/telemetry-worker/README.md +++ b/telemetry-worker/README.md @@ -31,15 +31,19 @@ Telemetry is stored in the `codegraph-telemetry` D1 database on the same account events are written in a single `batch()` (one implicit transaction) under `ctx.waitUntil`, so the write is off the response path. It is deliberately **fail-silent**: a D1 error is logged to Workers Logs (counts only, never the payload) and the client still gets its `204`, -because clients never retry — losing a datapoint beats losing availability. Alongside the -raw rows, the worker upserts `machine_days` and `machine_first_seen`; when a batch is emptied -by the allowlist, nothing at all is written, so those tables only ever describe stored events. - -The complete schema is [`migrations/0001_init.sql`](migrations/0001_init.sql) — +because clients never retry — losing a datapoint beats losing availability. Lifecycle events +(`install`, `index`, `uninstall`) are one `events` row each. `usage_rollup` counters are +**added** into `usage_daily`, one row per machine × day × tool: clients upload a counter per +process rather than per day, and storing a row per upload is what filled the database in +August 2026. Alongside those, the worker upserts `machine_days` and `machine_first_seen`; when +a batch is emptied by the allowlist, nothing at all is written, so those tables only ever +describe stored events. + +The complete schema is [`migrations/`](migrations/) (applied in order) — checked in for the same reason this worker's source is public: it is the entire list of what gets kept, with a comment on every column and on which dashboard chart each rollup table -serves. Shape: raw sanitized `events`, `daily_*` rollups recomputed nightly, and -`machine_days` / `machine_first_seen` for retention cohorts. The dashboard reads rollups; raw +serves. Shape: raw sanitized `events`, `usage_daily` counters, `daily_*` rollups recomputed +nightly, and `machine_days` / `machine_first_seen` for retention cohorts. The dashboard reads rollups; raw events exist for drill-down and are purged past the retention window. ```bash @@ -56,29 +60,47 @@ edit to a migration that has been applied. Volume, at ~97k accepted POSTs/day: ≈30M D1 row writes/month against the 50M included on Workers Paid, plus roughly as much again once the purge reaches steady state — a delete bills like an insert, and at steady state every row written is eventually deleted, so budget ≈48M. -D1 bills a row write per index touched on top of the table row, which is why `events` carries -only two indexes; dropping `events_machine_day` is the first lever if that gets tight. Storage -is the other constraint, and it is what sets the window: raw events grow ≈74 MB/day, so 90 days -lands at ≈6.7 GB against D1's 10 GB per-database cap, while 180 days would exceed it. Full +D1 bills a row write per index touched on top of the table row, which is why `events` now +carries one index (`events_machine_day` is dropped by `0004`, once the legacy usage rows are +folded out — on the full table the drop runs past D1's per-query limit). Storage is the other +constraint, and it is what sets the window: the estimate here was ≈74 MB/day of raw events +(≈6.7 GB at 90 days against D1's 10 GB per-database cap), but it assumed one usage row per +machine × day × tool. Clients actually sent one per process — ≈3.8M rows a day by early +August — and the database hit the cap on 2026-08-11. `usage_daily` bounds that by +construction; re-measure with `npx wrangler d1 info codegraph-telemetry` before widening the +window. Full arithmetic and the remaining levers are in the migration's footer comment. +**A full database fails quietly.** D1 caps a database at 10 GB on Workers Paid (500 MB on +Free), and upgrading cannot raise it. Once full, nearly every write fails with +`D1_ERROR: Exceeded maximum DB size`: ingest stops storing events (the client never retries, +so they are lost) and the nightly rollup stops too, while deletes and reads still work, so +nothing looks broken from the outside. That ran from 2026-08-11 to October 2026. +`npx wrangler d1 info codegraph-telemetry` shows the current size. + ## Rollups & retention (nightly cron) `src/rollup.ts` runs on a Cron Trigger at **00:30 UTC** and does two things. -**Rolls up** the day that just ended into `daily_machines`, `daily_event_counts` and -`daily_dim_counts`, then re-runs the two days before it — offline clients ship completed-day -rollups late, so a day keeps growing after it ends. The aggregation is one +**Rolls up** the day that just ended into `daily_machines`, `daily_event_counts`, +`daily_dim_counts` and `machine_first_seen.first_index_day`, then re-runs the two days before +it — offline clients ship completed-day rollups late, so a day keeps growing after it ends. +It then **catches up** on any earlier day that saw activity but never got a rollup (a +`machine_days` day with no `daily_machines` row — a night the run failed or the database +refused writes), newest first, up to 31 a night. An outage heals on the first good night +instead of leaving a hole someone has to notice. The aggregation is one `INSERT … SELECT … ON CONFLICT DO UPDATE` per table or dimension, so it happens inside D1 and no event row crosses the wire. Every write overwrites the recomputed value rather than adding to it: **re-running a day is a no-op, never a double count.** Two things the SQL is careful -about — a `usage_rollup` row is a counter the client pre-aggregated, so its `count` prop is -summed rather than the rows counted; and `index.languages` / `install.targets` are unnested -with `json_each`, one row per element. Adding a breakdown is a line in `ROLLUP_STATEMENTS`, +about — usage figures are sums of `usage_daily.count` (calls), not row counts; and +`index.languages` / `install.targets` are unnested with `json_each`, one row per element. +Before rolling up a day it folds any legacy `usage_rollup` rows still in `events` (how usage +was stored before `0003`) into `usage_daily`, 50k rows per transaction, so re-running the +rollup over old days is the whole migration. Adding a breakdown is a line in `ROLLUP_STATEMENTS`, never a migration — that is what the generic `(dim, value)` shape buys. -**Purges** raw `events` older than `RETENTION_DAYS` (90, a var in `wrangler.jsonc`) in bounded -`DELETE` batches, and logs one line of counts. `machine_days` and `machine_first_seen` are +**Purges** raw `events` and `usage_daily` rows older than `RETENTION_DAYS` (90, a var in +`wrangler.jsonc`) in bounded `DELETE` batches, and logs one line of counts. `machine_days` and `machine_first_seen` are never purged — retention cohorts need the full history and they are two orders of magnitude smaller. Rollups are kept forever, so shortening the window costs ad-hoc drill-back, never a chart. @@ -98,6 +120,37 @@ retention window, where it would delete rows and then find no events to rebuild the response says which days it refused. Keep manual ranges to a few days at production volume; each day is a full scan of that day's events, and the request has a wall-clock budget. +### Backfilling first_index_day + +`first_index_day` (migration `0002`) is what the dashboard's activation funnel reads, and only +the rollup writes it. Days rolled up before the migration left it NULL, and days stored before +`0003` still hold their usage as one `events` row per upload. Re-running the rollup over every +day that still has raw events fixes both — it folds that day's legacy usage rows into +`usage_daily`, sets `first_index_day`, and recomputes the day's rollups. It is idempotent, so +overlapping or repeating a range is harmless. A day with millions of legacy rows takes several +minutes, so go a day at a time: + +```bash +# One day per call, from the earliest raw event (2026-07-04) through yesterday. +for day in $(node -e 'for (let t = Date.parse("2026-07-04"); t < Date.now() - 864e5; t += 864e5) + console.log(new Date(t).toISOString().slice(0, 10))'); do + curl -sS -X POST -H "x-admin-token: $ADMIN_TOKEN" \ + "https://telemetry.getcodegraph.com/admin/rollup?day=$day"; echo +done +``` + +Move the start to the earliest day `events` still holds (`/api/meta` on the dashboard +reports it as `earliest_raw_day`). A machine whose index events were purged before the backfill stays NULL +and counts as not activated — so run it before the retention window passes those days. + +### When the dashboard says ingest stalled or the rollup is behind + +Both banners mean a writer stopped. In order: `npx wrangler tail codegraph-telemetry` (look +for `d1 write failed` / `rollup day failed` and the error text), then +`npx wrangler d1 info codegraph-telemetry` against the plan's database cap (see Storage +above). Once writes succeed again the next nightly run catches the rollup up on its own; the +events that arrived while writes were failing are gone. + ## Deploy Prereqs: the `getcodegraph.com` zone on the deploying Cloudflare account (the custom diff --git a/telemetry-worker/migrations/0002_first_index_day.sql b/telemetry-worker/migrations/0002_first_index_day.sql new file mode 100644 index 0000000000..37b2910a1f --- /dev/null +++ b/telemetry-worker/migrations/0002_first_index_day.sql @@ -0,0 +1,19 @@ +-- codegraph telemetry — the first day each machine ran an index. +-- +-- Serves the install → first-index activation funnel. Until this column, the +-- dashboard answered that funnel by joining every cohort machine against raw +-- `events`, which on production volume took ~55 s for ONE week of cohorts — long +-- enough to stall D1 (one query at a time per database) and fail every other +-- panel queued behind it. With the first index day stored next to the first-seen +-- day, the funnel is a range read over `machine_first_seen` alone. +-- +-- It is also what lets the funnel outlive the raw-event retention purge, like +-- every other panel: the nightly cron fills it from `events` while they exist. +-- +-- Written by the nightly rollup (telemetry-worker/src/rollup.ts), never by the +-- ingest path, and only ever lowered: NULL means "has not indexed (yet)". +-- Existing rows start NULL — backfill them by re-running the rollup over the +-- retained days (see "Backfilling first_index_day" in telemetry-worker/README.md). +-- +-- ADD COLUMN with no default is a schema-only change in SQLite: no row is rewritten. +ALTER TABLE machine_first_seen ADD COLUMN first_index_day TEXT; diff --git a/telemetry-worker/migrations/0003_usage_daily.sql b/telemetry-worker/migrations/0003_usage_daily.sql new file mode 100644 index 0000000000..3cfaa07357 --- /dev/null +++ b/telemetry-worker/migrations/0003_usage_daily.sql @@ -0,0 +1,35 @@ +-- codegraph telemetry — usage counters, one row per machine × day × tool. +-- +-- `usage_rollup` events were stored one row per event in `events`. The client was +-- meant to send one pre-aggregated counter per machine × day × tool, but it only +-- aggregates within a single process: every short-lived codegraph process (each +-- `serve` launch, CLI command or prompt hook) uploads its own `count: 1` line. By +-- 2026-08-10 that was ~3.8M rows a day, 98% of the database, and the database hit +-- D1's 10 GB cap — from 2026-08-11 nearly every write failed and was lost. +-- +-- This table makes storage independent of how a client batches: the ingest worker +-- ADDS each counter into the row for its key instead of inserting a new row, so a +-- machine that uploads the same counter 80,000 times still owns one row. Same +-- fields as the `usage_rollup` props plus the envelope the rollup breaks down by; +-- nothing new is collected. +-- +-- Every key column is NOT NULL with '' for "absent": a primary key treats NULLs as +-- distinct, which would quietly turn the upsert back into an insert. The envelope +-- columns are in the key so a machine that upgrades mid-day keeps exact per-version +-- counts. Purged at the same retention window as `events`. +CREATE TABLE usage_daily ( + day TEXT NOT NULL, -- UTC YYYY-MM-DD the counts belong to + machine_id TEXT NOT NULL, -- random UUIDv4, client-minted + kind TEXT NOT NULL, -- mcp_tool | cli_command + name TEXT NOT NULL, -- tool or command name + client_name TEXT NOT NULL DEFAULT '', + client_version TEXT NOT NULL DEFAULT '', + codegraph_version TEXT NOT NULL DEFAULT '', + os TEXT NOT NULL DEFAULT '', + arch TEXT NOT NULL DEFAULT '', + node_major TEXT NOT NULL DEFAULT '', + count INTEGER NOT NULL DEFAULT 0, -- calls + error_count INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (day, machine_id, kind, name, client_name, client_version, + codegraph_version, os, arch, node_major) +) WITHOUT ROWID; diff --git a/telemetry-worker/migrations/0004_drop_events_machine_day.sql b/telemetry-worker/migrations/0004_drop_events_machine_day.sql new file mode 100644 index 0000000000..3a637e8523 --- /dev/null +++ b/telemetry-worker/migrations/0004_drop_events_machine_day.sql @@ -0,0 +1,11 @@ +-- codegraph telemetry — drop the per-machine index on `events`. +-- +-- Its only query reader was the dashboard's old activation join, which now reads +-- machine_first_seen.first_index_day. Dropping it saves a row write per stored event, +-- and it was the first storage lever the 0001 footer names. +-- +-- A separate migration, applied AFTER the legacy usage rows are folded out of +-- `events` (see "Backfilling first_index_day" in README.md): with ~30M rows in the +-- table, dropping the index ran past D1's per-query time limit and was rolled back. +-- On the folded table it is quick. +DROP INDEX IF EXISTS events_machine_day; diff --git a/telemetry-worker/scripts/backfill-rollup.sh b/telemetry-worker/scripts/backfill-rollup.sh new file mode 100755 index 0000000000..4f9e24bfbe --- /dev/null +++ b/telemetry-worker/scripts/backfill-rollup.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +# Re-runs the nightly rollup over a range of days against the DEPLOYED ingest worker, +# one day per request. Each day: legacy usage rows still in `events` are folded into +# `usage_daily`, `first_index_day` is set, and the day's rollups are recomputed. All of +# it is idempotent, so a range can be repeated or overlapped safely. +# +# ./scripts/backfill-rollup.sh 2026-07-04 2026-10-02 [token-file] +# +# The admin token comes from the token file when one is given, else from $ADMIN_TOKEN. +# Stops at the first day that does not answer 200, so a rerun can start from there. +set -euo pipefail + +FROM=${1:?usage: backfill-rollup.sh FROM_DAY TO_DAY [TOKEN_FILE]} +TO=${2:?usage: backfill-rollup.sh FROM_DAY TO_DAY [TOKEN_FILE]} +if [ -n "${3:-}" ]; then ADMIN_TOKEN=$(tr -d '\n' < "$3"); fi +: "${ADMIN_TOKEN:?set ADMIN_TOKEN or pass a token file}" +BASE="${ROLLUP_BASE:-https://telemetry.getcodegraph.com}" + +days=$(node -e ' +const [from, to] = process.argv.slice(1); +for (let t = Date.parse(`${from}T00:00:00Z`); t <= Date.parse(`${to}T00:00:00Z`); t += 864e5) + console.log(new Date(t).toISOString().slice(0, 10));' "$FROM" "$TO") + +for day in $days; do + started=$(date +%s) + # A day with millions of legacy rows takes minutes; never let one hang forever. + response=$(curl -sS --max-time 1800 -X POST -H "x-admin-token: $ADMIN_TOKEN" \ + -w $'\n%{http_code}' "$BASE/admin/rollup?day=$day") || { echo "$day request failed"; exit 1; } + code=${response##*$'\n'} + body=${response%$'\n'*} + printf '%s %s %ss %s\n' "$day" "$code" "$(( $(date +%s) - started ))" "$body" + [ "$code" = 200 ] || { echo "stopping at $day"; exit 1; } +done diff --git a/telemetry-worker/scripts/smoke-cutover.sh b/telemetry-worker/scripts/smoke-cutover.sh index 47aed75370..f0dff12443 100755 --- a/telemetry-worker/scripts/smoke-cutover.sh +++ b/telemetry-worker/scripts/smoke-cutover.sh @@ -233,11 +233,13 @@ ts "indexing activity" indexing_activity '[3]' '[3]' ts "tool calls (sums the prop)" tool_calls '[20]' '[2]' MET=$(api "meta") -is "meta anchors on the rolled-up day" "$DAY" "$(jget "$MET" latest_day)" +is "meta reports the rolled-up day" "$DAY" "$(jget "$MET" latest_day)" is "meta reports the rollup ran" "$DAY" "$(jget "$MET" latest_rollup_day)" -# The funnel is the one panel that reads RAW events rather than a rollup, so it is -# also the one the retention purge can blind — worth pinning that it works today. +# The funnel reads machine_first_seen.first_index_day, which only the nightly +# rollup writes (from raw `index` events). This is the seam that pins it: ingest +# writes the events, the rollup sets the column, the dashboard reads it. If the +# rollup stopped setting it, "activated" here would read 0. # # Its denominator is FIRST-SEEN MACHINES, not `install` events (api.ts: "a machine # that reinstalls does not re-enter the funnel"). m3 is the discriminator: it never @@ -247,7 +249,7 @@ ACT=$(api "activation?$RANGE&window=1") is "funnel counts new machines, not install events" 3 "$(jget "$ACT" installs)" is "all three indexed within the window" 3 "$(jget "$ACT" activated)" is "nobody dropped out" 0 "$(jget "$ACT" dropped)" -is "raw-event floor is reported to the caller" "$DAY" "$(jget "$ACT" raw_events_from)" +is "cohorts are counted through the rolled-up day" "$DAY" "$(jget "$ACT" covered_through)" is "retention endpoint answers" 200 \ "$(curl -s -o /dev/null -w '%{http_code}' -b "$JAR" "$DASH/api/retention?$RANGE")" diff --git a/telemetry-worker/scripts/smoke-ingest.sh b/telemetry-worker/scripts/smoke-ingest.sh index f9dd4795be..458123443a 100755 --- a/telemetry-worker/scripts/smoke-ingest.sh +++ b/telemetry-worker/scripts/smoke-ingest.sh @@ -37,6 +37,9 @@ q() { # --------------------------------------------------------------------------- # Boot # --------------------------------------------------------------------------- +echo "applying migrations to local D1" +npx wrangler d1 migrations apply "$DB" --local >/dev/null 2>&1 + echo "booting wrangler dev on :$PORT" npx wrangler dev --port "$PORT" >/tmp/cg-smoke-ingest.log 2>&1 & DEV_PID=$! @@ -83,26 +86,31 @@ is "oversized body → 413" 413 "$(post "$BIG")" echo echo "ingest" -M_OK=$(uuid); M_DROP=$(uuid); M_CI=$(uuid); M_BACK=$(uuid) +M_OK=$(uuid); M_DROP=$(uuid); M_CI=$(uuid); M_BACK=$(uuid); M_USE=$(uuid) TODAY=$(date -u +%F) +# Client timestamps older than 30 days are rejected at ingest, so the dated events +# here are relative to today — fixed dates rot into "too old" a month after writing. +day_ago() { node -e 'console.log(new Date(Date.now()-process.argv[1]*864e5).toISOString().slice(0,10))' "$1"; } +RECENT_DAY=$(day_ago 2) +BACK_DAY=$(day_ago 9) # Three valid events + one unknown event + unknown/malformed props that must be stripped. is "valid batch → 204" 204 "$(post "$(node -e ' -const [m] = process.argv.slice(1); +const [m, day] = process.argv.slice(1); process.stdout.write(JSON.stringify({ machine_id: m, codegraph_version: "1.5.0", os: "darwin", arch: "arm64", node_major: 22, ci: false, schema_version: 1, secret_field: "must not be stored", events: [ - { event: "install", ts: "2026-07-27T10:00:00Z", + { event: "install", ts: `${day}T10:00:00Z`, props: { scope: "local", kind: "fresh", targets: ["claude", "cursor"], nope: "strip me" } }, - { event: "index", ts: "2026-07-27T10:01:00Z", + { event: "index", ts: `${day}T10:01:00Z`, props: { languages: ["typescript"], file_count_bucket: "100-1k", duration_bucket: "bogus-bucket", repo_path: "/Users/someone/secret" } }, { event: "usage_rollup", props: { kind: "mcp_tool", name: "codegraph_explore", count: 12, client_name: "Claude Code" } }, { event: "not_an_event", props: { count: 1 } }, ], -}));' "$M_OK")")" +}));' "$M_OK" "$RECENT_DAY")")" # Nothing survives the allowlist: unknown event + usage_rollup missing required props. is "all-dropped batch → 204" 204 "$(post "$(node -e ' @@ -113,6 +121,26 @@ process.stdout.write(JSON.stringify({ machine_id: m, os: "linux", events: [ { event: "install", props: { scope: "local" } }, ]}));' "$M_DROP")")" +# Usage counters ADD into one row per machine × day × tool — what old clients send +# is the same `count: 1` line once per process, many times over. Two batches: the +# first repeats one counter three times (one of them an error) beside a second tool, +# the second sends the first counter again. +USAGE_A=$(node -e ' +const [m] = process.argv.slice(1); +const explore = (errors) => ({ event: "usage_rollup", + props: { kind: "mcp_tool", name: "codegraph_explore", count: 1, error_count: errors, client_name: "Claude Code" } }); +process.stdout.write(JSON.stringify({ machine_id: m, os: "darwin", events: [ + explore(0), explore(1), explore(0), + { event: "usage_rollup", props: { kind: "cli_command", name: "index", count: 2, error_count: 0 } }, +]}));' "$M_USE") +USAGE_B=$(node -e ' +const [m] = process.argv.slice(1); +process.stdout.write(JSON.stringify({ machine_id: m, os: "darwin", events: [ + { event: "usage_rollup", props: { kind: "mcp_tool", name: "codegraph_explore", count: 1, error_count: 0, client_name: "Claude Code" } }, +]}));' "$M_USE") +is "usage batch → 204" 204 "$(post "$USAGE_A")" +is "the same counter again → 204" 204 "$(post "$USAGE_B")" + # NOTE: build every body into a variable first. Escaped quotes nested inside # "$(post "…\"…\"…")" break out of the quoting context and get brace-expanded. index_batch() { # [ci] [ts] @@ -130,8 +158,8 @@ is "ci batch → 204" 204 "$(post "$CI_ON")" is "same machine, non-ci → 204" 204 "$(post "$CI_OFF")" # A late offline buffer arriving second must move first_day EARLIER, never later. -RECENT=$(index_batch "$M_BACK" "" 2026-07-27T09:00:00Z) -BACKDATED=$(index_batch "$M_BACK" "" 2026-07-20T09:00:00Z) +RECENT=$(index_batch "$M_BACK" "" "${RECENT_DAY}T09:00:00Z") +BACKDATED=$(index_batch "$M_BACK" "" "${BACK_DAY}T09:00:00Z") is "recent batch → 204" 204 "$(post "$RECENT")" is "backdated batch → 204" 204 "$(post "$BACKDATED")" @@ -145,17 +173,22 @@ sleep 1 # and let miniflare release the local sqlite file echo echo "stored rows" -is "3 of 4 events stored (unknown dropped)" 3 "$(q "select count(*) from events where machine_id='$M_OK'")" +is "lifecycle events land in events (usage and the unknown event do not)" 2 \ + "$(q "select count(*) from events where machine_id='$M_OK'")" +is "the usage counter lands in usage_daily" "codegraph_explore|mcp_tool|12|0|Claude Code" \ + "$(q "select name||'|'||kind||'|'||count||'|'||error_count||'|'||client_name from usage_daily where machine_id='$M_OK'")" +is "…carrying the batch's envelope" "1.5.0|darwin|arm64|22" \ + "$(q "select codegraph_version||'|'||os||'|'||arch||'|'||node_major from usage_daily where machine_id='$M_OK'")" is "all-dropped batch stored nothing" 0 "$(q "select count(*) from events where machine_id='$M_DROP'")" is "…and no machine_days row for it" 0 "$(q "select count(*) from machine_days where machine_id='$M_DROP'")" is "envelope columns land in their own columns" "darwin|arm64|22|0|1.5.0" \ "$(q "select os||'|'||arch||'|'||node_major||'|'||ci||'|'||codegraph_version from events where machine_id='$M_OK' limit 1")" -is "day derived from the client ts" "2026-07-27" \ +is "day derived from the client ts" "$RECENT_DAY" \ "$(q "select day from events where machine_id='$M_OK' and event='install'")" is "day falls back to received_at when ts is absent" "$TODAY" \ - "$(q "select day from events where machine_id='$M_OK' and event='usage_rollup'")" + "$(q "select day from usage_daily where machine_id='$M_OK'")" is "ts is NULL when the client sent none" 1 \ - "$(q "select ts is null from events where machine_id='$M_OK' and event='usage_rollup'")" + "$(q "select ts is null from events where machine_id='$M_CI' limit 1")" is "allowlisted props stored" "local|fresh|2" \ "$(q "select json_extract(props,'\$.scope')||'|'||json_extract(props,'\$.kind')||'|'||json_array_length(props,'\$.targets') from events where machine_id='$M_OK' and event='install'")" is "unknown prop stripped" 0 \ @@ -167,7 +200,7 @@ is "path-shaped prop stripped" 0 \ is "unknown envelope field stored nowhere" 0 \ "$(q "select count(*) from events where props like '%must not be stored%'")" -# The valid batch mixes ts-dated events (2026-07-27) with an undated rollup (today), +# The valid batch mixes ts-dated events ($RECENT_DAY) with an undated rollup (today), # so it legitimately spans two days and must produce a machine_days row for each. is "machine_days: one row per distinct day in the batch" 2 \ "$(q "select count(*) from machine_days where machine_id='$M_OK'")" @@ -175,13 +208,24 @@ is "machine_days: non-ci machine is production" 1 \ "$(q "select min(prod) from machine_days where machine_id='$M_OK'")" is "machine_days: a later non-ci batch flips the day to production" 1 \ "$(q "select prod from machine_days where machine_id='$M_CI'")" -is "machine_days: each backdated batch gets its own day" "2026-07-20,2026-07-27" \ +is "machine_days: each backdated batch gets its own day" "$BACK_DAY,$RECENT_DAY" \ "$(q "select group_concat(day) from (select day from machine_days where machine_id='$M_BACK' order by day)")" -is "machine_first_seen recorded" "2026-07-27" "$(q "select first_day from machine_first_seen where machine_id='$M_OK'")" -is "machine_first_seen only moves earlier" "2026-07-20" \ +is "machine_first_seen recorded" "$RECENT_DAY" "$(q "select first_day from machine_first_seen where machine_id='$M_OK'")" +is "machine_first_seen only moves earlier" "$BACK_DAY" \ "$(q "select first_day from machine_first_seen where machine_id='$M_BACK'")" +echo +echo "usage counters add up instead of piling up" +is "no usage row ever lands in events" 0 "$(q "select count(*) from events where event='usage_rollup' and machine_id='$M_USE'")" +is "one row per tool, however many uploads" 2 "$(q "select count(*) from usage_daily where machine_id='$M_USE'")" +is "repeats within a batch and across batches are summed" "4/1" \ + "$(q "select count||'/'||error_count from usage_daily where machine_id='$M_USE' and name='codegraph_explore'")" +is "…and a different tool keeps its own count" "2/0" \ + "$(q "select count||'/'||error_count from usage_daily where machine_id='$M_USE' and name='index'")" +is "an absent client name is stored as empty, not NULL (it is part of the key)" "" \ + "$(q "select client_name from usage_daily where machine_id='$M_USE' and name='index'")" + echo echo "$pass passed, $fail failed" [ "$fail" -eq 0 ] diff --git a/telemetry-worker/scripts/smoke-rollup.sh b/telemetry-worker/scripts/smoke-rollup.sh index c0bfbef0b5..294b4378f8 100755 --- a/telemetry-worker/scripts/smoke-rollup.sh +++ b/telemetry-worker/scripts/smoke-rollup.sh @@ -15,6 +15,10 @@ # raw events are already purged # * the purge deletes only rows past the window, and leaves machine_days / # machine_first_seen alone +# * each machine's first_index_day is its earliest index, and survives the purge +# * the cron catches up on a day that saw activity but was never rolled up +# * usage comes from usage_daily; a legacy usage row still in `events` is folded +# into it (counts added, row deleted) before the day is rolled up # * /admin/rollup does not exist without ADMIN_TOKEN, and rejects a wrong one # # Re-runnable: it wipes its own synthetic days first, and they are chosen to sit @@ -39,10 +43,13 @@ is() { [ "$2" = "$3" ] && ok "$1" || bad "$1" "$2" "$3"; } day_ago() { node -e 'console.log(new Date(Date.now()-process.argv[1]*864e5).toISOString().slice(0,10))' "$1"; } # Synthetic days. MAIN/RESET sit inside the 90-day retention window but outside the -# cron's 3-day lookback; OLD sits past the window so the purge takes it. +# cron's 3-day lookback; OLD sits past the window so the purge takes it. MISSED is +# never rolled up by hand — the cron has to find it on its own. DAY_MAIN=$(day_ago 40) DAY_RESET=$(day_ago 41) +DAY_MISSED=$(day_ago 30) DAY_OLD=$(day_ago 200) +DAY_ANCIENT=$(day_ago 199) # a usage counter past the window, for the usage purge CUTOFF=$(day_ago 90) # First column of the first row of a query against the LOCAL D1 state. @@ -91,18 +98,18 @@ shutdown() { echo "applying migrations to local D1" npx wrangler d1 migrations apply "$DB" --local >/dev/null 2>&1 -echo "seeding $DAY_MAIN / $DAY_RESET / $DAY_OLD" +echo "seeding $DAY_MAIN / $DAY_RESET / $DAY_MISSED / $DAY_OLD" node -e ' -const [main, reset, old, seedFile] = process.argv.slice(1); +const [main, reset, old, missed, ancient, seedFile] = process.argv.slice(1); const sq = (v) => `'"'"'${String(v).replace(/'"'"'/g, "'"'"''"'"'")}'"'"'`; const M = ["11111111-1111-4111-8111-111111111111", "22222222-2222-4222-8222-222222222222", "33333333-3333-4333-8333-333333333333", "44444444-4444-4444-8444-444444444444", - "99999999-9999-4999-8999-999999999999"]; + "99999999-9999-4999-8999-999999999999", "55555555-5555-4555-8555-555555555555"]; const out = []; // Re-runnable: every table this script touches, scoped to its own synthetic days. -for (const t of ["events", "daily_event_counts", "daily_dim_counts", "daily_machines", "machine_days"]) { - out.push(`DELETE FROM ${t} WHERE day IN (${[main, reset, old].map(sq).join(", ")});`); +for (const t of ["events", "daily_event_counts", "daily_dim_counts", "daily_machines", "machine_days", "usage_daily"]) { + out.push(`DELETE FROM ${t} WHERE day IN (${[main, reset, old, missed, ancient].map(sq).join(", ")});`); } out.push(`DELETE FROM machine_first_seen WHERE machine_id IN (${M.map(sq).join(", ")});`); @@ -110,7 +117,6 @@ out.push(`DELETE FROM machine_first_seen WHERE machine_id IN (${M.map(sq).join(" const rows = [ [main, M[0], "install", "darwin", "arm64", "1.5.0", 22, 0, {targets:["claude","cursor"], scope:"local", kind:"fresh"}], [main, M[0], "index", "darwin", "arm64", "1.5.0", 22, 0, {languages:["typescript","go"], file_count_bucket:"100-1k", duration_bucket:"10-60s"}], - [main, M[0], "usage_rollup", "darwin", "arm64", "1.5.0", 22, 0, {kind:"mcp_tool", name:"codegraph_explore", count:10, error_count:2, client_name:"Claude Code"}], [main, M[1], "index", "darwin", "x64", "1.5.0", 20, 0, {languages:["typescript"], file_count_bucket:"1k-10k", duration_bucket:"10-60s"}], [main, M[1], "usage_rollup", "darwin", "x64", "1.5.0", 20, 0, {kind:"mcp_tool", name:"codegraph_explore", count:5, error_count:0, client_name:"Cursor"}], [main, M[2], "install", "linux", "x64", "1.4.1", 22, 1, {targets:["claude"], scope:"global", kind:"upgrade"}], @@ -118,6 +124,8 @@ const rows = [ [reset, M[3], "index", "darwin", "arm64", "1.5.0", 22, 0, {languages:["python"], file_count_bucket:"<100", duration_bucket:"<10s"}], [old, M[4], "install", "linux", "x64", "1.0.0", 20, 0, {targets:["codex"], scope:"local", kind:"fresh"}], [old, M[4], "index", "linux", "x64", "1.0.0", 20, 0, {languages:["rust"], file_count_bucket:"<100", duration_bucket:"<10s"}], + [missed, M[5], "install", "win32", "x64", "1.5.0", 22, 0, {targets:["claude"], scope:"local", kind:"fresh"}], + [missed, M[5], "index", "win32", "x64", "1.5.0", 22, 0, {languages:["go"], file_count_bucket:"<100", duration_bucket:"<10s"}], ]; for (const [day, m, event, os, arch, version, node, ci, props] of rows) { out.push(`INSERT INTO events (received_at, ts, day, event, machine_id, codegraph_version, os, arch, node_major, ci, schema_version, props) @@ -125,8 +133,19 @@ for (const [day, m, event, os, arch, version, node, ci, props] of rows) { ${sq(version)}, ${sq(os)}, ${sq(arch)}, ${node}, ${ci}, 1, ${sq(JSON.stringify(props))});`); } +// Usage for M[0] arrives the way the ingest path writes it now: added into usage_daily. +// Usage for M[1] (in the rows above) is a LEGACY row in `events`, as stored before +// migrations/0003 — the rollup has to fold it in before counting. +out.push(`INSERT INTO usage_daily (day, machine_id, kind, name, client_name, client_version, + codegraph_version, os, arch, node_major, count, error_count) + VALUES (${[main, M[0], "mcp_tool", "codegraph_explore", "Claude Code", "", + "1.5.0", "darwin", "arm64", "22"].map(sq).join(", ")}, 10, 2);`); +// A usage counter past the retention window: the purge must take it. +out.push(`INSERT INTO usage_daily (day, machine_id, kind, name, count, error_count) + VALUES (${[ancient, M[4], "cli_command", "index"].map(sq).join(", ")}, 3, 0);`); + // What the ingest path would have written alongside those events. -for (const [m, day, prod] of [[M[0], main, 1], [M[1], main, 1], [M[2], main, 0], [M[3], reset, 1], [M[4], old, 1]]) { +for (const [m, day, prod] of [[M[0], main, 1], [M[1], main, 1], [M[2], main, 0], [M[3], reset, 1], [M[4], old, 1], [M[5], missed, 1]]) { out.push(`INSERT INTO machine_days (machine_id, day, prod) VALUES (${sq(m)}, ${sq(day)}, ${prod});`); out.push(`INSERT INTO machine_first_seen (machine_id, first_day) VALUES (${sq(m)}, ${sq(day)}) ON CONFLICT (machine_id) DO UPDATE SET first_day = min(machine_first_seen.first_day, excluded.first_day);`); @@ -137,7 +156,7 @@ out.push(`INSERT INTO daily_dim_counts (day, event, dim, value, count, machines) VALUES (${sq(reset)}, ${sq("index")}, ${sq("obsolete_dim")}, ${sq("stale")}, 99, 99);`); require("fs").writeFileSync(seedFile, out.join("\n")); -' "$DAY_MAIN" "$DAY_RESET" "$DAY_OLD" "$SEED_SQL" +' "$DAY_MAIN" "$DAY_RESET" "$DAY_OLD" "$DAY_MISSED" "$DAY_ANCIENT" "$SEED_SQL" npx wrangler d1 execute "$DB" --local --file "$SEED_SQL" >/dev/null # --------------------------------------------------------------------------- @@ -173,7 +192,8 @@ is "rollup $DAY_MAIN again → 200" 200 "$(roll "day=$DAY_MAIN")" is "rollup $DAY_OLD, whose events are still there → 200" 200 "$(roll "day=$DAY_OLD")" is "rollup $DAY_RESET with reset → 200" 200 "$(roll "day=$DAY_RESET&reset=1")" -# The cron body: rolls up the last three days and purges everything past the window. +# The cron body: rolls up the last three days, catches up $DAY_MISSED (active, never +# rolled up), and purges everything past the window. is "cron trigger → 200" 200 "$(curl -s -o /dev/null -w '%{http_code}' "$BASE/__scheduled?cron=30+0+*+*+*")" sleep 2 @@ -184,6 +204,12 @@ is "…and reports the reset it refused to run" "[\"$DAY_OLD\"]" \ "$(curl -s -X POST -H "x-admin-token: $TOKEN" "$BASE/admin/rollup?day=$DAY_OLD&reset=1" | node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>console.log(JSON.stringify(JSON.parse(s).reset_ignored)))')" +# first_index_day only ever moves earlier. Give M[1] an index day before $DAY_MAIN, +# then re-roll $DAY_MAIN (where it indexed again): the earlier day must stand. +q "update machine_first_seen set first_index_day='$DAY_RESET' + where machine_id='22222222-2222-4222-8222-222222222222'" >/dev/null +is "re-roll $DAY_MAIN over an earlier first index day → 200" 200 "$(roll "day=$DAY_MAIN")" + shutdown # --------------------------------------------------------------------------- @@ -239,10 +265,19 @@ echo "cross-check against the raw events" is "machines matches count(distinct machine_id)" \ "$(q "select count(distinct machine_id) from events where day='$DAY_MAIN' and event='index'")" \ "$(q "select machines from daily_event_counts where day='$DAY_MAIN' and event='index'")" -is "usage count matches sum(props.count)" \ - "$(q "select sum(json_extract(props,'\$.count')) from events where day='$DAY_MAIN' and event='usage_rollup'")" \ +is "usage count matches sum(usage_daily.count)" \ + "$(q "select sum(count) from usage_daily where day='$DAY_MAIN'")" \ "$(q "select count from daily_event_counts where day='$DAY_MAIN' and event='usage_rollup'")" +echo +echo "legacy usage rows" +is "folded out of events" 0 "$(q "select count(*) from events where day='$DAY_MAIN' and event='usage_rollup'")" +is "…into usage_daily, counts and client intact" "5|0|Cursor|darwin|x64|20" \ + "$(q "select count||'|'||error_count||'|'||client_name||'|'||os||'|'||arch||'|'||node_major from usage_daily + where day='$DAY_MAIN' and machine_id='22222222-2222-4222-8222-222222222222'")" +is "…and re-rolling the day adds nothing (no double count)" 15 \ + "$(q "select sum(count) from usage_daily where day='$DAY_MAIN'")" + echo echo "reset" is "?reset=1 drops a rollup row whose dimension no longer exists" 0 \ @@ -255,12 +290,38 @@ echo echo "retention purge" is "raw events past the window are gone" 0 "$(q "select count(*) from events where day='$DAY_OLD'")" is "nothing older than the cutoff survives" 0 "$(q "select count(*) from events where day<'$CUTOFF'")" -is "events inside the window are untouched" 7 "$(q "select count(*) from events where day='$DAY_MAIN'")" +# 6 seeded, minus the legacy usage row the rollup folded into usage_daily. +is "events inside the window are untouched" 5 "$(q "select count(*) from events where day='$DAY_MAIN'")" +is "usage counters past the window are purged" 0 "$(q "select count(*) from usage_daily where day='$DAY_ANCIENT'")" +is "…and inside it they stay" 2 "$(q "select count(*) from usage_daily where day='$DAY_MAIN'")" is "machine_days is NOT purged (retention cohorts need it)" 1 \ "$(q "select count(*) from machine_days where day='$DAY_OLD'")" is "machine_first_seen is NOT purged" "$DAY_OLD" \ "$(q "select first_day from machine_first_seen where machine_id='99999999-9999-4999-8999-999999999999'")" +echo +echo "first_index_day — the activation funnel's input" +is "machines that indexed get that day" "$DAY_MAIN" \ + "$(q "select first_index_day from machine_first_seen where machine_id='11111111-1111-4111-8111-111111111111'")" +is "a machine that never indexed stays NULL" "null" \ + "$(q "select coalesce(first_index_day, 'null') from machine_first_seen where machine_id='33333333-3333-4333-8333-333333333333'")" +is "set on a reset run too" "$DAY_RESET" \ + "$(q "select first_index_day from machine_first_seen where machine_id='44444444-4444-4444-8444-444444444444'")" +is "outlives the purge of the events it came from" "$DAY_OLD" \ + "$(q "select first_index_day from machine_first_seen where machine_id='99999999-9999-4999-8999-999999999999'")" +is "only ever lowered: re-rolling a later index day leaves an earlier one" "$DAY_RESET" \ + "$(q "select first_index_day from machine_first_seen where machine_id='22222222-2222-4222-8222-222222222222'")" + +echo +echo "the cron catches up on a day nobody rolled up" +is "$DAY_MISSED got its daily_machines row" "1/1" \ + "$(q "select machines||'/'||prod_machines from daily_machines where day='$DAY_MISSED'")" +is "…and its event counts" "1/1" \ + "$(q "select count||'/'||machines from daily_event_counts where day='$DAY_MISSED' and event='index'")" +is "…and its breakdowns" "1/1" "$(dim "$DAY_MISSED" index language go)" +is "…and its first index day" "$DAY_MISSED" \ + "$(q "select first_index_day from machine_first_seen where machine_id='55555555-5555-4555-8555-555555555555'")" + echo echo "rollups outlive the events they came from" is "daily_event_counts survives the purge" "1/1" \ diff --git a/telemetry-worker/src/index.ts b/telemetry-worker/src/index.ts index 801d9a783e..9a6efa3394 100644 --- a/telemetry-worker/src/index.ts +++ b/telemetry-worker/src/index.ts @@ -4,7 +4,7 @@ * This file is public on purpose: it is the exact code that receives codegraph's * anonymous usage telemetry, so anyone can audit what is (and is not) stored. * The schema contract lives in docs/design/telemetry.md; the storage schema — the - * complete list of what is kept — is migrations/0001_init.sql. + * complete list of what is kept — is migrations/. * * Guarantees enforced here: * - strict allowlist: unknown events are dropped, unknown properties are stripped @@ -39,7 +39,7 @@ sent; the client IP is never read or stored; the machine ID is a random UUID the client mints locally and can delete at any time. Accepted events are stored in our own database on Cloudflare (D1) and are never forwarded to any third-party analytics vendor. The stored schema is the complete list of what is kept: -https://github.com/colbymchenry/codegraph/blob/main/telemetry-worker/migrations/0001_init.sql +https://github.com/colbymchenry/codegraph/tree/main/telemetry-worker/migrations Individual events are deleted after ${keepDays} days. What outlives them: anonymous daily totals (counts per day of things like operating system, version and language), @@ -204,10 +204,54 @@ const UPSERT_MACHINE_DAY = `INSERT INTO machine_days (machine_id, day, prod) VAL const UPSERT_FIRST_SEEN = `INSERT INTO machine_first_seen (machine_id, first_day) VALUES (?, ?) ON CONFLICT (machine_id) DO UPDATE SET first_day = min(machine_first_seen.first_day, excluded.first_day)`; +// usage_rollup counters ADD into one row per machine × day × tool (migrations/0003): +// clients upload the same counter many times over — once per process — so storing +// a row per upload grew without bound. +const UPSERT_USAGE = `INSERT INTO usage_daily ( + day, machine_id, kind, name, client_name, client_version, + codegraph_version, os, arch, node_major, count, error_count +) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT (day, machine_id, kind, name, client_name, client_version, codegraph_version, os, arch, node_major) + DO UPDATE SET count = usage_daily.count + excluded.count, + error_count = usage_daily.error_count + excluded.error_count`; + +interface UsageCounter { + day: string; + kind: string; + name: string; + clientName: string; + clientVersion: string; + count: number; + errors: number; +} + +/** + * Folds a batch's usage_rollup events into one counter per day × tool × client. + * Old clients send up to 100 copies of the same `count: 1` line per request; this + * turns them into one upsert instead of a hundred. + */ +function foldUsage(events: StoredEvent[], receivedAt: string): UsageCounter[] { + const byKey = new Map(); + for (const e of events) { + const day = (e.ts ?? receivedAt).slice(0, 10); + const kind = String(e.props.kind); + const name = String(e.props.name); + const clientName = typeof e.props.client_name === 'string' ? e.props.client_name : ''; + const clientVersion = typeof e.props.client_version === 'string' ? e.props.client_version : ''; + const key = [day, kind, name, clientName, clientVersion].join('\u0000'); + const counter = byKey.get(key) ?? { day, kind, name, clientName, clientVersion, count: 0, errors: 0 }; + counter.count += typeof e.props.count === 'number' ? e.props.count : 0; + counter.errors += typeof e.props.error_count === 'number' ? e.props.error_count : 0; + byKey.set(key, counter); + } + return [...byKey.values()]; +} + /** - * Persist a sanitized batch: one `events` row per event, plus the machine×day and - * first-seen bookkeeping the dashboard's retention/activation panels need. One D1 - * `batch()` = one implicit transaction = one round trip. + * Persist a sanitized batch: one `events` row per lifecycle event, usage counters + * added into `usage_daily`, plus the machine×day and first-seen bookkeeping the + * dashboard's retention/activation panels need. One D1 `batch()` = one implicit + * transaction = one round trip. * * Fail-silent by design: the client treats every response as final and never retries, * so a failed write loses a datapoint rather than costing availability. The error is @@ -236,7 +280,33 @@ async function writeToD1( // machine_days gets one row per distinct day rather than one per batch. const days = new Set(); + const usage = batch.filter((e) => e.event === 'usage_rollup'); + if (usage.length > 0) { + const upsertUsage = env.DB.prepare(UPSERT_USAGE); + const [version, os, arch, nodeMajor] = envelopeCols; + for (const u of foldUsage(usage, receivedAt)) { + days.add(u.day); + stmts.push( + upsertUsage.bind( + u.day, + machineId, + u.kind, + u.name, + u.clientName, + u.clientVersion, + version ?? '', + os ?? '', + arch ?? '', + nodeMajor === null ? '' : String(nodeMajor), + u.count, + u.errors, + ), + ); + } + } + for (const e of batch) { + if (e.event === 'usage_rollup') continue; const day = (e.ts ?? receivedAt).slice(0, 10); days.add(day); stmts.push( diff --git a/telemetry-worker/src/rollup.ts b/telemetry-worker/src/rollup.ts index abd1671c98..87072a59a9 100644 --- a/telemetry-worker/src/rollup.ts +++ b/telemetry-worker/src/rollup.ts @@ -6,11 +6,13 @@ * * Two jobs, both driven by the cron trigger in wrangler.jsonc (00:30 UTC daily): * - * 1. ROLL UP the just-completed UTC day into `daily_machines`, `daily_event_counts` - * and `daily_dim_counts` — plus the two days before it, because clients buffer - * offline and ship completed-day rollups late, so a day keeps growing after it - * ends. Every write is an upsert that OVERWRITES the recomputed value rather than - * adding to it, so re-running a day is a no-op and never double-counts. + * 1. ROLL UP the just-completed UTC day into `daily_machines`, `daily_event_counts`, + * `daily_dim_counts` and `machine_first_seen.first_index_day` — plus the two days + * before it, because clients buffer offline and ship completed-day rollups late, so + * a day keeps growing after it ends, plus any earlier day a failed run missed. Every + * write is an upsert that OVERWRITES the recomputed value rather than adding to it + * (or, for first_index_day, only ever lowers it), so re-running a day is a no-op + * and never double-counts. * * 2. PURGE raw `events` past the retention window, in bounded batches. Rollups are * kept forever, so only ad-hoc drill-down has a horizon; `machine_days` and @@ -28,6 +30,8 @@ export const DEFAULT_RETENTION_DAYS = 90; export const ROLLUP_LOOKBACK_DAYS = 3; /** Widest range one manual /admin/rollup call will attempt. */ export const MAX_MANUAL_DAYS = 31; +/** Most missed days one nightly run catches up on, newest first; the next night takes the rest. */ +export const MAX_CATCHUP_DAYS = 31; /** Rows per purge DELETE — bounded so one statement stays well inside D1's limits. */ const PURGE_BATCH_ROWS = 5_000; @@ -66,18 +70,17 @@ export function retentionCutoff(atMs: number, keepDays: number): string { // aggregation happens inside D1, so a day rolls up in one round trip and no event // row ever crosses the wire. Each takes exactly one bound parameter — the day. // +// Lifecycle events (install / index / uninstall) are one `events` row each, so their +// volume is count(*). Usage is NOT in `events`: the ingest worker adds every +// usage_rollup counter into `usage_daily` (one row per machine × day × tool), so the +// usage figures are sums over that table — its `count` is calls, and counting its +// rows would report "machines that used the tool" instead. +// // Adding a breakdown is a line in ROLLUP_STATEMENTS, never a migration — that is // what the generic (dim, value) shape of daily_dim_counts buys. -/** - * A group's event volume. For install/index/uninstall one row is one event, but a - * usage_rollup row is a counter the client pre-aggregated (one per machine × day × - * tool), so its `count` prop is what has to be summed — counting rows there would - * silently report "machines that used the tool" and undercount by an order of magnitude. - */ -const COUNT = `CASE WHEN e.event = 'usage_rollup' - THEN sum(coalesce(json_extract(e.props, '$.count'), 0)) - ELSE count(*) END`; +/** Lifecycle events only; usage comes from `usage_daily`. */ +const LIFECYCLE = ` AND e.event <> 'usage_rollup'`; const DIM_CONFLICT = `ON CONFLICT (day, event, dim, value) DO UPDATE SET count = excluded.count, machines = excluded.machines`; @@ -87,15 +90,29 @@ const quoted = (values: readonly string[]): string => values.map((v) => `'${v}'` const onlyEvents = (...events: readonly string[]): string => ` AND e.event IN (${quoted(events)})`; /** One dimension whose value is a scalar column or a scalar prop. */ -function dimStatement(dim: string, value: string, where = ''): string { +function dimStatement(dim: string, value: string, where = LIFECYCLE): string { return `INSERT INTO daily_dim_counts (day, event, dim, value, count, machines) - SELECT e.day, e.event, '${dim}', CAST(${value} AS TEXT), ${COUNT}, count(DISTINCT e.machine_id) + SELECT e.day, e.event, '${dim}', CAST(${value} AS TEXT), count(*), count(DISTINCT e.machine_id) FROM events e WHERE e.day = ? AND ${value} IS NOT NULL AND ${value} <> ''${where} GROUP BY e.day, e.event, ${value} ${DIM_CONFLICT}`; } +/** + * One usage dimension, from a `usage_daily` column. Absent values are stored as '' + * (they are part of the primary key), so '' is what gets skipped. `where` narrows + * the rows further — name_error keeps only counters that saw an error. + */ +function usageDimStatement(dim: string, column: string, total = 'count', where = ''): string { + return `INSERT INTO daily_dim_counts (day, event, dim, value, count, machines) + SELECT day, 'usage_rollup', '${dim}', ${column}, sum(${total}), count(DISTINCT machine_id) + FROM usage_daily + WHERE day = ? AND ${column} <> ''${where} + GROUP BY day, ${column} + ${DIM_CONFLICT}`; +} + /** * One dimension unnested from a JSON array prop — one row per element, so an index * of a TypeScript+Go repo counts once under each language. `json_each` over a path @@ -125,22 +142,48 @@ const DAILY_MACHINES = `INSERT INTO daily_machines (day, machines, prod_machines ON CONFLICT (day) DO UPDATE SET machines = excluded.machines, prod_machines = excluded.prod_machines`; +/** + * The activation funnel's input: each machine's first index day, lowered to this day + * if the machine indexed on it and has no earlier index on record. The dashboard reads + * it straight off `machine_first_seen`, which keeps the funnel off raw `events` (a + * cohort join there took most of a minute per week of cohorts) and past the purge. + * + * Only rows that actually move are written, so re-running a day is free. `?1` is the + * day — still the one bound parameter, used three times. + */ +const FIRST_INDEX_DAY = `UPDATE machine_first_seen + SET first_index_day = ?1 + WHERE (first_index_day IS NULL OR first_index_day > ?1) + AND machine_id IN (SELECT machine_id FROM events WHERE day = ?1 AND event = 'index')`; + const ROLLUP_STATEMENTS: readonly string[] = [ DAILY_MACHINES, + FIRST_INDEX_DAY, `INSERT INTO daily_event_counts (day, event, count, machines) - SELECT e.day, e.event, ${COUNT}, count(DISTINCT e.machine_id) + SELECT e.day, e.event, count(*), count(DISTINCT e.machine_id) FROM events e - WHERE e.day = ? + WHERE e.day = ?${LIFECYCLE} GROUP BY e.day, e.event ON CONFLICT (day, event) DO UPDATE SET count = excluded.count, machines = excluded.machines`, + `INSERT INTO daily_event_counts (day, event, count, machines) + SELECT day, 'usage_rollup', sum(count), count(DISTINCT machine_id) + FROM usage_daily + WHERE day = ? + GROUP BY day + ON CONFLICT (day, event) DO UPDATE + SET count = excluded.count, machines = excluded.machines`, // Envelope dimensions — every event type carries them. dimStatement('os', 'e.os'), dimStatement('arch', 'e.arch'), dimStatement('codegraph_version', 'e.codegraph_version'), dimStatement('node_major', 'e.node_major'), + usageDimStatement('os', 'os'), + usageDimStatement('arch', 'arch'), + usageDimStatement('codegraph_version', 'codegraph_version'), + usageDimStatement('node_major', 'node_major'), // Event-specific scalar props. dimStatement('file_count_bucket', prop('file_count_bucket'), onlyEvents('index')), @@ -148,9 +191,10 @@ const ROLLUP_STATEMENTS: readonly string[] = [ dimStatement('scope', prop('scope'), onlyEvents('install')), // `kind` is fresh/upgrade/reinstall on install and mcp_tool/cli_command on // usage_rollup; `event` is part of the primary key, so both live here without colliding. - dimStatement('kind', prop('kind'), onlyEvents('install', 'usage_rollup')), - dimStatement('name', prop('name'), onlyEvents('usage_rollup')), - dimStatement('client_name', prop('client_name'), onlyEvents('usage_rollup')), + dimStatement('kind', prop('kind'), onlyEvents('install')), + usageDimStatement('kind', 'kind'), + usageDimStatement('name', 'name'), + usageDimStatement('client_name', 'client_name'), // Array props. arrayDimStatement('language', '$.languages', ['index']), @@ -158,20 +202,13 @@ const ROLLUP_STATEMENTS: readonly string[] = [ // Errors per tool/command. Not in the migration's documented dim list because dims // are a cron concern rather than a schema one, but rolled up because it is the one - // usage number that is gone for good after the purge. Only groups with at least one - // error are stored, so `count` is errors and `machines` is the machines that saw one + // usage number that is gone for good after the purge. Only counters with at least + // one error count, so `count` is errors and `machines` is the machines that saw one // — NOT the machines that ran the tool (that is the `name` dim). - `INSERT INTO daily_dim_counts (day, event, dim, value, count, machines) - SELECT e.day, e.event, 'name_error', CAST(${prop('name')} AS TEXT), - sum(${prop('error_count')}), count(DISTINCT e.machine_id) - FROM events e - WHERE e.day = ? AND e.event = 'usage_rollup' - AND ${prop('name')} IS NOT NULL AND coalesce(${prop('error_count')}, 0) > 0 - GROUP BY e.day, e.event, ${prop('name')} - ${DIM_CONFLICT}`, + usageDimStatement('name_error', 'name', 'error_count', ' AND error_count > 0'), ]; -/** Rollup tables derived from raw `events` — the ones `reset` wipes before recomputing. */ +/** Rollup tables derived from raw `events` / `usage_daily` — the ones `reset` wipes before recomputing. */ const EVENT_DERIVED_TABLES = ['daily_event_counts', 'daily_dim_counts'] as const; // --------------------------------------------------------------------------- @@ -206,6 +243,7 @@ export async function rollupDay( opts: { cutoff: string; reset?: boolean }, ): Promise { const pastRetention = day < opts.cutoff; + await foldLegacyUsage(env, day); const statements: D1PreparedStatement[] = []; if (opts.reset && !pastRetention) { @@ -222,36 +260,147 @@ export async function rollupDay( return { day, rows, pastRetention }; } +// --------------------------------------------------------------------------- +// Legacy usage rows +// --------------------------------------------------------------------------- +// Before migrations/0003, every usage_rollup event was its own `events` row — about +// 3.8M a day by August 2026, because clients upload a counter per process rather +// than per day, and enough to fill the database. These statements fold one day's +// worth of those rows into `usage_daily`, adding counts the same way the ingest path +// does, and delete them in the same transaction, so a count is never in both places +// and never in neither. rollupDay runs it first, which makes re-running the rollup +// over old days the whole migration; once a day has no legacy rows left it costs +// one indexed probe. + +/** Rows folded per transaction — bounded so one GROUP BY + DELETE stays well inside D1's limits. */ +const LEGACY_CHUNK_ROWS = 50_000; +/** Ceiling per day (the worst day on record had ~3.9M rows). */ +const LEGACY_MAX_CHUNKS = 200; + +const LEGACY_FOLD = `INSERT INTO usage_daily ( + day, machine_id, kind, name, client_name, client_version, + codegraph_version, os, arch, node_major, count, error_count) + SELECT day, machine_id, + coalesce(json_extract(props, '$.kind'), ''), + coalesce(json_extract(props, '$.name'), ''), + coalesce(json_extract(props, '$.client_name'), ''), + coalesce(json_extract(props, '$.client_version'), ''), + coalesce(codegraph_version, ''), coalesce(os, ''), coalesce(arch, ''), + coalesce(CAST(node_major AS TEXT), ''), + sum(coalesce(json_extract(props, '$.count'), 0)), + sum(coalesce(json_extract(props, '$.error_count'), 0)) + FROM events + WHERE day = ?1 AND event = 'usage_rollup' AND id < ?2 + GROUP BY 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 + ON CONFLICT (day, machine_id, kind, name, client_name, client_version, codegraph_version, os, arch, node_major) + DO UPDATE SET count = usage_daily.count + excluded.count, + error_count = usage_daily.error_count + excluded.error_count`; + +const LEGACY_DELETE = `DELETE FROM events WHERE day = ?1 AND event = 'usage_rollup' AND id < ?2`; + +/** Folds any legacy usage rows for `day` into usage_daily. Returns how many rows it moved. */ +export async function foldLegacyUsage(env: Env, day: string): Promise { + const any = await env.DB.prepare(`SELECT 1 AS hit FROM events WHERE day = ? AND event = 'usage_rollup' LIMIT 1`) + .bind(day) + .first<{ hit: number }>(); + if (!any) return 0; + + let moved = 0; + for (let chunk = 0; chunk < LEGACY_MAX_CHUNKS; chunk++) { + // Always the oldest remaining rows: everything below the next chunk's first id. + // Rows folded by earlier chunks are gone, so there is no cursor to carry. + const next = await env.DB.prepare( + `SELECT id FROM events WHERE day = ? AND event = 'usage_rollup' ORDER BY id LIMIT 1 OFFSET ${LEGACY_CHUNK_ROWS}`, + ) + .bind(day) + .first<{ id: number }>(); + const below = next?.id ?? Number.MAX_SAFE_INTEGER; + const [, deleted] = await env.DB.batch([ + env.DB.prepare(LEGACY_FOLD).bind(day, below), + env.DB.prepare(LEGACY_DELETE).bind(day, below), + ]); + moved += deleted?.meta?.changes ?? 0; + if (!next) return moved; + } + throw new Error(`legacy usage fold for ${day} did not finish within ${LEGACY_MAX_CHUNKS} chunks`); +} + export interface PurgeResult { /** Everything strictly before this day was deleted. */ cutoff: string; deleted: number; + /** usage_daily rows past the window — per-machine counters, so they expire with the events. */ + usageDeleted: number; batches: number; /** Hit the per-run batch ceiling — more rows are still due, next run takes them. */ capped: boolean; } /** - * Delete raw events older than the window, oldest first, in bounded batches. - * `id` is a rowid alias and the purge only ever removes the oldest rows, so the - * keyset subquery stays a cheap index range scan on (day, event). + * Delete raw events older than the window, oldest first, in bounded batches, then + * the usage counters older than it, a day per statement (`usage_daily` is keyed by + * day first, so each is one primary-key range). `id` is a rowid alias and the purge + * only ever removes the oldest rows, so the keyset subquery stays a cheap index range + * scan on (day, event). */ export async function purgeOldEvents(env: Env, cutoff: string): Promise { const del = env.DB.prepare( `DELETE FROM events WHERE id IN (SELECT id FROM events WHERE day < ? LIMIT ${PURGE_BATCH_ROWS})`, ); let deleted = 0; - for (let batch = 1; batch <= PURGE_MAX_BATCHES; batch++) { + let batches = 0; + let capped = true; + while (batches < PURGE_MAX_BATCHES) { + batches++; const { meta } = await del.bind(cutoff).run(); const removed = meta?.changes ?? 0; deleted += removed; - if (removed < PURGE_BATCH_ROWS) return { cutoff, deleted, batches: batch, capped: false }; + if (removed < PURGE_BATCH_ROWS) { + capped = false; + break; + } } - return { cutoff, deleted, batches: PURGE_MAX_BATCHES, capped: true }; + + const delUsageDay = env.DB.prepare( + `DELETE FROM usage_daily WHERE day = (SELECT min(day) FROM usage_daily WHERE day < ?)`, + ); + let usageDeleted = 0; + for (let day = 0; day < PURGE_MAX_BATCHES; day++) { + const { meta } = await delUsageDay.bind(cutoff).run(); + const removed = meta?.changes ?? 0; + if (removed === 0) break; + usageDeleted += removed; + } + return { cutoff, deleted, usageDeleted, batches, capped }; +} + +/** + * Days that saw activity but never got a rollup: the nights a run failed, or the + * database refused writes. Every active day has `machine_days` rows and every + * rolled-up day has a `daily_machines` row, so the difference is exactly the missed + * days — a day nobody was active on is in neither, and needs nothing. + * + * Without this, the cron only ever looks three days back, so one bad week leaves a + * permanent hole that someone has to notice and backfill by hand. Bounded below by + * the retention cutoff, past which the raw events are gone and there is nothing left + * to roll up. + */ +async function missedDays(env: Env, cutoff: string, before: string): Promise { + const { results } = await env.DB.prepare( + `SELECT DISTINCT day FROM machine_days + WHERE day >= ? AND day < ? + AND day NOT IN (SELECT day FROM daily_machines) + ORDER BY day DESC + LIMIT ${MAX_CATCHUP_DAYS}`, + ) + .bind(cutoff, before) + .all<{ day: string }>(); + return results.map((r) => r.day); } /** - * The cron body: roll up the completed day and the two before it, then purge. + * The cron body: roll up the completed day and the two before it, catch up any + * earlier day a past run missed, then purge. * * Logs one line of counts — never a day's contents, never a machine id. Throws if * anything failed so the invocation is marked failed (and retried) rather than @@ -262,11 +411,23 @@ export async function runNightly(env: Env, atMs: number): Promise { const keepDays = retentionDays(env); const cutoff = retentionCutoff(atMs, keepDays); + const days: string[] = []; + for (let back = 1; back <= ROLLUP_LOOKBACK_DAYS; back++) days.push(utcDay(atMs - back * DAY_MS)); + + // A failure to find the missed days must not cost tonight's regular rollup. + let caughtUp = 0; + try { + const missed = await missedDays(env, cutoff, days[days.length - 1] ?? utcDay(atMs)); + days.push(...missed); + caughtUp = missed.length; + } catch (err) { + console.error(JSON.stringify({ msg: 'missed-day scan failed', err: String(err) })); + } + const rolled: string[] = []; const failed: string[] = []; let rows = 0; - for (let back = 1; back <= ROLLUP_LOOKBACK_DAYS; back++) { - const day = utcDay(atMs - back * DAY_MS); + for (const day of days) { try { rows += (await rollupDay(env, day, { cutoff })).rows; rolled.push(day); @@ -287,11 +448,13 @@ export async function runNightly(env: Env, atMs: number): Promise { JSON.stringify({ msg: 'nightly rollup', days: rolled, + caught_up: caughtUp, rows, failed: failed.length, retention_days: keepDays, purged_before: cutoff, purged: purge?.deleted ?? null, + usage_purged: purge?.usageDeleted ?? null, purge_batches: purge?.batches ?? null, purge_capped: purge?.capped ?? null, ms: Date.now() - started, From 0f0af7bf22a5083a47e34fcb1e1bf67355174ef8 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Sat, 3 Oct 2026 06:35:17 +0000 Subject: [PATCH 168/259] release: 1.6.2 (#2318) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps package.json and ui/package.json (the component library is versioned with the engine; ui-package.test.ts pins it) to 1.6.2, and the matching version fields in package-lock.json — the top level, packages[""] and the ui workspace, whose entry was still at 1.6.0. Only those three fields change; `npm ci` accepts the lock. Co-authored-by: Claude Opus 5.5 --- package-lock.json | 6 +++--- package.json | 2 +- ui/package.json | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/package-lock.json b/package-lock.json index e6a232af46..9591645893 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@colbymchenry/codegraph", - "version": "1.6.1", + "version": "1.6.2", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@colbymchenry/codegraph", - "version": "1.6.1", + "version": "1.6.2", "license": "MIT", "workspaces": [ "ui" @@ -3134,7 +3134,7 @@ }, "ui": { "name": "@colbymchenry/codegraph-ui", - "version": "1.6.0", + "version": "1.6.2", "license": "MIT", "dependencies": { "@xyflow/svelte": "^1.6.5" diff --git a/package.json b/package.json index 92d1aa6c7f..ab8ba5cca6 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@colbymchenry/codegraph", - "version": "1.6.1", + "version": "1.6.2", "description": "Supercharge AI coding agents with semantic code intelligence — surgical context, fewer tool calls, faster answers. 100% local.", "repository": { "type": "git", diff --git a/ui/package.json b/ui/package.json index 65cd32c89f..4543403516 100644 --- a/ui/package.json +++ b/ui/package.json @@ -1,7 +1,7 @@ { "name": "@colbymchenry/codegraph-ui", "private": true, - "version": "1.6.1", + "version": "1.6.2", "type": "module", "description": "The CodeGraph reader as Svelte components: Symbol view, Flow strip and architecture Map behind one data adapter.", "keywords": [ From 6560052a6f856855d3f71eee838fd66ccfa4285d Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Sat, 3 Oct 2026 06:38:45 +0000 Subject: [PATCH 169/259] docs(changelog): promote [Unreleased] into [1.6.2] [skip ci] Auto-generated by Release workflow. --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index e027687d6b..06ca319d98 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,9 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] + +## [1.6.2] - 2026-10-03 + ### Highlights - **Far fewer wrong links, in every language.** A call, import or type now resolves the way its language scopes names — through imports, packages, namespaces and the class it is written in — so it stops landing on an unrelated symbol that only shares its name. This covers TypeScript and JavaScript, Python, Java, Kotlin, Scala, C#, VB.NET, Swift, Objective-C, Go, Rust, C and C++, PHP, Ruby, Dart, Lua, R and more, and makes callers, impact and `codegraph_explore` answers more trustworthy. @@ -1313,3 +1316,4 @@ Thanks @andreinknv for the substantive draft this release was based on. [1.5.0]: https://github.com/colbymchenry/codegraph/releases/tag/v1.5.0 [1.6.0]: https://github.com/colbymchenry/codegraph/releases/tag/v1.6.0 [1.6.1]: https://github.com/colbymchenry/codegraph/releases/tag/v1.6.1 +[1.6.2]: https://github.com/colbymchenry/codegraph/releases/tag/v1.6.2 From 489b4749c725cc446df7245fa611d77dab9c301e Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 14:41:59 +0000 Subject: [PATCH 170/259] fix(vbnet,csharp): index every Structure member; walk property bodies and VB initializers (#2345) VB.NET's grammar tags each member of a `Structure` as its own `body` field, so extractAggregate's field lookup saw only the first member. SCrawler's UserMedia indexed none of its members, and a structure that opened with a nested enum indexed only the enum. Aggregates now resolve their body the way classes and enums do (resolveBody, which VB maps to the declaration itself). Objective-C, the only other struct language with that hook, is unchanged. Property bodies were never walked: the propertyTypes branch set skipChildren after extractProperty, so calls, instantiations and reads in a VB `Get`/`Set` block, `= initializer` or `As New`, and in a C# accessor body or `=> expr`, were lost. - propertyBodies() names the parts of a property that run code (VB: the whole declaration, as its methods are walked; C#: each accessor's body and an arrow value), walked with the property on the stack. The candidates-only fn-ref scan skips the walked subtrees by node id, so a function value is captured once, from the property. C# `= initializer`s stay unwalked, as C# field initializers are. - The C# kernel mirrors it (property_bodies, scan skip list). The parity suite and a new micro pass; a serilog sweep is 209/214 byte-identical with 0 diffs (5 deferred for parse errors). - VB field declarators (`= expr`, `As New T`) and Custom Event AddHandler/RemoveHandler/RaiseEvent blocks are walked as their field or event. Before/after (nodes / edges): SCrawler 9,950 -> 10,427 / 14,336 -> 15,092; staxrip 13,440 -> 13,644 / 24,120 -> 28,002, most of it its `Property X As New NumParam With {...}` settings; serilog edges 6,642 -> 6,654. All 23 removed edges are the same call sites re-resolved by VB's name-guessing resolver now that struct members are candidates: 10 wrong edges dropped, 9 correct ones now guessed wrong, 4 wrong either way. Designer -> Add/Size edge counts are unchanged. EXTRACTION_VERSION 27 -> 28. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 6 + __tests__/csharp-property-accessors.test.ts | 91 +++++++++ __tests__/kernel-csharp-parity.test.ts | 20 +- __tests__/vbnet-member-bodies.test.ts | 209 ++++++++++++++++++++ codegraph-kernel/src/csharp.rs | 80 ++++++-- docs/design/csharp-kernel-port-checklist.md | 22 ++- src/extraction/extraction-version.ts | 2 +- src/extraction/languages/vbnet.ts | 9 +- src/extraction/tree-sitter.ts | 62 ++++-- 9 files changed, 459 insertions(+), 42 deletions(-) create mode 100644 __tests__/csharp-property-accessors.test.ts create mode 100644 __tests__/vbnet-member-bodies.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 06ca319d98..44144e1606 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,12 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] +### Fixes + +- In VB.NET, every member of a `Structure` is now indexed, including its fields, properties, methods, constructors and nested enums. Before, only the first member was, so the rest could not be found and their callers looked empty. +- In VB.NET and C#, what a property's `Get` and `Set` code calls, creates and reads now belongs to that property, as do C#'s `get => …` accessors and `=> …` property bodies. Before, it was dropped, so a method used only from a property looked unused. +- In VB.NET, a field or property initializer like `= Compute()` or `As New List(Of Order)` now links what it calls and creates, and so do a `Custom Event`'s `AddHandler`, `RemoveHandler` and `RaiseEvent` blocks. Re-index VB.NET and C# projects after upgrading. + ## [1.6.2] - 2026-10-03 diff --git a/__tests__/csharp-property-accessors.test.ts b/__tests__/csharp-property-accessors.test.ts new file mode 100644 index 0000000000..b77d8549d6 --- /dev/null +++ b/__tests__/csharp-property-accessors.test.ts @@ -0,0 +1,91 @@ +/** + * C# property accessor bodies belong to the property. `get { … }`, + * `set { … }`, `get => …` and an expression-bodied `=> …` property used to + * be skipped by both extractors, so the calls, instantiations and static + * reads written there were lost and a getter's callee looked unused. + * + * Runs against the native kernel (when built) and the wasm extractor, which + * must agree. + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { Node } from '../src/types'; + +const FILES: Record = { + 'Helper.cs': `public static class Helper { + public static int Compute(int n) => n; + public static void Store(int n) {} + public static int Arrow() => 1; + public static int GetA() => 1; + public static void SetA(int v) {} +} +public class Widget {} +public static class Defaults { public const string Name = "x"; } +`, + 'Box.cs': `public class Box { + private int _x; + public int Value { + get { return Helper.Compute(_x); } + set { Helper.Store(value); _x = value; } + } + public int Arrow => Helper.Arrow(); + public int Both { get => Helper.GetA(); set => Helper.SetA(value); } + public Widget Made { get { return new Widget(); } } + public string Label { get { return Defaults.Name; } } + private void Handle(int v) { } + public System.Action Handler { get { return Pick(Handle); } } +} +`, +}; + +describe('C# property accessor bodies', () => { + let root = ''; + let cg: CodeGraph | undefined; + let kernel: string | undefined; + + beforeEach(() => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-cs-accessors-')); + for (const [rel, content] of Object.entries(FILES)) fs.writeFileSync(path.join(root, rel), content); + kernel = process.env.CODEGRAPH_KERNEL; + }); + + afterEach(() => { + cg?.close(); + cg = undefined; + fs.rmSync(root, { recursive: true, force: true }); + if (kernel === undefined) delete process.env.CODEGRAPH_KERNEL; + else process.env.CODEGRAPH_KERNEL = kernel; + }); + + it.each(['default', 'wasm'])('are walked as the property (%s)', async (backend) => { + if (backend === 'wasm') process.env.CODEGRAPH_KERNEL = '0'; + else delete process.env.CODEGRAPH_KERNEL; + cg = await CodeGraph.init(root, { index: true }); + const graph = cg; + const member = (qualifiedName: string): Node => { + const node = graph.getNodesInFile('Box.cs').find((n) => n.qualifiedName === qualifiedName); + expect(node, qualifiedName).toBeDefined(); + return node!; + }; + const targets = (node: Node, kind: string): string[] => + graph + .getOutgoingEdgesFrom([node.id]) + .filter((e) => e.kind === kind) + .map((e) => graph.getNode(e.target)!.qualifiedName) + .sort(); + + expect(targets(member('Box::Value'), 'calls')).toEqual(['Helper::Compute', 'Helper::Store']); + expect(targets(member('Box::Arrow'), 'calls')).toEqual(['Helper::Arrow']); + expect(targets(member('Box::Both'), 'calls')).toEqual(['Helper::GetA', 'Helper::SetA']); + expect(targets(member('Box::Made'), 'instantiates')).toEqual(['Widget']); + expect(targets(member('Box::Label'), 'references')).toContain('Defaults'); + // A method passed as a value inside an accessor is the property's + // reference, captured once — not the class's as well. + expect(targets(member('Box::Handler'), 'references')).toContain('Box::Handle'); + expect(targets(member('Box'), 'references')).not.toContain('Box::Handle'); + expect(targets(member('Box'), 'calls')).toEqual([]); + }); +}); diff --git a/__tests__/kernel-csharp-parity.test.ts b/__tests__/kernel-csharp-parity.test.ts index f26f5072df..3d09c19d98 100644 --- a/__tests__/kernel-csharp-parity.test.ts +++ b/__tests__/kernel-csharp-parity.test.ts @@ -8,7 +8,8 @@ * * - Torture.cs — block namespace + nested/second-namespace quirks, * base_list shapes, records, properties (incl. the bare-identifier - * signature loss and never-walked accessor bodies), fields/constants, + * signature loss, accessor bodies walked as the property and + * never-walked initializers), fields/constants, * events/operators/indexer/destructor (no nodes, calls → class), ctor * initializer hole, explicit interface impl, local functions, the call * zoo (raw member-access texts, chained re-encode, `(myDel)(x)` conv, @@ -167,6 +168,23 @@ describe.skipIf(!kernelBuilt)('kernel C# extraction parity', () => { source: 'public record Empty;\n', minNodes: 2, }, + { + // Accessor bodies and `=> expr` are walked as the property (calls, + // instantiations, static reads, fn-ref candidates); an `= initializer` + // is only scanned for candidates, attributed to the class. + name: 'property bodies are walked as the property; initializers only scanned', + source: [ + 'public class C {', + ' void H(int v) { }', + ' public Action P { get { return Make(H); } set { Register(H); } }', + ' public int Q => Run(H);', + ' public Action R { get; } = Wrap(H);', + ' public int S { get => Calc.Max; init => Store(new Widget()); }', + '}', + '', + ].join('\n'), + minNodes: 7, + }, ]; for (const m of MICROS) { diff --git a/__tests__/vbnet-member-bodies.test.ts b/__tests__/vbnet-member-bodies.test.ts new file mode 100644 index 0000000000..8c280a5b55 --- /dev/null +++ b/__tests__/vbnet-member-bodies.test.ts @@ -0,0 +1,209 @@ +/** + * VB.NET member bodies the extractor used to skip: + * + * - A `Structure` lists each member as its own `body` field, so reading the + * field returned only the first member: SCrawler's `UserMedia` indexed its + * nested `States` enum (declared first) and nothing else, and a structure + * that opened with a field indexed no members at all. + * - A property's `Get` / `Set` blocks, its `= initializer` and its + * `As New T` were never walked, nor a `Custom Event`'s accessors. + * - A field's `= initializer` and `As New T` were never walked. + * + * The calls, instantiations and reads in those places belong to the member + * that declares them. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { Node } from '../src/types'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-vb-bodies-')); + const files: Record = { + 'Helper.vb': `Public Class Helper + Public Sub New(ByVal n As Integer) + End Sub + Public Shared Function Compute(ByVal n As Integer) As Integer + Return n + End Function + Public Shared Sub Store(ByVal n As Integer) + End Sub + Public Shared Function Seed() As Integer + Return 1 + End Function + Public Shared Function Format(ByVal o As Object) As String + Return "" + End Function + Public Shared Sub Log(ByVal s As String) + End Sub + Public Shared Sub Attach(ByVal h As EventHandler) + End Sub +End Class +`, + 'Widget.vb': `Public Class Widget +End Class +`, + 'UserMedia.vb': `Public Structure UserMedia + Public State As States + Private Count As Integer + Public Enum States + Unknown + Downloaded + End Enum + Public ReadOnly Property Name As String + Get + Return Helper.Format(State) + End Get + End Property + Public Sub New(ByVal s As States) + State = s + Helper.Log("created") + End Sub + Public Function IsDone() As Boolean + Return State = States.Downloaded + End Function +End Structure +`, + 'Box.vb': `Public Class Box + Private _x As Integer + Private _items As New Widget + Private _helper As Helper = New Helper(1) + Private Shared ReadOnly Def As Integer = Helper.Compute(3), Other As Integer = Helper.Seed() + Public Property Value As Integer + Get + Return Helper.Compute(_x) + End Get + Set(ByVal v As Integer) + Helper.Store(v) + _x = v + End Set + End Property + Public Property Auto As Integer = Helper.Seed() + Public ReadOnly Property Lst As New Widget + Public Custom Event Changed As EventHandler + AddHandler(ByVal value As EventHandler) + Helper.Attach(value) + End AddHandler + RemoveHandler(ByVal value As EventHandler) + End RemoveHandler + RaiseEvent(ByVal sender As Object, ByVal e As EventArgs) + End RaiseEvent + End Event +End Class +`, + // The 1.6.2 receiver gate must hold in the newly walked places too: + // `Me.Panel.Controls.Add(…)` and `New System.Drawing.Size(…)` name no + // project member. + 'Collections/DataColorCollection.vb': `Friend Class DataColorCollection + Friend Sub Add(ByVal Item As Object) + End Sub +End Class +`, + 'Editors/UsersInfoForm.vb': `Friend Class UsersInfoForm + Private Enum EComparers + Name + Size + End Enum +End Class +`, + 'MainForm.vb': `Public Class MainForm + Private ReadOnly DefaultSize As System.Drawing.Size = New System.Drawing.Size(184, 25) + Public ReadOnly Property Ready As Boolean + Get + Me.Panel.Controls.Add(Me.Button1) + Return True + End Get + End Property +End Class +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +function member(file: string, qualifiedName: string): Node { + const node = cg.getNodesInFile(file).find((n) => n.qualifiedName === qualifiedName); + expect(node, `${qualifiedName} in ${file}`).toBeDefined(); + return node!; +} + +/** `kind` edges leaving `node`, as target qualified names. */ +function targets(node: Node, kind: string): string[] { + return cg + .getOutgoingEdgesFrom([node.id]) + .filter((e) => e.kind === kind) + .map((e) => cg.getNode(e.target)!.qualifiedName) + .sort(); +} + +describe('VB.NET member bodies', () => { + it('indexes every member of a Structure, not just the first', () => { + const struct = member('UserMedia.vb', 'UserMedia'); + expect(struct.kind).toBe('struct'); + const contained = cg + .getOutgoingEdgesFrom([struct.id]) + .filter((e) => e.kind === 'contains') + .map((e) => cg.getNode(e.target)!) + .map((n) => `${n.kind} ${n.qualifiedName}`) + .sort(); + expect(contained).toEqual([ + 'enum UserMedia::States', + 'field UserMedia::Count', + 'field UserMedia::State', + 'method UserMedia::IsDone', + 'method UserMedia::New', + 'property UserMedia::Name', + ]); + expect(member('UserMedia.vb', 'UserMedia::States::Downloaded').kind).toBe('enum_member'); + expect(targets(member('UserMedia.vb', 'UserMedia::New'), 'calls')).toEqual(['Helper::Log']); + }); + + it("attributes a property's Get and Set blocks to the property", () => { + expect(targets(member('Box.vb', 'Box::Value'), 'calls')).toEqual(['Helper::Compute', 'Helper::Store']); + expect(targets(member('UserMedia.vb', 'UserMedia::Name'), 'calls')).toEqual(['Helper::Format']); + }); + + it("walks a property's initializer and its As New", () => { + expect(targets(member('Box.vb', 'Box::Auto'), 'calls')).toEqual(['Helper::Seed']); + expect(targets(member('Box.vb', 'Box::Lst'), 'instantiates')).toEqual(['Widget']); + }); + + it("walks each field's initializer and its As New, per declarator", () => { + expect(targets(member('Box.vb', 'Box::Def'), 'calls')).toEqual(['Helper::Compute']); + expect(targets(member('Box.vb', 'Box::Other'), 'calls')).toEqual(['Helper::Seed']); + expect(targets(member('Box.vb', 'Box::_items'), 'instantiates')).toEqual(['Widget']); + expect(targets(member('Box.vb', 'Box::_helper'), 'instantiates')).toEqual(['Helper']); + }); + + it("attributes a Custom Event's accessors to the event", () => { + expect(targets(member('Box.vb', 'Box::Changed'), 'calls')).toEqual(['Helper::Attach']); + }); + + it('keeps the class out of what its members do', () => { + expect(targets(member('Box.vb', 'Box'), 'calls')).toEqual([]); + expect(targets(member('Box.vb', 'Box'), 'instantiates')).toEqual([]); + }); + + it('still links nothing through a receiver no name resolves', () => { + const ids = cg.getNodesInFile('MainForm.vb').map((n) => n.id); + const linked = cg + .getOutgoingEdgesFrom(ids) + .filter((e) => e.kind !== 'contains') + .map((e) => cg.getNode(e.target)!.qualifiedName); + expect(linked).not.toContain('DataColorCollection::Add'); + expect(linked).not.toContain('UsersInfoForm::EComparers::Size'); + }); +}); diff --git a/codegraph-kernel/src/csharp.rs b/codegraph-kernel/src/csharp.rs index 7d481662a7..04e20ca5a5 100644 --- a/codegraph-kernel/src/csharp.rs +++ b/codegraph-kernel/src/csharp.rs @@ -5,7 +5,7 @@ //! path, bug-for-bug, verified by scripts/kernel-parity.mjs and the full-index //! dump-diff gate. The authoritative quirk list is //! docs/design/csharp-kernel-port-checklist.md — including every deliberate -//! emission hole (property/accessor bodies, constructor initializers, +//! emission hole (field/property initializers, constructor initializers, //! delegates/events/operators/indexers, top-level locals) and garbage ref //! (`(repo)` primary-ctor extends, `: byte` enum extends, `nameof` calls) //! this file preserves on purpose. Positions in UTF-16 code units. Files whose @@ -572,21 +572,36 @@ impl<'t> Walker<'t> { self.extract_enum(node); skip_children = true; } else if kind == "property_declaration" && self.inside_class_like() { - // Property accessor/expression bodies are NEVER walked (calls - // inside are lost by design) — candidates-only scan. - self.extract_property(node); - self.scan_fn_ref_subtree(node, 0); + // The code a property runs — its accessor bodies and `=> expr` — + // is walked with the property on the stack (propertyBodies). The + // candidates-only scan covers the rest (an `= initializer`), + // skipping what the body walk captured. + let walked: Vec = match self.extract_property(node) { + Some((row, name)) => { + let bodies = property_bodies(node); + if !bodies.is_empty() { + self.stack.push(Scope { row, kind: "property", name }); + for body in &bodies { + self.visit_function_body(*body); + } + self.stack.pop(); + } + bodies.iter().map(|b| b.id()).collect() + } + None => Vec::new(), + }; + self.scan_fn_ref_subtree(node, 0, &walked); skip_children = true; } else if kind == "field_declaration" && self.inside_class_like() { self.extract_field(node); - self.scan_fn_ref_subtree(node, 0); + self.scan_fn_ref_subtree(node, 0, &[]); skip_children = true; } else if kind == "local_declaration_statement" && !self.inside_class_like() { // Top-level statements: extractVariable's generic fallback finds no // direct identifier/variable_declarator children (C# nests them in // variable_declaration) → ZERO nodes, zero refs. Candidates only. self.extract_variable(node); - self.scan_fn_ref_subtree(node, 0); + self.scan_fn_ref_subtree(node, 0, &[]); skip_children = true; } else if kind == "using_directive" { self.extract_import(node); @@ -789,9 +804,9 @@ impl<'t> Walker<'t> { } /// extractProperty (1986) — property_declaration only (dispatch-gated to - /// class-like scopes). Accessor bodies and `=>` value clauses are never - /// walked; type refs DO come from the `type` field. - fn extract_property(&mut self, node: Node<'t>) { + /// class-like scopes). Type refs come from the `type` field; the caller + /// walks the bodies (property_bodies) with the returned row on the stack. + fn extract_property(&mut self, node: Node<'t>) -> Option<(u32, String)> { let docstring = preceding_docstring(node, self.src); let visibility = Some(self.visibility_of(node)); let is_static = Some(self.is_static(node)); // ?? false — always concrete @@ -804,10 +819,10 @@ impl<'t> Walker<'t> { .filter_map(|i| node.named_child(i)) .find(|c| c.kind() == "identifier") }); - let Some(name_node) = name_node else { return }; + let name_node = name_node?; let name = self.text(name_node).to_string(); if name.is_empty() { - return; + return None; } // Generic scan (isTsJsField=false): FIRST namedChild that isn't a @@ -842,11 +857,10 @@ impl<'t> Walker<'t> { &name, node, Extra { docstring, signature: Some(signature), visibility, is_static, ..Extra::default() }, - ); - if let Some(row) = row { - // decorators: none for C#; then the csharp type-ref path. - self.extract_csharp_type_refs(node, row); - } + )?; + // decorators: none for C#; then the csharp type-ref path. + self.extract_csharp_type_refs(node, row); + Some((row, name)) } /// extractField (2046) — field_declaration; each declarator becomes a @@ -1468,11 +1482,15 @@ impl<'t> Walker<'t> { }); } - fn scan_fn_ref_subtree(&mut self, node: Node<'t>, depth: u32) { + fn scan_fn_ref_subtree(&mut self, node: Node<'t>, depth: u32, walked: &[usize]) { stack_guard!(); if depth > 12 { return; } + // Subtrees the body walker has already been through. + if walked.contains(&node.id()) { + return; + } // functionTypes is EMPTY for C#; the literal halt list applies — // lambda_expression IS C#'s lambda, so initializer lambdas stop the // scan; anonymous_method_expression is NOT listed and scans through. @@ -1487,7 +1505,7 @@ impl<'t> Walker<'t> { self.maybe_capture_fn_refs(node); for i in 0..node.named_child_count() { if let Some(c) = node.named_child(i) { - self.scan_fn_ref_subtree(c, depth + 1); + self.scan_fn_ref_subtree(c, depth + 1, walked); } } } @@ -1624,6 +1642,30 @@ impl<'t> Walker<'t> { } } +/// propertyBodies (tree-sitter.ts) — the parts of a property that run code: +/// each accessor's `body` (a block or `=> expr`) and an expression-bodied +/// property's `=> …` value. An `= initializer` value is not a body. +fn property_bodies(node: Node) -> Vec { + let mut bodies = Vec::new(); + if let Some(accessors) = node.child_by_field_name("accessors") { + for i in 0..accessors.named_child_count() { + let Some(accessor) = accessors.named_child(i) else { continue }; + if accessor.kind() != "accessor_declaration" { + continue; + } + if let Some(body) = accessor.child_by_field_name("body") { + bodies.push(body); + } + } + } + if let Some(value) = node.child_by_field_name("value") { + if value.kind() == "arrow_expression_clause" { + bodies.push(value); + } + } + bodies +} + fn find_anonymous_class_body(node: Node) -> Option { for i in 0..node.named_child_count() { if let Some(child) = node.named_child(i) { diff --git a/docs/design/csharp-kernel-port-checklist.md b/docs/design/csharp-kernel-port-checklist.md index 41894cb067..e817968c95 100644 --- a/docs/design/csharp-kernel-port-checklist.md +++ b/docs/design/csharp-kernel-port-checklist.md @@ -234,7 +234,7 @@ all PRESERVE): | `enum_declaration` | enumTypes:1064 → extractEnum:1914 | body `enum_member_declaration_list` required (bodiless → no node); extractInheritance sees `base_list` → **the underlying type `: byte` emits an `extends` ref named `byte`** (quirk, §inheritance); `enum_member_declaration` children → extractEnumMembers:1958 — `name` field path: ONE `enum_member` node per member, positioned at the member node (attributes included in its span), values/attributes ignored; non-member children (preproc_*, comment) → visitNode (no-op) | | `method_declaration` | methodTypes:1027 → extractMethod:1737 | classifyMethodNode absent → always extractMethod. Gate 1747 passes via class-like (a method_declaration outside a type does not occur in non-erroring C# — top-level `void M(){}` parses as local_function_statement, probed); bodyless interface/partial signatures mint nodes with no body walk; **expression-bodied methods have `body: arrow_expression_clause` (a real body FIELD, probed) → walked** | | `constructor_declaration` | methodTypes → extractMethod | name field = the class-name identifier → **method node named like the class**; returnType undefined; **`constructor_initializer` (`: base(args)` / `: this(args)`) is a sibling of the body field → NEVER walked → calls inside initializer args are LOST** (probed); expression-bodied ctor body = arrow_expression_clause → walked | -| `property_declaration` (inside class-like) | propertyTypes:1075 → extractProperty:1986 | property node + scanFnRefSubtree (capture-only) + skipChildren → **accessor bodies (`get { … }`, `get => …`) and the `=> expr` value clause are NEVER walked — calls inside property getters/setters/expression bodies emit NOTHING** (only fn-ref candidates). §property below | +| `property_declaration` (inside class-like) | propertyTypes:1075 → extractProperty:1986 | property node, then **propertyBodies — each accessor's `body` (`get { … }`, `set => …`) and an expression-bodied `=> expr` `value:` — walked by visitFunctionBody with the property pushed** (calls, instantiates, static reads and fn-ref candidates attribute to the property; changed 2026-10-04, previously never walked), then scanFnRefSubtree (capture-only, attributed to the class) over the rest — **the `= initializer` stays unwalked** — skipping the walked bodies, + skipChildren. §property below | | `field_declaration` (inside class-like) | fieldTypes:1084 → extractField:2046 | field/constant nodes per declarator + scanFnRefSubtree + skipChildren → **field initializers emit no calls/instantiates/static-member refs** (fn-ref candidates only). §field below | | `local_declaration_statement` | variableTypes:1098 (only reachable at top level — global statements; body locals go through visitFunctionBody instead) | not class-like → extractVariable:2538 → **generic fallback (2863-2881) finds no direct `identifier`/`variable_declarator` children (the declarator nests inside `variable_declaration`, probed) → ZERO nodes minted**; isClassScopeConstantAssignment (1508) needs node.type `assignment` → never true. skipChildren=true + scanFnRefSubtree → **a top-level `var builder = WebApplication.CreateBuilder(args);` produces NO node, NO calls ref, NO instantiates** — only fn-ref candidates. PRESERVE | | `using_directive` | importTypes:1209 → extractImport:3170 | hook (§config) → import node + ONE generic `imports` ref {fromNodeId: nodeStack top (namespace node if present, else file), referenceName: moduleName, line/col of the directive}; **no per-binding emitter** (the TS/py/rust/php/ruby ladder at 3197-3234 excludes csharp) | @@ -305,15 +305,22 @@ for C#) or bare `name`. QUIRKS (probed, PRESERVE): - An expression-bodied property (`public int Computed => MaxItems + 1;`) has children [modifier, predefined_type, identifier, **arrow_expression_clause (`value:` field)**] → typeNode = predefined_type → signature `"int Computed"`; - the arrow clause is NEVER walked (calls inside lost). -- `{ get; } = new();` initializers: the `value:` implicit_object_creation and - the accessor_list are both skipped/excluded → no refs, no instantiates. + the arrow clause is a property body (walked as the property, below). +- `{ get; } = new();` initializers: the `value:` implicit_object_creation is + NOT a body → no refs, no instantiates (candidates-only scan, attributed to + the class); the accessor_list is excluded from the type scan. Then extractDecoratorsFor (no-op) and **extractTypeAnnotations (2037) → extractCsharpTypeRefs** — the `type` field IS walked for refs (so `public List Items` emits references `List` + `Foo` even though the signature -kept the raw text). Return value feeds no body walk (the classifyMethodNode -initializer-walk path at 1031-1047 is TS-only). +kept the raw text). The returned node is pushed while propertyBodies +(tree-sitter.ts; csharp.rs `property_bodies`) are walked: every +`accessor_declaration`'s `body` field (block or arrow_expression_clause, in +accessor order), then the property's `value:` when it is an +arrow_expression_clause. The fn-ref scan that follows skips those subtrees +by node id, so a candidate is captured once — from the property when it sits +in a body, from the class when it sits in an initializer. (The +classifyMethodNode initializer-walk path at 1031-1047 is TS-only.) ### extractField (2046) — field_declaration @@ -758,7 +765,8 @@ AspNetCore refs / Program.cs / Startup.cs / controller-source scan. + multi-declarator + instance fields (signatures `Type name`); `protected internal` (→ protected); property shapes: predefined-type, bare-identifier type (signature loses type), generic type, expression-bodied - (`=>` calls LOST), `{ get; } = new();`, accessor bodies with calls (LOST); + (`=>` calls → the property), `{ get; } = new();` (initializer LOST), + accessor bodies with calls (→ the property); event_field_declaration + event_declaration with add/remove bodies (no nodes; accessor calls → class); operator + conversion operator + indexer + destructor (no nodes; body calls → class); constructor with diff --git a/src/extraction/extraction-version.ts b/src/extraction/extraction-version.ts index 3691e95f0a..e41b5f6149 100644 --- a/src/extraction/extraction-version.ts +++ b/src/extraction/extraction-version.ts @@ -21,4 +21,4 @@ * turns the re-index hint into noise — keep it honest (see CLAUDE.md, "Honesty * in the product is load-bearing"). */ -export const EXTRACTION_VERSION = 27; +export const EXTRACTION_VERSION = 28; diff --git a/src/extraction/languages/vbnet.ts b/src/extraction/languages/vbnet.ts index b2ec4aab50..e8502b2e30 100644 --- a/src/extraction/languages/vbnet.ts +++ b/src/extraction/languages/vbnet.ts @@ -116,8 +116,13 @@ export const vbnetExtractor: LanguageExtractor = { // findable declaration (WinForms/WPF code is built around them). if (node.type === 'event_declaration' || node.type === 'custom_event_declaration') { const nameNode = node.childForFieldName('name'); - if (nameNode) { - ctx.createNode('field', getNodeText(nameNode, ctx.source), node); + const event = nameNode ? ctx.createNode('field', getNodeText(nameNode, ctx.source), node) : null; + // A Custom Event's AddHandler / RemoveHandler / RaiseEvent blocks run + // code, which is the event's. + if (event && node.type === 'custom_event_declaration') { + ctx.pushScope(event.id); + ctx.visitFunctionBody(node, event.id); + ctx.popScope(); } return true; } diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index 5c27523acc..9a10862475 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -733,8 +733,10 @@ export class TreeSitterExtractor { * nested function definitions: their bodies are walked — and their * candidates attributed — by extractFunction's own body walk. */ - private scanFnRefSubtree(node: SyntaxNode, depth: number): void { + private scanFnRefSubtree(node: SyntaxNode, depth: number, walked?: ReadonlySet): void { if (!this.fnRefSpec || depth > 12) return; + // Subtrees the body walker has already been through. + if (walked?.has(node.id)) return; const nodeType = node.type; if (depth > 0 && ( this.extractor?.functionTypes.includes(nodeType) || @@ -749,7 +751,7 @@ export class TreeSitterExtractor { this.maybeCaptureFnRefs(node, nodeType); for (let i = 0; i < node.namedChildCount; i++) { const child = node.namedChild(i); - if (child) this.scanFnRefSubtree(child, depth + 1); + if (child) this.scanFnRefSubtree(child, depth + 1, walked); } } @@ -1279,11 +1281,19 @@ export class TreeSitterExtractor { } // Check for class properties (e.g. C# property_declaration) else if (this.extractor.propertyTypes?.includes(nodeType) && this.isInsideClassLikeNode()) { - this.extractProperty(node); - // Property initializers aren't walked — scan for function-as-value - // candidates (#756): Scala `val table = Seq(targetCb)` in an object, - // Kotlin `val cb = ::handler` class properties. - this.scanFnRefSubtree(node, 0); + const propNode = this.extractProperty(node); + // The code a property runs is the property's: its calls, + // instantiations and reads attribute to it, as a method's do. + const bodies = propNode ? this.propertyBodies(node) : []; + if (propNode && bodies.length > 0) { + this.nodeStack.push(propNode.id); + for (const body of bodies) this.visitFunctionBody(body, propNode.id); + this.nodeStack.pop(); + } + // Whatever the body walk didn't cover (a C# `= initializer`, any other + // language's whole declaration) is scanned for function-as-value + // candidates (#756); the bodies captured their own. + this.scanFnRefSubtree(node, 0, new Set(bodies.map((b) => b.id))); skipChildren = true; } // Check for class fields (e.g. Java field_declaration, C# field_declaration) @@ -2113,7 +2123,12 @@ export class TreeSitterExtractor { // (#1093) because the two defaults differ — a bodiless CLASS is kept // unless a language opts into skipping, a bodiless STRUCT is skipped // unless a language opts into keeping. - const body = getChildByField(node, this.extractor.bodyField); + // + // resolveBody first, as for classes and enums: a VB.NET Structure tags + // every member as its own `body` field, so the field alone yields only + // the first member. + const body = this.extractor.resolveBody?.(node, this.extractor.bodyField) + ?? getChildByField(node, this.extractor.bodyField); if (!body && node.type !== 'record_declaration' && !this.extractor.allowBodilessStruct) return; @@ -2304,6 +2319,28 @@ export class TreeSitterExtractor { return propNode; } + /** + * The parts of a property declaration that run code, for the body walker. + * VB.NET writes its `Get` / `Set` blocks, `= initializer` and `As New T` + * as children of the declaration itself, so the declaration is walked + * whole, the way its methods are (resolveBody). C# runs code in each + * accessor's body (`get { … }`, `set => …`) and in an expression-bodied + * property's `=> …`; its `= initializer`, like a field's, stays unwalked. + * Mirrored in the kernel (csharp.rs property_bodies). + */ + private propertyBodies(node: SyntaxNode): SyntaxNode[] { + if (this.language === 'vbnet') return [node]; + if (this.language !== 'csharp') return []; + const bodies: SyntaxNode[] = []; + for (const accessor of getChildByField(node, 'accessors')?.namedChildren ?? []) { + const body = accessor.type === 'accessor_declaration' ? getChildByField(accessor, 'body') : null; + if (body) bodies.push(body); + } + const value = getChildByField(node, 'value'); + if (value?.type === 'arrow_expression_clause') bodies.push(value); + return bodies; + } + /** * Extract a class field declaration (e.g. Java field_declaration, C# field_declaration). * Extracts each declarator as a 'field' kind node inside the owning class. @@ -2408,10 +2445,11 @@ export class TreeSitterExtractor { // candidates, so a lambda / method reference / anonymous class in // `private final Runnable r = () -> target();` contributed NO call // edge at all and `target` looked callerless. Keyed on the `value` - // FIELD, which only Java's `variable_declarator` carries — C#, - // VB.NET and PHP spell their initializer differently and are - // deliberately untouched here. - const valueNode = getChildByField(decl, 'value'); + // FIELD, which only Java's `variable_declarator` carries. VB.NET + // writes `= expr` (the declarator's `initializer`) or `As New T(…)` + // (inside its as_clause), so its whole declarator is walked. C# + // and PHP spell their initializer differently and are untouched. + const valueNode = this.language === 'vbnet' ? decl : getChildByField(decl, 'value'); if (valueNode) { this.nodeStack.push(fieldNode.id); this.visitFunctionBody(valueNode, fieldNode.id); From 511d86e94dc9f26e74e32741477b6ffaae1695a7 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 14:45:10 +0000 Subject: [PATCH 171/259] fix(mcp): replace a daemon left running by an older install (#2335) (#2343) A daemon keeps running the code it started with. The 1.6.1 daemon in #2335 outlived an upgrade, kept loading grammars from the install the upgrade removed and wrote every file it re-indexed as empty, while every session from the new install served itself in-process, read-only, because the old daemon held the writer lock. Daemons from before this change cannot notice an upgrade, so the launcher acts: - A launcher that finds a daemon of an older plain X.Y.Z release stops it the way `codegraph daemon` does (daemon.pid and the socket hello must agree; SIGTERM, SIGKILL only if it still answers) and starts one from its own install. It finds the daemon by its hello on the launcher's own socket, or by the project lock when it listens elsewhere: on Windows, daemons before #2278 named their pipe after the root as typed. A daemon of the same, a newer, a prerelease or an unknown version is never touched, so two installs cannot take turns stopping each other's. - The writer slot never falls free on the way. The launcher swaps writer.pid to itself (mode `handover`) before the signal, clears the old daemon's leftovers while it holds the slot, and the daemon it spawns takes the slot over (CODEGRAPH_DAEMON_HANDOVER). Sessions of the old daemon that fall back in-process find the slot held and serve reads only, instead of claiming it with the removed install's code. A daemon that does not exit gets the slot back, and a successor outwaits a racing candidate that took daemon.pid first. Validated on Windows with real npm upgrades from the published 1.6.1 and 1.6.2 to this build packed as 1.6.3, with the old session calling throughout. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/daemon-older-version.test.ts | 716 +++++++++++++++++++++++++ __tests__/fixtures/older-daemon.cjs | 36 ++ src/mcp/daemon-registry.ts | 173 +++++- src/mcp/daemon.ts | 19 +- src/mcp/index.ts | 144 ++++- src/mcp/project-lifecycle.ts | 3 +- src/mcp/proxy.ts | 24 +- src/mcp/version.ts | 25 +- src/mcp/writer-lock.ts | 38 ++ 10 files changed, 1120 insertions(+), 59 deletions(-) create mode 100644 __tests__/daemon-older-version.test.ts create mode 100644 __tests__/fixtures/older-daemon.cjs diff --git a/CHANGELOG.md b/CHANGELOG.md index 44144e1606..32db6b23ab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In VB.NET, every member of a `Structure` is now indexed, including its fields, properties, methods, constructors and nested enums. Before, only the first member was, so the rest could not be found and their callers looked empty. - In VB.NET and C#, what a property's `Get` and `Set` code calls, creates and reads now belongs to that property, as do C#'s `get => …` accessors and `=> …` property bodies. Before, it was dropped, so a method used only from a property looked unused. - In VB.NET, a field or property initializer like `= Compute()` or `As New List(Of Order)` now links what it calls and creates, and so do a `Custom Event`'s `AddHandler`, `RemoveHandler` and `RaiseEvent` blocks. Re-index VB.NET and C# projects after upgrading. +- Upgrading CodeGraph while an agent session is open no longer leaves the old version's background server in charge of your project: the first session started from the new install stops it and starts a current one in its place, even while sessions opened before the upgrade are still running. That old server could no longer load the language parsers the upgrade removed, so it saved every file it re-indexed with no symbols, while new sessions could only read the index beside it without keeping it up to date. A background server from a newer install is never stopped, and sessions opened before the upgrade keep the old version until you restart them. Thanks @lipchey for the report. (#2335) ## [1.6.2] - 2026-10-03 diff --git a/__tests__/daemon-older-version.test.ts b/__tests__/daemon-older-version.test.ts new file mode 100644 index 0000000000..188cd6cf82 --- /dev/null +++ b/__tests__/daemon-older-version.test.ts @@ -0,0 +1,716 @@ +/** + * A launcher replaces a daemon from an older install (#2335). + * + * A daemon keeps running the code it started with. The 1.6.1 daemon in #2335 + * outlived an upgrade to 1.6.2, went on loading grammars from the install the + * upgrade removed, and wrote every file its watcher re-indexed as empty. A + * launcher that found it could only serve its own session in-process, read-only: + * the old daemon held the project's writer lock and file watcher, so no daemon + * from the new install could start. Daemons from before the fix cannot notice + * the upgrade themselves, so the launcher acts: a daemon of an OLDER release is + * stopped the way `codegraph daemon` stops one (identity-checked SIGTERM, a + * graceful shutdown on POSIX) and one from the launcher's own install takes its + * place. The launcher hears an older daemon's hello on its own socket, or — + * when it listens elsewhere (on Windows, daemons before #2278 named their pipe + * after the root as typed) — finds it by the project lock. A daemon of the + * same, a newer or an unknown version is never touched, so two installed + * versions cannot fight over the daemon. + * + * The end-to-end cases run real daemons of other versions: this build's dist/ + * linked into a temp "install" whose package.json carries another version. A + * daemon resolves its version, and every grammar it loads, from its own files, + * so that install's daemon advertises that version in its lock and its hello. + */ +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'; +import { ChildProcessWithoutNullStreams, execFileSync, spawn } from 'child_process'; +import { once } from 'events'; +import * as fs from 'fs'; +import * as net from 'net'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import { encodeLockInfo, getDaemonPidPath, getDaemonSocketPath } from '../src/mcp/daemon-paths'; +import { isProcessAlive, stopOlderDaemon } from '../src/mcp/daemon-registry'; +import { connectWithHello } from '../src/mcp/proxy'; +import { CodeGraphPackageVersion, isOlderRelease } from '../src/mcp/version'; +import { getWriterPidPath, readWriterLock, swapWriterLock } from '../src/mcp/writer-lock'; +import { WASM_RUNTIME_FLAGS } from '../src/extraction/wasm-runtime-flags'; +import { recordSpawns, removeSpawnLog, settleLosingCandidates } from './daemon-candidates'; + +const BIN = path.resolve(__dirname, '../dist/bin/codegraph.js'); +const DIST = path.resolve(__dirname, '../dist'); +const NODE_MODULES = path.resolve(__dirname, '../node_modules'); +const OLDER_DAEMON = path.resolve(__dirname, 'fixtures/older-daemon.cjs'); + +/** Plain releases always below / above any version this repo will carry. */ +const OLDER = '0.0.1'; +const NEWER = '999.0.0'; + +describe('isOlderRelease', () => { + it('orders plain releases numerically', () => { + expect(isOlderRelease('1.6.1', '1.6.2')).toBe(true); + expect(isOlderRelease('1.6.9', '1.6.10')).toBe(true); + expect(isOlderRelease('1.9.0', '1.10.0')).toBe(true); + expect(isOlderRelease('0.99.99', '1.0.0')).toBe(true); + expect(isOlderRelease('1.6.2', '1.6.2')).toBe(false); + expect(isOlderRelease('1.6.3', '1.6.2')).toBe(false); + expect(isOlderRelease('2.0.0', '1.99.99')).toBe(false); + }); + + it('never calls a prerelease, a build or an unknown version older, in either position', () => { + for (const odd of ['0.0.0-unknown', '0.0.0-mismatch', '1.6.1-beta.1', '1.6.1+local', 'v1.6.1', 'unknown', 'test', '', '1.6']) { + expect(isOlderRelease(odd, '1.6.2'), odd).toBe(false); + expect(isOlderRelease('1.6.1', odd), odd).toBe(false); + } + }); + + it('lets at most one of two versions replace the other', () => { + const versions = ['0.9.0', '1.6.1', '1.6.2', '1.6.10', '1.7.0', '2.0.0', '1.6.2-rc.1', '0.0.0-unknown']; + for (const a of versions) { + for (const b of versions) { + expect(isOlderRelease(a, b) && isOlderRelease(b, a), `${a} / ${b}`).toBe(false); + } + } + }); +}); + +/** A path a test server can listen on: a named pipe on Windows, a socket file elsewhere. */ +function listenPath(dir: string, name: string): string { + return process.platform === 'win32' + ? `\\\\.\\pipe\\${path.basename(dir)}-${name}` + : path.join(dir, `${name}.sock`); +} + +async function listen(server: net.Server, socketPath: string): Promise { + await new Promise((resolve, reject) => { + server.once('error', reject); + server.listen(socketPath, resolve); + }); +} + +describe('connectWithHello', () => { + let dir: string; + let server: net.Server | null = null; + + beforeEach(() => { dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-hello-')); }); + afterEach(async () => { + if (server) await new Promise((resolve) => server!.close(() => resolve())); + server = null; + fs.rmSync(dir, { recursive: true, force: true }); + }); + + it.each([ + [OLDER, 'older-version', false], + [NEWER, 'version-mismatch', true], + ['0.0.0-unknown', 'version-mismatch', true], + ])('reports a %s daemon as %s', async (version, expected, logged) => { + const socketPath = listenPath(dir, 'daemon'); + server = net.createServer((socket) => { + socket.write(JSON.stringify({ codegraph: version, pid: process.pid, socketPath, protocol: 1 }) + '\n'); + }); + await listen(server, socketPath); + const stderr = vi.spyOn(process.stderr, 'write').mockImplementation(() => true); + try { + expect(await connectWithHello(socketPath, '1.6.2')).toBe(expected); + // An older daemon is the launcher's to report; it is not a mismatch to serve around. + expect(stderr.mock.calls.some(([line]) => String(line).includes('differs from ours'))).toBe(logged); + } finally { + stderr.mockRestore(); + } + }); +}); + +describe('swapWriterLock', () => { + let root: string; + beforeEach(() => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-writer-swap-')); + fs.mkdirSync(path.join(root, '.codegraph')); + }); + afterEach(() => { fs.rmSync(root, { recursive: true, force: true }); }); + + const record = (pid: number, mode: string) => ({ pid, mode, startedAt: 1, ready: false }); + + it('replaces the record of the given pid, and only that record', () => { + const file = getWriterPidPath(root); + fs.writeFileSync(file, JSON.stringify(record(111, 'daemon')) + '\n'); + expect(swapWriterLock(root, 222, record(333, 'handover'))).toBe(false); + expect(readWriterLock(root)).toMatchObject({ pid: 111, mode: 'daemon' }); + expect(swapWriterLock(root, 111, record(333, 'handover'))).toBe(true); + expect(readWriterLock(root)).toMatchObject({ pid: 333, mode: 'handover' }); + // A slot nobody holds is not handed over: it is acquired the usual way. + fs.rmSync(file); + expect(swapWriterLock(root, 333, record(444, 'daemon'))).toBe(false); + expect(fs.existsSync(file)).toBe(false); + expect(fs.readdirSync(path.join(root, '.codegraph'))).toEqual([]); + }); +}); + +/** A live process to stand in for a daemon; signals reach it, never the test. */ +function startDetachedProcess(): number { + const source = [ + "const { spawn } = require('child_process');", + "const child = spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)'], { detached: true, stdio: 'ignore' });", + 'child.unref();', + 'console.log(child.pid);', + ].join(' '); + const pid = Number(execFileSync(process.execPath, ['-e', source], { encoding: 'utf8' }).trim()); + if (!Number.isInteger(pid) || pid <= 0) throw new Error('could not start daemon fixture'); + return pid; +} + +async function deadPid(): Promise { + const child = spawn(process.execPath, ['-e', 'process.exit(0)']); + await once(child, 'exit'); + await new Promise((r) => setTimeout(r, 50)); + return child.pid!; +} + +describe('stopOlderDaemon', () => { + let root: string; + let pidPath: string; + let socketPath: string; + let server: net.Server | null; + let pids: number[]; + + beforeEach(() => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-older-daemon-')); + fs.mkdirSync(path.join(root, '.codegraph')); + pidPath = getDaemonPidPath(root); + socketPath = getDaemonSocketPath(root); + server = null; + pids = []; + }); + + afterEach(async () => { + vi.restoreAllMocks(); + for (const pid of pids) if (isProcessAlive(pid)) process.kill(pid, 'SIGKILL'); + if (server) await new Promise((resolve) => server!.close(() => resolve())); + fs.rmSync(root, { recursive: true, force: true }); + }); + + /** A daemon of `version`: its lock, and a socket answering the hello as `helloPid`. */ + async function fakeDaemon(version: string, helloPid?: number): Promise<{ pid: number; lock: string }> { + const pid = startDetachedProcess(); + pids.push(pid); + server = net.createServer((socket) => { + socket.end(JSON.stringify({ protocol: 1, codegraph: version, pid: helloPid ?? pid, socketPath }) + '\n'); + }); + await listen(server, socketPath); + const lock = encodeLockInfo({ pid, version, socketPath, startedAt: Date.now() }); + fs.writeFileSync(pidPath, lock); + return { pid, lock }; + } + + /** Signals sent to `pid`, other than the signal-0 liveness probe. */ + function signalsTo(kill: ReturnType, pid: number): unknown[] { + return kill.mock.calls.filter(([target, signal]) => target === pid && signal !== 0).map(([, signal]) => signal); + } + + /** The writer record, decoded. */ + const writerRecord = (): any => { + try { return JSON.parse(fs.readFileSync(getWriterPidPath(root), 'utf8')); } catch { return null; } + }; + + it('stops a verified daemon of an older release and clears its lock', async () => { + const { pid } = await fakeDaemon('1.6.1'); + const result = await stopOlderDaemon(root, '1.6.2'); + expect(result).toMatchObject({ pid, outcome: 'term', version: '1.6.1' }); + expect(isProcessAlive(pid)).toBe(false); + expect(fs.existsSync(pidPath)).toBe(false); + // Kept for the daemon the caller starts next. + expect(writerRecord()).toMatchObject({ pid: process.pid, mode: 'handover' }); + }, 15000); + + it('holds the writer slot from before the signal, so it is never free or stale', async () => { + const { pid } = await fakeDaemon('1.6.1'); + fs.writeFileSync(getWriterPidPath(root), JSON.stringify({ pid, mode: 'daemon', startedAt: 1, ready: true }) + '\n'); + let atSignal: unknown = null; + const originalKill = process.kill.bind(process); + vi.spyOn(process, 'kill').mockImplementation((target, signal) => { + if (target === pid && signal === 'SIGTERM') atSignal = writerRecord(); + return originalKill(target, signal); + }); + expect(await stopOlderDaemon(root, '1.6.2')).toMatchObject({ pid, outcome: 'term' }); + // A session of the old daemon serving itself the moment it went found the + // slot held, and so served reads only. + expect(atSignal).toMatchObject({ pid: process.pid, mode: 'handover' }); + expect(writerRecord()).toMatchObject({ pid: process.pid, mode: 'handover' }); + expect(fs.existsSync(pidPath)).toBe(false); + }, 15000); + + it('gives the writer slot back to an older daemon that does not exit', async () => { + const { pid, lock } = await fakeDaemon('1.6.1'); + const record = { pid, mode: 'daemon', startedAt: 1, ready: true }; + fs.writeFileSync(getWriterPidPath(root), JSON.stringify(record) + '\n'); + // The signal reaches nothing: the daemon closes its socket, as its own + // shutdown does first, and then never exits. + const originalKill = process.kill.bind(process); + vi.spyOn(process, 'kill').mockImplementation((target, signal) => { + if (target === pid && signal === 'SIGTERM') { server?.close(); return true; } + return originalKill(target, signal); + }); + expect(await stopOlderDaemon(root, '1.6.2', { shutdownGraceMs: 300 })).toMatchObject({ pid, outcome: 'still-running' }); + expect(writerRecord()).toEqual(record); + expect(fs.readFileSync(pidPath, 'utf8')).toBe(lock); + expect(isProcessAlive(pid)).toBe(true); + }, 15000); + + it('leaves the daemon to another launcher already holding the slot to replace it', async () => { + const { pid, lock } = await fakeDaemon('1.6.1'); + const other = startDetachedProcess(); + pids.push(other); + const claim = { pid: other, mode: 'handover', startedAt: 1, ready: false }; + fs.writeFileSync(getWriterPidPath(root), JSON.stringify(claim) + '\n'); + const kill = vi.spyOn(process, 'kill'); + expect(await stopOlderDaemon(root, '1.6.2')).toBeNull(); + expect(signalsTo(kill, pid)).toEqual([]); + expect(writerRecord()).toEqual(claim); + expect(fs.readFileSync(pidPath, 'utf8')).toBe(lock); + }); + + it.each(['1.6.2', '1.6.3', '2.0.0', '1.6.1-beta.1', '0.0.0-unknown', 'test'])( + 'never signals a daemon of version %s', + async (version) => { + const { pid, lock } = await fakeDaemon(version); + const kill = vi.spyOn(process, 'kill'); + expect(await stopOlderDaemon(root, '1.6.2')).toBeNull(); + expect(signalsTo(kill, pid)).toEqual([]); + expect(isProcessAlive(pid)).toBe(true); + expect(fs.readFileSync(pidPath, 'utf8')).toBe(lock); + }, + ); + + it('leaves an older lock alone when its socket answers for another process (#1553)', async () => { + const { pid, lock } = await fakeDaemon('1.6.1', process.pid); + const kill = vi.spyOn(process, 'kill'); + expect(await stopOlderDaemon(root, '1.6.2')).toMatchObject({ pid, outcome: 'unverified' }); + expect(signalsTo(kill, pid)).toEqual([]); + expect(signalsTo(kill, process.pid)).toEqual([]); + expect(isProcessAlive(pid)).toBe(true); + expect(fs.readFileSync(pidPath, 'utf8')).toBe(lock); + }); + + it('reports an older daemon that already exited, leaving its lock to its successor', async () => { + const lock = encodeLockInfo({ pid: await deadPid(), version: '1.6.1', socketPath, startedAt: Date.now() }); + fs.writeFileSync(pidPath, lock); + expect(await stopOlderDaemon(root, '1.6.2')).toMatchObject({ outcome: 'not-running', version: '1.6.1' }); + expect(fs.readFileSync(pidPath, 'utf8')).toBe(lock); + }); + + it('has nothing to stop without a lock, and leaves a listening socket alone', async () => { + server = net.createServer((socket) => { + socket.end(JSON.stringify({ protocol: 1, codegraph: '1.6.1', pid: process.pid, socketPath }) + '\n'); + }); + await listen(server, socketPath); + expect(await stopOlderDaemon(root, '1.6.2')).toBeNull(); + expect(await connectWithHello(socketPath, '1.6.2')).toBe('older-version'); + }); + + it('does not act on an unreadable lock', async () => { + fs.mkdirSync(pidPath); + expect(await stopOlderDaemon(root, '1.6.2')).toEqual({ root, pid: null, outcome: 'unverified' }); + }); +}); + +/** + * A CodeGraph install of `version`: this build's dist/, hard-linked file by + * file (copied where a link is refused), next to a package.json of that version + * and a link to this checkout's node_modules. + */ +function makeInstall(version: string): string { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), `cg-install-${version}-`)); + const linkTree = (from: string, to: string): void => { + fs.mkdirSync(to, { recursive: true }); + for (const entry of fs.readdirSync(from, { withFileTypes: true })) { + const source = path.join(from, entry.name); + const target = path.join(to, entry.name); + if (entry.isDirectory()) linkTree(source, target); + else { + try { fs.linkSync(source, target); } catch { fs.copyFileSync(source, target); } + } + } + }; + linkTree(DIST, path.join(dir, 'dist')); + fs.symlinkSync(NODE_MODULES, path.join(dir, 'node_modules'), 'junction'); + fs.writeFileSync(path.join(dir, 'package.json'), JSON.stringify({ name: '@colbymchenry/codegraph', version }) + '\n'); + return dir; +} + +interface Session { + child: ChildProcessWithoutNullStreams; + stdout: string[]; + stderr: string[]; +} + +function lines(stream: NodeJS.ReadableStream, into: string[]): void { + let buffer = ''; + stream.on('data', (chunk: Buffer) => { + buffer += chunk.toString('utf8'); + let idx: number; + while ((idx = buffer.indexOf('\n')) !== -1) { + into.push(buffer.slice(0, idx)); + buffer = buffer.slice(idx + 1); + } + }); +} + +function findResponse(stdout: string[], id: number): any | null { + for (const line of stdout) { + try { + const parsed = JSON.parse(line); + if (parsed && parsed.id === id && (parsed.result !== undefined || parsed.error !== undefined)) return parsed; + } catch { /* not JSON */ } + } + return null; +} + +function waitFor(predicate: () => T | undefined | null | false, timeoutMs: number, label: string): Promise { + return new Promise((resolve, reject) => { + const started = Date.now(); + const tick = (): void => { + let value: T | undefined | null | false; + try { value = predicate(); } catch (e) { reject(e); return; } + if (value) { resolve(value as T); return; } + if (Date.now() - started > timeoutMs) { reject(new Error(`Timed out after ${timeoutMs}ms waiting for: ${label}`)); return; } + setTimeout(tick, 25); + }; + tick(); + }); +} + +function isAlive(pid: number): boolean { + try { process.kill(pid, 0); return true; } catch { return false; } +} + +async function waitProcessExit(pid: number, timeoutMs: number): Promise { + return waitFor(() => !isAlive(pid), timeoutMs, `pid ${pid} to exit`).then(() => true, () => false); +} + +function readJson(file: string): any | null { + try { return JSON.parse(fs.readFileSync(file, 'utf8')); } catch { return null; } +} + +describe('a launcher meeting a daemon of another version (#2335)', () => { + let olderInstall: string; + let newerInstall: string; + let tempDir: string; + let realRoot: string; + const sessions: Session[] = []; + const daemonPids = new Set(); + + beforeAll(() => { + olderInstall = makeInstall(OLDER); + newerInstall = makeInstall(NEWER); + }); + + afterAll(async () => { + for (const dir of [olderInstall, newerInstall]) { + if (!dir) continue; + // The link first: removing an install must never reach this checkout's modules. + try { fs.unlinkSync(path.join(dir, 'node_modules')); } catch { /* not created */ } + await fs.promises.rm(dir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); + } + }); + + beforeEach(async () => { + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-older-version-')); + fs.writeFileSync(path.join(tempDir, 'app.ts'), 'export function appMain() { return 1; }\n'); + const cg = await CodeGraph.init(tempDir); + try { await cg.indexAll(); } finally { cg.close(); } + realRoot = fs.realpathSync(tempDir); + }); + + afterEach(async () => { + // Every launcher first (with the runtime flags there is no relaunch child), + // then the losing daemon candidates, then the daemons themselves — and only + // then the fixture: on Windows a live process inside it blocks the removal. + await Promise.all(sessions.map(async ({ child }) => { + if (child.exitCode !== null || child.signalCode !== null) return; + const exited = once(child, 'exit'); + child.kill('SIGKILL'); + await exited; + })); + sessions.length = 0; + const lockPid = (): number | null => readJson(path.join(realRoot, '.codegraph', 'daemon.pid'))?.pid ?? null; + await settleLosingCandidates(tempDir, lockPid); + const holder = lockPid(); + if (holder) daemonPids.add(holder); + for (const pid of daemonPids) { + if (pid === process.pid || !isAlive(pid)) continue; + try { process.kill(pid, 'SIGKILL'); } catch { /* raced to exit */ } + await waitProcessExit(pid, 5000); + } + daemonPids.clear(); + removeSpawnLog(tempDir); + await fs.promises.rm(tempDir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); + }, 45_000); + + /** An agent session: `install`'s launcher, `serve --mcp` in the project. */ + function startSession(install: string | null, env: NodeJS.ProcessEnv = {}): Session { + const bin = install ? path.join(install, 'dist', 'bin', 'codegraph.js') : BIN; + const recorder = recordSpawns(tempDir); + const child = spawn(process.execPath, [...WASM_RUNTIME_FLAGS, ...recorder.args, bin, 'serve', '--mcp'], { + cwd: tempDir, + stdio: ['pipe', 'pipe', 'pipe'], + env: { + ...process.env, + CODEGRAPH_MCP_LOG_ATTACH: '1', + CODEGRAPH_DAEMON_IDLE_TIMEOUT_MS: '30000', + ...recorder.env, + ...env, + }, + }) as ChildProcessWithoutNullStreams; + child.on('error', () => { /* ignore */ }); + child.stdin.on('error', () => { /* ignore */ }); + const session: Session = { child, stdout: [], stderr: [] }; + lines(child.stdout, session.stdout); + lines(child.stderr, session.stderr); + sessions.push(session); + send(session, { + jsonrpc: '2.0', id: 1, method: 'initialize', + params: { protocolVersion: '2024-11-05', capabilities: {}, clientInfo: { name: 'test', version: '0.0.0' }, rootUri: `file://${tempDir}` }, + }); + return session; + } + + function send(session: Session, msg: unknown): void { + try { session.child.stdin.write(JSON.stringify(msg) + '\n'); } catch { /* gone */ } + } + + async function status(session: Session, id: number, label: string): Promise { + send(session, { jsonrpc: '2.0', id, method: 'tools/call', params: { name: 'codegraph_status', arguments: {} } }); + const reply = await waitFor(() => findResponse(session.stdout, id), 20000, label).catch((err: Error) => { + throw new Error(`${err.message}\n--- stderr ---\n${session.stderr.join('\n')}\n--- daemon.log ---\n${daemonLog()}`); + }); + expect(reply.error, label).toBeUndefined(); + expect(reply.result?.isError, label).not.toBe(true); + expect(JSON.stringify(reply.result), label).toContain('CodeGraph Status'); + return reply; + } + + const attachedLine = (session: Session): string | undefined => + session.stderr.find((l) => l.includes('Attached to shared daemon')); + const daemonLock = (): any => readJson(path.join(realRoot, '.codegraph', 'daemon.pid')); + const writerLock = (): any => readJson(path.join(realRoot, '.codegraph', 'writer.pid')); + const daemonLog = (): string => { + try { return fs.readFileSync(path.join(realRoot, '.codegraph', 'daemon.log'), 'utf8'); } catch { return ''; } + }; + + /** A session of `install` attached to a daemon of its own, which owns the project. */ + async function sessionWithDaemon(install: string, version: string, env: NodeJS.ProcessEnv = {}): Promise<{ session: Session; pid: number }> { + const session = startSession(install, env); + await waitFor(() => attachedLine(session), 20000, `the ${version} session to attach`); + // Attached means listening, not initialized: let the engine open the + // database before anything stops this daemon. + await status(session, 2, `a ${version} tool call`); + const lock = daemonLock(); + expect(lock).toMatchObject({ version }); + expect(writerLock()).toMatchObject({ pid: lock.pid, mode: 'daemon' }); + daemonPids.add(lock.pid); + return { session, pid: lock.pid }; + } + + /** A daemon process for the project, started the way a launcher starts one. */ + function startDaemon(env: NodeJS.ProcessEnv): { child: ChildProcessWithoutNullStreams; log: string[] } { + const child = spawn(process.execPath, [...WASM_RUNTIME_FLAGS, BIN, 'serve', '--mcp', '--path', realRoot], { + cwd: tempDir, + stdio: ['pipe', 'pipe', 'pipe'], + env: { ...process.env, CODEGRAPH_DAEMON_INTERNAL: '1', CODEGRAPH_DAEMON_IDLE_TIMEOUT_MS: '30000', ...env }, + }) as ChildProcessWithoutNullStreams; + child.on('error', () => { /* ignore */ }); + child.stdout.resume(); + const log: string[] = []; + lines(child.stderr, log); + daemonPids.add(child.pid!); + return { child, log }; + } + + /** The slot as a launcher that just stopped an older daemon holds it: this test process stands in. */ + function holdWriterSlotForSuccessor(): string { + const claim = JSON.stringify({ pid: process.pid, mode: 'handover', startedAt: Date.now(), ready: false }) + '\n'; + fs.writeFileSync(path.join(realRoot, '.codegraph', 'writer.pid'), claim); + return claim; + } + + it('a daemon takes over the writer slot held for it, and only that one', async () => { + const claim = holdWriterSlotForSuccessor(); + + // A daemon the slot is not held for finds it held, as by any live writer. + const stranger = startDaemon({}); + const [strangerCode] = await once(stranger.child, 'exit'); + expect(strangerCode).toBe(1); + expect(stranger.log.join('\n')).toContain(`writer lock held by PID ${process.pid} (handover mode)`); + expect(fs.readFileSync(path.join(realRoot, '.codegraph', 'writer.pid'), 'utf8')).toBe(claim); + + const successor = startDaemon({ CODEGRAPH_DAEMON_HANDOVER: String(process.pid) }); + await waitFor( + () => daemonLock()?.pid === successor.child.pid && successor.log.some((l) => l.includes('Listening on')), + 20000, 'the successor to listen', + ); + expect(writerLock()).toMatchObject({ pid: successor.child.pid, mode: 'daemon' }); + expect(successor.log.join('\n')).toContain(`Took over the writer lock from launcher pid ${process.pid}`); + }, 60000); + + it('a daemon the slot is held for outwaits a candidate still starting in its way', async () => { + holdWriterSlotForSuccessor(); + // A racing launcher's candidate holds the daemon lock, still starting; it + // cannot get the writer slot, so it will give up. + const candidate = startDetachedProcess(); + daemonPids.add(candidate); + const pidPath = path.join(realRoot, '.codegraph', 'daemon.pid'); + const candidateLock = encodeLockInfo({ + pid: candidate, version: CodeGraphPackageVersion, socketPath: getDaemonSocketPath(realRoot), startedAt: Date.now(), + }); + fs.writeFileSync(pidPath, candidateLock); + + // Any other daemon yields to it at once. + const yielding = startDaemon({}); + const [yieldingCode] = await once(yielding.child, 'exit'); + expect(yieldingCode).toBe(0); + expect(yielding.log.join('\n')).toContain(`Another daemon (pid ${candidate}) already holds the lock`); + + const successor = startDaemon({ CODEGRAPH_DAEMON_HANDOVER: String(process.pid) }); + await waitFor( + () => successor.log.some((l) => l.includes(`Waiting for daemon candidate pid ${candidate}`)), + 20000, 'the successor to wait for the candidate', + ); + expect(successor.child.exitCode).toBeNull(); + expect(fs.readFileSync(pidPath, 'utf8')).toBe(candidateLock); + // The candidate gives up: its lock goes, then so does it. + fs.rmSync(pidPath); + process.kill(candidate, 'SIGKILL'); + await waitFor( + () => daemonLock()?.pid === successor.child.pid && successor.log.some((l) => l.includes('Listening on')), + 20000, 'the successor to listen', + ); + expect(writerLock()).toMatchObject({ pid: successor.child.pid, mode: 'daemon' }); + }, 60000); + + it('keeps the older daemon\'s sessions read-only through the replacement, even mid-call', async () => { + const before = await sessionWithDaemon(olderInstall, OLDER); + // The older session calls every 20ms throughout, so it serves itself + // in-process the instant its daemon goes. Finding the writer slot free + // then, it would claim it — with the code of the install being replaced. + let next = 100; + const calling = setInterval(() => { + send(before.session, { jsonrpc: '2.0', id: next++, method: 'tools/call', params: { name: 'codegraph_status', arguments: {} } }); + }, 20); + let after: Session; + try { + after = startSession(null); + await waitFor(() => attachedLine(after), 30000, 'the new session to attach'); + await new Promise((r) => setTimeout(r, 500)); + } finally { + clearInterval(calling); + } + const lock = daemonLock(); + daemonPids.add(lock.pid); + expect(lock.version).toBe(CodeGraphPackageVersion); + expect(writerLock()).toMatchObject({ pid: lock.pid, mode: 'daemon' }); + await status(after, 2, 'a tool call through the new daemon'); + + const last = next - 1; + await waitFor(() => findResponse(before.session.stdout, last), 20000, 'the older session to answer every call'); + for (let id = 100; id <= last; id++) expect(findResponse(before.session.stdout, id), `call ${id}`).toBeTruthy(); + const stderr = before.session.stderr.join('\n'); + expect(stderr).toContain('Serving reads in-process without auto-sync'); + expect(stderr).not.toContain('File watcher active'); + expect(writerLock()).toMatchObject({ pid: lock.pid, mode: 'daemon' }); + }, 120000); + + it('stops a daemon from an older install and starts one from its own', async () => { + const before = await sessionWithDaemon(olderInstall, OLDER); + + const after = startSession(null); + const stopped = await waitFor( + () => after.stderr.find((l) => l.includes('[CodeGraph MCP] Stopped')), + 30000, 'the older daemon to be stopped', + ); + expect(stopped).toContain(`the CodeGraph ${OLDER} daemon (pid ${before.pid})`); + expect(stopped).toContain(`from this install (${CodeGraphPackageVersion})`); + expect(await waitProcessExit(before.pid, 10000)).toBe(true); + + const attached = await waitFor(() => attachedLine(after), 20000, 'the new session to attach'); + const lock = daemonLock(); + daemonPids.add(lock.pid); + expect(lock.version).toBe(CodeGraphPackageVersion); + expect(lock.pid).not.toBe(before.pid); + expect(attached).toContain(`(pid ${lock.pid}, v${CodeGraphPackageVersion})`); + await status(after, 2, 'a tool call through the new daemon'); + // The replacement owns updates for the project: writer lock and watcher, + // handed to it by the launcher that held the slot through the stop. + expect(writerLock()).toMatchObject({ pid: lock.pid, mode: 'daemon' }); + expect(daemonLog()).toContain(`Took over the writer lock from launcher pid ${after.child.pid}`); + if (process.platform !== 'win32') { + // SIGTERM ran the old daemon's own shutdown. (On Windows it is + // TerminateProcess, and the stop clears what is left.) + expect(daemonLog()).toContain(`Shutting down (SIGTERM`); + } + + // The session from before the upgrade keeps answering, read-only, and + // leaves the new daemon its writer lock. + await waitFor( + () => before.session.stderr.some((l) => l.includes('Shared daemon connection lost')), + 10000, 'the older session to lose its daemon', + ); + await status(before.session, 3, 'a tool call in the older session'); + expect(before.session.stderr.some((l) => l.includes('Serving reads in-process without auto-sync'))).toBe(true); + expect(writerLock()).toMatchObject({ pid: lock.pid, mode: 'daemon' }); + expect(daemonLock()).toEqual(lock); + expect(isAlive(lock.pid)).toBe(true); + }, 120000); + + it('finds an older daemon its probe cannot reach by the project lock, and stops it', async () => { + // On Windows a 1.6.1 daemon named its pipe after the root as typed (#2278), + // so a newer launcher's probe never hears its hello; only its lock says + // where it listens. Stand one up on a socket no launcher probes. + const socketDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-sock-')); + try { + const socketPath = listenPath(socketDir, 'older'); + const older = spawn(process.execPath, [OLDER_DAEMON, realRoot, OLDER, socketPath], { detached: true, stdio: 'ignore' }); + older.unref(); + daemonPids.add(older.pid!); + await waitFor(() => daemonLock()?.pid === older.pid, 10000, 'the older daemon to hold the project'); + expect(writerLock()).toMatchObject({ pid: older.pid, mode: 'daemon' }); + + const session = startSession(null); + const stopped = await waitFor( + () => session.stderr.find((l) => l.includes('[CodeGraph MCP] Stopped')), + 30000, 'the older daemon to be stopped', + ); + expect(stopped).toContain(`the CodeGraph ${OLDER} daemon (pid ${older.pid})`); + expect(await waitProcessExit(older.pid!, 10000)).toBe(true); + const attached = await waitFor(() => attachedLine(session), 20000, 'the session to attach'); + const lock = daemonLock(); + daemonPids.add(lock.pid); + expect(lock.version).toBe(CodeGraphPackageVersion); + expect(attached).toContain(`(pid ${lock.pid}, v${CodeGraphPackageVersion})`); + expect(writerLock()).toMatchObject({ pid: lock.pid, mode: 'daemon' }); + expect(daemonLog()).toContain(`Took over the writer lock from launcher pid ${session.child.pid}`); + await status(session, 2, 'a tool call through the new daemon'); + } finally { + fs.rmSync(socketDir, { recursive: true, force: true }); + } + }, 90000); + + it('never stops a daemon from a newer install', async () => { + const retry = { CODEGRAPH_DAEMON_RETRY_MS: '300', CODEGRAPH_DAEMON_RETRY_MAX_MS: '600' }; + const newer = await sessionWithDaemon(newerInstall, NEWER); + const lock = daemonLock(); + + const current = startSession(null, retry); + // The first probe and the retries that follow all find the newer daemon. + await waitFor( + () => current.stderr.filter((l) => l.includes(`version (${NEWER}) differs from ours`)).length >= 3, + 20000, 'repeated probes of the newer daemon', + ); + await status(current, 2, 'a tool call served in-process'); + expect(current.stderr.some((l) => l.includes('Serving reads in-process without auto-sync'))).toBe(true); + expect(current.stderr.some((l) => l.includes('[CodeGraph MCP] Stopped'))).toBe(false); + + expect(isAlive(newer.pid)).toBe(true); + expect(daemonLock()).toEqual(lock); + expect(writerLock()).toMatchObject({ pid: newer.pid, mode: 'daemon' }); + await status(newer.session, 3, 'a tool call through the newer daemon'); + }, 90000); +}); diff --git a/__tests__/fixtures/older-daemon.cjs b/__tests__/fixtures/older-daemon.cjs new file mode 100644 index 0000000000..a6535f4d9b --- /dev/null +++ b/__tests__/fixtures/older-daemon.cjs @@ -0,0 +1,36 @@ +/** + * A daemon of an older release, listening where a current launcher does not + * look (#2335). On Windows, daemons before #2278 named their pipe after the + * project root as typed, so a newer launcher's probe never reaches them and + * only the project lock says where they are. Like a real daemon it holds the + * project's daemon and writer locks and answers the hello; on SIGTERM it lets + * go of whichever of the two locks still names it and exits, the shape of a + * real daemon's shutdown. (On Windows SIGTERM is TerminateProcess, and the + * stop clears what is left.) + * + * node older-daemon.cjs + */ +const fs = require('fs'); +const net = require('net'); +const path = require('path'); + +const [root, version, socketPath] = process.argv.slice(2); +const dir = path.join(root, '.codegraph'); +const writeLock = (name, record) => fs.writeFileSync(path.join(dir, name), JSON.stringify(record) + '\n'); + +const server = net.createServer((socket) => { + socket.end(JSON.stringify({ codegraph: version, pid: process.pid, socketPath, protocol: 1 }) + '\n'); +}); +server.listen(socketPath, () => { + writeLock('writer.pid', { pid: process.pid, mode: 'daemon', startedAt: Date.now(), ready: true }); + writeLock('daemon.pid', { pid: process.pid, version, socketPath, startedAt: Date.now() }); +}); +process.on('SIGTERM', () => { + for (const name of ['writer.pid', 'daemon.pid']) { + const file = path.join(dir, name); + try { + if (JSON.parse(fs.readFileSync(file, 'utf8')).pid === process.pid) fs.unlinkSync(file); + } catch { /* already gone */ } + } + server.close(() => process.exit(0)); +}); diff --git a/src/mcp/daemon-registry.ts b/src/mcp/daemon-registry.ts index aabb79139b..a9e5860116 100644 --- a/src/mcp/daemon-registry.ts +++ b/src/mcp/daemon-registry.ts @@ -31,7 +31,14 @@ import { probeDaemonIdentity, type DaemonLockInfo, } from './daemon-paths'; -import { readWriterLock, releaseWriterLock, tryAcquireWriterLock } from './writer-lock'; +import { + readWriterLock, + releaseWriterLock, + swapWriterLock, + tryAcquireWriterLock, + type WriterLockInfo, +} from './writer-lock'; +import { isOlderRelease } from './version'; import { WORKER_START_SETTLE_MS } from '../worker-teardown'; export interface DaemonRecord { @@ -155,7 +162,6 @@ function cleanupDaemonArtifacts( root: string, expectedLockContents: string | null, ): boolean { - const pidPath = getDaemonPidPath(root); // A daemon owns writer.pid before binding or relocating its socket. Claiming // the writer slot therefore freezes every legitimate daemon artifact writer // while we compare the inspected lock snapshot and clean it up. @@ -164,33 +170,39 @@ function cleanupDaemonArtifacts( if (claim.kind === 'taken') return false; try { - if (expectedLockContents === null) { - if (fs.existsSync(pidPath)) return false; - } else { - try { - if (fs.readFileSync(pidPath, 'utf8') !== expectedLockContents) return false; - } catch { - return false; - } - } - // POSIX sockets are real files; Windows named pipes vanish with the process. - // Sweep every candidate before releasing daemon.pid, so no successor can - // acquire the lock and bind a socket that this cleanup then removes. - if (process.platform !== 'win32') { - for (const candidate of getDaemonSocketCandidates(root)) { - try { fs.unlinkSync(candidate); } catch { /* gone */ } - } - } - deregisterDaemon(root); - try { fs.unlinkSync(pidPath); } catch (err) { - if ((err as NodeJS.ErrnoException).code !== 'ENOENT') return false; - } - return true; + return removeDaemonArtifacts(root, expectedLockContents); } finally { releaseWriterLock(root); } } +/** The removal behind {@link cleanupDaemonArtifacts}, for a caller already holding the writer slot. */ +function removeDaemonArtifacts(root: string, expectedLockContents: string | null): boolean { + const pidPath = getDaemonPidPath(root); + if (expectedLockContents === null) { + if (fs.existsSync(pidPath)) return false; + } else { + try { + if (fs.readFileSync(pidPath, 'utf8') !== expectedLockContents) return false; + } catch { + return false; + } + } + // POSIX sockets are real files; Windows named pipes vanish with the process. + // Sweep every candidate before releasing daemon.pid, so no successor can + // acquire the lock and bind a socket that this cleanup then removes. + if (process.platform !== 'win32') { + for (const candidate of getDaemonSocketCandidates(root)) { + try { fs.unlinkSync(candidate); } catch { /* gone */ } + } + } + deregisterDaemon(root); + try { fs.unlinkSync(pidPath); } catch (err) { + if ((err as NodeJS.ErrnoException).code !== 'ENOENT') return false; + } + return true; +} + /** Remove daemon artifacts only when no matching daemon answers the socket hello. */ export async function clearStaleDaemonArtifacts(root: string): Promise { const pidPath = getDaemonPidPath(root); @@ -240,6 +252,8 @@ export interface StopResult { pid: number | null; /** 'term' graceful, 'kill' force, 'still-running' refused, 'not-running' stale, 'no-daemon' absent, 'unverified' preserved. */ outcome: 'term' | 'kill' | 'still-running' | 'not-running' | 'no-daemon' | 'unverified'; + /** The daemon's version as its lock recorded it, when the stop got that far. */ + version?: string; } /** @@ -292,13 +306,27 @@ export async function stopDaemonAt( const removed = cleanupDaemonArtifacts(root, lockContents); return { root, pid, outcome: removed ? 'not-running' : 'unverified' }; } + return stopVerifiedDaemon(root, identity, lockContents, options.shutdownGraceMs); +} +/** + * The signalling half of a stop, for a daemon whose socket hello just proved + * `identity` against the lock read as `lockContents`: SIGTERM, wait, SIGKILL + * only one that still answers, then sweep its artifacts. + */ +async function stopVerifiedDaemon( + root: string, + identity: DaemonLockInfo, + lockContents: string | null, + shutdownGraceMs = DAEMON_SHUTDOWN_GRACE_MS, +): Promise { + const { pid, version } = identity; // Identity probing awaits I/O: never act on a superseded ownership record. const sameLock = (): boolean => { try { return fs.readFileSync(getDaemonPidPath(root), 'utf8') === lockContents; } catch { return lockContents === null; } }; - if (!sameLock()) return { root, pid, outcome: 'unverified' }; + if (!sameLock()) return { root, pid, outcome: 'unverified', version }; // POSIX: SIGTERM runs the daemon's graceful shutdown. Windows: TerminateProcess // (no graceful path), so we always sweep artifacts ourselves below. @@ -309,7 +337,7 @@ export async function stopDaemonAt( if (sameLock() && await probeDaemonIdentity(identity) && sameLock()) { try { process.kill(pid, 'SIGKILL'); } catch { /* raced to exit */ } if (!(await waitForDeath(pid, 2000))) { - return { root, pid, outcome: 'still-running' }; + return { root, pid, outcome: 'still-running', version }; } outcome = 'kill'; } else { @@ -321,14 +349,101 @@ export async function stopDaemonAt( // Its identity can't be re-proven without the socket, so signal nothing // more; just wait, bounded, for the PID to go. One that outlives the // wait is reported, as before. - if (!(await waitForDeath(pid, options.shutdownGraceMs ?? DAEMON_SHUTDOWN_GRACE_MS))) { - return { root, pid, outcome: 'still-running' }; + if (!(await waitForDeath(pid, shutdownGraceMs))) { + return { root, pid, outcome: 'still-running', version }; } } } // Compares the lock with the one we signalled, so a successor's is kept. cleanupDaemonArtifacts(root, lockContents); - return { root, pid, outcome }; + return { root, pid, outcome, version }; +} + +/** + * Stop the daemon serving `root` when it runs an older CodeGraph release than + * `version`, so the caller can start one from its own install (#2335). A + * daemon keeps running the code it started with: one that outlived an upgrade + * goes on loading grammars from files the upgrade removed, while it holds the + * project's writer lock and file watcher. Only a daemon whose lock records an + * older release ({@link isOlderRelease}) and whose socket hello confirms that + * lock is signalled, exactly as {@link stopDaemonAt} would; one of the same, a + * newer or an unknown version is never touched, so two installs cannot take + * turns stopping each other's daemon. + * + * The project's writer slot never falls free on the way: before the signal, + * this process takes it over from the old daemon (mode `handover`), and the + * caller hands it on to the daemon it starts, which takes it over in turn + * ({@link swapWriterLock}). The old daemon's sessions serve themselves + * in-process the moment it goes, and one that found the slot free would claim + * it as their writer — with the code of an install the upgrade removed — and + * keep the new daemon from starting. When the old daemon is not stopped, the + * slot goes back to it. + * + * Resolves null when the lock names no older daemon (there is none, or another + * launcher already replaced it) or the slot could not be taken (another + * launcher holds it to replace the same daemon). Otherwise says what became of + * the daemon: `term` or `kill` when it was stopped, and this process now holds + * the slot for its successor (release it with {@link releaseWriterLock} once + * that one has taken over or failed); `not-running` when it had already exited + * (its successor clears the stale lock); `unverified` or `still-running` when + * it is still there. + */ +export async function stopOlderDaemon( + root: string, + version: string, + options: { shutdownGraceMs?: number } = {}, +): Promise { + let lockContents: string; + try { + lockContents = fs.readFileSync(getDaemonPidPath(root), 'utf8'); + } catch (err) { + if ((err as NodeJS.ErrnoException).code === 'ENOENT') return null; + return { root, pid: null, outcome: 'unverified' }; + } + const identity = decodeLockInfo(lockContents); + if (!identity) return { root, pid: null, outcome: 'unverified' }; + if (!isOlderRelease(identity.version, version)) return null; + const { pid } = identity; + if (!isProcessAlive(pid)) return { root, pid, outcome: 'not-running', version: identity.version }; + // The hello must name this pid and version (#1553): a reused pid is no daemon. + if (!canProbeDaemonIdentity(identity) || !await probeDaemonIdentity(identity)) { + return { root, pid, outcome: 'unverified', version: identity.version }; + } + const slot = takeWriterSlotFrom(root, pid); + if (!slot) return null; + const result = await stopVerifiedDaemon(root, identity, lockContents, options.shutdownGraceMs); + if (result.outcome === 'term' || result.outcome === 'kill') { + // The stop's own sweep needs the slot this process now holds: clear what + // the old daemon left (all of it on Windows, where the stop is + // TerminateProcess), so its successor starts on a clean lock. + removeDaemonArtifacts(root, lockContents); + } else { + slot.giveBack(); + } + return result; +} + +/** + * Take the project's writer slot from the daemon `pid` this process is about + * to stop (see {@link stopOlderDaemon}), whether the record is that daemon's, + * stale, or absent. Returns how to give it back to that daemon, or null when + * the slot could not be taken: another live process holds it (another + * launcher replacing the same daemon), or the swap lost a race. + */ +function takeWriterSlotFrom(root: string, pid: number): { giveBack(): void } | null { + const previous = readWriterLock(root); + if (previous && previous.pid !== pid && previous.pid !== process.pid && isProcessAlive(previous.pid)) return null; + const claim: WriterLockInfo = { pid: process.pid, mode: 'handover', startedAt: Date.now(), ready: false }; + const held = previous + ? swapWriterLock(root, previous.pid, claim) + : tryAcquireWriterLock(root, 'handover').kind === 'acquired'; + if (!held) return null; + return { + giveBack: () => { + if (previous?.pid === pid) swapWriterLock(root, process.pid, previous); + else releaseWriterLock(root); + }, + }; } /** Stop every registered, live daemon. */ diff --git a/src/mcp/daemon.ts b/src/mcp/daemon.ts index 5f44ff6e85..73a4969598 100644 --- a/src/mcp/daemon.ts +++ b/src/mcp/daemon.ts @@ -57,9 +57,11 @@ import { import { CodeGraphPackageVersion } from './version'; import { releaseWriterLock, + swapWriterLock, tryAcquireWriterLock, assertNoRebuild, writerLockHeldMessage, + type WriterLockInfo, } from './writer-lock'; import { registerDaemon, deregisterDaemon } from './daemon-registry'; @@ -187,15 +189,18 @@ export class Daemon { private stopping = false; private socketPath: string; private pidPath: string; + /** The launcher holding the writer slot for this daemon, when it replaced an older one (#2335). */ + private handoverFrom: number | null; constructor( private projectRoot: string, - opts: { idleTimeoutMs?: number; maxIdleMs?: number } = {}, + opts: { idleTimeoutMs?: number; maxIdleMs?: number; handoverFrom?: number | null } = {}, ) { this.socketPath = getDaemonSocketPath(projectRoot); this.pidPath = getDaemonPidPath(projectRoot); this.idleTimeoutMs = opts.idleTimeoutMs ?? resolveIdleTimeoutMs(); this.maxIdleMs = opts.maxIdleMs ?? resolveMaxIdleMs(); + this.handoverFrom = opts.handoverFrom ?? null; // Daemon mode serves many concurrent clients on one event loop, so off-load // read-tool dispatch to a worker pool — otherwise concurrent explores // serialize and starve the MCP transport (clients time out). Direct mode @@ -214,7 +219,17 @@ export class Daemon { // #1740: claim the project writer lock before opening/watching so a // concurrent direct-mode serve --mcp cannot start a second watcher. assertNoRebuild(this.projectRoot); - const writer = tryAcquireWriterLock(this.projectRoot, 'daemon'); + let writer = tryAcquireWriterLock(this.projectRoot, 'daemon'); + // The launcher that stopped an older daemon has held the slot for us since + // (#2335); take it over without letting it fall free. + if (writer.kind === 'taken' && this.handoverFrom !== null && + writer.existing?.pid === this.handoverFrom && writer.existing.mode === 'handover') { + const info: WriterLockInfo = { pid: process.pid, mode: 'daemon', startedAt: Date.now(), ready: false }; + if (swapWriterLock(this.projectRoot, this.handoverFrom, info)) { + writer = { kind: 'acquired', pidPath: writer.pidPath, info }; + process.stderr.write(`[CodeGraph daemon] Took over the writer lock from launcher pid ${this.handoverFrom}, which stopped an older daemon.\n`); + } + } if (writer.kind === 'taken') { const msg = writerLockHeldMessage(writer.existing, writer.pidPath); process.stderr.write(`[CodeGraph daemon] ${msg}\n`); diff --git a/src/mcp/index.ts b/src/mcp/index.ts index 824132d629..ac7aa1d0ad 100644 --- a/src/mcp/index.ts +++ b/src/mcp/index.ts @@ -47,8 +47,9 @@ import { isProcessAlive, tryAcquireDaemonLock, } from './daemon'; -import { clearStaleDaemonArtifacts } from './daemon-registry'; +import { clearStaleDaemonArtifacts, stopOlderDaemon } from './daemon-registry'; import { connectWithHello, runLocalHandshakeProxy } from './proxy'; +import { CodeGraphPackageVersion } from './version'; import { readWriterLock, assertNoRebuild, @@ -83,6 +84,13 @@ const DIRECT_QUERY_POOL_MAX = 2; */ const DAEMON_INTERNAL_ENV = 'CODEGRAPH_DAEMON_INTERNAL'; +/** + * Env var naming the launcher that holds the project's writer slot for the + * daemon it spawns, after stopping an older one (#2335). That daemon takes the + * slot over instead of finding it held. + */ +const DAEMON_HANDOVER_ENV = 'CODEGRAPH_DAEMON_HANDOVER'; + /** * Retries for the detached daemon arbitrating the O_EXCL lock against a racing * sibling. Tiny — the lock resolves on the first round in practice; the retries @@ -91,6 +99,13 @@ const DAEMON_INTERNAL_ENV = 'CODEGRAPH_DAEMON_INTERNAL'; const TAKEOVER_MAX_RETRIES = 5; const TAKEOVER_RETRY_DELAY_MS = 100; +/** + * How long a daemon the writer slot is held for (#2335) waits out a racing + * candidate that took the daemon lock first. That candidate cannot get the + * slot, so it leaves; this bounds a wait that never ends. + */ +const HANDOVER_CANDIDATE_WAIT_MS = 5_000; + /** * A fallback that serves reads without a watcher or a writer lock (#1963). * Say so on stderr: otherwise a session that quietly stopped syncing looks @@ -243,9 +258,10 @@ function resolveDaemonRoot(explicitPath: string | null): string | null { * `process.execArgv` (carries `--liftoff-only`, so the daemon never re-execs) * and `process.argv[1]` (this script). The spawned process self-arbitrates the * O_EXCL lock, so racing launchers may each spawn one — losers exit and every - * launcher proxies through the single winner. + * launcher proxies through the single winner. `handover` tells the daemon this + * launcher holds the writer slot for it (see {@link DAEMON_HANDOVER_ENV}). */ -function spawnDetachedDaemon(root: string): void { +function spawnDetachedDaemon(root: string, handover = false): void { const scriptPath = process.argv[1]; if (!scriptPath) { // No resolvable CLI entry point to re-invoke — let the caller fall back to @@ -267,6 +283,8 @@ function spawnDetachedDaemon(root: string): void { // where a long-dead session's host pid would trigger spurious shutdowns. const env: NodeJS.ProcessEnv = { ...process.env, [DAEMON_INTERNAL_ENV]: '1' }; delete env[HOST_PPID_ENV]; + if (handover) env[DAEMON_HANDOVER_ENV] = String(process.pid); + else delete env[DAEMON_HANDOVER_ENV]; const child = spawn( process.execPath, [...process.execArgv, scriptPath, 'serve', '--mcp', '--path', root], @@ -286,6 +304,76 @@ function spawnDetachedDaemon(root: string): void { } } +/** + * How long a launcher replacing an older daemon waits for one that is partway + * through its shutdown, past the stop's own 3s wait. Tool calls are held + * meanwhile, so it is short: a daemon that takes longer leaves the session + * served in-process until the proxy's next retry (#2277) finds it gone. + */ +const OLDER_DAEMON_SHUTDOWN_GRACE_MS = 5_000; + +/** + * Stop the daemon serving `root` from an older install so the caller can start + * one from this install (#2335). A daemon keeps the code it started with: one + * that outlived an upgrade went on loading grammars from the removed install + * and wrote every file it re-indexed as empty, and as the project's writer it + * kept every newer session from syncing. {@link stopOlderDaemon} finds it by + * the project lock, verifies it, and never stops one of the same, a newer or an + * unknown version. `answered` says an older daemon answered this launcher's + * own probe. Resolves `stopped` (this launcher now holds the project's writer + * slot for the daemon it starts next), `none` (no older daemon holds the + * project), or `kept` (one does, unverified or still shutting down). + */ +async function replaceOlderDaemon(root: string, answered: boolean): Promise<'stopped' | 'none' | 'kept'> { + const result = await stopOlderDaemon(root, CodeGraphPackageVersion, { + shutdownGraceMs: OLDER_DAEMON_SHUTDOWN_GRACE_MS, + }); + // No older daemon in the lock (none, or already replaced), or it had exited. + if (!result || result.outcome === 'not-running') return 'none'; + const daemon = result.version + ? `the CodeGraph ${result.version} daemon (pid ${result.pid})` + : 'an older CodeGraph daemon'; + if (result.outcome === 'term' || result.outcome === 'kill') { + process.stderr.write( + `[CodeGraph MCP] Stopped ${daemon} serving this project; starting one from this install (${CodeGraphPackageVersion}).\n` + ); + return 'stopped'; + } + if (result.outcome === 'still-running') { + process.stderr.write(`[CodeGraph MCP] Asked ${daemon} serving this project to stop, but it has not exited yet.\n`); + } else if (answered) { + process.stderr.write( + `[CodeGraph MCP] Found ${daemon} serving this project but could not verify it, so left it running; serving this session in-process.\n` + ); + } + return 'kept'; +} + +/** + * Wait out the daemon candidate `pid` holding the daemon lock while launcher + * `handoverFrom` holds the writer slot for this daemon (#2335): the candidate + * cannot get the slot, so it releases the lock and exits. Resolves true once + * `pid` no longer holds the lock; false at once when no slot is held for this + * daemon, and false when that stops being so or the wait runs out. + */ +async function outwaitCandidate(root: string, handoverFrom: number | null, pid: number): Promise { + const heldForUs = (): boolean => { + const writer = readWriterLock(root); + return handoverFrom !== null && writer?.pid === handoverFrom && writer.mode === 'handover'; + }; + if (!heldForUs()) return false; + process.stderr.write(`[CodeGraph daemon] Waiting for daemon candidate pid ${pid} to give up the lock; the writer lock is held for this daemon.\n`); + const deadline = Date.now() + HANDOVER_CANDIDATE_WAIT_MS; + while (Date.now() < deadline) { + if (!heldForUs()) return false; + let holder: number | undefined; + try { holder = decodeLockInfo(fs.readFileSync(getDaemonPidPath(root), 'utf8'))?.pid; } catch { holder = undefined; } + if (holder !== pid || !isProcessAlive(pid)) return true; + await sleep(25); + } + return false; +} + /** * MCP Server for CodeGraph * @@ -495,11 +583,15 @@ export class MCPServer { // kills/restarts can be placed in time (#1431 — the log was undatable). timestampStderrLines(); const root = resolveDaemonRoot(this.projectPath) ?? this.projectPath ?? process.cwd(); + // Read once and dropped, so nothing this daemon spawns inherits it. + const handoverPid = Number(process.env[DAEMON_HANDOVER_ENV]); + const handoverFrom = Number.isInteger(handoverPid) && handoverPid > 0 ? handoverPid : null; + delete process.env[DAEMON_HANDOVER_ENV]; for (let attempt = 0; attempt < TAKEOVER_MAX_RETRIES; attempt++) { const lock = tryAcquireDaemonLock(root); if (lock.kind === 'acquired') { - const daemon = new Daemon(root); + const daemon = new Daemon(root, { handoverFrom }); await daemon.start(); this.daemon = daemon; this.mode = 'daemon'; @@ -529,6 +621,9 @@ export class MCPServer { stillStarting || await probeDaemonIdentity(existing) ) { + // A daemon the writer slot is held for outwaits a candidate still + // starting in its way, which cannot get the slot and leaves (#2335). + if (stillStarting && await outwaitCandidate(root, handoverFrom, existing.pid)) continue; process.stderr.write( `[CodeGraph daemon] Another daemon (pid ${existing.pid}) already holds the lock; exiting.\n` ); @@ -560,7 +655,8 @@ export class MCPServer { /** * Proxy mode (the common case). Serve the MCP handshake LOCALLY for instant * tool registration, forwarding tool calls to the shared daemon — which is - * connected in the background (probed, then spawned + polled if absent) so the + * connected in the background (probed, then spawned + polled if absent; one of + * an older release is stopped first, #2335) so the * handshake never waits ~600ms on it. Runs until the host disconnects; the * proxy falls back to an in-process engine if the daemon never binds, so this * never wedges a session. @@ -577,9 +673,8 @@ export class MCPServer { for (const candidate of candidates) { const s = await connectWithHello(candidate); // A wrong-version daemon IS up — definitive; propagate so the caller - // serves in-process instead of spawning + polling for 6s. Don't keep - // probing fallbacks past it. - if (s === 'version-mismatch') return s; + // replaces it (an older release) or serves in-process instead of + // spawning + polling for 6s. Don't keep probing fallbacks past it. if (s) return s; } return null; @@ -588,16 +683,31 @@ export class MCPServer { // Fast path: a daemon may already be listening (on either candidate). const probe = await connectAnyCandidate(); if (probe === 'version-mismatch') return null; // definitive — serve in-process, don't poll for 6s - if (probe) return probe; - // None reachable — spawn one (detached) and poll for its bind. - spawnDetachedDaemon(root); - for (let attempt = 0; attempt < DAEMON_CONNECT_MAX_RETRIES; attempt++) { - await sleep(DAEMON_CONNECT_RETRY_DELAY_MS); - const s = await connectAnyCandidate(); - if (s === 'version-mismatch') return null; - if (s) return s; + if (probe && probe !== 'older-version') return probe; + // A daemon of an older release answered — or none did, while the project + // lock may still name one on a socket we don't probe (on Windows, daemons + // before #2278 named their pipe after the root as typed). Either way it + // makes way for one from this install (#2335). + const answered = probe === 'older-version'; + const older = await replaceOlderDaemon(root, answered); + if (older === 'kept' && answered) return null; + // None reachable — spawn one (detached) and poll for its bind. After a + // replacement this launcher holds the writer slot for it meanwhile. + try { + spawnDetachedDaemon(root, older === 'stopped'); + for (let attempt = 0; attempt < DAEMON_CONNECT_MAX_RETRIES; attempt++) { + await sleep(DAEMON_CONNECT_RETRY_DELAY_MS); + const s = await connectAnyCandidate(); + // Another version answering now, older or not, waits for the next + // retry: one replacement per attempt, so this can never loop. + if (s === 'older-version' || s === 'version-mismatch') return null; + if (s) return s; + } + return null; // never bound — the proxy serves this session in-process + } finally { + // The new daemon has taken the slot over, or never came; a no-op unless still ours. + if (older === 'stopped') releaseWriterLock(root); } - return null; // never bound — the proxy serves this session in-process }; await runLocalHandshakeProxy({ getDaemonSocket, makeEngine: () => makeFallbackEngine(root), root }); } diff --git a/src/mcp/project-lifecycle.ts b/src/mcp/project-lifecycle.ts index b7a52c1a5b..1a634815eb 100644 --- a/src/mcp/project-lifecycle.ts +++ b/src/mcp/project-lifecycle.ts @@ -118,7 +118,8 @@ async function activate(root: string, project: Project): Promise { if (writer.existing?.mode === 'daemon') { for (const candidate of getDaemonSocketCandidates(root)) { const socket = await connectWithHello(candidate); - if (!socket || socket === 'version-mismatch') continue; + // A daemon of another version, older or not, is left to the project's own launchers. + if (!socket || typeof socket === 'string') continue; if (project.refs === 0) { socket.destroy(); return; } project.socket = socket; socket.once('close', () => { diff --git a/src/mcp/proxy.ts b/src/mcp/proxy.ts index 52e6b77fd5..c0d9f6cafb 100644 --- a/src/mcp/proxy.ts +++ b/src/mcp/proxy.ts @@ -26,7 +26,7 @@ import { EARLY_PPID } from './early-ppid'; import { supervisionLostReason } from './ppid-watchdog'; import { armStartupHandshakeTimeout } from './startup-handshake'; import { treatStdinFailureAsShutdown } from './stdin-teardown'; -import { CodeGraphPackageVersion } from './version'; +import { CodeGraphPackageVersion, isOlderRelease } from './version'; import { SERVER_INFO, PROTOCOL_VERSION, initializeInstructions } from './session'; import { SERVER_INSTRUCTIONS } from './server-instructions'; import { getStaticTools } from './tools'; @@ -148,14 +148,17 @@ export async function runProxy( /** * Connect to a daemon at `socketPath` and verify its hello (exact version match). - * Returns the live socket (hello already consumed) or null if unreachable / stale - * / version-mismatched. Unlike {@link runProxy} it does NOT pipe — the caller - * owns the socket. Used by the local-handshake proxy's background connect. + * Returns the live socket (hello already consumed), null if unreachable / stale, + * or — for a daemon of another version — `'older-version'` when it runs an older + * release ({@link isOlderRelease}; the launcher replaces it, #2335) and + * `'version-mismatch'` for any other version, which is left alone. Unlike + * {@link runProxy} it does NOT pipe — the caller owns the socket. Used by the + * local-handshake proxy's background connect. */ export async function connectWithHello( socketPath: string, expectedVersion: string = CodeGraphPackageVersion, -): Promise { +): Promise { if (process.platform !== 'win32' && !fs.existsSync(socketPath)) return null; const socket = net.createConnection(socketPath); socket.setEncoding('utf8'); @@ -176,12 +179,14 @@ export async function connectWithHello( } if (hello.codegraph !== expectedVersion) { // A daemon IS up but it's the wrong version — definitive, not a "not yet". - // Don't poll; the caller serves in-process so we never run stale-vs-new. + socket.destroy(); + // An older release is the launcher's to replace (#2335); it says what it did. + if (isOlderRelease(hello.codegraph, expectedVersion)) return 'older-version'; + // Any other: don't poll; the caller serves in-process so we never run stale-vs-new. process.stderr.write( `[CodeGraph MCP] Found a daemon on ${socketPath} but version (${hello.codegraph}) ` + `differs from ours (${expectedVersion}); serving this session in-process.\n` ); - socket.destroy(); return 'version-mismatch'; } logAttachedDaemon(socketPath, hello); @@ -212,8 +217,9 @@ type JsonRpc = Record; /** Dependencies the local-handshake proxy needs, injected by MCPServer (which * owns the daemon-spawn machinery and the engine factory). */ export interface LocalHandshakeDeps { - /** Probe → spawn → retry → hello-verify; resolves a connected daemon socket, - * or null when the daemon path is genuinely unavailable (→ in-process fallback). */ + /** Probe (replacing a daemon of an older release, #2335) → spawn → retry → + * hello-verify; resolves a connected daemon socket, or null when the daemon + * path is genuinely unavailable (→ in-process fallback). */ getDaemonSocket(): Promise; /** Lazily create an in-process engine — used only while the daemon is * unreachable, preserving the "a broken daemon never wedges a session" diff --git a/src/mcp/version.ts b/src/mcp/version.ts index cef1b7834a..b6c919f0f0 100644 --- a/src/mcp/version.ts +++ b/src/mcp/version.ts @@ -4,7 +4,8 @@ * The version string is the rendezvous datum between cooperating daemon and * proxy processes: the daemon advertises its version in the hello line, and * the proxy refuses to share IPC across a mismatch (falls back to direct - * mode). Keeping the resolution in one place avoids drift between the CLI + * mode), or replaces a daemon of an older release ({@link isOlderRelease}). + * Keeping the resolution in one place avoids drift between the CLI * `--version` output (which reads `package.json` directly) and the daemon * handshake. * @@ -34,3 +35,25 @@ function readPackageVersion(): string { } export const CodeGraphPackageVersion = readPackageVersion(); + +/** + * Whether `version` is a CodeGraph release older than `than` — the test a + * launcher applies before it replaces a running daemon (#2335). Only plain + * `MAJOR.MINOR.PATCH` releases compare: a prerelease, a build suffix or the + * "0.0.0-unknown" sentinel is never older. A daemon is replaced only by a + * strictly newer release, so two installed versions can never take turns + * stopping each other's daemon. + */ +export function isOlderRelease(version: string, than: string): boolean { + const a = parseRelease(version); + const b = parseRelease(than); + if (!a || !b) return false; + if (a[0] !== b[0]) return a[0] < b[0]; + if (a[1] !== b[1]) return a[1] < b[1]; + return a[2] < b[2]; +} + +function parseRelease(version: string): [number, number, number] | null { + const m = /^(\d+)\.(\d+)\.(\d+)$/.exec(version); + return m ? [Number(m[1]), Number(m[2]), Number(m[3])] : null; +} diff --git a/src/mcp/writer-lock.ts b/src/mcp/writer-lock.ts index 9acbdb9a46..7785cde674 100644 --- a/src/mcp/writer-lock.ts +++ b/src/mcp/writer-lock.ts @@ -165,6 +165,44 @@ export function markWriterReady(projectRoot: string): void { } } +/** Waits between retries of a writer-record swap refused by Windows (another process has the file open). */ +const SWAP_RETRY_DELAYS_MS = [25, 50, 100, 200]; + +/** + * Replace the writer record of `fromPid` with `next`, and only that record, so + * the slot passes between two processes without ever falling free (#2335). A + * stopped daemon's sessions serve themselves in-process the moment it goes, and + * one that found the slot free or stale would claim it as their own writer. + * Compare and rename are separate steps, so callers check the result: whether + * the record was replaced. Never throws. + */ +export function swapWriterLock(projectRoot: string, fromPid: number, next: WriterLockInfo): boolean { + const pidPath = getWriterPidPath(projectRoot); + const tmp = `${pidPath}.${process.pid}.swap.tmp`; + try { + if (readWriterLock(projectRoot)?.pid !== fromPid) return false; + fs.writeFileSync(tmp, encode(next), { mode: 0o600 }); + for (let attempt = 0; ; attempt++) { + try { + fs.renameSync(tmp, pidPath); + return true; + } catch (err) { + const code = (err as NodeJS.ErrnoException).code; + const delay = SWAP_RETRY_DELAYS_MS[attempt]; + if (process.platform !== 'win32' || !['EPERM', 'EACCES', 'EBUSY'].includes(code ?? '') || delay === undefined) { + return false; + } + Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, delay); + if (readWriterLock(projectRoot)?.pid !== fromPid) return false; + } + } + } catch { + return false; + } finally { + try { fs.unlinkSync(tmp); } catch { /* renamed, or never written */ } + } +} + /** Release if we still own the lock (pid match). */ export function releaseWriterLock(projectRoot: string, lockName: 'writer.pid' | 'rebuild.pid' = 'writer.pid'): void { const pidPath = getWriterPidPath(projectRoot, lockName); From dea076fd1e9fa3e236e93fa915bb0feebe73579d Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 15:08:14 +0000 Subject: [PATCH 172/259] fix(mcp,extraction): exit a daemon whose install changed; never store a grammar-load failure (#2335, #2336) (#2346) * fix(mcp,extraction): a daemon whose install changed exits; grammar-load failures are never stored (#2335, #2336) A daemon that outlived an upgrade loaded grammars lazily from an install that no longer existed, and stored every file its watcher re-indexed as a zero-node row under the file's new content hash: no hash-based sync ever revisited those rows, and `codegraph status` called the index up to date. - The daemon checks its own package.json every 30 s (CODEGRAPH_DAEMON_INSTALL_CHECK_MS, 0 turns it off) and exits once the file is gone or carries another version, so the next session starts a daemon from the current install. - A result whose grammar failed to load (`parser_error`) is never stored: the file keeps its previous data and the next sync or index retries it, and a row an older engine stored that way is re-indexed even though its hash matches. The tag-based CFML path reports a missing grammar with the same code. - A launcher whose own install was deleted no longer crashes on the spawn error of the daemon it tried to start. - `codegraph status` and `status --json` (filesNeedingReindex, filesWithParseErrors) report indexed files whose symbols are missing although their content is current (#2336), and `files --json` carries each file's recorded errors. Co-Authored-By: Claude Opus 5.5 * fix(mcp): a session whose own install changed never claims the writer lock (#2335) A daemon now exits once its install is upgraded or removed, and nobody holds the writer slot for a successor at that point. Its sessions run the same replaced code and serve themselves in-process the moment it goes; one that found the slot free claimed it as its writer even though its engine could not load (a real npm upgrade: "Cannot find module './sqlite-adapter'"), and so kept a daemon from the current install from starting. Such a session now falls back read-only and says to restart it. Also adds the CHANGELOG entries for the install check and grammar-load failures (#2335) and for the status report (#2336). Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 2 + __tests__/daemon-install-check.test.ts | 83 ++++++++++++ __tests__/daemon-older-version.test.ts | 45 +++++++ __tests__/grammar-load-failure.test.ts | 167 +++++++++++++++++++++++++ __tests__/status-index-health.test.ts | 159 +++++++++++++++++++++++ src/bin/codegraph.ts | 49 +++++++- src/db/queries.ts | 12 ++ src/extraction/cfml-extractor.ts | 4 +- src/extraction/grammars.ts | 17 ++- src/extraction/index.ts | 40 ++++-- src/index.ts | 30 +++++ src/mcp/daemon.ts | 72 ++++++++++- src/mcp/index.ts | 17 ++- src/mcp/version.ts | 6 +- src/types.ts | 18 +++ 15 files changed, 706 insertions(+), 15 deletions(-) create mode 100644 __tests__/daemon-install-check.test.ts create mode 100644 __tests__/grammar-load-failure.test.ts create mode 100644 __tests__/status-index-health.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 32db6b23ab..f280890b4f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,8 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In VB.NET and C#, what a property's `Get` and `Set` code calls, creates and reads now belongs to that property, as do C#'s `get => …` accessors and `=> …` property bodies. Before, it was dropped, so a method used only from a property looked unused. - In VB.NET, a field or property initializer like `= Compute()` or `As New List(Of Order)` now links what it calls and creates, and so do a `Custom Event`'s `AddHandler`, `RemoveHandler` and `RaiseEvent` blocks. Re-index VB.NET and C# projects after upgrading. - Upgrading CodeGraph while an agent session is open no longer leaves the old version's background server in charge of your project: the first session started from the new install stops it and starts a current one in its place, even while sessions opened before the upgrade are still running. That old server could no longer load the language parsers the upgrade removed, so it saved every file it re-indexed with no symbols, while new sessions could only read the index beside it without keeping it up to date. A background server from a newer install is never stopped, and sessions opened before the upgrade keep the old version until you restart them. Thanks @lipchey for the report. (#2335) +- A file is no longer saved with no symbols when its language parser can't be loaded, which is what happened to every file a background server re-indexed after an upgrade removed its install: the file keeps what it had and is indexed again once the parser loads, and files an earlier version emptied this way are re-indexed by the next sync. A background server also exits on its own once its install is upgraded or removed, so the next session starts one from the current install. Thanks @lipchey for the report. (#2335) +- `codegraph status` no longer says the index is up to date while indexed files are missing their symbols: it now names files the parser couldn't read and files stored without their symbols (which `codegraph sync` repairs), `status --json` counts both, and `codegraph files --json` lists each file's recorded errors. Thanks @lipchey for the report. (#2336) ## [1.6.2] - 2026-10-03 diff --git a/__tests__/daemon-install-check.test.ts b/__tests__/daemon-install-check.test.ts new file mode 100644 index 0000000000..614afb6e4a --- /dev/null +++ b/__tests__/daemon-install-check.test.ts @@ -0,0 +1,83 @@ +/** + * A daemon whose install was upgraded or deleted underneath it exits (#2335). + * + * The daemon loads grammars (and the native kernel, and worker scripts) lazily + * from the directory it was started from. An upgrade replaces or deletes that + * directory while the daemon keeps running, so every later load fails; the + * 1.6.1 daemon in #2335 went on re-indexing files as empty for as long as it + * lived. It now checks its own package.json periodically and makes way for a + * daemon started from the current install. The check is pure and the Daemon's + * reaction is tested with it injected; neither starts a real daemon. + */ +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { Daemon, installChangedReason } from '../src/mcp/daemon'; + +describe('installChangedReason', () => { + let dir: string; + let pkg: string; + + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-install-check-')); + pkg = path.join(dir, 'package.json'); + fs.writeFileSync(pkg, JSON.stringify({ name: '@colbymchenry/codegraph', version: '1.6.2' })); + }); + + afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('is null while the install still carries the running version', () => { + expect(installChangedReason(pkg, '1.6.2')).toBeNull(); + }); + + it('reports an install upgraded in place', () => { + fs.writeFileSync(pkg, JSON.stringify({ version: '1.6.3' })); + expect(installChangedReason(pkg, '1.6.2')).toMatch(/replaced by v1\.6\.3/); + }); + + it('reports an install that was deleted, file or whole directory', () => { + fs.rmSync(pkg); + expect(installChangedReason(pkg, '1.6.2')).toMatch(/removed/); + fs.rmSync(dir, { recursive: true, force: true }); + expect(installChangedReason(pkg, '1.6.2')).toMatch(/removed/); + }); + + it('waits for a package.json caught mid-write instead of exiting on it', () => { + fs.writeFileSync(pkg, '{"name": "@colbymchenry/codeg'); + expect(installChangedReason(pkg, '1.6.2')).toBeNull(); + fs.writeFileSync(pkg, ''); + expect(installChangedReason(pkg, '1.6.2')).toBeNull(); + }); +}); + +describe('Daemon.checkInstall', () => { + // A Daemon is safe to construct (nothing binds until start()); stop() is + // replaced so a positive check can't tear down the test process. + const makeDaemon = () => { + const d = new Daemon('/tmp/codegraph-install-check-unit-test', { idleTimeoutMs: 0 }) as any; + d.stop = vi.fn(async () => {}); + return d; + }; + + it('stops the daemon when its install changed', () => { + const d = makeDaemon(); + expect(d.checkInstall(() => 'Install removed')).toBe(true); + expect(d.stop).toHaveBeenCalledWith('install changed'); + }); + + it('keeps running while the install is unchanged', () => { + const d = makeDaemon(); + expect(d.checkInstall(() => null)).toBe(false); + expect(d.stop).not.toHaveBeenCalled(); + }); + + it('does nothing once the daemon is already stopping', () => { + const d = makeDaemon(); + d.stopping = true; + expect(d.checkInstall(() => 'Install removed')).toBe(false); + expect(d.stop).not.toHaveBeenCalled(); + }); +}); diff --git a/__tests__/daemon-older-version.test.ts b/__tests__/daemon-older-version.test.ts index 188cd6cf82..b29cbba3c6 100644 --- a/__tests__/daemon-older-version.test.ts +++ b/__tests__/daemon-older-version.test.ts @@ -713,4 +713,49 @@ describe('a launcher meeting a daemon of another version (#2335)', () => { expect(writerLock()).toMatchObject({ pid: newer.pid, mode: 'daemon' }); await status(newer.session, 3, 'a tool call through the newer daemon'); }, 90000); + + it('a session whose own install was upgraded serves reads only once its daemon exits', async () => { + // An upgrade in place: the daemon notices its package.json changed and + // exits (its install check), and its session falls back in-process. As + // the project's writer that session — running code the upgrade replaced — + // would only keep a daemon from the new install from starting. + const install = makeInstall('0.0.2'); + try { + const before = await sessionWithDaemon(install, '0.0.2', { + CODEGRAPH_DAEMON_INSTALL_CHECK_MS: '200', + CODEGRAPH_DAEMON_RETRY_MS: '0', + }); + fs.writeFileSync(path.join(install, 'package.json'), JSON.stringify({ name: '@colbymchenry/codegraph', version: '0.0.3' }) + '\n'); + expect(await waitProcessExit(before.pid, 10000)).toBe(true); + expect(daemonLog()).toContain('Install replaced by v0.0.3'); + await waitFor( + () => before.session.stderr.some((l) => l.includes('Shared daemon connection lost')), + 10000, 'the session to lose its daemon', + ); + + await status(before.session, 3, 'a tool call after the daemon exited'); + const stderr = before.session.stderr.join('\n'); + expect(stderr).toContain("this session's CodeGraph install was upgraded or removed"); + expect(stderr).not.toContain('File watcher active'); + expect(writerLock()).toBeNull(); + + // So a session from the current install gets a daemon straight away. + const after = startSession(null); + await waitFor(() => attachedLine(after), 20000, 'the new session to attach'); + const lock = daemonLock(); + daemonPids.add(lock.pid); + expect(writerLock()).toMatchObject({ pid: lock.pid, mode: 'daemon' }); + } finally { + // On Windows a process still running from the install blocks its removal. + for (const { child } of sessions) { + if (child.exitCode === null && child.signalCode === null) { + const exited = once(child, 'exit'); + child.kill('SIGKILL'); + await exited; + } + } + try { fs.unlinkSync(path.join(install, 'node_modules')); } catch { /* not created */ } + await fs.promises.rm(install, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); + } + }, 90000); }); diff --git a/__tests__/grammar-load-failure.test.ts b/__tests__/grammar-load-failure.test.ts new file mode 100644 index 0000000000..1f285f0fd4 --- /dev/null +++ b/__tests__/grammar-load-failure.test.ts @@ -0,0 +1,167 @@ +/** + * A grammar that fails to load is an environment failure, not a parse result + * (#2335). + * + * A long-running daemon loads tree-sitter grammars lazily from its install + * directory. When an upgrade deletes that directory underneath it, every later + * grammar load fails (ENOENT) and the extractor answers `parser_error` with no + * nodes. That used to be STORED: the file's good symbols were replaced by a + * zero-node row carrying the new content hash, so `sync` never revisited it + * and `status` stayed green. Now nothing is stored for such a result — the + * file keeps its previous index data and the next sync or index retries it — + * and rows an older engine already wrote that way are re-indexed by `sync`. + * + * The failure is simulated at `getParser` (what a failed load leaves behind: + * no parser for the language), with the native kernel switched off so every + * file goes through the grammar path regardless of whether a kernel is built. + */ +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { CodeGraph } from '../src'; +import type { Language } from '../src/types'; + +const { failing } = vi.hoisted(() => ({ failing: new Set() })); + +vi.mock('../src/extraction/grammars', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + getParser: (language: Language) => (failing.has(language) ? null : actual.getParser(language)), + }; +}); + +const MATH_TS = + 'export function add(a: number, b: number): number {\n' + + ' return a + b;\n' + + '}\n' + + 'export class Calculator {\n' + + ' total = 0;\n' + + ' plus(n: number): this { this.total = add(this.total, n); return this; }\n' + + '}\n'; + +const VIEW_TS = + "import { Calculator } from './math';\n" + + 'export function total(n: number): number {\n' + + ' return new Calculator().plus(n).total;\n' + + '}\n'; + +const SUB_TS = '\nexport function sub(a: number, b: number): number {\n return a - b;\n}\n'; + +function names(cg: CodeGraph, file: string): string[] { + return cg.getNodesInFile(file).map((n) => n.name); +} + +describe('grammar load failures are never stored (#2335)', () => { + let dir: string; + let cg: CodeGraph; + let kernelEnv: string | undefined; + + beforeEach(() => { + kernelEnv = process.env.CODEGRAPH_KERNEL; + process.env.CODEGRAPH_KERNEL = '0'; + failing.clear(); + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-grammar-load-failure-')); + fs.mkdirSync(path.join(dir, 'src')); + fs.writeFileSync(path.join(dir, 'src', 'math.ts'), MATH_TS); + fs.writeFileSync(path.join(dir, 'src', 'view.ts'), VIEW_TS); + }); + + afterEach(() => { + failing.clear(); + cg?.destroy(); + if (kernelEnv === undefined) delete process.env.CODEGRAPH_KERNEL; + else process.env.CODEGRAPH_KERNEL = kernelEnv; + fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('sync keeps the previous symbols of a file whose grammar failed to load, then retries it', async () => { + cg = await CodeGraph.init(dir); + await cg.indexAll(); + const before = cg.getFile('src/math.ts')!; + expect(before.nodeCount).toBeGreaterThan(1); + expect(names(cg, 'src/math.ts')).toEqual(expect.arrayContaining(['add', 'Calculator'])); + + // The upgrade removed the grammar; the watcher then re-indexes an edit. + failing.add('typescript'); + fs.appendFileSync(path.join(dir, 'src', 'math.ts'), SUB_TS); + const failed = await cg.sync(); + + expect(failed.failedFilePaths).toContain('src/math.ts'); + const kept = cg.getFile('src/math.ts')!; + expect(kept.nodeCount).toBe(before.nodeCount); + expect(kept.contentHash).toBe(before.contentHash); + expect(kept.errors ?? []).toEqual([]); + expect(names(cg, 'src/math.ts')).toEqual(expect.arrayContaining(['add', 'Calculator'])); + + // Grammar available again (a fresh process from the new install): the + // next sync sees the file as still changed and indexes the edit. + failing.clear(); + const retried = await cg.sync(); + expect(retried.filesModified).toBe(1); + expect(names(cg, 'src/math.ts')).toEqual(expect.arrayContaining(['add', 'Calculator', 'sub'])); + expect(cg.getFile('src/math.ts')!.nodeCount).toBeGreaterThan(before.nodeCount); + }); + + it('a full index stores nothing for files whose grammar failed to load, and sync adds them later', async () => { + failing.add('typescript'); + cg = await CodeGraph.init(dir); + const result = await cg.indexAll(); + + expect(result.filesErrored).toBe(2); + expect(result.errors.filter((e) => e.code === 'parser_error')).toHaveLength(2); + expect(cg.getFile('src/math.ts')).toBeNull(); + expect(cg.getFile('src/view.ts')).toBeNull(); + + failing.clear(); + const synced = await cg.sync(); + expect(synced.filesAdded).toBe(2); + expect(names(cg, 'src/math.ts')).toEqual(expect.arrayContaining(['add', 'Calculator'])); + expect(names(cg, 'src/view.ts')).toContain('total'); + }); + + it('sync re-indexes rows an older engine stored while the grammar was missing', async () => { + cg = await CodeGraph.init(dir); + await cg.indexAll(); + const before = cg.getFile('src/math.ts')!; + + // What v1.6.2 wrote for a file it re-indexed with no grammar: no nodes, + // the parser error, and the CURRENT content hash — so a hash-based + // reconcile considers it up to date forever. + const db = (cg as unknown as { db: { getDb(): { prepare(sql: string): { run(...args: unknown[]): unknown } } } }).db.getDb(); + db.prepare('DELETE FROM nodes WHERE file_path = ?').run('src/math.ts'); + db.prepare('UPDATE files SET node_count = 0, errors = ? WHERE path = ?').run( + JSON.stringify([{ message: 'Failed to get parser for language: typescript', filePath: 'src/math.ts', severity: 'error', code: 'parser_error' }]), + 'src/math.ts' + ); + expect(cg.getFile('src/math.ts')!.nodeCount).toBe(0); + + await cg.sync(); + + const healed = cg.getFile('src/math.ts')!; + expect(healed.nodeCount).toBe(before.nodeCount); + expect(healed.errors ?? []).toEqual([]); + expect(names(cg, 'src/math.ts')).toEqual(expect.arrayContaining(['add', 'Calculator'])); + }); + + it('a component whose script grammar failed keeps its previous symbols too', async () => { + fs.writeFileSync( + path.join(dir, 'src', 'Counter.svelte'), + '\n\n' + ); + cg = await CodeGraph.init(dir); + await cg.indexAll(); + const before = cg.getFile('src/Counter.svelte')!; + expect(names(cg, 'src/Counter.svelte')).toContain('increment'); + + failing.add('typescript'); + fs.appendFileSync(path.join(dir, 'src', 'Counter.svelte'), '

    edited

    \n'); + await cg.sync(); + + // The component node alone would have been stored before: the script's + // symbols gone, with a nonzero node count hiding it. + expect(cg.getFile('src/Counter.svelte')!.nodeCount).toBe(before.nodeCount); + expect(names(cg, 'src/Counter.svelte')).toContain('increment'); + }); +}); diff --git a/__tests__/status-index-health.test.ts b/__tests__/status-index-health.test.ts new file mode 100644 index 0000000000..b94562c8eb --- /dev/null +++ b/__tests__/status-index-health.test.ts @@ -0,0 +1,159 @@ +/** + * `codegraph status` must not call an index "up to date" while indexed files + * are missing their symbols (#2336, #2335). + * + * "Up to date" used to mean only "every file's content hash matches the + * index". Two kinds of file pass that check with nothing usable in the graph: + * + * - a file the parser could not read — `export type * from` is valid + * TypeScript 5.0 the bundled grammar has no rule for, so the file is stored + * with a parse error and no symbols (#2336); + * - a row stored without its symbols — by an engine whose grammar failed to + * load (#2335), or the #1541 wipe — which `codegraph sync` re-indexes. + * + * Both are counted in `status` (human and `--json`), and `files --json` carries + * each file's recorded errors. Exercised against the built CLI so the output + * and the JSON field names are what users and scripts actually see. + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { spawnSync } from 'child_process'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +const BIN = path.resolve(__dirname, '../dist/bin/codegraph.js'); + +function spawn(cwd: string, args: string[]) { + return spawnSync(process.execPath, [BIN, ...args], { + cwd, + encoding: 'utf-8', + timeout: 30_000, + env: { + ...process.env, + CODEGRAPH_NO_DAEMON: '1', + CODEGRAPH_TELEMETRY: '0', + CODEGRAPH_NO_UPDATE_CHECK: '1', + NO_COLOR: '1', + }, + }); +} + +function run(cwd: string, args: string[]): { status: number | null; out: string } { + const result = spawn(cwd, args); + return { status: result.status, out: (result.stdout ?? '') + (result.stderr ?? '') }; +} + +function runJson(cwd: string, args: string[]): any { + const result = spawn(cwd, args); + expect(result.status, (result.stdout ?? '') + (result.stderr ?? '')).toBe(0); + return JSON.parse(result.stdout); +} + +describe('status reports files indexed without their symbols (#2336, #2335)', () => { + let dir: string; + + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-status-health-')); + fs.mkdirSync(path.join(dir, 'src', 'constants'), { recursive: true }); + fs.mkdirSync(path.join(dir, 'src', 'types'), { recursive: true }); + fs.writeFileSync( + path.join(dir, 'src', 'constants', 'index.ts'), + 'export const MAX_ITEMS = 10;\nexport function clamp(n: number): number { return Math.min(n, MAX_ITEMS); }\n' + ); + fs.writeFileSync( + path.join(dir, 'src', 'types', 'database.ts'), + 'export interface Row { id: number; name: string }\nexport type RowId = Row["id"];\n' + ); + }); + + afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); + }); + + async function index(): Promise { + const cg = CodeGraph.initSync(dir); + await cg.indexAll(); + cg.destroy(); + } + + it('a clean index is up to date, with zero counts and empty error lists', async () => { + await index(); + + const human = run(dir, ['status']); + expect(human.status, human.out).toBe(0); + expect(human.out).toContain('Index is up to date'); + + const json = runJson(dir, ['status', '--json']); + expect(json.index.filesNeedingReindex).toBe(0); + expect(json.index.filesWithParseErrors).toBe(0); + + const files = runJson(dir, ['files', '--json']); + expect(files).toHaveLength(2); + for (const f of files) expect(f.errors).toEqual([]); + }); + + it('a file the parser could not read keeps status from reporting "up to date"', async () => { + fs.writeFileSync( + path.join(dir, 'src', 'index.ts'), + 'export * from "./constants/index.js";\nexport type * from "./types/database.js";\n' + ); + await index(); + + const human = run(dir, ['status']); + expect(human.status, human.out).toBe(0); + expect(human.out).not.toContain('Index is up to date'); + expect(human.out).toMatch(/1 file could not be parsed/); + expect(human.out).toContain('src/index.ts'); + + const json = runJson(dir, ['status', '--json']); + expect(json.index.filesWithParseErrors).toBe(1); + expect(json.index.filesNeedingReindex).toBe(0); + + const files = runJson(dir, ['files', '--json']); + const broken = files.find((f: any) => f.path === 'src/index.ts'); + expect(broken.errors).toHaveLength(1); + expect(broken.errors[0]).toMatchObject({ severity: 'warning', code: 'parse_error' }); + expect(broken.errors[0].message).toContain('parse produced no symbols'); + expect(files.find((f: any) => f.path === 'src/constants/index.ts').errors).toEqual([]); + }); + + it('rows stored without their symbols ask for a sync, and the sync repairs them', async () => { + await index(); + + // A row v1.6.2 wrote with its grammar missing (#2335), and a #1541 wipe: + // both carry the current content hash, so the hash check calls them current. + const { DatabaseSync } = require('node:sqlite'); + const db = new DatabaseSync(path.join(dir, '.codegraph', 'codegraph.db')); + db.prepare('DELETE FROM nodes WHERE file_path IN (?, ?)').run('src/constants/index.ts', 'src/types/database.ts'); + db.prepare('UPDATE files SET node_count = 0, errors = ? WHERE path = ?').run( + JSON.stringify([{ message: 'Failed to get parser for language: typescript', severity: 'error', code: 'parser_error' }]), + 'src/constants/index.ts' + ); + db.prepare('UPDATE files SET node_count = 0 WHERE path = ?').run('src/types/database.ts'); + db.close(); + + const human = run(dir, ['status']); + expect(human.status, human.out).toBe(0); + expect(human.out).not.toContain('Index is up to date'); + expect(human.out).toMatch(/2 files are missing their symbols/); + expect(human.out).toContain('codegraph sync'); + + const json = runJson(dir, ['status', '--json']); + expect(json.index.filesNeedingReindex).toBe(2); + expect(json.index.filesWithParseErrors).toBe(0); + expect(json.pendingChanges).toEqual({ added: 0, modified: 0, removed: 0 }); + + const synced = run(dir, ['sync']); + expect(synced.status, synced.out).toBe(0); + + const after = runJson(dir, ['status', '--json']); + expect(after.index.filesNeedingReindex).toBe(0); + expect(run(dir, ['status']).out).toContain('Index is up to date'); + const files = runJson(dir, ['files', '--json']); + for (const f of files) { + expect(f.nodeCount).toBeGreaterThan(1); + expect(f.errors).toEqual([]); + } + }); +}); diff --git a/src/bin/codegraph.ts b/src/bin/codegraph.ts index e5f8eacfc6..de33c1a47e 100644 --- a/src/bin/codegraph.ts +++ b/src/bin/codegraph.ts @@ -1056,6 +1056,33 @@ program } }); +/** + * The `status` warnings for indexed files whose symbols are missing although + * their content is current (#2335, #2336): one line per group, saying what is + * wrong, what to do, and naming up to three of the files. + */ +function describeFilesMissingSymbols(health: { needsReindex: string[]; parseErrors: string[] }): string[] { + const dash = getGlyphs().dash; + const sample = (paths: string[]): string => + paths.slice(0, 3).join(', ') + (paths.length > 3 ? ` (+${formatNumber(paths.length - 3)} more)` : ''); + const lines: string[] = []; + const reindex = health.needsReindex.length; + if (reindex > 0) { + lines.push( + `${formatNumber(reindex)} ${reindex === 1 ? 'file is missing its' : 'files are missing their'} symbols ${dash} ` + + `run "codegraph sync" to re-index ${reindex === 1 ? 'it' : 'them'}: ${sample(health.needsReindex)}` + ); + } + const unparsed = health.parseErrors.length; + if (unparsed > 0) { + lines.push( + `${formatNumber(unparsed)} ${unparsed === 1 ? 'file could not be parsed, so its symbols are' : 'files could not be parsed, so their symbols are'} ` + + `missing: ${sample(health.parseErrors)} ${dash} "codegraph files --json" shows the errors` + ); + } + return lines; +} + /** * codegraph status [path] */ @@ -1103,6 +1130,9 @@ program // Zero on a healthy index; non-zero at rest means a resolution pass was // interrupted, so some files' call edges are missing (#1187). const pendingRefs = cg.getPendingReferenceCount(); + // Files whose content is current but whose symbols are missing — the + // content-hash check behind "up to date" cannot see them (#2336). + const health = cg.getIndexHealth(); // JSON output mode if (options.json) { @@ -1143,6 +1173,12 @@ program // interrupted resolution pass left edges missing; the next // sync sweeps them (#1187). pendingRefs, + // Files stored without their symbols (e.g. while their grammar + // could not load, #2335); the next sync re-indexes them. + filesNeedingReindex: health.needsReindex.length, + // Files with a recorded parse error and no symbols from it + // (#2336); unchanged until the file or the parser changes. + filesWithParseErrors: health.parseErrors.length, }, })); cg.destroy(); @@ -1235,7 +1271,10 @@ program console.log(` Removed: ${changes.removed.length} files`); } info('Run "codegraph sync" to update the index'); - } else { + } + const missingSymbols = describeFilesMissingSymbols(health); + for (const line of missingSymbols) warn(line); + if (totalChanges === 0 && missingSymbols.length === 0) { success('Index is up to date'); } console.log(); @@ -1751,6 +1790,14 @@ program language: f.language, nodeCount: f.nodeCount, size: f.size, + // What extraction recorded for the file — a parse error, a skip + // reason — so a health check needn't read the database (#2336). + errors: (f.errors ?? []).map((e) => ({ + severity: e.severity, + code: e.code, + message: e.message, + line: e.line, + })), })); console.log(JSON.stringify(output, null, 2)); cg.destroy(); diff --git a/src/db/queries.ts b/src/db/queries.ts index a12e7c2643..704cfd5bcd 100644 --- a/src/db/queries.ts +++ b/src/db/queries.ts @@ -3253,6 +3253,18 @@ export class QueryBuilder { return rows.map(rowToFileRecord); } + /** + * Files stored with no nodes or with recorded extraction errors — the only + * rows that can be missing their symbols (`CodeGraph.getIndexHealth`). A + * healthy index returns few or none, so this stays cheap on a large one. + */ + getFilesWithoutNodesOrWithErrors(): FileRecord[] { + const rows = this.db + .prepare('SELECT * FROM files WHERE node_count = 0 OR errors IS NOT NULL ORDER BY path') + .all() as FileRow[]; + return rows.map(rowToFileRecord); + } + /** * Most recent index timestamp (ms since epoch) across all tracked files, or * null when nothing is indexed yet. One indexed aggregate, no per-row scan. (#329) diff --git a/src/extraction/cfml-extractor.ts b/src/extraction/cfml-extractor.ts index efd2ea6ea6..a1a668706a 100644 --- a/src/extraction/cfml-extractor.ts +++ b/src/extraction/cfml-extractor.ts @@ -147,10 +147,12 @@ export class CfmlExtractor { private extractTagBased(): void { const parser = getParser('cfml'); if (!parser) { + // Same code as TreeSitterExtractor's missing parser, so a grammar that + // failed to load is never stored as this file's index data (#2335). this.errors.push({ message: 'cfml grammar not loaded', severity: 'error', - code: 'unsupported_language', + code: 'parser_error', }); return; } diff --git a/src/extraction/grammars.ts b/src/extraction/grammars.ts index 6a05b96c0d..0a12e61d7d 100644 --- a/src/extraction/grammars.ts +++ b/src/extraction/grammars.ts @@ -9,7 +9,7 @@ import * as path from 'path'; import * as fsp from 'fs/promises'; import { Parser, Language as WasmLanguage } from 'web-tree-sitter'; -import { Language } from '../types'; +import { ExtractionError, Language } from '../types'; export type GrammarLanguage = Exclude; @@ -828,6 +828,21 @@ export function getUnavailableGrammarErrors(): Partial> return out; } +/** + * Whether these extraction errors say the file was never parsed because its + * grammar was not available (`parser_error`, recorded by the extractor when + * `getParser` has nothing for the language). + * + * That is a fact about the running process, not about the file: a daemon whose + * install was upgraded or deleted underneath it fails every lazy grammar load + * (#2335). Such a result must never replace a file's index data, and a row an + * older engine stored that way must be re-indexed — not treated as current + * because its content hash still matches. + */ +export function hasGrammarLoadFailure(errors: readonly ExtractionError[] | undefined): boolean { + return !!errors && errors.some((e) => e.code === 'parser_error'); +} + /** * Get language display name */ diff --git a/src/extraction/index.ts b/src/extraction/index.ts index 0ca3145be2..ecb946eafb 100644 --- a/src/extraction/index.ts +++ b/src/extraction/index.ts @@ -25,7 +25,7 @@ import { ParseWorkerPool, resolveParsePoolSize, resolveParseTimeoutMs } from './ import { StoreWriter, StoreBundle, finalizeStoreBundle } from './store-writer'; import { materializeKernelResult } from './kernel'; import { detectGeneratedFile } from './generated-detection'; -import { detectLanguage, isSourceFile, isLanguageSupported, isFileLevelOnlyLanguage, initGrammars, loadGrammarsForLanguages, readGrammarWasmBytes, isMpegTransportStream, hasMpegTsExtension, MPEG_TS_SNIFF_BYTES } from './grammars'; +import { detectLanguage, isSourceFile, isLanguageSupported, isFileLevelOnlyLanguage, initGrammars, loadGrammarsForLanguages, readGrammarWasmBytes, isMpegTransportStream, hasMpegTsExtension, MPEG_TS_SNIFF_BYTES, hasGrammarLoadFailure } from './grammars'; import { loadExtensionOverrides, loadIncludeIgnoredPatterns, loadExcludePatterns, loadIncludePatterns, PROJECT_CONFIG_FILENAME } from '../project-config'; import { isCodeGraphDataDir } from '../directory'; import { logDebug, logWarn } from '../errors'; @@ -2289,11 +2289,17 @@ export class ExtractionOrchestrator { const nodeCount = result.kernelCounts?.nodes ?? result.nodes.length; const edgeCount = result.kernelCounts?.edges ?? result.edges.length; + // A file whose grammar failed to load was never parsed (#2335). + const grammarUnavailable = hasGrammarLoadFailure(result.errors); + // Store: on the writer thread when active (fresh DB — bundles applied // in the same file order this chain dispatches them), else on the main // thread (SQLite connections are per-thread). const language = detectLanguage(filePath, content, overrides); - if (storeWriter) { + if (grammarUnavailable) { + // Store nothing: a row from an earlier run keeps its data, and with no + // row (or an older hash) the next sync or index retries the file. + } else if (storeWriter) { if (result.kernelBuffers) { // Buffers go to the writer as-is; the worker decodes + finalizes. // The main thread's only per-file work stays O(1) + the content hash. @@ -2321,7 +2327,9 @@ export class ExtractionOrchestrator { errors.push(...result.errors); } - if (nodeCount > 0) { + if (grammarUnavailable) { + filesErrored++; + } else if (nodeCount > 0) { filesIndexed++; totalNodes += nodeCount; totalEdges += edgeCount; @@ -2735,7 +2743,9 @@ export class ExtractionOrchestrator { errors.push(...result.errors); } - if (result.nodes.length > 0) { + if (hasGrammarLoadFailure(result.errors)) { + filesErrored++; // nothing was stored (#2335) + } else if (result.nodes.length > 0) { filesIndexed++; totalNodes += result.nodes.length; totalEdges += result.edges.length; @@ -2918,6 +2928,12 @@ export class ExtractionOrchestrator { result: ExtractionResult, onYield?: MaybeYield ): Promise { + // The file was never parsed: its grammar failed to load (#2335). Storing + // this would replace the file's symbols with an empty row under the new + // content hash, which no hash-based sync revisits. Keep whatever the index + // has; the stale (or missing) row makes the next sync retry the file. + if (hasGrammarLoadFailure(result.errors)) return; + // A kernel result can arrive as an undecoded buffer transport (empty // node/edge arrays, tables riding in kernelBuffers). Decode it before // storing — persisting the transport as-is records the file as having no @@ -2944,7 +2960,10 @@ export class ExtractionOrchestrator { const existingIsMarker = existingFile.nodeCount === 0 && (existingFile.errors?.length ?? 0) > 0; const incomingHasContent = result.nodes.length > 0; - if (!existingIsMarker || !incomingHasContent) { + // A row an older engine stored while the grammar could not load + // (#2335) records no parse at all: any real result replaces it. + const existingNeverParsed = hasGrammarLoadFailure(existingFile.errors); + if (!existingNeverParsed && (!existingIsMarker || !incomingHasContent)) { return; // No changes } } @@ -3400,13 +3419,18 @@ export class ExtractionOrchestrator { } const fullPath = path.join(this.rootDir, filePath); const tracked = trackedMap.get(filePath); + // A row an older engine stored while the file's grammar could not load + // (#2335) holds no parse of these bytes: re-index it even though its + // size, mtime and hash all match. Rows with a real parse error are + // deterministic and are not retried. + const neverParsed = tracked !== undefined && hasGrammarLoadFailure(tracked.errors); // Cheap pre-filter: an already-indexed file whose size AND mtime both match // the DB is unchanged — skip it without reading or hashing. (A content // change that preserves both exactly is the blind spot every mtime-based // incremental tool accepts; `index --force` is the escape hatch. Git bumps // mtime on every file it writes during checkout/merge, so pulls are caught.) - if (tracked) { + if (tracked && !neverParsed) { try { const stat = fs.statSync(fullPath); if (stat.size === tracked.size && Math.floor(stat.mtimeMs) === Math.floor(tracked.modifiedAt)) { @@ -3442,7 +3466,7 @@ export class ExtractionOrchestrator { filesToIndex.push(filePath); changedFilePaths.push(filePath); filesAdded++; - } else if (tracked.contentHash !== contentHash) { + } else if (tracked.contentHash !== contentHash || neverParsed) { onFileChange?.(filePath, content); filesToIndex.push(filePath); changedFilePaths.push(filePath); @@ -3477,7 +3501,7 @@ export class ExtractionOrchestrator { const result = await this.indexFile(filePath); if (result.errors.some(e => e.severity === 'error')) failedFilePaths.push(filePath); - nodesUpdated += result.nodes.length; + if (!hasGrammarLoadFailure(result.errors)) nodesUpdated += result.nodes.length; // else nothing was stored (#2335) const pause = backpressure?.(); if (pause) await pause; diff --git a/src/index.ts b/src/index.ts index 47330503aa..c1e5d25e58 100644 --- a/src/index.ts +++ b/src/index.ts @@ -25,6 +25,7 @@ import { BuildContextOptions, FindRelevantContextOptions, UnresolvedReference, + IndexHealth, } from './types'; import { DatabaseConnection, getDatabasePath, removeDatabaseFiles } from './db'; import { WalCheckpointValve, resolveWalValveMb } from './db/wal-valve'; @@ -43,6 +44,7 @@ import { extractFromSource, initGrammars, } from './extraction'; +import { hasGrammarLoadFailure, isFileLevelOnlyLanguage } from './extraction/grammars'; import { ReferenceResolver, createResolver, @@ -2065,6 +2067,34 @@ export class CodeGraph { return this.queries.countGeneratedFiles(); } + /** + * Indexed files whose symbols are missing although their content is current + * — rows a content-hash comparison calls up to date (#2336). Reported by + * `status`; `sync` repairs the `needsReindex` group. + */ + getIndexHealth(): IndexHealth { + const needsReindex: string[] = []; + const parseErrors: string[] = []; + for (const file of this.queries.getFilesWithoutNodesOrWithErrors()) { + const errors = file.errors ?? []; + if (hasGrammarLoadFailure(errors)) { + // Stored without being parsed — its grammar could not load (#2335). + needsReindex.push(file.path); + } else if (file.nodeCount === 0) { + // Every parse stores at least the file node, so zero nodes means a + // wiped row (#1541) or a recorded failure — except file-level-only + // languages and files over the size limit, which are empty on purpose. + if (isFileLevelOnlyLanguage(file.language)) continue; + if (errors.length === 0) needsReindex.push(file.path); + else if (errors.some((e) => e.severity === 'error' || e.code === 'parse_error')) parseErrors.push(file.path); + } else if (errors.some((e) => e.code === 'parse_error')) { + // Parsed, but the tree had errors and no symbols survived. + parseErrors.push(file.path); + } + } + return { needsReindex, parseErrors }; + } + // =========================================================================== // Graph Query Methods // =========================================================================== diff --git a/src/mcp/daemon.ts b/src/mcp/daemon.ts index 73a4969598..db38f97f5e 100644 --- a/src/mcp/daemon.ts +++ b/src/mcp/daemon.ts @@ -54,7 +54,7 @@ import { getDaemonSocketCandidates, getDaemonSocketPath, } from './daemon-paths'; -import { CodeGraphPackageVersion } from './version'; +import { CodeGraphPackageJsonPath, CodeGraphPackageVersion } from './version'; import { releaseWriterLock, swapWriterLock, @@ -120,6 +120,38 @@ export function finalizeDaemonExit( /** How often the daemon sweeps connected clients for a dead peer process (#692). */ const DEFAULT_CLIENT_SWEEP_MS = 30_000; +/** How often the daemon checks that the install it runs from is still in place (#2335). */ +const DEFAULT_INSTALL_CHECK_MS = 30_000; + +/** + * Why the install this daemon was started from no longer matches it, or null + * while it still does. An upgrade replaces or deletes the package directory + * under a running daemon (npm reinstalling the package, the launcher pruning + * an old bundle), and everything the daemon loads lazily from there afterwards + * fails — grammars among them, so every file its watcher re-indexed came out + * empty (#2335). Such a daemon should make way for one started from the + * current install. Any other read failure, or a package.json caught mid-write, + * answers null: the next check looks again. Exported for testing. + */ +export function installChangedReason(packageJsonPath: string, runningVersion: string): string | null { + let raw: string; + try { + raw = fs.readFileSync(packageJsonPath, 'utf8'); + } catch (err) { + const code = (err as NodeJS.ErrnoException).code; + return code === 'ENOENT' || code === 'ENOTDIR' ? `Install removed (${packageJsonPath} is gone)` : null; + } + let version: unknown; + try { + version = (JSON.parse(raw) as { version?: unknown } | null)?.version; + } catch { + return null; + } + return typeof version === 'string' && version !== runningVersion + ? `Install replaced by v${version} (this daemon is v${runningVersion})` + : null; +} + /** How long the daemon waits for the optional client-hello before proceeding without it. */ const CLIENT_HELLO_TIMEOUT_MS = 3_000; @@ -185,6 +217,7 @@ export class Daemon { private lastActivityAt = Date.now(); private maxIdleTimer: NodeJS.Timeout | null = null; private clientSweepTimer: NodeJS.Timeout | null = null; + private installCheckTimer: NodeJS.Timeout | null = null; private engine: MCPEngine; private stopping = false; private socketPath: string; @@ -372,6 +405,10 @@ export class Daemon { clearInterval(this.clientSweepTimer); this.clientSweepTimer = null; } + if (this.installCheckTimer) { + clearInterval(this.installCheckTimer); + this.installCheckTimer = null; + } process.stderr.write(`[CodeGraph daemon] Shutting down (${reason}; clients=${this.clients.size}).\n`); for (const session of [...this.clients]) { try { session.stop(); } catch { /* best-effort */ } @@ -491,6 +528,31 @@ export class Daemon { this.clientSweepTimer = setInterval(() => this.reapDeadClients(isProcessAlive), sweepMs); this.clientSweepTimer.unref?.(); } + // An install whose version could not be read at startup has nothing to + // compare against; leave that daemon alone. + const installMs = resolveInstallCheckMs(); + if (installMs > 0 && CodeGraphPackageVersion !== '0.0.0-unknown') { + this.installCheckTimer = setInterval(() => { this.checkInstall(); }, installMs); + this.installCheckTimer.unref?.(); + } + } + + /** + * Exit once the install this daemon runs from has been upgraded or removed + * (#2335; see {@link installChangedReason}). Its clients' proxies fall back + * and reconnect, and the next launch starts a daemon from the current + * install. Returns whether the daemon is exiting. The check is injected for + * tests; the timer passes the real one. + */ + checkInstall( + changed: () => string | null = () => installChangedReason(CodeGraphPackageJsonPath, CodeGraphPackageVersion), + ): boolean { + if (this.stopping) return false; + const reason = changed(); + if (!reason) return false; + process.stderr.write(`[CodeGraph daemon] ${reason}; exiting so the next session starts a daemon from the current install.\n`); + void this.stop('install changed'); + return true; } /** @@ -865,6 +927,14 @@ function resolveClientSweepMs(): number { return Math.floor(parsed); // 0 disables the sweep } +function resolveInstallCheckMs(): number { + const raw = process.env.CODEGRAPH_DAEMON_INSTALL_CHECK_MS; + if (raw === undefined || raw === '') return DEFAULT_INSTALL_CHECK_MS; + const parsed = Number(raw); + if (!Number.isFinite(parsed) || parsed < 0) return DEFAULT_INSTALL_CHECK_MS; + return Math.floor(parsed); // 0 disables the check +} + /** * Parse one client-hello line. Returns the peer pids if `line` is a well-formed * client-hello (carries the `codegraph_client` marker), or null otherwise — in diff --git a/src/mcp/index.ts b/src/mcp/index.ts index ac7aa1d0ad..e41bd43e05 100644 --- a/src/mcp/index.ts +++ b/src/mcp/index.ts @@ -44,12 +44,13 @@ import { MCPSession } from './session'; import { Daemon, clearStaleDaemonLock, + installChangedReason, isProcessAlive, tryAcquireDaemonLock, } from './daemon'; import { clearStaleDaemonArtifacts, stopOlderDaemon } from './daemon-registry'; import { connectWithHello, runLocalHandshakeProxy } from './proxy'; -import { CodeGraphPackageVersion } from './version'; +import { CodeGraphPackageJsonPath, CodeGraphPackageVersion } from './version'; import { readWriterLock, assertNoRebuild, @@ -150,6 +151,14 @@ function makeFallbackEngine(root: string): MCPEngine { if (existing && isProcessAlive(existing.pid)) { return readOnlyFallback(`live daemon PID ${existing.pid} holds the project lock`); } + // This session's own install was upgraded or removed underneath it (#2335): + // its code no longer loads whole, so as the project's writer it would only + // keep a daemon from the current install from starting. Its daemon exits + // for the same reason, which is how it got here. + if (CodeGraphPackageVersion !== '0.0.0-unknown' && + installChangedReason(CodeGraphPackageJsonPath, CodeGraphPackageVersion) !== null) { + return readOnlyFallback('this session\'s CodeGraph install was upgraded or removed; restart the session to use the current one'); + } return new MCPEngine({ writerLockRoot: root, queryPool: true, queryPoolDefaultMax: DIRECT_QUERY_POOL_MAX }); } @@ -295,6 +304,12 @@ function spawnDetachedDaemon(root: string, handover = false): void { env, }, ); + // An upgrade can delete the install this launcher runs from — its daemon + // then exits (#2335) and a respawn finds no executable. spawn reports that + // as an asynchronous 'error' event, which would crash this process if + // nobody listened. No daemon binds either way; the caller's poll gives up + // and the session is served in-process. + child.on('error', () => { /* no daemon — see above */ }); child.unref(); } finally { // The child holds its own dup of the log fd now; the launcher doesn't need it. diff --git a/src/mcp/version.ts b/src/mcp/version.ts index b6c919f0f0..e2ee11ea4d 100644 --- a/src/mcp/version.ts +++ b/src/mcp/version.ts @@ -20,10 +20,12 @@ import * as fs from 'fs'; import * as path from 'path'; +/** The `package.json` of the install this process runs from. */ +export const CodeGraphPackageJsonPath = path.join(__dirname, '..', '..', 'package.json'); + function readPackageVersion(): string { try { - const pkgPath = path.join(__dirname, '..', '..', 'package.json'); - const raw = fs.readFileSync(pkgPath, 'utf8'); + const raw = fs.readFileSync(CodeGraphPackageJsonPath, 'utf8'); const parsed = JSON.parse(raw); if (typeof parsed?.version === 'string' && parsed.version.length > 0) { return parsed.version; diff --git a/src/types.ts b/src/types.ts index 44ffaf4e43..f30080002f 100644 --- a/src/types.ts +++ b/src/types.ts @@ -265,6 +265,24 @@ export interface FileRecord { generated?: boolean; } +/** + * Indexed files whose symbols are missing even though their content hash is + * current — what a hash comparison alone calls "up to date" (#2336). Paths are + * project-relative and sorted. + */ +export interface IndexHealth { + /** + * Files `sync` re-indexes: stored while their grammar could not load (#2335) + * or stored with no nodes and no recorded reason (#1541). + */ + needsReindex: string[]; + /** + * Files with a recorded parse error and no symbols from it. Deterministic: + * they stay this way until the file (or the parser) changes. + */ + parseErrors: string[]; +} + // ============================================================================= // Extraction Types // ============================================================================= From 0c7d0f7d422aadbe5b78366e926823afaa5e7bd0 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 16:09:30 +0000 Subject: [PATCH 173/259] =?UTF-8?q?perf(resolution):=20remove=20the=20Pyth?= =?UTF-8?q?on=20indexing=20regressions=20since=201.6.2=20=E2=80=94=20graph?= =?UTF-8?q?=20byte-identical=20(#2332)=20(#2344)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Indexing CPython took about three times as long on 1.6.2 as at 290e03f (193 s vs 59 s wall, 8.3 GB vs 3.5 GB peak on a 16-thread Windows box). A CPU profile put ~290 s of the resolver workers' 412 s on two Python checks that first shipped in 1.6.2: - pythonExternalWrites (#2291) read every .py file once per module global its own module types. CPython has 2,375 Python files against the resolver's 1,000-entry content cache, so every read went to disk (184 s). - isPythonLocallyBound (#2198, #2219) re-stripped the calling file for every (function, name), and, asked once per same-named candidate by fitsPythonCallShape, re-derived the enclosing function each time: 3.55M calls for 53K distinct answers (104 s). Now: - A global's own module is read first; when its bindings already leave the type unknown (319 of 379 globals on CPython), no other module is read. - One pass per resolution indexes which Python files spell a name after a dot or a quote, and which write through .__dict__; the write scan visits only those files, in the same path order. Comment stripping only blanks text, so any write the scan accepts is spelled that way in the raw text; a non-ASCII name still reads every file. Index keys are flat copies (a regex capture would pin its whole file). - The repo files an import names are resolved once per mapping, in a WeakMap that lives as long as the resolver's import cache keeps it. - isPythonLocallyBound keeps the current file's stripped lines and per-name module-level answer, and answers a ref once. Verification: nodes, edges and unresolved_refs dumps of CPython and pretix are byte-identical to 6560052a. Medians of 3 interleaved CPython runs on a loaded machine: 290e03f 100 s / 271 s CPU / 3.3 GB, 6560052a 322 s / 801 s / 7.9 GB, this change 123 s / 361 s / 5.5 GB. In the CPU profile pythonExternalWrites drops 184 s -> 16 s and isPythonLocallyBound 104 s -> 13 s. Most of the remaining gap to 290e03f on CPython is JS checks running on its bundled minified d3 (matchDestructuredCallResult, #2334; jsFunctionLocalScope from #2226). __tests__/python-resolution-work.test.ts counts the work instead of timing it: reads of untouched files, strips of the calling file, and node lookups with 16 same-named candidates (4 / 2 / 8 with the fix, 19 / 18 / 128 on main). Each count fails when only its own part of the fix is reverted. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 2 +- __tests__/python-resolution-work.test.ts | 102 +++++++++++++++ src/resolution/name-matcher.ts | 155 +++++++++++++++++++---- 3 files changed, 231 insertions(+), 28 deletions(-) create mode 100644 __tests__/python-resolution-work.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index f280890b4f..68d7cc062b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,7 +20,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - Upgrading CodeGraph while an agent session is open no longer leaves the old version's background server in charge of your project: the first session started from the new install stops it and starts a current one in its place, even while sessions opened before the upgrade are still running. That old server could no longer load the language parsers the upgrade removed, so it saved every file it re-indexed with no symbols, while new sessions could only read the index beside it without keeping it up to date. A background server from a newer install is never stopped, and sessions opened before the upgrade keep the old version until you restart them. Thanks @lipchey for the report. (#2335) - A file is no longer saved with no symbols when its language parser can't be loaded, which is what happened to every file a background server re-indexed after an upgrade removed its install: the file keeps what it had and is indexed again once the parser loads, and files an earlier version emptied this way are re-indexed by the next sync. A background server also exits on its own once its install is upgraded or removed, so the next session starts one from the current install. Thanks @lipchey for the report. (#2335) - `codegraph status` no longer says the index is up to date while indexed files are missing their symbols: it now names files the parser couldn't read and files stored without their symbols (which `codegraph sync` repairs), `status --json` counts both, and `codegraph files --json` lists each file's recorded errors. Thanks @lipchey for the report. (#2336) - +- Indexing large Python projects is much faster again and needs less memory: since 1.6.2, resolving Python references re-read source files over and over, so a project the size of CPython took several times as long to index. The graph it builds is unchanged. Thanks @bompus for the report. (#2332) ## [1.6.2] - 2026-10-03 diff --git a/__tests__/python-resolution-work.test.ts b/__tests__/python-resolution-work.test.ts new file mode 100644 index 0000000000..f4faf445b5 --- /dev/null +++ b/__tests__/python-resolution-work.test.ts @@ -0,0 +1,102 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { ReferenceResolver } from '../src/resolution'; +import { stripCommentsForRegex } from '../src/resolution/strip-comments'; + +// Pass-through, so a test can count how often a file's text is stripped. +vi.mock('../src/resolution/strip-comments', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, stripCommentsForRegex: vi.fn(actual.stripCommentsForRegex) }; +}); + +/** + * #2332: resolving Python does a bounded amount of work per file and per call. + * The cross-module write scan for a module global read every Python file once + * per global; the local-binding check stripped the calling file once per + * function, and looked up the function around a call once per same-named + * candidate. On CPython that was minutes of CPU. Counted, never timed. + */ +describe('Python resolution work (#2332)', () => { + let tmpDir: string | undefined; + let cg: CodeGraph | undefined; + + afterEach(() => { + vi.restoreAllMocks(); + vi.mocked(stripCommentsForRegex).mockClear(); + cg?.close(); + cg = undefined; + if (tmpDir) fs.rmSync(tmpDir, { recursive: true, force: true, maxRetries: 5 }); + tmpDir = undefined; + }); + + /** Index `files`, counting per file what resolution reads, strips and looks up. */ + async function indexCounting(files: Record) { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-py-work-')); + for (const [file, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(tmpDir, file)), { recursive: true }); + fs.writeFileSync(path.join(tmpDir, file), content); + } + cg = CodeGraph.initSync(tmpDir); + const context = (cg as unknown as { resolver: ReferenceResolver }).resolver.getResolutionContext(); + const reads = vi.spyOn(context, 'readFile'); + const lookups = vi.spyOn(context, 'getNodesInFile'); + await cg.indexAll(); + return { + graph: cg, + reads: (file: string) => reads.mock.calls.filter(([f]) => f === file).length, + lookups: (file: string) => lookups.mock.calls.filter(([f]) => f === file).length, + strips: (file: string) => vi.mocked(stripCommentsForRegex).mock.calls.filter(([text]) => text === files[file]).length, + }; + } + + it('reads no file once per module global, and strips no file once per function', async () => { + const SCALE = 16; + const conns = Array.from({ length: SCALE }, (_, i) => `conn${i}`); + const files: Record = { + // Globals typed by their own module's writes: each one's type also + // depends on what every other module writes to it. + 'store.py': 'class Store:\n def fetch(self, ids):\n return ids\n', + 'settings.py': `from store import Store\n${conns.map(c => `${c} = None\n`).join('')}\n` + + `def init():\n global ${conns.join(', ')}\n${conns.map(c => ` ${c} = Store()\n`).join('')}`, + 'consumer.py': `import settings\n${conns.map((c, i) => `\ndef cb${i}(pool):\n pool.submit(settings.${c}.fetch)\n`).join('')}`, + // A bare call in every function: each asks whether its function binds the name. + 'helpers.py': 'def helper():\n return 1\n', + 'callers.py': Array.from({ length: SCALE }, (_, i) => `def f${i}():\n return helper()\n\n`).join('') + + 'def shadowed(make):\n helper = make()\n return helper()\n', + }; + // Files no reference touches. + const fillers = Array.from({ length: 20 }, (_, i) => `pkg/filler${i}.py`); + fillers.forEach((file, i) => { files[file] = `def filler${i}(x):\n return x + ${i}\n`; }); + const { graph, reads, strips } = await indexCounting(files); + + // Both paths ran: every global's method value reached Store.fetch, and the + // function that binds `helper` itself calls its own value. + const fetch = graph.getNodesByName('fetch').find(n => n.qualifiedName === 'Store::fetch')!; + expect(graph.getIncomingEdges(fetch.id).filter(e => e.metadata?.fnRef === true) + .map(e => graph.getNode(e.source)?.name).sort()).toEqual(conns.map((_, i) => `cb${i}`).sort()); + const helper = graph.getNodesByName('helper').find(n => n.kind === 'function')!; + expect(graph.getIncomingEdges(helper.id).filter(e => e.kind === 'calls') + .map(e => graph.getNode(e.source)?.name).sort()).toEqual(Array.from({ length: SCALE }, (_, i) => `f${i}`).sort()); + // A few whole-project passes read each file once; the write scan read every + // file once more per global. + expect(Math.max(...fillers.map(reads))).toBeLessThan(SCALE / 2); + // Stripped once for the pass, not once per function. + expect(strips('callers.py')).toBeLessThan(SCALE / 2); + }); + + it('finds the function around a call once, however many functions share its name', async () => { + const CANDIDATES = 16; + const CALLS = 8; + const files: Record = { + 'callers.py': Array.from({ length: CALLS }, (_, i) => `def f${i}():\n return helper()\n\n`).join(''), + }; + for (let k = 0; k < CANDIDATES; k++) files[`lib${k}/helpers.py`] = 'def helper():\n return 1\n'; + const { lookups } = await indexCounting(files); + // Each same-named candidate asks whether the caller binds `helper`; the + // answer is the call's, so the caller's nodes are read per call. + expect(lookups('callers.py')).toBeLessThanOrEqual(2 * CALLS); + }); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index e0b4ad28b8..2984a43464 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -7,7 +7,7 @@ import * as fs from 'fs'; import * as path from 'path'; import { Language, Node } from '../types'; -import { UnresolvedRef, ResolvedRef, ResolutionContext, isSupertypeTarget, CPP_DEFINE_SIGNATURE, isInheritanceRef, isImportableKind } from './types'; +import { UnresolvedRef, ResolvedRef, ResolutionContext, ImportMapping, isSupertypeTarget, CPP_DEFINE_SIGNATURE, isInheritanceRef, isImportableKind } from './types'; import { blankStringContents, stripCommentsForRegex } from './strip-comments'; import { JS_BUILT_INS, JS_BUILTIN_METHODS, TS_PRIMITIVE_TYPES } from './js-builtins'; import { SWIFT_TYPE_PATH_CALL, resolveSwiftTypePathCall } from './swift-type-visibility'; @@ -659,26 +659,13 @@ function pythonGlobalClasses(global: Node, ref: UnresolvedRef, context: Resoluti let memo = PYTHON_GLOBAL_CLASSES.get(context); if (!memo) { memo = new Map(); PYTHON_GLOBAL_CLASSES.set(context, memo); } if (memo.has(global.id)) return memo.get(global.id)!; - const file = global.filePath; - const external = pythonExternalWrites(global, context); + const own = pythonOwnGlobalWrites(global, context); + const external = own && pythonExternalWrites(global, context); // Each type is resolved in the file that wrote it (its imports name the class). - const writes: Array<{ type: string; file: string }> = [...external.writes]; - let classes: Node[] | null = external.unknown || pythonDynamicGlobalWrite(global.name, file, context) ? null : []; - for (const b of classes ? pythonGlobalBindings(global.name, file, context) : []) { - if (b.kind !== 'assign') { classes = null; break; } - const constructor = b.value && b.value !== 'None' ? pythonConstructorCall(b.value) : null; - if (b.type) { - // `conn: Base = make()` trusts the annotation; `conn: A = B()` contradicts it. - if (constructor && constructor.split('.').pop() !== b.type.split('.').pop()) { classes = null; break; } - writes.push({ type: b.type, file }); - continue; - } - if (b.value === 'None') continue; - if (!constructor) { classes = null; break; } - writes.push({ type: constructor, file }); - } + const writes = external && !external.unknown ? [...external.writes, ...own!] : null; + let classes: Node[] | null = writes && []; const seen = new Set(); - for (const write of classes ? writes : []) { + for (const write of writes ?? []) { const cls = pythonRefClass(write.type, { ...ref, filePath: write.file }, context); if (!cls) { classes = null; break; } if (!seen.has(cls.id)) { seen.add(cls.id); classes!.push(cls); } @@ -687,6 +674,33 @@ function pythonGlobalClasses(global: Node, ref: UnresolvedRef, context: Resoluti return classes; } +/** + * The types the global's own module writes to it, or null when a binding + * there leaves its type unknown — whatever other modules write, so they are + * not read (#2332). + */ +function pythonOwnGlobalWrites(global: Node, context: ResolutionContext): Array<{ type: string; file: string }> | null { + return pythonNameScan(context, `own\0${global.id}`, () => { + const file = global.filePath; + if (pythonDynamicGlobalWrite(global.name, file, context)) return null; + const writes: Array<{ type: string; file: string }> = []; + for (const b of pythonGlobalBindings(global.name, file, context)) { + if (b.kind !== 'assign') return null; + const constructor = b.value && b.value !== 'None' ? pythonConstructorCall(b.value) : null; + if (b.type) { + // `conn: Base = make()` trusts the annotation; `conn: A = B()` contradicts it. + if (constructor && constructor.split('.').pop() !== b.type.split('.').pop()) return null; + writes.push({ type: b.type, file }); + continue; + } + if (b.value === 'None') continue; + if (!constructor) return null; + writes.push({ type: constructor, file }); + } + return writes; + }); +} + /** * The repo files a Python module path can name from `fromFile`. Relative * paths (`..settings`) resolve exactly; absolute ones match a file path @@ -722,9 +736,7 @@ function pythonModuleAliases(filePath: string, moduleFile: string, context: Reso const aliases = new Set(); const ambiguous = new Set(); for (const m of context.getImportMappings(filePath, 'python')) { - const dotted = m.isNamespace ? m.source - : /^\.+$/.test(m.source) ? `${m.source}${m.exportedName}` : `${m.source}.${m.exportedName}`; - const files = pythonModuleFiles(dotted, filePath, context); + const files = pythonImportedFiles(m, filePath, context); if (!files.includes(moduleFile)) continue; // The mapping cannot tell `import a.b` from `import a.b as b`; the source line can. // Exactly this module (not `other.a.b`), outside string literals. @@ -736,6 +748,24 @@ function pythonModuleAliases(filePath: string, moduleFile: string, context: Reso return { aliases: [...aliases], ambiguous: [...ambiguous] }; } +const PYTHON_IMPORTED_FILES = new WeakMap>(); +/** + * The repo files an import of `filePath` can name. Every global's write scan + * asks again of the same files (#2332); kept for as long as the resolver keeps + * the mapping itself, so the memo never outlives its import cache. + */ +function pythonImportedFiles(m: ImportMapping, filePath: string, context: ResolutionContext): string[] { + let memo = PYTHON_IMPORTED_FILES.get(context); + if (!memo) PYTHON_IMPORTED_FILES.set(context, (memo = new WeakMap())); + let files = memo.get(m); + if (!files) { + const dotted = m.isNamespace ? m.source + : /^\.+$/.test(m.source) ? `${m.source}${m.exportedName}` : `${m.source}.${m.exportedName}`; + memo.set(m, (files = pythonModuleFiles(dotted, filePath, context))); + } + return files; +} + const PYTHON_MAIN_GUARD = /^if\s+(?:__name__\s*==\s*(['"])__main__\1|(['"])__main__\2\s*==\s*__name__)\s*:/; /** Line indexes inside a top-level `if __name__ == "__main__":` block — script code, not module state. */ function pythonMainBlockLines(filePath: string, context: ResolutionContext): Set { @@ -775,8 +805,8 @@ function pythonExternalWrites(global: Node, context: ResolutionContext): { write const out = { writes: [] as Array<{ type: string; file: string }>, unknown: false, writers: new Set() }; const name = global.name; const escape = (s: string) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); - for (const file of context.getAllFiles()) { - if (file === global.filePath || !/\.pyi?$/.test(file) || !context.readFile(file)?.includes(name)) continue; + for (const file of pythonWriteCandidates(name, context)) { + if (file === global.filePath) continue; const test = isPythonTestFile(file); const { aliases, ambiguous } = pythonModuleAliases(file, global.filePath, context); if (!test && !aliases.length && !ambiguous.length) continue; @@ -812,6 +842,41 @@ function pythonExternalWrites(global: Node, context: ResolutionContext): { write }); } +/** + * The Python files that can write a module global named `name`, in path + * order: those that spell it after a dot (`settings.conn = X`) or a quote + * (`setattr(settings, "conn", X)`), and those that write through `.__dict__`, + * whose statement may spell it anywhere. Comment stripping only blanks text, + * so whatever a stripped statement spells, the file's text spells too. The + * files are indexed once per pass, instead of every global reading every + * Python file (#2332). + */ +function pythonWriteCandidates(name: string, context: ResolutionContext): string[] { + const index = pythonNameScan(context, 'write-candidates', () => { + const files = context.getAllFiles().filter(f => /\.pyi?$/.test(f)); + const spelled = new Map(); + const dict: Array<{ at: number; words: string }> = []; + files.forEach((file, at) => { + const source = context.readFile(file) ?? ''; + const names = new Set(); + for (const m of source.matchAll(/[.'"](\w+)/g)) names.add(m[1]!); + for (const n of names) { + const list = spelled.get(n); + // A capture is a sliced view that would pin the file's whole text: key a flat copy. + if (list) list.push(at); else spelled.set(Buffer.from(n).toString(), [at]); + } + // A word-only name is in the text exactly when it is in one of its words. + if (names.has('__dict__')) dict.push({ at, words: [...new Set(source.match(/\w+/g))].join('\n') }); + }); + return { files, spelled, dict }; + }); + // The index holds ASCII words; any other name is looked for in every file's text. + if (!/^\w+$/.test(name)) return index.files.filter(f => context.readFile(f)?.includes(name)); + const hits = new Set(index.spelled.get(name)); + for (const { at, words } of index.dict) if (words.includes(name)) hits.add(at); + return [...hits].sort((a, b) => a - b).map(at => index.files[at]!); +} + /** * Whether the global's own module can write it through its namespace dict: * `globals()` / `vars()` / `sys.modules[__name__]` used as anything but a @@ -853,6 +918,8 @@ function pythonDynamicGlobalWrite(name: string, filePath: string, context: Resol * as a base-typed receiver does. Otherwise, no edge. */ function pythonGlobalMembers(global: Node, member: string, ref: UnresolvedRef, context: ResolutionContext): Node[] { + // Unknown from its own module alone: no edge, and no other module to read. + if (!pythonOwnGlobalWrites(global, context)) return []; // A test that installs its own double sees the double, not the production type. if (pythonExternalWrites(global, context).writers.has(ref.filePath)) return []; const classes = pythonGlobalClasses(global, ref, context); @@ -2538,6 +2605,14 @@ function isDecoratedFixture(n: Node, context: ResolutionContext): boolean { } const PY_LOCAL_BINDS = new WeakMap>(); +/** + * The file isPythonLocallyBound last read: its code lines, and per name + * whether its module binds it. Refs arrive grouped by file, so one file per + * context spares re-stripping the file for every function and name (#2332). + */ +const PY_LOCAL_FILE = new WeakMap }>(); +/** The ref isPythonLocallyBound last answered, and the answer. */ +const PY_LOCAL_LAST = new WeakMap(); /** * Whether the function around a Python call — or its module, at top level — @@ -2547,6 +2622,17 @@ const PY_LOCAL_BINDS = new WeakMap>(); * every such call went to one test file's `def view`. */ function isPythonLocallyBound(name: string, ref: UnresolvedRef, context: ResolutionContext): boolean { + // fitsPythonCallShape asks once per same-named candidate, and finding the + // function around the call reads every node in the file: answer a ref once (#2332). + const last = PY_LOCAL_LAST.get(context); + if (last?.ref === ref && last.name === name) return last.bound; + const bound = pythonLocalBinding(name, ref, context); + PY_LOCAL_LAST.set(context, { ref, name, bound }); + return bound; +} + +/** isPythonLocallyBound's answer, kept per calling function and name. */ +function pythonLocalBinding(name: string, ref: UnresolvedRef, context: ResolutionContext): boolean { const fn = context.getNodesInFile(ref.filePath) .filter((f) => (f.kind === 'function' || f.kind === 'method') && f.startLine <= ref.line && f.endLine >= ref.line) .sort((a, b) => (a.endLine - a.startLine) - (b.endLine - b.startLine))[0]; @@ -2561,7 +2647,12 @@ function isPythonLocallyBound(name: string, ref: UnresolvedRef, context: Resolut return false; } // Code only: `{% user_display user as user_display %}` in a docstring binds nothing. - const lines = stripCommentsForRegex(context.readFile(ref.filePath) ?? '', 'python').split(/\r?\n/); + let file = PY_LOCAL_FILE.get(context); + if (file?.filePath !== ref.filePath) { + const lines = stripCommentsForRegex(context.readFile(ref.filePath) ?? '', 'python').split(/\r?\n/); + PY_LOCAL_FILE.set(context, (file = { filePath: ref.filePath, lines, module: new Map() })); + } + const lines = file.lines; const n = name; const assigns = new RegExp(`^\\s*(?:[\\w\\s,*()\\[\\]]*,\\s*)?\\(?\\*?${n}\\)?\\s*(?:,[\\w\\s,*()\\[\\]]*)?(?::[^=]+)?=(?!=)`); const targets = new RegExp(`\\bfor\\s+[\\w\\s,()]*\\b${n}\\b[\\w\\s,()]*\\s+in\\b|\\bas\\s+${n}\\b`); @@ -2582,8 +2673,15 @@ function isPythonLocallyBound(name: string, ref: UnresolvedRef, context: Resolut } } // A module-level binding (`view = api_view(['GET'])(handler)`). - const top = new RegExp(`^(?:[\\w,\\s]*,\\s*)?${n}\\s*(?:,[\\w\\s,]*)?(?::[^=]+)?=(?!=)`); - for (let line = 0; !bound && line < lines.length; line++) bound = top.test(lines[line] ?? ''); + if (!bound) { + let module = file.module.get(n); + if (module === undefined) { + const top = new RegExp(`^(?:[\\w,\\s]*,\\s*)?${n}\\s*(?:,[\\w\\s,]*)?(?::[^=]+)?=(?!=)`); + module = lines.some(line => top.test(line)); + file.module.set(n, module); + } + bound = module; + } memo.set(key, bound); return bound; } @@ -6606,6 +6704,7 @@ export function clearNameMatcherMemos(context: ResolutionContext): void { PYTHON_STATEMENT_STARTS.delete(context); PYTHON_GLOBAL_CLASSES.delete(context); PYTHON_NAME_SCANS.delete(context); + PYTHON_IMPORTED_FILES.delete(context); AWAITED_TYPE_MEMO.delete(context); AWAITED_FILES.delete(context); C_STATIC_MEMO.delete(context); @@ -6664,6 +6763,8 @@ export function clearNameMatcherMemos(context: ResolutionContext): void { JVM_PACKAGES.delete(context); MINIFIED_SCRIPTS.delete(context); PY_LOCAL_BINDS.delete(context); + PY_LOCAL_FILE.delete(context); + PY_LOCAL_LAST.delete(context); OVERLOAD_SETS.delete(context); PHP_FILE_SCOPES.delete(context); JAVA_STATIC_IMPORTS.delete(context); From 355afc70b14c22713fee491eafb881b577d34ef1 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 17:20:15 +0000 Subject: [PATCH 174/259] fix(csharp): walk field and property initializers; a constant never names a type (#2337) (#2348) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(csharp): walk field and property initializers as the member Field declarators and property `= initializer` values were skipped by both extractors, so `private readonly ILogger _log = LogManager.GetLogger(typeof(X));` linked nothing, and a method or class used only from an initializer looked unused. - tree-sitter.ts / csharp.rs: walk each C# field declarator whole with its field on the stack, and every property `value:` (not only `=> expr`) with the property on the stack. Calls, instantiations, static reads, lambda bodies and fn-ref candidates (varinit included) belong to the member; the fn-ref scan skips the walked subtrees, so each candidate is captured once. Attributes stay unwalked. - A target-typed `new()` that is the initializer instantiates the declared type (`List _items = new();` -> List). In bodies it stays invisible, as before. - name-matcher.ts: a .NET type position never names a `constant` (the kind `const` / `static readonly` fields get). Walking initializers exposed `new Version(5, 18)` binding to a `const string Version` and `static readonly Meter Meter = new(...)` instantiating itself. Kernel/wasm parity: 0 diffs on serilog, Newtonsoft.Json and jellyfin, and full-index dumps byte-identical between the two arms. No EXTRACTION_VERSION bump: #2345 already moved it to 28 this release. Co-Authored-By: Claude Opus 5.5 * docs(changelog): credit the report and contributor behind the constant change The type-position change here is what issue #2337 reports (`new Station { … }` unresolved beside a `private const string Station`) and the same one-line change as @drakeo338's #2339. Verified on the issue's own three-file repro: the `instantiates` edge to `Demo.Core::Station`, missing on main, is present with this branch. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 2 + __tests__/csharp-member-initializers.test.ts | 155 +++++++++++++++++++ __tests__/csharp-type-position-refs.test.ts | 42 +++++ __tests__/kernel-csharp-parity.test.ts | 40 ++++- codegraph-kernel/src/csharp.rs | 94 +++++++++-- docs/design/csharp-kernel-port-checklist.md | 96 ++++++++---- src/extraction/tree-sitter.ts | 103 +++++++++--- src/resolution/name-matcher.ts | 8 +- 8 files changed, 469 insertions(+), 71 deletions(-) create mode 100644 __tests__/csharp-member-initializers.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 68d7cc062b..da460b5b77 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,8 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In VB.NET, every member of a `Structure` is now indexed, including its fields, properties, methods, constructors and nested enums. Before, only the first member was, so the rest could not be found and their callers looked empty. - In VB.NET and C#, what a property's `Get` and `Set` code calls, creates and reads now belongs to that property, as do C#'s `get => …` accessors and `=> …` property bodies. Before, it was dropped, so a method used only from a property looked unused. - In VB.NET, a field or property initializer like `= Compute()` or `As New List(Of Order)` now links what it calls and creates, and so do a `Custom Event`'s `AddHandler`, `RemoveHandler` and `RaiseEvent` blocks. Re-index VB.NET and C# projects after upgrading. +- In C#, a field or property initializer like `private readonly ILogger _log = LogManager.GetLogger(typeof(X));` or `public List Items { get; } = new();` now links what it calls, creates and reads, including inside a lambda, and a target-typed `new()` there counts as creating the declared type. Before, initializers were skipped, so a method or class used only from one — like a converter created in a static list — looked unused, and a method passed as a value there was credited to the whole class instead of the field or property. +- In C# and VB.NET, a constant no longer stands in for a type with the same name: `new Station { … }` links to class `Station` even when another class declares `private const string Station`, `new Version(…)` no longer links to a `const string Version`, and a field like `static readonly Meter Meter` no longer points at itself. Thanks @EvanYu1980 for the report and @drakeo338. (#2337) - Upgrading CodeGraph while an agent session is open no longer leaves the old version's background server in charge of your project: the first session started from the new install stops it and starts a current one in its place, even while sessions opened before the upgrade are still running. That old server could no longer load the language parsers the upgrade removed, so it saved every file it re-indexed with no symbols, while new sessions could only read the index beside it without keeping it up to date. A background server from a newer install is never stopped, and sessions opened before the upgrade keep the old version until you restart them. Thanks @lipchey for the report. (#2335) - A file is no longer saved with no symbols when its language parser can't be loaded, which is what happened to every file a background server re-indexed after an upgrade removed its install: the file keeps what it had and is indexed again once the parser loads, and files an earlier version emptied this way are re-indexed by the next sync. A background server also exits on its own once its install is upgraded or removed, so the next session starts one from the current install. Thanks @lipchey for the report. (#2335) - `codegraph status` no longer says the index is up to date while indexed files are missing their symbols: it now names files the parser couldn't read and files stored without their symbols (which `codegraph sync` repairs), `status --json` counts both, and `codegraph files --json` lists each file's recorded errors. Thanks @lipchey for the report. (#2336) diff --git a/__tests__/csharp-member-initializers.test.ts b/__tests__/csharp-member-initializers.test.ts new file mode 100644 index 0000000000..f5125b8321 --- /dev/null +++ b/__tests__/csharp-member-initializers.test.ts @@ -0,0 +1,155 @@ +/** + * C# field and property initializers belong to the member they initialize. + * `private readonly ILogger _log = LogManager.GetLogger(typeof(X));` and + * `public List Items { get; } = new List();` used to be skipped by + * both extractors, so the calls, instantiations and static reads written + * there were lost, and a method passed as a value there was the class's + * reference rather than the member's. + * + * A target-typed `new()` names no type of its own; as an initializer it + * constructs the member's declared type. + * + * Runs against the native kernel (when built) and the wasm extractor, which + * must agree. + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { Node } from '../src/types'; + +const FILES: Record = { + 'Lib.cs': `namespace Lib +{ + public class Crate { } +} +public interface ILogger { } +public static class LogManager +{ + public static ILogger GetLogger(System.Type t) => null; +} +public static class Helper +{ + public static int Compute(int n) => n; + public static int Seed() => 1; + public static void Register(System.Action a) { } + public static System.Action Wrap(System.Action a) => a; +} +public class Widget +{ + public Widget() { } + public Widget(int n) { } + public int Size { get; set; } +} +public class Bag { } +public static class Defaults +{ + public const string Name = "x"; + public static readonly Widget Empty = new Widget(); +} +`, + 'Box.cs': `using Lib; + +public class Box +{ + private readonly ILogger _log = LogManager.GetLogger(typeof(Box)); + private int _a = 1, _b = Helper.Compute(2); + private static readonly int Max = Helper.Seed(); + private readonly Widget _made = new Widget(3); + private readonly Widget _typed = new() { Size = Helper.Seed() }; + private Widget? _maybe = new(); + private readonly Bag _bag = new(); + private readonly Lib.Crate _crate = new(); + private readonly string _label = Defaults.Name; + private readonly System.Func _square = x => Helper.Compute(x * x); + private readonly System.Action _later = () => Helper.Register(Handle); + private readonly System.Action _direct = Handle; + [System.Obsolete("use Helper.Seed()")] private int _flagged; + public Bag Items { get; } = new Bag(); + public Widget Made { get; set; } = new(); + public Widget Copy { get; } = Defaults.Empty; + public System.Action Wrapped { get; } = Helper.Wrap(Handle); + public int Seeded { get; } = Helper.Seed(); + private static void Handle() { } +} +`, +}; + +describe('C# field and property initializers', () => { + let root = ''; + let cg: CodeGraph | undefined; + let kernel: string | undefined; + + beforeEach(() => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-cs-initializers-')); + for (const [rel, content] of Object.entries(FILES)) fs.writeFileSync(path.join(root, rel), content); + kernel = process.env.CODEGRAPH_KERNEL; + }); + + afterEach(() => { + cg?.close(); + cg = undefined; + fs.rmSync(root, { recursive: true, force: true }); + if (kernel === undefined) delete process.env.CODEGRAPH_KERNEL; + else process.env.CODEGRAPH_KERNEL = kernel; + }); + + it.each(['default', 'wasm'])('are walked as the member they initialize (%s)', async (backend) => { + if (backend === 'wasm') process.env.CODEGRAPH_KERNEL = '0'; + else delete process.env.CODEGRAPH_KERNEL; + cg = await CodeGraph.init(root, { index: true }); + const graph = cg; + const member = (qualifiedName: string): Node => { + const node = graph.getNodesInFile('Box.cs').find((n) => n.qualifiedName === qualifiedName); + expect(node, qualifiedName).toBeDefined(); + return node!; + }; + const targets = (node: Node, kind: string): string[] => + graph + .getOutgoingEdgesFrom([node.id]) + .filter((e) => e.kind === kind) + .map((e) => graph.getNode(e.target)!.qualifiedName) + .sort(); + + // Calls, per declarator. + expect(targets(member('Box::_log'), 'calls')).toEqual(['LogManager::GetLogger']); + expect(targets(member('Box::_a'), 'calls')).toEqual([]); + expect(targets(member('Box::_b'), 'calls')).toEqual(['Helper::Compute']); + expect(member('Box::Max').kind).toBe('constant'); + expect(targets(member('Box::Max'), 'calls')).toEqual(['Helper::Seed']); + expect(targets(member('Box::Seeded'), 'calls')).toEqual(['Helper::Seed']); + + // Instantiations, including a target-typed `new()` of the declared type. + expect(targets(member('Box::_made'), 'instantiates')).toEqual(['Widget']); + expect(targets(member('Box::_typed'), 'instantiates')).toEqual(['Widget']); + expect(targets(member('Box::_typed'), 'calls')).toEqual(['Helper::Seed']); + expect(targets(member('Box::_maybe'), 'instantiates')).toEqual(['Widget']); + expect(targets(member('Box::_bag'), 'instantiates')).toEqual(['Bag']); + expect(targets(member('Box::_crate'), 'instantiates')).toEqual(['Lib::Crate']); + expect(targets(member('Box::Items'), 'instantiates')).toEqual(['Bag']); + expect(targets(member('Box::Made'), 'instantiates')).toEqual(['Widget']); + + // Static reads. + expect(targets(member('Box::_label'), 'references')).toContain('Defaults'); + expect(targets(member('Box::Copy'), 'references')).toContain('Defaults'); + + // Lambda bodies belong to the member the lambda initializes. + expect(targets(member('Box::_square'), 'calls')).toEqual(['Helper::Compute']); + expect(targets(member('Box::_later'), 'calls')).toEqual(['Helper::Register']); + + // A method passed as a value is the member's reference, captured once — + // not the class's as well. + expect(targets(member('Box::_later'), 'references')).toContain('Box::Handle'); + expect(targets(member('Box::_direct'), 'references')).toContain('Box::Handle'); + expect(targets(member('Box::Wrapped'), 'calls')).toEqual(['Helper::Wrap']); + expect(targets(member('Box::Wrapped'), 'references')).toContain('Box::Handle'); + expect(targets(member('Box'), 'references')).not.toContain('Box::Handle'); + + // Attribute arguments are not an initializer, and the class itself + // calls and creates nothing. + expect(targets(member('Box::_flagged'), 'calls')).toEqual([]); + expect(targets(member('Box'), 'calls')).toEqual([]); + expect(targets(member('Box'), 'instantiates')).toEqual([]); + }); +}); diff --git a/__tests__/csharp-type-position-refs.test.ts b/__tests__/csharp-type-position-refs.test.ts index c000d92105..ccfba566b9 100644 --- a/__tests__/csharp-type-position-refs.test.ts +++ b/__tests__/csharp-type-position-refs.test.ts @@ -69,4 +69,46 @@ describe('C#: a type position names a type', () => { cg.close(); } }); + + it('never a constant that shares the name', async () => { + // `const` and `static readonly` fields are constants: jellyfin's `new + // Version(5, 18)` (System.Version) bound to a `const string Version` + // claim name, and serilog's `static readonly Meter Meter = new(…)` + // instantiated itself. + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-cs-type-refs-')); + roots.push(root); + const files: Record = { + 'src/Claims.cs': `namespace App; +public static class ClaimTypes { + public const string Version = "v"; +} +`, + 'src/Encoder.cs': `namespace App; +using System; +public class ValueFormatter { } +public class Encoder { + private static readonly Version MinVersion = new Version(5, 18); + private static readonly Meter Meter = new("app"); + private static readonly ValueFormatter ValueFormatter = new(); + public void Check() { var v = new Version(1, 0); } +} +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + const cg = await CodeGraph.init(root, { index: true }); + try { + const members = cg.getNodesInFile('src/Encoder.cs').filter((n) => n.kind !== 'file'); + const targets = cg + .getOutgoingEdgesFrom(members.map((n) => n.id), ['references', 'instantiates', 'type_of']) + .map((e) => `${cg.getNode(e.source)!.name} ${e.kind} ${cg.getNode(e.target)!.kind}:${cg.getNode(e.target)!.qualifiedName}`) + .sort(); + expect(targets.filter((t) => t.includes(' constant:'))).toEqual([]); + expect(targets).toContain('ValueFormatter instantiates class:App::ValueFormatter'); + } finally { + cg.close(); + } + }); }); diff --git a/__tests__/kernel-csharp-parity.test.ts b/__tests__/kernel-csharp-parity.test.ts index 3d09c19d98..7a15371891 100644 --- a/__tests__/kernel-csharp-parity.test.ts +++ b/__tests__/kernel-csharp-parity.test.ts @@ -8,12 +8,12 @@ * * - Torture.cs — block namespace + nested/second-namespace quirks, * base_list shapes, records, properties (incl. the bare-identifier - * signature loss, accessor bodies walked as the property and - * never-walked initializers), fields/constants, + * signature loss; accessor bodies and initializers walked as the + * property), fields/constants (initializers walked as the field), * events/operators/indexer/destructor (no nodes, calls → class), ctor * initializer hole, explicit interface impl, local functions, the call * zoo (raw member-access texts, chained re-encode, `(myDel)(x)` conv, - * `nameof`), instantiation shapes (incl. invisible `new()`/`new {}`/ + * `nameof`), instantiation shapes (incl. invisible body `new()`/`new {}`/ * arrays), static value reads, C# type refs, fn-ref candidates * (`+=` subscription, `this.X` bare-name form, initializer lists), * value-ref targets + local shadow prune, preprocessor passthrough. @@ -169,10 +169,9 @@ describe.skipIf(!kernelBuilt)('kernel C# extraction parity', () => { minNodes: 2, }, { - // Accessor bodies and `=> expr` are walked as the property (calls, - // instantiations, static reads, fn-ref candidates); an `= initializer` - // is only scanned for candidates, attributed to the class. - name: 'property bodies are walked as the property; initializers only scanned', + // Accessor bodies, `=> expr` and `= initializer` are walked as the + // property (calls, instantiations, static reads, fn-ref candidates). + name: 'property bodies and initializers are walked as the property', source: [ 'public class C {', ' void H(int v) { }', @@ -185,6 +184,33 @@ describe.skipIf(!kernelBuilt)('kernel C# extraction parity', () => { ].join('\n'), minNodes: 7, }, + { + // Each field declarator and each property `= initializer` is walked + // with its member on the stack — lambdas included — and its fn-ref + // candidates (varinit too) are the member's, captured once. A + // target-typed `new()` there instantiates the declared type; attribute + // arguments stay with the class's candidates-only scan. + name: 'field and property initializers are walked as the member', + source: [ + 'public class C {', + ' static void H() { }', + ' private readonly ILogger _log = LogManager.GetLogger(typeof(C));', + ' int a = 1, b = Calc.Max(2);', + ' private static readonly List Table = new() { H };', + ' private Widget? _w = new(1) { Size = Make() };', + ' private global::Ns.Box _g = new(), _h = (new());', + ' private (int, int) _t = new();', + ' Action g = () => Register(H);', + ' Del d = H;', + ' public Widget P { get; } = new();', + ' public Action R { get; } = Wrap(H);', + ' public string S { get; } = Defaults.Name;', + ' [Attr(Make(H))] public int T { get; set; } = 3;', + '}', + '', + ].join('\n'), + minNodes: 17, + }, ]; for (const m of MICROS) { diff --git a/codegraph-kernel/src/csharp.rs b/codegraph-kernel/src/csharp.rs index 04e20ca5a5..c10696ec99 100644 --- a/codegraph-kernel/src/csharp.rs +++ b/codegraph-kernel/src/csharp.rs @@ -5,8 +5,8 @@ //! path, bug-for-bug, verified by scripts/kernel-parity.mjs and the full-index //! dump-diff gate. The authoritative quirk list is //! docs/design/csharp-kernel-port-checklist.md — including every deliberate -//! emission hole (field/property initializers, constructor initializers, -//! delegates/events/operators/indexers, top-level locals) and garbage ref +//! emission hole (constructor initializers, delegates/events/operators/ +//! indexers, top-level locals) and garbage ref //! (`(repo)` primary-ctor extends, `: byte` enum extends, `nameof` calls) //! this file preserves on purpose. Positions in UTF-16 code units. Files whose //! parse tree contains ERRORS defer to the wasm extractor. @@ -572,16 +572,18 @@ impl<'t> Walker<'t> { self.extract_enum(node); skip_children = true; } else if kind == "property_declaration" && self.inside_class_like() { - // The code a property runs — its accessor bodies and `=> expr` — - // is walked with the property on the stack (propertyBodies). The - // candidates-only scan covers the rest (an `= initializer`), + // The code a property runs — its accessor bodies, `=> expr` and + // `= initializer` — is walked with the property on the stack + // (propertyBodies). The candidates-only scan covers the rest, // skipping what the body walk captured. let walked: Vec = match self.extract_property(node) { Some((row, name)) => { let bodies = property_bodies(node); if !bodies.is_empty() { self.stack.push(Scope { row, kind: "property", name }); + let declared = node.child_by_field_name("type"); for body in &bodies { + self.extract_target_typed_new(Some(*body), declared); self.visit_function_body(*body); } self.stack.pop(); @@ -593,8 +595,11 @@ impl<'t> Walker<'t> { self.scan_fn_ref_subtree(node, 0, &walked); skip_children = true; } else if kind == "field_declaration" && self.inside_class_like() { - self.extract_field(node); - self.scan_fn_ref_subtree(node, 0, &[]); + // Each declarator, `= initializer` included, is walked with its + // field on the stack (extract_field); the candidates-only scan + // covers the rest, skipping the declarators that walk captured. + let walked = self.extract_field(node); + self.scan_fn_ref_subtree(node, 0, &walked); skip_children = true; } else if kind == "local_declaration_statement" && !self.inside_class_like() { // Top-level statements: extractVariable's generic fallback finds no @@ -864,8 +869,11 @@ impl<'t> Walker<'t> { } /// extractField (2046) — field_declaration; each declarator becomes a - /// field/constant node anchored at the DECLARATOR. - fn extract_field(&mut self, node: Node<'t>) { + /// field/constant node anchored at the DECLARATOR, and the declarator — + /// its `= initializer` — is walked with that node on the stack. Returns + /// the walked declarators, for the fn-ref scan to skip. + fn extract_field(&mut self, node: Node<'t>) -> Vec { + let mut walked = Vec::new(); let docstring = preceding_docstring(node, self.src); let visibility = Some(self.visibility_of(node)); let is_static = Some(self.is_static(node)); @@ -932,6 +940,15 @@ impl<'t> Walker<'t> { // multi-declarator fields emit the type refs once PER // declarator, each from its own field node. self.extract_csharp_type_refs(node, row); + // The initializer is the declarator's last, unnamed child: + // its calls, instantiations, static reads and fn-ref + // candidates (varinit included) are the field's. + self.stack.push(Scope { row, kind: field_kind, name }); + let declared = var_decl.and_then(|vd| vd.child_by_field_name("type")); + self.extract_target_typed_new(last_named_child(decl), declared); + self.visit_function_body(decl); + self.stack.pop(); + walked.push(decl.id()); } } } else { @@ -951,6 +968,44 @@ impl<'t> Walker<'t> { ); } } + walked + } + + /// extractTargetTypedNew (tree-sitter.ts) — a target-typed `new()` + /// (implicit_object_creation_expression) names no type, so it is no + /// instantiation kind; as a field's or property's initializer it + /// constructs the declared type (`List _items = new();` → List). + /// Emitted from the stack top (the member) at the `new()`. + fn extract_target_typed_new(&mut self, value: Option>, declared: Option>) { + let Some(value) = value else { return }; + if value.kind() != "implicit_object_creation_expression" || self.stack.is_empty() { + return; + } + let Some(class_name) = declared.and_then(|t| self.class_type_name(t)) else { return }; + let from = self.top_row(); + self.push_ref_at(from, &class_name, edge_kind_index("instantiates").unwrap(), value); + } + + /// csharpClassTypeName (tree-sitter.ts) — the class a declared type + /// names, as `new T()` would name it: `List` → List, `Ns.Foo` / + /// `global::Foo` → Foo, `Foo?` → Foo. Predefined, array, tuple and pointer + /// types name no class. + fn class_type_name(&self, node: Node) -> Option { + match node.kind() { + "identifier" => { + let text = self.text(node); + (!text.is_empty()).then(|| text.to_string()) + } + "generic_name" => { + let ident = (0..node.named_child_count()) + .filter_map(|i| node.named_child(i)) + .find(|c| c.kind() == "identifier")?; + self.class_type_name(ident) + } + "qualified_name" | "alias_qualified_name" => self.class_type_name(node.child_by_field_name("name")?), + "nullable_type" => self.class_type_name(node.child_by_field_name("type")?), + _ => None, + } } /// extractMethod (1737) — method_declaration + constructor_declaration. @@ -1142,7 +1197,8 @@ impl<'t> Walker<'t> { let Some(ctor) = ctor else { return }; // `new List()` → `List`; `new Ns.Foo()` → `Foo`. Target-typed // `new()` / anonymous `new { }` / arrays `new T[n]` never reach here - // (not in INSTANTIATION_KINDS) — invisible by design. + // (not in INSTANTIATION_KINDS) — invisible by design, except a `new()` + // that initializes a field or property (extract_target_typed_new). let class_name = strip_generic_and_qualifier(self.text(ctor)); if !class_name.is_empty() { let from = self.top_row(); @@ -1492,8 +1548,9 @@ impl<'t> Walker<'t> { return; } // functionTypes is EMPTY for C#; the literal halt list applies — - // lambda_expression IS C#'s lambda, so initializer lambdas stop the - // scan; anonymous_method_expression is NOT listed and scans through. + // lambda_expression IS C#'s lambda, so a lambda the scan reaches (a + // top-level statement's) stops it; anonymous_method_expression is NOT + // listed and scans through. Member initializers are walked instead. if depth > 0 && matches!( node.kind(), @@ -1643,8 +1700,9 @@ impl<'t> Walker<'t> { } /// propertyBodies (tree-sitter.ts) — the parts of a property that run code: -/// each accessor's `body` (a block or `=> expr`) and an expression-bodied -/// property's `=> …` value. An `= initializer` value is not a body. +/// each accessor's `body` (a block or `=> expr`), then the property's +/// `value` — an expression body's `=> …` or an `= initializer`. Attributes +/// are not walked. fn property_bodies(node: Node) -> Vec { let mut bodies = Vec::new(); if let Some(accessors) = node.child_by_field_name("accessors") { @@ -1659,13 +1717,15 @@ fn property_bodies(node: Node) -> Vec { } } if let Some(value) = node.child_by_field_name("value") { - if value.kind() == "arrow_expression_clause" { - bodies.push(value); - } + bodies.push(value); } bodies } +fn last_named_child(node: Node) -> Option { + node.named_child(node.named_child_count().checked_sub(1)?) +} + fn find_anonymous_class_body(node: Node) -> Option { for i in 0..node.named_child_count() { if let Some(child) = node.named_child(i) { diff --git a/docs/design/csharp-kernel-port-checklist.md b/docs/design/csharp-kernel-port-checklist.md index e817968c95..919e767687 100644 --- a/docs/design/csharp-kernel-port-checklist.md +++ b/docs/design/csharp-kernel-port-checklist.md @@ -234,8 +234,8 @@ all PRESERVE): | `enum_declaration` | enumTypes:1064 → extractEnum:1914 | body `enum_member_declaration_list` required (bodiless → no node); extractInheritance sees `base_list` → **the underlying type `: byte` emits an `extends` ref named `byte`** (quirk, §inheritance); `enum_member_declaration` children → extractEnumMembers:1958 — `name` field path: ONE `enum_member` node per member, positioned at the member node (attributes included in its span), values/attributes ignored; non-member children (preproc_*, comment) → visitNode (no-op) | | `method_declaration` | methodTypes:1027 → extractMethod:1737 | classifyMethodNode absent → always extractMethod. Gate 1747 passes via class-like (a method_declaration outside a type does not occur in non-erroring C# — top-level `void M(){}` parses as local_function_statement, probed); bodyless interface/partial signatures mint nodes with no body walk; **expression-bodied methods have `body: arrow_expression_clause` (a real body FIELD, probed) → walked** | | `constructor_declaration` | methodTypes → extractMethod | name field = the class-name identifier → **method node named like the class**; returnType undefined; **`constructor_initializer` (`: base(args)` / `: this(args)`) is a sibling of the body field → NEVER walked → calls inside initializer args are LOST** (probed); expression-bodied ctor body = arrow_expression_clause → walked | -| `property_declaration` (inside class-like) | propertyTypes:1075 → extractProperty:1986 | property node, then **propertyBodies — each accessor's `body` (`get { … }`, `set => …`) and an expression-bodied `=> expr` `value:` — walked by visitFunctionBody with the property pushed** (calls, instantiates, static reads and fn-ref candidates attribute to the property; changed 2026-10-04, previously never walked), then scanFnRefSubtree (capture-only, attributed to the class) over the rest — **the `= initializer` stays unwalked** — skipping the walked bodies, + skipChildren. §property below | -| `field_declaration` (inside class-like) | fieldTypes:1084 → extractField:2046 | field/constant nodes per declarator + scanFnRefSubtree + skipChildren → **field initializers emit no calls/instantiates/static-member refs** (fn-ref candidates only). §field below | +| `property_declaration` (inside class-like) | propertyTypes:1075 → extractProperty:1986 | property node, then **propertyBodies — each accessor's `body` (`get { … }`, `set => …`), then the `value:` (an expression body's `=> expr` or an `= initializer`) — walked by visitFunctionBody with the property pushed** (calls, instantiates, static reads and fn-ref candidates attribute to the property; accessor/arrow bodies changed 2026-10-04, `= initializer` 2026-10-05 — previously never walked); a target-typed `= new()` value instantiates the declared type (extractTargetTypedNew); then scanFnRefSubtree (capture-only, attributed to the class) over the rest (attributes, type) skipping the walked bodies, + skipChildren. §property below | +| `field_declaration` (inside class-like) | fieldTypes:1084 → extractField:2046 | field/constant nodes per declarator, **each declarator walked whole by visitFunctionBody with its field pushed** (its `= initializer`: calls, instantiates, static reads, fn-ref candidates incl. varinit attribute to the field; changed 2026-10-05, previously never walked); a target-typed `= new()` instantiates the declared type; then scanFnRefSubtree (capture-only, attributed to the class) skipping the walked declarators, + skipChildren. §field below | | `local_declaration_statement` | variableTypes:1098 (only reachable at top level — global statements; body locals go through visitFunctionBody instead) | not class-like → extractVariable:2538 → **generic fallback (2863-2881) finds no direct `identifier`/`variable_declarator` children (the declarator nests inside `variable_declaration`, probed) → ZERO nodes minted**; isClassScopeConstantAssignment (1508) needs node.type `assignment` → never true. skipChildren=true + scanFnRefSubtree → **a top-level `var builder = WebApplication.CreateBuilder(args);` produces NO node, NO calls ref, NO instantiates** — only fn-ref candidates. PRESERVE | | `using_directive` | importTypes:1209 → extractImport:3170 | hook (§config) → import node + ONE generic `imports` ref {fromNodeId: nodeStack top (namespace node if present, else file), referenceName: moduleName, line/col of the directive}; **no per-binding emitter** (the TS/py/rust/php/ruby ladder at 3197-3234 excludes csharp) | | `invocation_expression` (top level — global statements) | callTypes:1248 → extractCall:3684 | fires with caller = file/namespace node; children still visited (no skipChildren) so nested invocations recurse | @@ -306,9 +306,10 @@ for C#) or bare `name`. QUIRKS (probed, PRESERVE): children [modifier, predefined_type, identifier, **arrow_expression_clause (`value:` field)**] → typeNode = predefined_type → signature `"int Computed"`; the arrow clause is a property body (walked as the property, below). -- `{ get; } = new();` initializers: the `value:` implicit_object_creation is - NOT a body → no refs, no instantiates (candidates-only scan, attributed to - the class); the accessor_list is excluded from the type scan. +- `{ get; } = new();` initializers: the `value:` is the + implicit_object_creation_expression itself (no type child) → walked as a + body, and extractTargetTypedNew emits `instantiates` named for the + declared `type` (below); the accessor_list is excluded from the type scan. Then extractDecoratorsFor (no-op) and **extractTypeAnnotations (2037) → extractCsharpTypeRefs** — the `type` field IS walked for refs (so `public @@ -316,10 +317,22 @@ List Items` emits references `List` + `Foo` even though the signature kept the raw text). The returned node is pushed while propertyBodies (tree-sitter.ts; csharp.rs `property_bodies`) are walked: every `accessor_declaration`'s `body` field (block or arrow_expression_clause, in -accessor order), then the property's `value:` when it is an -arrow_expression_clause. The fn-ref scan that follows skips those subtrees -by node id, so a candidate is captured once — from the property when it sits -in a body, from the class when it sits in an initializer. (The +accessor order), then the property's `value:` whatever it is (an +arrow_expression_clause or an `= initializer` expression). Each body is +preceded by **extractTargetTypedNew(body, `type` field)** (csharp.rs +`extract_target_typed_new`): when the body IS an +`implicit_object_creation_expression` it emits ONE `instantiates` ref from +the property at the `new()`'s position, named by csharpClassTypeName +(csharp.rs `class_type_name`): `identifier` → its text; `generic_name` → its +identifier (`List` → `List`); `qualified_name` / `alias_qualified_name` +→ recurse into the `name` field (`Ns.Foo` → `Foo`, `global::Foo` → +`Foo`, `Outer.Inner` → `Inner`); `nullable_type` → recurse into `type` +(`Foo?` → `Foo`); anything else (predefined/array/tuple/pointer) → nothing. +Only a `new()` that IS the value counts — `(new())`, `c ? new() : null` and +nested `new()`s emit nothing. The fn-ref scan that follows skips the walked +bodies by node id, so a candidate is captured once — from the property; only +what is left (attribute arguments, e.g. `[Attr(Make(H))]` → `H` from the +class) is the class's. Attributes are never walked for calls. (The classifyMethodNode initializer-walk path at 1031-1047 is TS-only.) ### extractField (2046) — field_declaration @@ -342,6 +355,23 @@ classifyMethodNode initializer-walk path at 1031-1047 is TS-only.) the variable_declaration's `type` field (5905-5909) → **multi-declarator fields (`Foo A, B;`) emit the type refs ONCE PER DECLARATOR**, each from its own field node. +- Then, per created field node (changed 2026-10-05 — initializers used to + emit nothing but fn-ref candidates, attributed to the class): push the + field, **extractTargetTypedNew(the declarator's LAST named child, the + variable_declaration's `type` field)** (§property above — a target-typed + `private readonly List _items = new();` instantiates `List`), then + **visitFunctionBody(the whole declarator)**, pop. The C# declarator has no + `value` field — the initializer is its last, unnamed child (probed: + [name, expression]; a fixed buffer is [name, bracketed_argument_list]) — + so the declarator is walked whole, like VB.NET's: maybeCaptureFnRefs fires + on it first (varinit: `Del d = Handler;` → candidate `Handler` FROM THE + FIELD), then the name (inert) and the initializer — calls, instantiates, + static reads (`Defaults.Name` → `Defaults`) and lambda bodies (no halt: + `Func f = x => Compute(x)` → the field calls `Compute`) attribute + to the field. extractField returns the walked declarator ids and the + dispatcher's scanFnRefSubtree skips them, so each candidate is captured + once, by the field. (Java's walk of its `value` field reports nothing back + — its class-level scan still captures those candidates too, unchanged.) - docstring/visibility/isStatic computed once from the outer declaration, shared by all declarators. The PHP property_element and bare-fallback branches (2078-2154) are unreachable for C#. @@ -425,16 +455,20 @@ statements at top level) AND visitFunctionBody:5145. QUIRKS, PRESERVE: **`implicit_object_creation_expression` (`new()`) and `anonymous_object_creation_expression` (`new { X = 1 }`) and `array_creation_expression` (`new Widget[10]`) are NOT in INSTANTIATION_KINDS -→ no instantiates refs** (target-typed `new()` — everywhere in modern C# — is -invisible); object/collection initializer args and `new[] { Mk() }` contents -still recurse to their own calls. Top-level `var w = new Widget();` emits -nothing at all (§local_declaration_statement). +→ no instantiates refs** (a target-typed `new()` in a body — `Widget c = +new();`, an argument, a return — is invisible); the one exception is a +`new()` that IS a field's or property's initializer, which +extractTargetTypedNew names for the declared type (§property, §field). +Object/collection initializer args and `new[] { Mk() }` contents still +recurse to their own calls. Top-level `var w = new Widget();` emits nothing +at all (§local_declaration_statement). ### extractStaticMemberRef (4750) — csharp ∈ STATIC_MEMBER_LANGS (345) Called for EVERY node in visitFunctionBody (5218) — body walker only (never -visitNode, so class-level field initializers and top-level statements emit no -static refs). Node gate: MEMBER_ACCESS_TYPES (323) contains +visitNode, so top-level statements emit no static refs; field and property +initializers ARE body-walked, so `= Defaults.Name` reads attribute to the +member). Node gate: MEMBER_ACCESS_TYPES (323) contains `member_access_expression`. Skip when the access IS a call's callee (4772-4779: parent ∈ callTypes && callee.startIndex === node.startIndex — so `Console.WriteLine(…)`'s access is skipped but `DoThing(Constants.MAX)`'s @@ -621,14 +655,17 @@ layers: `argument`→null (descend named children). special: nothing (not idTypes/special). - Capture fires from visitNode:990, visitFunctionBody:5137, and scanFnRefSubtree (property/field/variable declarations + top-level - statements, depth ≤12, capture-only). **scanFnRefSubtree's halt list - (tree-sitter.ts:606-612) includes the literal type `lambda_expression` — - which IS C#'s lambda node** — so at depth>0 the scan STOPS at a lambda in a - field/property initializer (`Action A = () => Register(H);` yields no - candidates from inside the lambda), while `anonymous_method_expression` - (`delegate() { … }`) is NOT in the list and is scanned through. Method-BODY - lambdas are unaffected (visitFunctionBody recursion has no halt — capture - fires per node). + statements, depth ≤12, capture-only — skipping the subtrees the body + walker already went through: property bodies/initializers and field + declarators, whose candidates are the member's). **scanFnRefSubtree's halt + list (tree-sitter.ts:606-612) includes the literal type `lambda_expression` + — which IS C#'s lambda node** — so at depth>0 the scan STOPS at a lambda it + reaches (a top-level statement's), while `anonymous_method_expression` + (`delegate() { … }`) is NOT in the list and is scanned through. Body-walked + lambdas — methods, accessors, and since 2026-10-05 field/property + initializers (`Action A = () => Register(H);` → candidate `H` from `A`) — + are unaffected (visitFunctionBody recursion has no halt — capture fires per + node). - Flush gate (flushFnRefCandidates:639): generated-file skip; `this.`-prefixed names skip the gate (C# never produces them — its this-forms are bare); otherwise name ∈ definedHere ∪ importedNames. **definedHere = same-file @@ -762,10 +799,14 @@ AspNetCore refs / Program.cs / Startup.cs / controller-source scan. `struct Fwd;` (NO node); interface with bodyless method + property + default-impl arrow method; enum with `: byte` (extends `byte` quirk), attributed member, valued members; const + static-readonly (→ `constant`) - + multi-declarator + instance fields (signatures `Type name`); + + multi-declarator + instance fields (signatures `Type name`), field + initializers (→ the field: `= new() { TargetCb }` instantiates the declared + `List` and the list candidate is the field's); `protected internal` (→ protected); property shapes: predefined-type, bare-identifier type (signature loses type), generic type, expression-bodied - (`=>` calls → the property), `{ get; } = new();` (initializer LOST), + (`=>` calls → the property), `{ get; } = new();` (→ the property, + instantiates the declared `Widget`), `{ get; } = () => Register(H)` (the + lambda's call and candidate → the property), accessor bodies with calls (→ the property); event_field_declaration + event_declaration with add/remove bodies (no nodes; accessor calls → class); operator + conversion operator + indexer + @@ -779,7 +820,7 @@ AspNetCore refs / Program.cs / Startup.cs / controller-source scan. `Foo.Create(1).Bar()` (re-encode + inner both) + `GetThing().Bar()`, `(myDel)(x)` (conv regex → `myDel`), `nameof(Widget)` (calls ref `nameof`); `new Widget(…) { … }` (instantiates + initializer calls) + `new Ns.Foo()` - (strip both) + `new()` / `new { }` / `new Widget[10]` (all NOTHING); + (strip both) + body `new()` / `new { }` / `new Widget[10]` (all NOTHING); static value reads (`ReadType.ReadAsDouble` → `ReadType`; `Outer.Inner.DEEP` → `Outer`; skip-as-callee; lowercase skip); type refs: params (nullable/array/tuple-element/generic/qualified/`dynamic`), returns @@ -792,7 +833,8 @@ AspNetCore refs / Program.cs / Startup.cs / controller-source scan. function + trailing `partial class Program`; fn-refs: `Register(HandleThing)`, `Register(this.HandleThing)` (bare name), `Register(C.StaticHandler)` (nothing), `Click += OnClick`, initializer_expression list, varinit - (`Action g = () => …` no candidate; `Del d = Handler;` candidate), + (`Action g = () => …` no candidate; `Del d = Handler;` candidate — from + the method for a local, from the field for a field initializer), `this.x = x` param-storage skip; value-refs: const target + reader methods + a `var MaxItems = …` local shadow (prune) + `static readonly` multi-target; preprocessor: `#region`/`#endregion`/`#pragma`/`#nullable`/`#define` diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index 9a10862475..222291f14c 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -239,6 +239,29 @@ function scalaBaseTypeName(node: SyntaxNode | null, source: string): string | nu } } +/** + * The class a C# declared type names, as `new T()` would name it: `Foo`, + * `List` → `List`, `Ns.Foo` / `global::Foo` → `Foo`, `Foo?` → `Foo`. + * Predefined, array, tuple and pointer types name no class → null. Mirrored + * in the kernel (csharp.rs class_type_name). + */ +function csharpClassTypeName(node: SyntaxNode | null, source: string): string | null { + if (!node) return null; + switch (node.type) { + case 'identifier': + return getNodeText(node, source) || null; + case 'generic_name': + return csharpClassTypeName(node.namedChildren.find((c: SyntaxNode) => c.type === 'identifier') ?? null, source); + case 'qualified_name': + case 'alias_qualified_name': + return csharpClassTypeName(getChildByField(node, 'name'), source); + case 'nullable_type': + return csharpClassTypeName(getChildByField(node, 'type'), source); + default: + return null; + } +} + /** * Resolve the declared identifier inside a C declarator. A `declaration`'s * `declarator` field nests the name through `init_declarator` (with value), @@ -1287,22 +1310,27 @@ export class TreeSitterExtractor { const bodies = propNode ? this.propertyBodies(node) : []; if (propNode && bodies.length > 0) { this.nodeStack.push(propNode.id); - for (const body of bodies) this.visitFunctionBody(body, propNode.id); + const declaredType = getChildByField(node, 'type'); + for (const body of bodies) { + this.extractTargetTypedNew(body, declaredType); + this.visitFunctionBody(body, propNode.id); + } this.nodeStack.pop(); } - // Whatever the body walk didn't cover (a C# `= initializer`, any other - // language's whole declaration) is scanned for function-as-value + // Whatever the body walk didn't cover (a C# property's attributes, any + // other language's whole declaration) is scanned for function-as-value // candidates (#756); the bodies captured their own. this.scanFnRefSubtree(node, 0, new Set(bodies.map((b) => b.id))); skipChildren = true; } // Check for class fields (e.g. Java field_declaration, C# field_declaration) else if (this.extractor.fieldTypes?.includes(nodeType) && this.isInsideClassLikeNode()) { - this.extractField(node); - // Field initializers aren't walked — scan for function-as-value - // candidates (#756): Java `List table = List.of(Main::cb)`, - // C# `List> table = new() { TargetCb }`. - this.scanFnRefSubtree(node, 0); + const walked = this.extractField(node); + // Scan the declaration for function-as-value candidates (#756): Java + // `List table = List.of(Main::cb)`. A C# declarator + // extractField walked captured its own (`List> table = + // new() { TargetCb }` is the field's), so the scan skips it. + this.scanFnRefSubtree(node, 0, walked); skipChildren = true; } // Check for variable declarations (const, let, var, etc.) @@ -2324,9 +2352,9 @@ export class TreeSitterExtractor { * VB.NET writes its `Get` / `Set` blocks, `= initializer` and `As New T` * as children of the declaration itself, so the declaration is walked * whole, the way its methods are (resolveBody). C# runs code in each - * accessor's body (`get { … }`, `set => …`) and in an expression-bodied - * property's `=> …`; its `= initializer`, like a field's, stays unwalked. - * Mirrored in the kernel (csharp.rs property_bodies). + * accessor's body (`get { … }`, `set => …`) and in the property's `value`: + * an expression body's `=> …` or an `= initializer`. Its attributes are + * not walked. Mirrored in the kernel (csharp.rs property_bodies). */ private propertyBodies(node: SyntaxNode): SyntaxNode[] { if (this.language === 'vbnet') return [node]; @@ -2337,16 +2365,41 @@ export class TreeSitterExtractor { if (body) bodies.push(body); } const value = getChildByField(node, 'value'); - if (value?.type === 'arrow_expression_clause') bodies.push(value); + if (value) bodies.push(value); return bodies; } + /** + * A C# target-typed `new()` names no type, which is why INSTANTIATION_KINDS + * leaves `implicit_object_creation_expression` out. As a field's or + * property's initializer, though, it constructs the declared type: + * `private readonly List _items = new();` instantiates List, as + * `new List()` does. Emitted from the node-stack top (the member). + * Mirrored in the kernel (csharp.rs extract_target_typed_new). + */ + private extractTargetTypedNew(value: SyntaxNode | null, declaredType: SyntaxNode | null): void { + if (this.language !== 'csharp' || value?.type !== 'implicit_object_creation_expression') return; + const className = declaredType ? csharpClassTypeName(declaredType, this.source) : null; + const fromNodeId = this.nodeStack[this.nodeStack.length - 1]; + if (!className || !fromNodeId) return; + this.unresolvedReferences.push({ + fromNodeId, + referenceName: className, + referenceKind: 'instantiates', + line: value.startPosition.row + 1, + column: value.startPosition.column, + }); + } + /** * Extract a class field declaration (e.g. Java field_declaration, C# field_declaration). * Extracts each declarator as a 'field' kind node inside the owning class. + * Returns the C# declarators it walked, for the function-as-value scan to + * skip. */ - private extractField(node: SyntaxNode): void { - if (!this.extractor) return; + private extractField(node: SyntaxNode): Set { + const walked = new Set(); + if (!this.extractor) return walked; const docstring = this.docstringFor(node); const visibility = this.extractor.getVisibility?.(node); @@ -2401,7 +2454,7 @@ export class TreeSitterExtractor { isStatic, }); } - return; + return walked; } } @@ -2447,11 +2500,24 @@ export class TreeSitterExtractor { // edge at all and `target` looked callerless. Keyed on the `value` // FIELD, which only Java's `variable_declarator` carries. VB.NET // writes `= expr` (the declarator's `initializer`) or `As New T(…)` - // (inside its as_clause), so its whole declarator is walked. C# - // and PHP spell their initializer differently and are untouched. - const valueNode = this.language === 'vbnet' ? decl : getChildByField(decl, 'value'); + // (inside its as_clause), and C# writes `= expr` as the + // declarator's last, unnamed child, so both walk the whole + // declarator. PHP spells its initializer differently and is + // untouched. + const wholeDeclarator = this.language === 'vbnet' || this.language === 'csharp'; + const valueNode = wholeDeclarator ? decl : getChildByField(decl, 'value'); if (valueNode) { this.nodeStack.push(fieldNode.id); + if (this.language === 'csharp') { + this.extractTargetTypedNew( + decl.namedChild(decl.namedChildCount - 1), + varDecl ? getChildByField(varDecl, 'type') : null, + ); + // Its function-as-value candidates are the field's, captured + // here once. (VB.NET captures none; Java's scan still takes + // its initializers for the class as well.) + walked.add(decl.id); + } this.visitFunctionBody(valueNode, fieldNode.id); this.nodeStack.pop(); } @@ -2470,6 +2536,7 @@ export class TreeSitterExtractor { }); } } + return walked; } /** diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 2984a43464..f721a2cc43 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -2438,10 +2438,14 @@ function isDotNetTypeRef(ref: UnresolvedRef, context: ResolutionContext): boolea * method (a constructor is one) or an enum case shares the type's name, not * its meaning: AutoMapper's `Type sourceType` bound to an attribute's `Type` * property and `TypeMap typeMap` to a `TypeMap` property beside the `TypeMap` - * class. A field never names a type either. + * class. A field never names a type either, nor does a constant — the kind a + * C# `const` / `static readonly` field gets: jellyfin's `new Version(5, 18)` + * bound to a claim-name `const string Version`, and serilog's `static + * readonly Meter Meter = new(…)` to itself. */ function canNameInTypePosition(n: Node): boolean { - return !(n.kind === 'property' || n.kind === 'method' || n.kind === 'enum_member' || n.kind === 'field'); + return !(n.kind === 'property' || n.kind === 'method' || n.kind === 'enum_member' || n.kind === 'field' || + n.kind === 'constant'); } /** From c9944147bbd579869a6f3eee3cbc3533e3167f44 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 17:32:53 +0000 Subject: [PATCH 175/259] fix(vbnet): resolve receiver calls by declared type; guard .NET names, bare calls and ties (#2351) VB.NET's resolver guessed a receiver call's target from the receiver's name whenever no class matched it, without any of the guards C# has. Indexing Structure members (#2345) gave those guesses new targets: 23 SCrawler `xxxFile.Delete()` calls on PersonalUtilities' SFile went to a nested struct's `TempFileConversion::Delete`, and staxrip's main app sent 90 calls to its AutoCrop tool's duplicate `ColorHSL`. - Declared types (new src/resolution/vbnet-receivers.ts): a receiver's `As` type decides the call (local, parameter, field, property, `For Each`, `As New`, ...). The candidates are that type's member, one it inherits, or an extension method written for it. A type from outside the project gets no link. - .NET standard method names (`Add`, `Contains`, `Clear`, `Dispose`, `ToString`, ...) on an untyped receiver need a receiver named after the owner, as in C#. VB.NET matches them case-insensitively. - A bare call, or one on `Me`, reaches the enclosing class, what it inherits, or a Module. A nested type is matched by its bare name only from inside its owner. Types resolve through enclosing namespaces, `Imports` (aliases included) and the caller's project. A tie between equally good guesses goes to the caller's file, then its project, then the nearer directory, or gets no link. - vbReceiverOf reads the name at or after the reference's column, so `Me.Size = New System.Drawing.Size(...)` is read through `System.Drawing`, not `Me`. Before/after on main (nodes unchanged): SCrawler 15,092 -> 14,893 edges, staxrip 28,002 -> 26,647. Of the non-containment edges removed, 855 / 2,802 are the same link with new resolver metadata. The rest: - .NET names on untyped receivers no longer guessed: 196 / 1,378. 731 of staxrip's are StringBuilder `sb.Append` calls that had gone to `LogBuilder::Append`. - Other receiver guesses dropped: 86 / 103, including the SFile.Delete calls. - Bare calls out of scope: 38 / 834. 590 of staxrip's are `New Point(...)` that had gone to a class nested in `ButtonEx`; others are WinForms `Refresh()`, `Focus()` and `Activate()`. - Retargeted: 227 / 532, e.g. 208 bare `Add(...)` calls in encoder classes now reach the inherited `CommandLineParams::Add`. 121 / 958 call sites resolve that did not before, e.g. `cms.Add` -> ContextMenuStripEx and `td.AddButton` -> TaskDialog. Designer -> Size links fall from 5 / 1 to 0; designer -> Add stays 0. Known trade-off, shared with C#'s guard: about 31 staxrip calls like `switches.Join(BR)` that may be the project's own `MiscExtensions` extension methods are no longer linked. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 5 + __tests__/vbnet-bare-member-scope.test.ts | 100 ++ __tests__/vbnet-member-access.test.ts | 35 + __tests__/vbnet-project-ties.test.ts | 143 +++ __tests__/vbnet-receiver-types.test.ts | 465 +++++++ __tests__/vbnet-std-methods.test.ts | 75 ++ src/resolution/index.ts | 2 + src/resolution/name-matcher.ts | 81 +- src/resolution/vbnet-receivers.ts | 1335 +++++++++++++++++++++ 9 files changed, 2235 insertions(+), 6 deletions(-) create mode 100644 __tests__/vbnet-bare-member-scope.test.ts create mode 100644 __tests__/vbnet-project-ties.test.ts create mode 100644 __tests__/vbnet-receiver-types.test.ts create mode 100644 __tests__/vbnet-std-methods.test.ts create mode 100644 src/resolution/vbnet-receivers.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index da460b5b77..b5476b128f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,11 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In VB.NET, a field or property initializer like `= Compute()` or `As New List(Of Order)` now links what it calls and creates, and so do a `Custom Event`'s `AddHandler`, `RemoveHandler` and `RaiseEvent` blocks. Re-index VB.NET and C# projects after upgrading. - In C#, a field or property initializer like `private readonly ILogger _log = LogManager.GetLogger(typeof(X));` or `public List Items { get; } = new();` now links what it calls, creates and reads, including inside a lambda, and a target-typed `new()` there counts as creating the declared type. Before, initializers were skipped, so a method or class used only from one — like a converter created in a static list — looked unused, and a method passed as a value there was credited to the whole class instead of the field or property. - In C# and VB.NET, a constant no longer stands in for a type with the same name: `new Station { … }` links to class `Station` even when another class declares `private const string Station`, `new Version(…)` no longer links to a `const string Version`, and a field like `static readonly Meter Meter` no longer points at itself. Thanks @EvanYu1980 for the report and @drakeo338. (#2337) +- In VB.NET, a call on a variable, parameter, field or property now reaches a method of the type it is declared with — one that type inherits, or an extension method written for it — and nothing when that type comes from outside your project, instead of any project method that merely shares the name. +- In VB.NET, a call to one of .NET's own methods such as `Add`, `Contains`, `Clear` or `Dispose` on a value whose type isn't known is no longer linked to a project method of that name, unless the value is named after that method's class. +- In VB.NET, a type name is looked up the way VB.NET does it — through the namespaces around it, the file's and project's `Imports` (aliases included), and the caller's own project — so a class declared in several namespaces or projects no longer draws every call to whichever copy was indexed first, and two candidates nothing tells apart get no link at all. +- In VB.NET, a call with no receiver, or on `Me`, now reaches only a member of the class it is written in, of a class it inherits, or of a Module, and no longer a nearby class's member of the same name. +- VB.NET designer code such as `New System.Drawing.Point(…)` or `Me.Size = New System.Drawing.Size(…)` is no longer linked to a project type or member that only shares the name, and a class nested inside another is matched by its bare name only from inside that class or one that inherits it. - Upgrading CodeGraph while an agent session is open no longer leaves the old version's background server in charge of your project: the first session started from the new install stops it and starts a current one in its place, even while sessions opened before the upgrade are still running. That old server could no longer load the language parsers the upgrade removed, so it saved every file it re-indexed with no symbols, while new sessions could only read the index beside it without keeping it up to date. A background server from a newer install is never stopped, and sessions opened before the upgrade keep the old version until you restart them. Thanks @lipchey for the report. (#2335) - A file is no longer saved with no symbols when its language parser can't be loaded, which is what happened to every file a background server re-indexed after an upgrade removed its install: the file keeps what it had and is indexed again once the parser loads, and files an earlier version emptied this way are re-indexed by the next sync. A background server also exits on its own once its install is upgraded or removed, so the next session starts one from the current install. Thanks @lipchey for the report. (#2335) - `codegraph status` no longer says the index is up to date while indexed files are missing their symbols: it now names files the parser couldn't read and files stored without their symbols (which `codegraph sync` repairs), `status --json` counts both, and `codegraph files --json` lists each file's recorded errors. Thanks @lipchey for the report. (#2336) diff --git a/__tests__/vbnet-bare-member-scope.test.ts b/__tests__/vbnet-bare-member-scope.test.ts new file mode 100644 index 0000000000..a2601671b1 --- /dev/null +++ b/__tests__/vbnet-bare-member-scope.test.ts @@ -0,0 +1,100 @@ +/** + * A VB.NET call with no receiver — or with `Me` / `MyClass` / `MyBase`, + * which the extractor drops — is a member of the type it is written in, of a + * type around it, or of one they inherit; a `Module`'s members are reached + * from anywhere. Never another class's same-named member, however near its + * file: SCrawler's `{ToString()}` in the Instagram `UserData` (which + * `Inherits UserDataBase`) went to a nearby structure's `ToString`. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-vb-bare-')); + const files: Record = { + 'API/Base/UserDataBase.vb': `Namespace API.Base + Friend MustInherit Class UserDataBase + Public Overrides Function ToString() As String + Return "user" + End Function + Protected Sub Refresh() + End Sub + End Class +End Namespace +`, + 'API/Instagram/MediaItem.vb': `Namespace API.Instagram + Friend Class MediaItem + Public Overrides Function ToString() As String + Return "media" + End Function + Friend Sub Refresh() + End Sub + End Class +End Namespace +`, + 'API/Instagram/UserData.vb': `Namespace API.Instagram + Friend Class UserData : Inherits API.Base.UserDataBase + Friend Sub SetTagsLimit() + Dim aStr$ = $"Enter the number of posts from user {ToString()}" + Me.Refresh() + Log(aStr) + End Sub + Private Class Counter + Friend Sub Tick() + Report() + End Sub + End Class + Private Shared Sub Report() + End Sub + End Class +End Namespace +`, + 'Tools/Logger.vb': `Public Module Logger + Public Sub Log(ByVal Text As String) + End Sub +End Module +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +/** `Owner::member` of every call edge out of one method. */ +function callsOf(file: string, qualifiedName: string): string[] { + const from = cg.getNodesInFile(file).find((n) => n.qualifiedName === qualifiedName)!; + return cg + .getOutgoingEdgesFrom([from.id]) + .filter((e) => e.kind === 'calls') + .map((e) => cg.getNode(e.target)!.qualifiedName) + .sort(); +} + +describe('VB.NET receiver-less calls', () => { + it('reach the members the class inherits, not a nearer class’s, and a module’s', () => { + expect(callsOf('API/Instagram/UserData.vb', 'API.Instagram::UserData::SetTagsLimit')).toEqual([ + 'API.Base::UserDataBase::Refresh', + 'API.Base::UserDataBase::ToString', + 'Logger::Log', + ]); + }); + + it('reach a member of the class around a nested one', () => { + expect(callsOf('API/Instagram/UserData.vb', 'API.Instagram::UserData::Counter::Tick')).toEqual([ + 'API.Instagram::UserData::Report', + ]); + }); +}); diff --git a/__tests__/vbnet-member-access.test.ts b/__tests__/vbnet-member-access.test.ts index d8643ecc52..668cc18aac 100644 --- a/__tests__/vbnet-member-access.test.ts +++ b/__tests__/vbnet-member-access.test.ts @@ -7,6 +7,12 @@ * (`GetService(Of Notifier).Notify()`). SCrawler's designer code sent * `New System.Drawing.Size(…)` to a nested enum's `Size` case 713 times and * `Controls.Add(…)` to a collection class's `Add` 547 times. + * + * The name read is the one the reference starts at (or after) — in + * `Me.Size = New System.Drawing.Size(…)` that is `System.Drawing.Size`, not + * `Me.Size` — and a type is one its written qualifier names: staxrip's + * designer `New System.Drawing.Point(…)` went to a nested `Point` class 550 + * times. A type nested in a class is named bare only inside it. */ import { describe, it, expect, afterAll, beforeAll } from 'vitest'; import * as fs from 'fs'; @@ -56,6 +62,24 @@ End Module Private Sub Refresh() End Sub End Class +`, + 'UI/ButtonEx.vb': `Public Class ButtonEx + Public Class SymbolDrawer + Public Class Point + End Class + Public Sub Draw() + Dim p As New Point() + End Sub + End Class +End Class +`, + 'Forms/AppsForm.vb': `Public Class AppsForm + Private Sub InitializeComponent() + Me.Size = New System.Drawing.Size(315, 205) + Me.ToolStrip.Location = New System.Drawing.Point(473, 10) + Me.lDescription.Location = New Point(4, 285) + End Sub +End Class `, }; for (const [rel, content] of Object.entries(files)) { @@ -84,4 +108,15 @@ describe('VB.NET member access', () => { expect(targets).toContain('Notifier::Notify'); expect(targets).toContain('MainForm::Refresh'); }); + + it('reads the name the reference starts at, and a type its qualifier names', () => { + const targetsOf = (file: string) => cg + .getOutgoingEdgesFrom(cg.getNodesInFile(file).map((n) => n.id)) + .filter((e) => e.kind === 'calls' || e.kind === 'instantiates') + .map((e) => `${e.kind} ${cg.getNode(e.target)!.qualifiedName}`) + .sort(); + // `Me.Size = New System.Drawing.Size(…)`, `New System.Drawing.Point(…)`, `New Point(…)` outside ButtonEx. + expect(targetsOf('Forms/AppsForm.vb')).toEqual([]); + expect(targetsOf('UI/ButtonEx.vb')).toEqual(['instantiates ButtonEx::SymbolDrawer::Point']); + }); }); diff --git a/__tests__/vbnet-project-ties.test.ts b/__tests__/vbnet-project-ties.test.ts new file mode 100644 index 0000000000..9d8e2fcc0e --- /dev/null +++ b/__tests__/vbnet-project-ties.test.ts @@ -0,0 +1,143 @@ +/** + * A VB.NET solution often carries the same type in two projects: staxrip's + * main app and its separate AutoCrop tool (its own AutoCrop.vbproj) each + * declare `ColorHSL`, `FrameServerFactory` and `DirectFrameServer`. A call + * in the main app means its own project's copy — about 90 calls such as + * `_backColor.AddLuminance(0.025)` went to the AutoCrop copy, whichever was + * indexed first. Between equally good guesses, the caller's project decides, + * then the nearer directory; when neither does, there is no guess. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +const VBPROJ = ` + + WinExe + + +`; + +const COLOR_HSL = `Public Class ColorHSL + Public Sub New(ByVal h As Double, ByVal s As Double, ByVal l As Double, ByVal a As Double) + End Sub + Public Function AddLuminance(ByVal offset As Single) As ColorHSL + Return Me + End Function +End Class +`; + +const FRAME_SERVER_FACTORY = `Public Class FrameServerFactory + Public Shared Function Create(ByVal path As String) As Object + Return Nothing + End Function +End Class +`; + +const HELPER = `Public Class Helper + Public Sub Run() + End Sub +End Class +`; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-vb-ties-')); + const files: Record = { + 'Source/StaxRip.vbproj': VBPROJ, + 'Source/Tools/AutoCrop/AutoCrop.vbproj': VBPROJ, + 'Source/Tools/AutoCrop/Main.vb': COLOR_HSL + FRAME_SERVER_FACTORY, + 'Source/UI/ColorHSL.vb': COLOR_HSL, + 'Source/Video/FrameServer.vb': FRAME_SERVER_FACTORY, + 'Source/General/ThemeManager.vb': `Public Class ThemeManager + Public Sub Apply(ByVal palette As Object) + Dim _backColor As ColorHSL = New ColorHSL(0, 0.01, 0.1, 1) + Dim _controlBackColor As ColorHSL = _backColor.AddLuminance(0.025) + For Each backgroundColor In palette + backgroundColor.AddLuminance(-0.1) + Next + Dim server = FrameServerFactory.Create("video.mkv") + End Sub +End Class +`, + 'Source/UI/TipProvider.vb': `Public Class TipProvider + Public Sub SetTip(ByVal text As String) + End Sub + Public Sub SetTip(ByVal text As String, ByVal title As String) + End Sub +End Class +`, + 'Source/General/Tips.vb': `Public Class Tips + Public Sub Apply(ByVal providers As Object) + For Each tipProvider In providers + tipProvider.SetTip("x") + Next + End Sub +End Class +`, + 'Plugins/A/A.vbproj': VBPROJ, + 'Plugins/A/Helper.vb': HELPER, + 'Plugins/B/B.vbproj': VBPROJ, + 'Plugins/B/Helper.vb': HELPER, + 'App/App.vbproj': VBPROJ, + 'App/Main.vb': `Public Class Main + Public Sub Start(ByVal helpers As Object) + For Each runHelper In helpers + runHelper.Run() + Next + End Sub +End Class +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +/** `Owner::member @ file` of every call / instantiation edge out of a file. */ +function targetsFrom(file: string, kind: 'calls' | 'instantiates'): string[] { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg + .getOutgoingEdgesFrom(ids) + .filter((e) => e.kind === kind) + .map((e) => { + const target = cg.getNode(e.target)!; + return `${target.qualifiedName} @ ${target.filePath}`; + }) + .sort(); +} + +describe('VB.NET duplicate types across projects', () => { + it('resolve a call to the caller’s own project’s copy', () => { + expect(targetsFrom('Source/General/ThemeManager.vb', 'calls')).toEqual([ + 'ColorHSL::AddLuminance @ Source/UI/ColorHSL.vb', + 'ColorHSL::AddLuminance @ Source/UI/ColorHSL.vb', + 'FrameServerFactory::Create @ Source/Video/FrameServer.vb', + ]); + }); + + it('resolve a construction to the caller’s own project’s copy', () => { + expect(targetsFrom('Source/General/ThemeManager.vb', 'instantiates')).toEqual([ + 'ColorHSL @ Source/UI/ColorHSL.vb', + ]); + }); + + it('make no guess between copies nothing tells apart', () => { + expect(targetsFrom('App/Main.vb', 'calls')).toEqual([]); + }); + + it('take one type’s overloads as one guess, not a tie', () => { + expect(targetsFrom('Source/General/Tips.vb', 'calls')).toEqual(['TipProvider::SetTip @ Source/UI/TipProvider.vb']); + }); +}); diff --git a/__tests__/vbnet-receiver-types.test.ts b/__tests__/vbnet-receiver-types.test.ts new file mode 100644 index 0000000000..0a81309747 --- /dev/null +++ b/__tests__/vbnet-receiver-types.test.ts @@ -0,0 +1,465 @@ +/** + * A VB.NET call through a local, a parameter, a field or a property is a call + * on the type it is declared with — `Dim x As T`, `ByVal x As T`, + * `x As New T`, `Private File As SFile`, `Property File As SFile`, a type + * character (`Dim name$`), or what `Dim x = obj.GetString(…)` returns — + * never a guess at a same-named method of some project class. Keywords are + * matched without regard to case. A declared type the project does not define + * means nothing of the project's, except an extension method declared for it. + * + * - SCrawler's `ThumbnailFile.Delete(SFO.File, …)` on an `SFile` (from the + * external PersonalUtilities library) went to a nested `TempFileConversion` + * class's `Delete`, `GroupFile.Delete()` to a download group's, + * `UserUpdatedEventHandlers.Add(e)` on a `List(Of …)` to `UserDataBind.Add` + * and `TotalSize.CompareTo(…)` on a `Double` to a plugin's `VSize.CompareTo`; + * - staxrip's `timestampFontColorValue.ToColor(…)`, a String from + * `settings.GetString(…)`, went to a `ColorHSL.ToColor()` instead of the + * String extension `StringExtensions.ToColor`. + * + * The type a name means is the one VB.NET's lookup finds where it is written: + * the namespaces around it (SCrawler declares a `SiteSettings` and an `M3U8` + * per site namespace), a base class's nested types, the file's `Imports` + * aliases, and the type arguments a subclass gives its base. The declared + * type wins over whatever is assigned later, a `For Each` variable is an + * element of what it loops over, and an interface a class implements lends + * it no members to read bare. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-vb-receiver-')); + const files: Record = { + 'Download/Groups/DownloadGroup.vb': `Namespace DownloadObjects.Groups + Friend Class DownloadGroup + Friend Sub Delete() + End Sub + End Class +End Namespace +`, + 'YouTube/Objects/YouTubeMediaContainerBase.vb': `Namespace API.YouTube.Objects + Public MustInherit Class YouTubeMediaContainerBase + Public ReadOnly Property ThumbnailFile As SFile + Private Class TempFileConversion + Friend Sub Delete(ByVal Mode As Integer) + End Sub + End Class + End Class +End Namespace +`, + 'Download/Groups/DownloadGroupCollection.vb': `Namespace DownloadObjects.Groups + Friend Class DownloadGroupCollection + Private ReadOnly GroupFile As SFile = "Settings\\Groups.xml" + Friend Sub Update() + GroupFile.Delete() + End Sub + Friend Sub Reset(byval groupFile as SFile) + groupFile.Delete() + End Sub + End Class +End Namespace +`, + 'Hosts/DownloadableMediaHost.vb': `Namespace Hosts + Friend Class DownloadableMediaHost : Inherits API.YouTube.Objects.YouTubeMediaContainerBase + Friend Sub Clean() + ThumbnailFile.Delete(1) + End Sub + End Class +End Namespace +`, + 'API/UserDataBind.vb': `Namespace API + Friend Class UserDataBind + Friend Sub Add(ByVal User As Object) + End Sub + End Class +End Namespace +`, + 'API/Base/UserDataBase.vb': `Namespace API.Base + Friend MustInherit Class UserDataBase + Private ReadOnly UserUpdatedEventHandlers As List(Of UserUpdatedEventHandler) + Friend Sub AddUpdateHandler(ByVal e As UserUpdatedEventHandler) + UserUpdatedEventHandlers.Add(e) + End Sub + Friend Sub Download() + End Sub + End Class +End Namespace +`, + 'API/Instagram/UserData.vb': `Namespace API.Instagram + Friend Class UserData : Inherits API.Base.UserDataBase + End Class +End Namespace +`, + 'API/Instagram/Downloader.vb': `Namespace API.Instagram + Friend Class Downloader + Friend Sub Run(ByVal u As UserData) + u.Download() + End Sub + End Class +End Namespace +`, + 'Plugin/UserData.vb': `Friend Class VSize + Public Function CompareTo(ByVal Other As VSize) As Integer + Return 0 + End Function +End Class +`, + 'Editors/UsersInfoForm.vb': `Friend Class UsersInfoForm + Private NotInheritable Class UserOpt + Friend Property TotalSize As Double = 0 + Friend Function CompareTo(ByVal Other As UserOpt) As Integer + Return TotalSize.CompareTo(Other.TotalSize) * -1 + End Function + End Class +End Class +`, + 'UI/ColorHSL.vb': `Public Class ColorHSL + Public Sub New(ByVal h As Double, ByVal s As Double, ByVal l As Double, ByVal a As Double) + End Sub + Public Function AddLuminance(ByVal offset As Single) As ColorHSL + Return Me + End Function + Public Function ToColor() As Integer + Return 0 + End Function +End Class +`, + 'UI/BackColorAdjuster.vb': `Public Class BackColorAdjuster + Public Function AddLuminance(ByVal offset As Single) As Integer + Return 0 + End Function +End Class +`, + 'UI/ThemeManager.vb': `Public Class ThemeManager + Public Sub Apply() + Dim _backColor As ColorHSL = New ColorHSL(0, 0.01, 0.1, 1) + Dim _controlBackColor As ColorHSL = _backColor.AddLuminance(0.025) + End Sub +End Class +`, + 'UI/Popup.vb': `Public Class Popup + Public Sub Show() + End Sub +End Class +`, + 'UI/MainForm.vb': `Public Class MainForm + Inherits Form + + Public Sub OpenSettings() + Dim settingsForm As New MainForm + settingsForm.Show() + End Sub +End Class +`, + 'General/Extensions.vb': `Imports System.Runtime.CompilerServices + +Module StringExtensions + + Function ToColor(ByVal str As String, Optional ByVal defaultColor As Integer = 0) As Integer + Return 0 + End Function + + Function Join(ByVal instance As IEnumerable(Of String), ByVal delimiter As String) As String + Return "" + End Function + + Function Sort(Of T)(ByVal instance As IEnumerable(Of T)) As IEnumerable(Of T) + Return instance + End Function +End Module +`, + 'General/ObjectStorage.vb': `Public Class ObjectStorage + Public Function GetString(ByVal key As String, Optional ByVal defaultValue As String = Nothing) As String + Return defaultValue + End Function +End Class +`, + 'General/Thumbnailer.vb': `Public Class Thumbnailer + Public Sub Run(ByVal settings As ObjectStorage) + Dim timestampFontColorValue = settings.GetString("TimestampFontColor", "#fff") + Dim timestampFontColor = timestampFontColorValue.ToColor(1) + Dim colorText As String = "#000" + Dim outline = colorText.ToColor() + Dim name$ = "x" + Dim named = name.ToColor() + End Sub +End Class +`, + 'General/Lists.vb': `Public Class Lists + Public Sub Run() + Dim names As New List(Of String) + names.Sort() + Dim joined = names.Join(", ") + End Sub +End Class +`, + 'Base/DownDetector.vb': `Namespace API.Base + Friend NotInheritable Class DownDetector + Friend MustInherit Class Checker(Of T) + Protected ReadOnly Property Source As T + End Class + End Class +End Namespace +`, + 'Sites/Bluesky/SiteSettings.vb': `Namespace API.Bluesky + Friend Class SiteSettings + Friend Function IsMyUser(ByVal url As String) As Boolean + Return False + End Function + Friend Function AvailableTrueValue() As Boolean + Return False + End Function + End Class + Friend NotInheritable Class M3U8 + Friend Shared Sub Download(ByVal url As String) + End Sub + End Class +End Namespace +`, + 'Sites/Reddit/SiteSettings.vb': `Namespace API.Reddit + Friend Class SiteSettings + Friend Function IsMyUser(ByVal url As String) As Boolean + Return True + End Function + Friend Function AvailableTrueValue() As Boolean + Return True + End Function + Private Class MyDownDetector : Inherits API.Base.DownDetector.Checker(Of SiteSettings) + Friend Sub Check() + Source.AvailableTrueValue() + End Sub + End Class + End Class + Friend NotInheritable Class M3U8 + Friend Shared Sub Download(ByVal url As String) + End Sub + End Class +End Namespace +`, + 'Sites/Reddit/UserData.vb': `Namespace API.Reddit + Friend Class UserData + Private ReadOnly Property MySettings As SiteSettings + Friend Sub Check(ByVal url As String) + MySettings.IsMyUser(url) + M3U8.Download(url) + End Sub + End Class +End Namespace +`, + 'Encoding/VideoEncoder.vb': `Public MustInherit Class VideoEncoder + Public Class MenuList + Public Sub Add(ByVal text As String) + End Sub + End Class +End Class +Public Class BatchEncoder + Inherits VideoEncoder + Public Function GetMenu() As Object + Dim ret As New MenuList + ret.Add("Codec Configuration") + Return ret + End Function +End Class +`, + 'YouTube/YouTubeFunctions.vb': `Friend Interface IContainer + Sub Parse() +End Interface +Friend Class Channel : Implements IContainer + Friend Sub Parse() Implements IContainer.Parse + End Sub +End Class +Friend Module YouTubeFunctions + Friend Sub Load() + Dim item As IContainer + item = New Channel + item.Parse() + End Sub +End Module +`, + 'UI/Labels.vb': `Public Class LabelUI + Public Function AddLabel(ByVal text As String) As Object + Return Nothing + End Function +End Class +Public Class SimpleUI + Public Function AddLabel(ByVal text As String) As Object + Return Nothing + End Function + Public Function AddLabel(ByVal text As String, ByVal width As Integer) As Object + Return Nothing + End Function +End Class +Public Class SimpleSettingsForm + Friend WithEvents SimpleUI As SimpleUI +End Class +`, + 'UI/AudioForm.vb': `Public Class AudioForm + Public Sub ShowAdvanced() + Using form As New SimpleSettingsForm() + Dim ui = form.SimpleUI + ui.AddLabel("EBU R128") + End Using + End Sub +End Class +`, + 'UI/Theme.vb': `Public Class ButtonLabel + Public Sub ApplyTheme() + End Sub +End Class +Public Class ToggleButtonLabel + Public Sub ApplyTheme() + End Sub +End Class +Public Class ThemeApplier + Public Sub Apply(ByVal controls As Object) + For Each control In controls.OfType(Of ToggleButtonLabel) + control.ApplyTheme() + Next + End Sub + Public Sub ApplyAll(ByVal labels As List(Of ToggleButtonLabel)) + For Each label In labels + label.ApplyTheme() + Next + End Sub +End Class +`, + 'Download/TDownloader.vb': `Namespace App.Download + Friend Class TDownloader + Friend Class Job + Friend Sub Start() + End Sub + End Class + End Class + Friend Class Worker + Friend Sub Start() + End Sub + End Class +End Namespace +`, + 'Download/DownloadProgress.vb': `Imports TDJob = App.Download.TDownloader.Job + +Namespace App.Download + Friend Class DownloadProgress + Friend ReadOnly Property Worker As TDJob + Friend Sub Run() + Worker.Start() + End Sub + End Class +End Namespace +`, + 'Plugin/IPluginContentProvider.vb': `Namespace Plugin + Public Interface ISiteSettings + End Interface + Public Interface IPluginContentProvider + Property Settings As ISiteSettings + End Interface +End Namespace +`, + 'MainMod.vb': `Friend Module MainMod + Friend Settings As SettingsCLS +End Module +`, + 'SettingsCLS.vb': `Friend Class SettingsCLS + Friend Sub UpdateUsersList() + End Sub +End Class +`, + 'API/Base/UserDataProvider.vb': `Namespace API.Base + Friend MustInherit Class UserDataProvider : Implements Plugin.IPluginContentProvider + Friend Property MySettings As Plugin.ISiteSettings Implements Plugin.IPluginContentProvider.Settings + Friend Sub Delete() + Settings.UpdateUsersList() + End Sub + End Class +End Namespace +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +/** `Owner::member` of every call edge out of a file. */ +function callsFrom(file: string): string[] { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg + .getOutgoingEdgesFrom(ids) + .filter((e) => e.kind === 'calls') + .map((e) => cg.getNode(e.target)!.qualifiedName) + .sort(); +} + +describe('VB.NET calls through declared receivers', () => { + it('reach nothing of the project on a type it does not define', () => { + expect(callsFrom('Download/Groups/DownloadGroupCollection.vb')).toEqual([]); + expect(callsFrom('Hosts/DownloadableMediaHost.vb')).toEqual([]); + expect(callsFrom('API/Base/UserDataBase.vb')).toEqual([]); + }); + + it('reach nothing of the project on a VB.NET built-in type', () => { + expect(callsFrom('Editors/UsersInfoForm.vb')).toEqual([]); + }); + + it('reach the declared type’s own method, or the one it inherits', () => { + expect(callsFrom('UI/ThemeManager.vb')).toEqual(['ColorHSL::AddLuminance']); + expect(callsFrom('API/Instagram/Downloader.vb')).toEqual(['API.Base::UserDataBase::Download']); + }); + + it('reach nothing of the project for a method the declared type inherits from outside it', () => { + expect(callsFrom('UI/MainForm.vb')).toEqual([]); + }); + + it('reach an extension method declared for the type, through what a call returns too', () => { + expect(callsFrom('General/Thumbnailer.vb')).toEqual([ + 'ObjectStorage::GetString', + 'StringExtensions::ToColor', + 'StringExtensions::ToColor', + 'StringExtensions::ToColor', + ]); + }); + + it('reach a .NET type’s own method before an extension of that name, and an extension it lacks', () => { + // `List(Of T)` has its own `Sort()`, but no `Join`. + expect(callsFrom('General/Lists.vb')).toEqual(['StringExtensions::Join']); + }); + + it('name the type their declaration’s namespace sees, through a base class’s type argument too', () => { + // SCrawler declares a `SiteSettings` and an `M3U8` in each site's namespace. + expect(callsFrom('Sites/Reddit/UserData.vb')).toEqual([ + 'API.Reddit::M3U8::Download', + 'API.Reddit::SiteSettings::IsMyUser', + ]); + // `Inherits DownDetector.Checker(Of SiteSettings)` makes its `Source As T` a SiteSettings. + expect(callsFrom('Sites/Reddit/SiteSettings.vb')).toEqual(['API.Reddit::SiteSettings::AvailableTrueValue']); + // A type nested in a base class is named bare in a subclass: staxrip's `Dim ret As New MenuList`. + expect(callsFrom('Encoding/VideoEncoder.vb')).toEqual(['VideoEncoder::MenuList::Add']); + }); + + it('take the declared type over what is assigned, and an import alias for what it names', () => { + expect(callsFrom('YouTube/YouTubeFunctions.vb')).toEqual(['IContainer::Parse']); + expect(callsFrom('Download/DownloadProgress.vb')).toEqual(['App.Download::TDownloader::Job::Start']); + }); + + it('read a field through what a call returns, and a loop variable through what it loops over', () => { + expect(callsFrom('UI/AudioForm.vb')).toEqual(['SimpleUI::AddLabel']); + // `For Each control In controls.OfType(Of ToggleButtonLabel)`, `For Each label In labels` over a `List(Of ToggleButtonLabel)`. + expect(callsFrom('UI/Theme.vb')).toEqual(['ToggleButtonLabel::ApplyTheme', 'ToggleButtonLabel::ApplyTheme']); + }); + + it('see a module’s field, not the property of an interface the class implements', () => { + expect(callsFrom('API/Base/UserDataProvider.vb')).toEqual(['SettingsCLS::UpdateUsersList']); + }); +}); diff --git a/__tests__/vbnet-std-methods.test.ts b/__tests__/vbnet-std-methods.test.ts new file mode 100644 index 0000000000..8c29b1e775 --- /dev/null +++ b/__tests__/vbnet-std-methods.test.ts @@ -0,0 +1,75 @@ +/** + * VB.NET runs on the same .NET base library as C#, so a call such as + * `x.Contains(…)`, `x.Add(…)` or `x.Dispose()` on a receiver whose type is not + * known is far more likely the library's method than the one project method + * that happens to share the name. As in C#, such a guess is kept only when the + * receiver is named after the method's owner. staxrip's + * `SupportedInput.Contains(ret)` — a String array — went to `VideoScript`'s + * `Contains`. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-vb-std-')); + const files: Record = { + 'Video/VideoScript.vb': `Public Class VideoScript + Public Function Contains(ByVal value As String) As Boolean + Return False + End Function +End Class +`, + 'Audio/AudioProfile.vb': `Public Class AudioProfile + Public Function IsInputSupported(ByVal inputs As Object, ByVal ext As String) As Boolean + For Each supported In inputs + If supported.Contains(ext) Then Return True + Next + Return False + End Function + + Public Function HasScript(ByVal scripts As Object, ByVal name As String) As Boolean + For Each script In scripts + If script.Contains(name) Then Return True + Next + Return False + End Function +End Class +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +/** `Owner::member` of every call edge out of one method. */ +function callsOf(qualifiedName: string): string[] { + const from = cg.getNodesInFile('Audio/AudioProfile.vb').find((n) => n.qualifiedName === qualifiedName)!; + return cg + .getOutgoingEdgesFrom([from.id]) + .filter((e) => e.kind === 'calls') + .map((e) => cg.getNode(e.target)!.qualifiedName) + .sort(); +} + +describe('VB.NET standard-library method names', () => { + it('are not a project method on a receiver that does not name its owner', () => { + expect(callsOf('AudioProfile::IsInputSupported')).toEqual([]); + }); + + it('still reach the project method when the receiver names its owner', () => { + expect(callsOf('AudioProfile::HasScript')).toEqual(['VideoScript::Contains']); + }); +}); diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 1eaf14b145..4c9fb45f2c 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -25,6 +25,7 @@ import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, is import { isVisibleCppMacro, clearCppMacroVisibility } from './cpp-macro-visibility'; import { isCppConstructorRef, matchCppConstructor } from './cpp-constructor'; import { gateSwiftTypeTarget, clearSwiftTypeVisibility, swiftExtendedConformances } from './swift-type-visibility'; +import { clearVbnetReceiverMemos } from './vbnet-receivers'; import { gateTypeParameter, clearTypeParameterMemos } from './type-parameters'; import { resolveViaImport, resolvePhpImportedStaticCall, resolvePhpQualifiedClassRef, resolveJvmImport, extractImportMappings, extractReExports, loadCppIncludeDirs, isPhpIncludePathRef, isCobolCopybookRef, isNixPathImportRef, isJsPathImportRef, isBoundToOutOfRepoImport, clearImportResolverMemos, resolveImportPath, isExternalImport } from './import-resolver'; import { ResolverPool, minRefsForPool, shouldEngageAdaptively } from './resolver-pool'; @@ -457,6 +458,7 @@ export class ReferenceResolver { clearNameMatcherMemos(this.context); clearCppMacroVisibility(this.context); clearSwiftTypeVisibility(this.context); + clearVbnetReceiverMemos(this.context); clearTypeParameterMemos(this.context); } } diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index f721a2cc43..0ba9c946d4 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -11,6 +11,7 @@ import { UnresolvedRef, ResolvedRef, ResolutionContext, ImportMapping, isSuperty import { blankStringContents, stripCommentsForRegex } from './strip-comments'; import { JS_BUILT_INS, JS_BUILTIN_METHODS, TS_PRIMITIVE_TYPES } from './js-builtins'; import { SWIFT_TYPE_PATH_CALL, resolveSwiftTypePathCall } from './swift-type-visibility'; +import { breakVbTie, isVbMemberInScope, isVbNestedTypeInScope, isVbTypeQualifiedBy, matchVbTypedCall, preferVbProject, sameVbProject } from './vbnet-receivers'; import { isTestPath } from '../search/query-utils'; import { isMinifiedContent } from '../extraction/generated-detection'; import { getCargoWorkspaceCrateMap } from './frameworks/cargo-workspace'; @@ -3112,6 +3113,9 @@ const CSHARP_STD_METHODS: ReadonlySet = new Set([ 'ForEach', 'Sort', 'Reverse', 'Clone', 'Seek', 'SetLength', ]); +/** The same .NET names as VB.NET writes them — in any case. */ +const VBNET_STD_METHODS: ReadonlySet = new Set([...CSHARP_STD_METHODS].map((m) => m.toLowerCase())); + /** * A request handler the web framework dispatches to: a Django / DRF / Flask * view's `get` / `post` / …, a controller's `index` / `store` / `update` / @@ -3156,11 +3160,17 @@ function stdMethodNames(language: string): ReadonlySet | null { case 'rust': return RUST_STD_METHODS; case 'kotlin': return KOTLIN_STD_METHODS; case 'csharp': return CSHARP_STD_METHODS; + case 'vbnet': return VBNET_STD_METHODS; case 'dart': return DART_STD_METHODS; default: return null; } } +/** Whether `name` is one of `language`'s standard-library method names (VB.NET's in any case). */ +function isStdMethodName(language: string, name: string): boolean { + return stdMethodNames(language)?.has(language === 'vbnet' ? name.toLowerCase() : name) ?? false; +} + /** Methods of Dart's String, List, Iterable, Map and Set — names a project type rarely carries itself. */ const DART_STD_METHODS: ReadonlySet = new Set([ 'endsWith', 'startsWith', 'contains', 'split', 'substring', 'trim', 'trimLeft', 'trimRight', 'toLowerCase', @@ -4657,7 +4667,13 @@ function vbReceiverOf(ref: UnresolvedRef, context: ResolutionContext): string | const name = ref.referenceName.toLowerCase(); let start = lower.startsWith(name, ref.column) ? ref.column : -1; if (start < 0) { - const m = new RegExp(`(?>>(); /** @@ -5536,6 +5568,8 @@ export function matchByExactName( const cfmlBare = (ref.language === 'cfml' || ref.language === 'cfscript') && ref.referenceKind === 'calls' && /^[A-Za-z_]\w*$/.test(ref.referenceName); const vbReceiver = ref.language === 'vbnet' && (ref.referenceKind === 'calls' || ref.referenceKind === 'instantiates') && /^\w+$/.test(ref.referenceName) ? vbReceiverOf(ref, context) : null; + const vbScoped = isVbScopedCall(ref, vbReceiver, context); + const vbUnqualified = isVbUnqualifiedName(ref, vbReceiver, context); const objcShape = ref.language === 'objc' && ref.referenceKind === 'calls' && /^[A-Za-z_]\w*:*(?:\w+:)*$/.test(ref.referenceName) ? objcCallShape(ref, context) : null; const csharpBare = ref.language === 'csharp' && (ref.referenceKind === 'calls' || ref.referenceKind === 'references') && /^[A-Za-z_]\w*$/.test(ref.referenceName); @@ -5557,6 +5591,9 @@ export function matchByExactName( !(objcShape === 'self-send' && !isObjcSelfSendTarget(n, ref, context)) && !(objcShape === 'super-send' && !isObjcSelfSendTarget(n, ref, context, true)) && !(vbReceiver !== null && !isVbMemberReachable(n, vbReceiver)) && + !(vbReceiver && !/^(?:me|mybase|myclass)$/i.test(vbReceiver) && !isVbTypeQualifiedBy(n, vbReceiver, ref.filePath, context)) && + !(vbScoped && !isVbMemberInScope(n, ref, context)) && + !(vbUnqualified && !isVbNestedTypeInScope(n, ref, context)) && !(rubyBare && n.kind === 'method' && !isRubyMethodInScope(n, ref, context)) && !(cfmlBare && n.kind === 'method' && !isCfmlMethodInScope(n, ref, context)) && !(javaBare && n.kind === 'method' && !isJavaMethodInScope(n, ref, context)) && @@ -7817,6 +7854,14 @@ export function matchMethodCall( // shared source-based inferrer. resolveMethodOnType validates the method // exists on the inferred type, so a mis-inference produces no edge. if (inferableReceiver) { + // A VB.NET receiver's declared type decides the call, or that there is no + // project method to call: SCrawler's `ThumbnailFile.Delete(…)` on an + // external `SFile` went to a nested class's `Delete` by a shared word. + if (ref.language === 'vbnet' && dotMatch) { + const typed = nmTimedT('mc-vbtyped', ref, () => + matchVbTypedCall(objectOrClass!, methodName!, ref, context, (name) => isStdMethodName('vbnet', name))); + if (typed !== undefined) return typed; + } let inferredType = nmTimedT('mc-infer', ref, () => ref.language === 'cpp' ? inferCppReceiverType(objectOrClass!, ref, context) @@ -8024,6 +8069,9 @@ export function matchMethodCall( const visible = classCandidates.filter((c) => c.language !== 'csharp' || isCsharpTypeVisible(c, typeRef, context)); classCandidates = [...visible, ...classCandidates.filter((c) => !visible.includes(c))]; } + // A VB.NET type declared in two projects is the caller's own project's: + // staxrip's `FrameServerFactory.Create(…)` went to its AutoCrop tool's copy. + if (ref.language === 'vbnet') classCandidates = preferVbProject(classCandidates, ref, context); for (const classNode of classCandidates) { // Skip cross-language class matches @@ -8213,10 +8261,10 @@ export function matchMethodCall( // `json` of its `MockedResponse`. !isUnnamedTestDouble(targetMethods[0]!, objectOrClass!, ref, context) && !((ref.language === 'lua' || ref.language === 'luau') && isLuaLibraryCall(objectOrClass!, methodName!, ref, targetMethods[0]!)) && - // Rust / Go / Kotlin / C#: a standard-library method name on an - // untyped receiver (`sym.map(…)`, `w.Header().Get(…)`, + // Rust / Go / Kotlin / C# / VB.NET: a standard-library method name on + // an untyped receiver (`sym.map(…)`, `w.Header().Get(…)`, // `reader.Value.ToString()`) is the library type's. - !(stdMethodNames(ref.language)?.has(methodName!) && + !(isStdMethodName(ref.language, methodName!) && !/^(?:self|Self|this|base)$/.test(objectOrClass!) && !receiverNamesOwner(receiverLink(objectOrClass!), targetMethods[0]!, context)) && !(UNTYPED_RECEIVER_LANGUAGES.has(ref.language) && !/^(?:self|self\.class|this|super|weak_?self|strong_?self)$/i.test(objectOrClass!) && !sharesReceiverWord(objectOrClass!, targetMethods[0]!) && @@ -8238,11 +8286,12 @@ export function matchMethodCall( const head = receiverWords[receiverWords.length - 1]?.toLowerCase(); let bestMatch: typeof targetMethods[0] | undefined; let bestScore = 0; + let tied: typeof targetMethods = []; // Same-file candidates first, so a score tie (`score > bestScore` keeps // the first seen) resolves to the call site's own file rather than the // first-indexed duplicate (#1079). - const std = stdMethodNames(ref.language)?.has(methodName!) && !/^(?:self|Self|this|base)$/.test(objectOrClass!); + const std = isStdMethodName(ref.language, methodName!) && !/^(?:self|Self|this|base)$/.test(objectOrClass!); for (const method of preferCallSiteFile(targetMethods, ref.filePath)) { if (std && !receiverNamesOwner(receiverLink(objectOrClass!), method, context)) continue; // The owner type's own name — not its namespace (`eShop.ClientApp…` @@ -8265,8 +8314,16 @@ export function matchMethodCall( if (score > bestScore) { bestScore = score; bestMatch = method; + tied = [method]; + } else if (score === bestScore) { + tied.push(method); } } + // VB.NET: between equally good guesses, the caller's own file, then its + // project, then the nearer directory — and no guess when none of them + // decides. staxrip's main app and its AutoCrop tool each declare a + // `ColorHSL`, and the first indexed took about 90 of the app's calls. + if (ref.language === 'vbnet' && tied.length > 1 && bestScore >= 2) bestMatch = breakVbTie(tied, ref, context) ?? undefined; // A wrapper handing its call on — BookStack's `FileStorage::delete` doing // `$storage->delete($path)`, `CommentRepo::delete` doing @@ -9229,7 +9286,7 @@ function computePathProximity(filePath1: string, filePath2: string): number { function findBestMatch( ref: UnresolvedRef, candidates: Node[], - _context: ResolutionContext + context: ResolutionContext ): Node | null { // Prioritization rules: // 1. Same file > different file @@ -9269,6 +9326,13 @@ function findBestMatch( // Directory proximity bonus — strongly prefer same module/package score += pathProximityFromDirs(refDirs, candidate.filePath); + // A VB.NET project compiles its own files: the caller's project weighs as + // much as the nearest a directory can be. staxrip's `New ColorHSL(…)` went + // to its AutoCrop tool's copy. + if (ref.language === 'vbnet' && candidate.language === 'vbnet' && sameVbProject(candidate.filePath, ref.filePath, context)) { + score += 80; + } + // Language matching: strongly prefer same language, penalize cross-language if (candidate.language === ref.language) { score += 50; @@ -9353,6 +9417,8 @@ export function matchFuzzy( const cfmlBare = (ref.language === 'cfml' || ref.language === 'cfscript') && ref.referenceKind === 'calls' && /^[A-Za-z_]\w*$/.test(ref.referenceName); const vbReceiver = ref.language === 'vbnet' && (ref.referenceKind === 'calls' || ref.referenceKind === 'instantiates') && /^\w+$/.test(ref.referenceName) ? vbReceiverOf(ref, context) : null; + const vbScoped = isVbScopedCall(ref, vbReceiver, context); + const vbUnqualified = isVbUnqualifiedName(ref, vbReceiver, context); const objcShape = ref.language === 'objc' && ref.referenceKind === 'calls' && /^[A-Za-z_]\w*:*(?:\w+:)*$/.test(ref.referenceName) ? objcCallShape(ref, context) : null; const csharpBare = ref.language === 'csharp' && (ref.referenceKind === 'calls' || ref.referenceKind === 'references') && /^[A-Za-z_]\w*$/.test(ref.referenceName); @@ -9388,6 +9454,9 @@ export function matchFuzzy( !(rubyBare && n.kind === 'method' && !isRubyMethodInScope(n, ref, context)) && !(cfmlBare && n.kind === 'method' && !isCfmlMethodInScope(n, ref, context)) && !(vbReceiver !== null && !isVbMemberReachable(n, vbReceiver)) && + !(vbReceiver && !/^(?:me|mybase|myclass)$/i.test(vbReceiver) && !isVbTypeQualifiedBy(n, vbReceiver, ref.filePath, context)) && + !(vbScoped && !isVbMemberInScope(n, ref, context)) && + !(vbUnqualified && !isVbNestedTypeInScope(n, ref, context)) && !(objcShape === 'c-call' && OBJC_MEMBER_KINDS.has(n.kind)) && !(objcShape === 'self-send' && !isObjcSelfSendTarget(n, ref, context)) && !(objcShape === 'super-send' && !isObjcSelfSendTarget(n, ref, context, true)) && diff --git a/src/resolution/vbnet-receivers.ts b/src/resolution/vbnet-receivers.ts new file mode 100644 index 0000000000..07732927e1 --- /dev/null +++ b/src/resolution/vbnet-receivers.ts @@ -0,0 +1,1335 @@ +/** + * VB.NET receivers, projects and receiver-less calls. + * + * The name matcher guesses a `receiver.Method()` call it cannot type by the + * method's name alone (matchMethodCall's Strategy 3: a unique name, or the + * receiver sharing words with the owner), and a receiver-less call by the + * nearest same-named member. VB.NET brought none of the evidence C# does to + * those guesses, so a call landed on whichever project method shared its + * name: SCrawler's `ThumbnailFile.Delete(…)` on an external `SFile` went to a + * nested `TempFileConversion.Delete`, staxrip's main app called its AutoCrop + * tool's copy of `ColorHSL`. What this module reads, from the source: + * + * 1. A receiver's declared type — `Dim x As T`, `ByVal x As T`, `x As New T`, + * a type character (`Dim name$`), `Dim x = New T(…)`, a `For Each` over a + * typed collection, a field or property of the class, of one it inherits + * (with the type arguments the subclass gives) or of a `Module`, or what + * `Dim x = obj.GetString(…)` returns. A typed receiver's call is the + * type's own method or one it inherits, else an extension method declared + * for the type, else nothing of the project's: a type the project does + * not define (`SFile`, `String`, `List(Of T)`) has none of its methods. + * 2. Which type a name means where it is written: the namespaces around it, + * outward, then its file's and project's `Imports` (aliases included), + * then the project a file belongs to — the `.vbproj` above it, whose + * `RootNamespace` its namespaces are inside. Between same-named types, and + * between guesses nothing else tells apart, the caller's own project's + * comes first; a written qualifier (`System.Drawing.Point`) must match. + * 3. A receiver-less call (or one on `Me` / `MyClass` / `MyBase`, which the + * extractor drops) is a member of a type around it, of what those inherit, + * or of a `Module`; a type nested in a class is named bare only inside it. + * + * VB.NET's keywords and names are matched without regard to case. Nothing + * here runs for another language. + */ +import * as fs from 'fs'; +import * as path from 'path'; +import type { Node } from '../types'; +import type { ResolutionContext, ResolvedRef, UnresolvedRef } from './types'; + +/** A VB.NET declared type: its simple name (no namespace, no type arguments) and whether it is an array. */ +interface VbType { + name: string; + array: boolean; + /** The namespaces or types written before the name (`API.Base` in `API.Base.UserDataBase`), lowercased. */ + qualifier?: string[]; + /** Where the name is written, which decides the namespaces it is looked up in. */ + file?: string; + line?: number; + /** A supertype named in an `Implements` statement: its members are not the type's own. */ + implemented?: boolean; + /** The type arguments written with it: `SiteSettings` in `Checker(Of SiteSettings)`. */ + args?: VbType[]; +} + +/** What a statement says a local is: its type, the call whose result it holds, or a binding that names no type. */ +type VbBinding = + | { kind: 'type'; type: VbType } + | { kind: 'call'; receiver: string | null; member: string } + | { kind: 'each'; collection: string } + | { kind: 'unknown' }; + +/** .NET's collections of one element type: `For Each x In list` over a `List(Of T)` binds a `T`. */ +const VB_ELEMENT_COLLECTIONS = /^(?:List|IList|IEnumerable|ICollection|IReadOnlyList|IReadOnlyCollection|HashSet|SortedSet|Queue|Stack|LinkedList|ObservableCollection|Collection|ReadOnlyCollection|BindingList|ConcurrentBag|ConcurrentQueue|ConcurrentStack|BlockingCollection)$/i; + +/** The type of what a `For Each` over a value of type `t` binds: an array's or a .NET collection's element type. */ +function elementType(t: VbType): VbType | null { + if (t.array) return { ...t, array: false }; + return t.args?.length === 1 && VB_ELEMENT_COLLECTIONS.test(t.name) && t.args[0]!.name !== '?' ? t.args[0]! : null; +} + +/** VB.NET's built-in types, by keyword and by .NET name, keyed to the keyword. */ +const VB_BUILTIN_TYPES: ReadonlyMap = new Map([ + ...['boolean', 'byte', 'char', 'date', 'decimal', 'double', 'integer', 'long', 'object', 'sbyte', 'short', 'single', + 'string', 'uinteger', 'ulong', 'ushort'].map((k): [string, string] => [k, k]), + ['int16', 'short'], ['int32', 'integer'], ['int64', 'long'], ['uint16', 'ushort'], ['uint32', 'uinteger'], + ['uint64', 'ulong'], ['datetime', 'date'], +]); + +/** The type a type character declares: `Dim aStr$`, `For i% = 0 …`. */ +const VB_TYPE_CHARS: Readonly> = { + $: 'String', '%': 'Integer', '&': 'Long', '!': 'Single', '#': 'Double', '@': 'Decimal', +}; + +/** VB.NET's conversion functions and the built-in type each returns. */ +const VB_CONVERSIONS: Readonly> = { + cbool: 'Boolean', cbyte: 'Byte', cchar: 'Char', cdate: 'Date', cdbl: 'Double', cdec: 'Decimal', cint: 'Integer', + clng: 'Long', cobj: 'Object', csbyte: 'SByte', cshort: 'Short', csng: 'Single', cstr: 'String', cuint: 'UInteger', + culng: 'ULong', cushort: 'UShort', +}; + +/** VB.NET type nodes a member can belong to (a `Module` is indexed as a class). */ +const VB_TYPE_KINDS: ReadonlySet = new Set(['class', 'struct', 'interface']); +const VB_VALUE_KINDS: ReadonlySet = new Set(['field', 'property', 'constant', 'variable']); +const VB_MEMBER_KINDS: ReadonlySet = new Set(['method', 'property', 'field', 'enum_member', 'constant', 'variable']); + +/** A declaration keyword right before a name: the name is a member being declared, not a variable. */ +const VB_MEMBER_HEAD = /\b(?:Function|Sub|Property|Event|Operator|Declare|Delegate|Class|Structure|Module|Interface|Enum|Namespace)\s+$/i; + +function escapeRegex(s: string): string { + return s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +} + +/** A type's key for comparing two of them: the built-in keyword or the lowercased name, `[]` for an array. */ +function typeKey(t: VbType): string { + const lower = t.name.toLowerCase(); + return `${VB_BUILTIN_TYPES.get(lower) ?? lower}${t.array ? '[]' : ''}`; +} + +function isBuiltin(t: VbType): boolean { + return !t.array && VB_BUILTIN_TYPES.has(t.name.toLowerCase()); +} + +/** The index of the `)` closing the `(` at `open`, or -1. */ +function closeParen(text: string, open: number): number { + let depth = 0; + for (let i = open; i < text.length; i++) { + if (text[i] === '(') depth++; + else if (text[i] === ')' && --depth === 0) return i; + } + return -1; +} + +/** The top-level items of the parenthesized list opening at `open`, or null when it does not close. */ +function splitArgs(text: string, open: number): string[] | null { + const close = closeParen(text, open); + if (close < 0) return null; + const args: string[] = []; + let depth = 0; + let cur = ''; + for (const ch of text.slice(open + 1, close)) { + if (ch === '(' || ch === '{') depth++; + else if (ch === ')' || ch === '}') depth--; + if (ch === ',' && depth === 0) { + args.push(cur); + cur = ''; + } else cur += ch; + } + args.push(cur); + return args.map((a) => a.trim()); +} + +/** + * The type written at `text[at]` — `List(Of Foo)`, `String()`, `Integer?`, + * `Global.A.B` — by its last name; null when no type starts there. + */ +function readType(text: string, at: number): VbType | null { + const head = /^\s*([A-Za-z_][\w.]*)/.exec(text.slice(at)); + if (!head) return null; + let i = at + head[0].length; + let args: VbType[] | undefined; + if (/^\s*\(\s*Of\b/i.test(text.slice(i))) { + const open = text.indexOf('(', i); + const close = closeParen(text, open); + if (close < 0) return null; + args = (splitArgs(text, open) ?? []).map((a) => readType(a.replace(/^Of\s+/i, ''), 0) ?? { name: '?', array: false }); + i = close + 1; + } + const segments = head[1]!.split('.'); + const name = segments.pop()!; + if (!/^[A-Za-z_]\w*$/.test(name) || /^(?:New|Of|As|In|Out|From|With)$/i.test(name)) return null; + const qualifier = segments.map((s) => s.toLowerCase()).filter((s, i) => !(i === 0 && s === 'global')); + return { + name, + array: /^\s*\??\s*\(\s*,*\s*\)/.test(text.slice(i)), + ...(qualifier.length > 0 ? { qualifier } : {}), + ...(args ? { args } : {}), + }; +} + +/** `t` — and the type arguments written inside it — written at `file:line`. */ +function sited(t: VbType | null, file: string, line: number): VbType | null { + return t ? { ...t, file, line, ...(t.args ? { args: t.args.map((a) => sited(a, file, line)!) } : {}) } : null; +} + +/** A VB.NET line's code: its comment dropped and every string literal emptied to `""`. */ +function vbCode(line: string): string { + // Most lines have neither: the line itself, without a copy. + if (!/["'‘’\r]/.test(line)) return line; + let out = ''; + for (let i = 0; i < line.length; i++) { + const ch = line[i]!; + if (ch === '"') { + let j = i + 1; + for (; j < line.length; j++) { + if (line[j] !== '"') continue; + if (line[j + 1] === '"') j++; + else break; + } + out += '""'; + i = j; + continue; + } + if (ch === "'" || ch === '\u2018' || ch === '\u2019' || ch === '\r') break; + out += ch; + } + return out; +} + +const VB_CODE_LINES = new WeakMap>(); + +/** A file's lines as code (see vbCode), read once per file. */ +function codeLines(file: string, context: ResolutionContext): string[] { + let memo = VB_CODE_LINES.get(context); + if (!memo) VB_CODE_LINES.set(context, (memo = new Map())); + const hit = memo.get(file); + if (hit) return hit; + const lines = (context.getFileLines?.(file) ?? context.readFile(file)?.split(/\r?\n/) ?? []).map(vbCode); + if (memo.size >= 1024) memo.delete(memo.keys().next().value!); + memo.set(file, lines); + return lines; +} + +interface VbPatterns { + /** The name standing where a binding names it — not as `name.Member` nor `name(args)`, which only use it. */ + declared: RegExp; + /** `name As T`, `name() As T`, `name As New T`: a variable, parameter, loop or catch variable. */ + asClause: RegExp; + /** `Dim name, other As T`: a declarator sharing the next one's type. */ + listed: RegExp; + /** `Dim name$`, `ByVal name%`, `For i% = …`. */ + typeChar: RegExp; + /** `Dim name = …`: the value's type. */ + inferred: RegExp; + /** `For Each name In …`: the collection's element type, when it says it. */ + forEach: RegExp; + /** A binding that names no type: `For name = …`, `Function(name)`, `From name In`, `Catch name`. */ + loose: RegExp; +} + +const VB_PATTERNS = new Map(); + +function patternsFor(name: string): VbPatterns { + const key = name.toLowerCase(); + const hit = VB_PATTERNS.get(key); + if (hit) return hit; + const r = escapeRegex(key); + const patterns: VbPatterns = { + declared: new RegExp(`(?= 4096) VB_PATTERNS.delete(VB_PATTERNS.keys().next().value!); + VB_PATTERNS.set(key, patterns); + return patterns; +} + +/** + * What every binding of a name has on its line: a declaring keyword, or a + * type character. Most lines naming a variable just use it (`sb.Append(…)`). + */ +const VB_MAY_BIND = /\b(?:As|Dim|Static|Const|For|Each|Function|Sub|Catch|Using|From|Let|Aggregate)\b|\w[$%&!#@]/i; + +/** What one line of code says `name` is, or undefined when it doesn't bind the name. */ +function bindingOn(code: string, name: string): VbBinding | undefined { + if (!VB_MAY_BIND.test(code)) return undefined; + const p = patternsFor(name); + if (!p.declared.test(code)) return undefined; + p.asClause.lastIndex = 0; + for (let m = p.asClause.exec(code); m; m = p.asClause.exec(code)) { + if (VB_MEMBER_HEAD.test(code.slice(0, m.index))) continue; + const type = readType(code, m.index + m[0].length); + if (!type) return { kind: 'unknown' }; + // `As New T(…)`: the parentheses are the constructor's. + return { kind: 'type', type: m[2] ? { ...type, array: false } : { ...type, array: type.array || !!m[1] } }; + } + const listed = p.listed.exec(code); + if (listed) { + const type = readType(code, listed.index + listed[0].length); + return type ? { kind: 'type', type } : { kind: 'unknown' }; + } + const typeChar = p.typeChar.exec(code); + if (typeChar) return { kind: 'type', type: { name: VB_TYPE_CHARS[typeChar[1]!]!, array: !!typeChar[2] } }; + // An assignment (`x = New Channel`) says nothing: the declared type + // (`Dim x As IYouTubeMediaContainer`) is what the call binds to. + const inferred = p.inferred.exec(code); + if (inferred) return valueBinding(code.slice(inferred.index + inferred[0].length)); + // `For Each c In controls.OfType(Of ButtonLabel)`: the loop's elements are + // that type; `For Each user In users`: the elements of what `users` is. + const loop = p.forEach.exec(code); + if (loop) { + const collection = loop[1]!.trim(); + const elements = /\.\s*(?:OfType|Cast)\s*\(\s*Of\s+/i.exec(collection); + const type = elements && /^[^()]*\)\s*(?:\(\s*\))?\s*$/.test(collection.slice(elements.index + elements[0].length)) + ? readType(collection, elements.index + elements[0].length) : null; + if (type) return { kind: 'type', type }; + return /^[A-Za-z_]\w*$/.test(collection) ? { kind: 'each', collection } : { kind: 'unknown' }; + } + return p.loose.test(code) ? { kind: 'unknown' } : undefined; +} + +/** What a local initialized with `expr` (`Dim x = expr`) is. */ +function valueBinding(expr: string): VbBinding { + const e = expr.split(/\s:(?!=)/)[0]!.trim(); + const unknown: VbBinding = { kind: 'unknown' }; + const created = /^New\s+/i.exec(e); + if (created) { + const type = readType(e, created[0].length); + return type ? { kind: 'type', type: { ...type, array: false } } : unknown; + } + const cast = /^(?:DirectCast|TryCast|CType)\s*\(/i.exec(e); + if (cast) { + const args = splitArgs(e, cast[0].length - 1); + const type = args?.length === 2 ? readType(args[1]!, 0) : null; + return type ? { kind: 'type', type } : unknown; + } + if (/^\$?""/.test(e)) return { kind: 'type', type: { name: 'String', array: false } }; + const conversion = /^(C[A-Za-z]+)\s*\(/.exec(e); + const converted = conversion ? VB_CONVERSIONS[conversion[1]!.toLowerCase()] : undefined; + if (converted) return { kind: 'type', type: { name: converted, array: false } }; + // `obj.Member(…)`, `Member(…)`, `Me.Member`: a call or a read as the whole value. + const call = /^(?:([A-Za-z_]\w*)\s*\.\s*)?([A-Za-z_]\w*)\s*/.exec(e); + if (call) { + let rest = e.slice(call[0].length); + if (rest.startsWith('(')) { + const close = closeParen(rest, 0); + rest = close < 0 ? 'unclosed' : rest.slice(close + 1).trim(); + } + if (rest === '') return { kind: 'call', receiver: call[1] ?? null, member: call[2]! }; + } + return unknown; +} + +/** The 0-based line a call's own declarations start on: its member's first line, which holds a method's parameters. */ +function scopeStartLine(ref: UnresolvedRef, context: ResolutionContext): number { + const from = context.getNodeById?.(ref.fromNodeId); + if (from && from.filePath === ref.filePath && from.startLine <= ref.line && from.endLine >= ref.line && + (from.kind === 'method' || from.kind === 'function' || from.kind === 'property' || from.kind === 'field')) { + return from.startLine - 1; + } + let start = -1; + for (const n of context.getNodesInFile(ref.filePath)) { + if (n.kind !== 'method' && n.kind !== 'function' && n.kind !== 'property') continue; + if (n.startLine <= ref.line && n.endLine >= ref.line && n.startLine - 1 > start) start = n.startLine - 1; + } + return start < 0 ? ref.line - 1 : start; +} + +const VB_WORD_LINES = new WeakMap>>(); + +/** + * The 0-based lines of a file's code each lowercased word appears on, in + * order: a call's receiver is looked for only on the lines that name it, + * not on every line back to its method's start (a long method's calls each + * rescanned it). + */ +function wordLines(file: string, context: ResolutionContext): Map { + let memo = VB_WORD_LINES.get(context); + if (!memo) VB_WORD_LINES.set(context, (memo = new Map())); + const hit = memo.get(file); + if (hit) return hit; + const index = new Map(); + const words = /[A-Za-z_]\w*/g; + const lines = codeLines(file, context); + for (let i = 0; i < lines.length; i++) { + const code = lines[i]!.toLowerCase(); + words.lastIndex = 0; + for (let m = words.exec(code); m; m = words.exec(code)) { + let at = index.get(m[0]); + if (!at) index.set(m[0], (at = [])); + if (at[at.length - 1] !== i) at.push(i); + } + } + if (memo.size >= 256) memo.delete(memo.keys().next().value!); + memo.set(file, index); + return index; +} + +const VB_BINDINGS = new WeakMap>>(); + +/** + * Every statement of a file that binds `name`, by 0-based line — read once: + * staxrip's `GetArgs` methods call `sb.Append(…)` on hundreds of lines, each + * of which would otherwise walk back over all the others to the `Dim sb`. + */ +function bindingsOf(name: string, file: string, context: ResolutionContext): Array<{ line: number; binding: VbBinding }> { + let memo = VB_BINDINGS.get(context); + if (!memo) VB_BINDINGS.set(context, (memo = new Map())); + const key = `${file}|${name.toLowerCase()}`; + const hit = memo.get(key); + if (hit) return hit; + const lines = codeLines(file, context); + const found: Array<{ line: number; binding: VbBinding }> = []; + for (const i of wordLines(file, context).get(name.toLowerCase()) ?? []) { + const binding = bindingOn(lines[i]!, name); + if (binding) found.push({ line: i, binding }); + } + if (memo.size >= 65536) memo.delete(memo.keys().next().value!); + memo.set(key, found); + return found; +} + +/** The nearest statement above a call (in its member) that binds `name`, and its 1-based line. */ +function localBinding(name: string, ref: UnresolvedRef, context: ResolutionContext): { binding: VbBinding; line: number } | undefined { + const found = bindingsOf(name, ref.filePath, context); + // The last binding at or above the call's line. + let lo = 0; + let hi = found.length; + while (lo < hi) { + const mid = (lo + hi) >>> 1; + if (found[mid]!.line <= ref.line - 1) lo = mid + 1; + else hi = mid; + } + const nearest = found[lo - 1]; + return nearest && nearest.line >= scopeStartLine(ref, context) ? { binding: nearest.binding, line: nearest.line + 1 } : undefined; +} + +const VB_FILE_TYPES = new WeakMap>(); + +/** The VB.NET types a line is written inside, innermost first. */ +function typesAround(file: string, line: number, context: ResolutionContext): Node[] { + let memo = VB_FILE_TYPES.get(context); + if (!memo) VB_FILE_TYPES.set(context, (memo = new Map())); + let types = memo.get(file); + if (!types) { + types = context.getNodesInFile(file) + .filter((n) => n.language === 'vbnet' && VB_TYPE_KINDS.has(n.kind)) + .sort((a, b) => b.startLine - a.startLine); + if (memo.size >= 1024) memo.delete(memo.keys().next().value!); + memo.set(file, types); + } + return types.filter((n) => n.startLine <= line && n.endLine >= line); +} + +/** The project's VB.NET types (classes, modules, structures, interfaces) named `name`, case aside. */ +function projectTypesNamed(name: string, context: ResolutionContext): Node[] { + return context.getNodesByLowerName(name.toLowerCase()).filter((n) => n.language === 'vbnet' && VB_TYPE_KINDS.has(n.kind)); +} + +const VB_MEMBERS = new WeakMap>(); + +/** Members named `name` declared directly in `type` (any partial part of it), case aside. */ +function membersNamed(type: Node, name: string, context: ResolutionContext): Node[] { + let memo = VB_MEMBERS.get(context); + if (!memo) VB_MEMBERS.set(context, (memo = new Map())); + const qn = `${type.qualifiedName}::${name}`.toLowerCase(); + const hit = memo.get(qn); + if (hit) return hit; + const members = context.getNodesByLowerName(name.toLowerCase()).filter((n) => n.language === 'vbnet' && n.qualifiedName.toLowerCase() === qn); + if (memo.size >= 65536) memo.delete(memo.keys().next().value!); + memo.set(qn, members); + return members; +} + +const VB_MODULES = new WeakMap>(); + +/** Whether a VB.NET owner (by qualified name) is a `Module`, whose members are reached without a qualifier. */ +function isModule(ownerQn: string, context: ResolutionContext): boolean { + let memo = VB_MODULES.get(context); + if (!memo) VB_MODULES.set(context, (memo = new Map())); + const hit = memo.get(ownerQn); + if (hit !== undefined) return hit; + // Its own head line or the two after it (attributes may come first); the raw + // lines, as the whole file need not be read as code for this. + const module = context.getNodesByQualifiedName(ownerQn).some((n) => n.language === 'vbnet' && n.kind === 'class' && + (context.getFileLines?.(n.filePath) ?? context.readFile(n.filePath)?.split(/\r?\n/) ?? []).slice(n.startLine - 1, n.startLine + 2) + .some((l) => /^\s*(?:<[^>]*>\s*)*(?:(?:Public|Friend|Private|Partial)\s+)*Module\s/i.test(l))); + memo.set(ownerQn, module); + return module; +} + +/** The members named `name` of the project's Modules, which code reaches without a qualifier — the caller's project's first. */ +function moduleMembersNamed(name: string, ref: UnresolvedRef, context: ResolutionContext): Node[] { + const members = context.getNodesByLowerName(name.toLowerCase()).filter((n) => { + const cut = n.qualifiedName.lastIndexOf('::'); + return n.language === 'vbnet' && cut > 0 && isModule(n.qualifiedName.slice(0, cut), context); + }); + return preferVbProject(members, ref, context); +} + +const VB_SUPERS = new WeakMap>(); + +/** + * The types a VB.NET type's declarations (every `Partial` part) inherit or + * implement: the `Inherits` / `Implements` statements that open its body, + * including the `Class X : Inherits Y` form. + */ +function supertypesOf(type: Node, context: ResolutionContext): VbType[] { + let memo = VB_SUPERS.get(context); + if (!memo) VB_SUPERS.set(context, (memo = new Map())); + const key = type.qualifiedName.toLowerCase(); + const hit = memo.get(key); + if (hit) return hit; + const supers: VbType[] = []; + const parts = context.getNodesByQualifiedName(type.qualifiedName).filter((n) => n.language === 'vbnet' && VB_TYPE_KINDS.has(n.kind)); + for (const part of parts.length > 0 ? parts : [type]) { + const lines = codeLines(part.filePath, context); + scan: for (let i = part.startLine - 1; i < Math.min(lines.length, part.startLine + 12, part.endLine); i++) { + for (const statement of lines[i]!.split(/:(?!=)/)) { + const s = statement.trim(); + // Blank, an attribute, a directive, or the declaration's own head. + if (s === '' || /^<.*>$/.test(s) || s.startsWith('#') || /\b(?:Class|Structure|Interface|Module)\s+[A-Za-z_]/i.test(s) && !/^(?:Inherits|Implements)\b/i.test(s)) continue; + const clause = /^(Inherits|Implements)\s+(.+)$/i.exec(s); + if (!clause) break scan; + for (const sup of splitArgs(`(${clause[2]})`, 0) ?? []) { + const t = sited(readType(sup, 0), part.filePath, part.startLine); + if (t) supers.push(/^Implements$/i.test(clause[1]!) ? { ...t, implemented: true } : t); + } + } + } + } + memo.set(key, supers); + return supers; +} + +/** A type a VB.NET type inherits, with what its type parameters stand for there (`T` → `SiteSettings`). */ +interface VbAncestor { + node: Node; + args: Map; +} + +const VB_ANCESTRIES = new WeakMap>(); + +/** `t` with a type parameter replaced by what `args` says it stands for. */ +function substitute(t: VbType, args: ReadonlyMap): VbType { + return (!t.qualifier && !t.array && args.get(t.name.toLowerCase())) || t; +} + +/** + * A VB.NET type and the project types it inherits, nearest first: a class's + * base classes, an interface's base interfaces — whose members are the + * type's own. Each carries the type arguments `Inherits Checker(Of + * SiteSettings)` gives its parameters. An interface a class implements lends + * it no members: they are reached through the class's own `… Implements + * IFoo.Bar` ones. + */ +function ancestry(type: Node, context: ResolutionContext): VbAncestor[] { + let memo = VB_ANCESTRIES.get(context); + if (!memo) VB_ANCESTRIES.set(context, (memo = new Map())); + const hit = memo.get(type.id); + if (hit) return hit; + const out: VbAncestor[] = []; + const seen = new Set(); + const queue: VbAncestor[] = [{ node: type, args: new Map() }]; + while (queue.length > 0 && out.length < 16) { + const current = queue.shift()!; + const key = current.node.qualifiedName.toLowerCase(); + if (seen.has(key)) continue; + seen.add(key); + out.push(current); + for (const sup of supertypesOf(current.node, context)) { + if (sup.implemented) continue; + const found = typesNamedAt(sup, context); + if (found.ambiguous) continue; + const given = (sup.args ?? []).map((a) => substitute(a, current.args)); + for (const base of found.owners) { + const params = [...typeParametersOf(base, context).keys()]; + queue.push({ node: base, args: new Map(params.flatMap((p, i): Array<[string, VbType]> => (given[i] ? [[p, given[i]!]] : []))) }); + } + } + } + memo.set(type.id, out); + return out; +} + +/** A VB.NET type and the project types it inherits, nearest first (see ancestry). */ +function hierarchy(type: Node, context: ResolutionContext): Node[] { + return ancestry(type, context).map((a) => a.node); +} + +const VB_PROJECTS = new WeakMap>(); + +/** The directory of the nearest `.vbproj` at or above `dir` (`''` for the root), or null outside any project. */ +function projectOfDir(dir: string, context: ResolutionContext, memo: Map): string | null { + const hit = memo.get(dir); + if (hit !== undefined) return hit; + let found: string | null = null; + try { + if (fs.readdirSync(path.join(context.getProjectRoot(), dir)).some((e) => /\.vbproj$/i.test(e))) found = dir; + } catch { + // Unreadable: no project file here. + } + if (found === null && dir !== '') { + const cut = dir.lastIndexOf('/'); + found = projectOfDir(cut < 0 ? '' : dir.slice(0, cut), context, memo); + } + memo.set(dir, found); + return found; +} + +/** The project (the directory of the nearest `.vbproj` above it) a file belongs to, or null. */ +function projectOf(file: string, context: ResolutionContext): string | null { + let memo = VB_PROJECTS.get(context); + if (!memo) VB_PROJECTS.set(context, (memo = new Map())); + const cut = file.lastIndexOf('/'); + return projectOfDir(cut < 0 ? '' : file.slice(0, cut), context, memo); +} + +/** Whether two files are in the same VB.NET project — false when either is in none. */ +export function sameVbProject(a: string, b: string, context: ResolutionContext): boolean { + const project = projectOf(a, context); + return project !== null && project === projectOf(b, context); +} + +interface VbProjectInfo { + /** The root namespace every file of the project declares its namespaces inside, as segments. */ + root: string[]; + /** The namespaces the project imports into every file (``). */ + imports: string[]; +} + +const VB_PROJECT_INFO = new WeakMap>(); + +/** A file's project's root namespace (``, else the project's name) and project-wide imports, lowercased. */ +function projectInfo(file: string, context: ResolutionContext): VbProjectInfo { + const dir = projectOf(file, context); + if (dir === null) return { root: [], imports: [] }; + let memo = VB_PROJECT_INFO.get(context); + if (!memo) VB_PROJECT_INFO.set(context, (memo = new Map())); + const hit = memo.get(dir); + if (hit) return hit; + const info: VbProjectInfo = { root: [], imports: [] }; + try { + const abs = path.join(context.getProjectRoot(), dir); + const project = fs.readdirSync(abs).find((e) => /\.vbproj$/i.test(e)); + if (project) { + const text = fs.readFileSync(path.join(abs, project), 'utf8'); + const root = /\s*([\w.]*)\s*<\/RootNamespace>/i.exec(text)?.[1] ?? project.replace(/\.vbproj$/i, ''); + info.root = root.toLowerCase().split('.').filter((s) => s !== ''); + for (const m of text.matchAll(/ s.split('.'))]; +} + +/** A file's lines before its first declaration: where its `Option` and `Imports` statements are. */ +function headerLines(file: string, context: ResolutionContext): string[] { + const lines = codeLines(file, context); + const end = lines.findIndex((l) => + /^\s*(?:Namespace|Module|Class|Structure|Interface|Enum|Delegate|Public|Friend|Private|Protected|Partial|NotInheritable|MustInherit)\b/i.test(l)); + return end < 0 ? lines : lines.slice(0, end); +} + +const VB_FILE_IMPORTS = new WeakMap>(); + +/** The namespaces a file imports — its own `Imports` and its project's — lowercased. */ +function importedNamespaces(file: string, context: ResolutionContext): string[] { + let memo = VB_FILE_IMPORTS.get(context); + if (!memo) VB_FILE_IMPORTS.set(context, (memo = new Map())); + const hit = memo.get(file); + if (hit) return hit; + const imports = [...projectInfo(file, context).imports]; + for (const line of headerLines(file, context)) { + const m = /^\s*Imports\s+([\w.]+)\s*$/i.exec(line); + if (m) imports.push(m[1]!.toLowerCase().replace(/^global\./, '')); + } + memo.set(file, imports); + return imports; +} + +const VB_ALIASES = new WeakMap>>(); + +/** A file's import aliases: `Imports TDJob = SCrawler.DownloadObjects.TDownloader.Job`, by lowercased alias. */ +function importAliases(file: string, context: ResolutionContext): Map { + let memo = VB_ALIASES.get(context); + if (!memo) VB_ALIASES.set(context, (memo = new Map())); + const hit = memo.get(file); + if (hit) return hit; + const aliases = new Map(); + for (const line of headerLines(file, context)) { + const m = /^\s*Imports\s+([A-Za-z_]\w*)\s*=\s*(.+)$/i.exec(line); + const target = m ? readType(m[2]!, 0) : null; + if (target) aliases.set(m![1]!.toLowerCase(), target); + } + memo.set(file, aliases); + return aliases; +} + +/** `t` with an import alias it is written with — as itself or as its first qualifier — spelled out. */ +function unalias(t: VbType, file: string, context: ResolutionContext): VbType { + const aliases = importAliases(file, context); + if (aliases.size === 0) return t; + if (!t.qualifier) { + const target = aliases.get(t.name.toLowerCase()); + return target ? { ...t, name: target.name, ...(target.qualifier ? { qualifier: target.qualifier } : {}) } : t; + } + const target = aliases.get(t.qualifier[0]!); + return target ? { ...t, qualifier: [...(target.qualifier ?? []), target.name.toLowerCase(), ...t.qualifier.slice(1)] } : t; +} + +/** Whether `tail` is the end of `full`. */ +function endsWith(full: string[], tail: string[]): boolean { + return tail.length <= full.length && tail.every((s, i) => full[full.length - tail.length + i] === s); +} + +const VB_TYPE_LIKE_KINDS: ReadonlySet = new Set(['class', 'struct', 'interface', 'enum', 'type_alias']); + +/** The class, structure or interface (not a `Module`) a VB.NET type is nested in, or null. */ +function enclosingType(n: Node, context: ResolutionContext): Node | null { + const cut = n.qualifiedName.lastIndexOf('::'); + if (cut < 0) return null; + const parentQn = n.qualifiedName.slice(0, cut); + const parent = context.getNodesByQualifiedName(parentQn).find((p) => p.language === 'vbnet' && VB_TYPE_KINDS.has(p.kind)); + return parent && !isModule(parentQn, context) ? parent : null; +} + +function isNestedInType(n: Node, context: ResolutionContext): boolean { + return enclosingType(n, context) !== null; +} + +const VB_BASE_NAMES = new WeakMap>>(); + +/** + * The lowercased names of the types around a line and of the classes they + * inherit, walked by name (a few levels): the types whose nested types code + * there names bare — `Dim ret As New MenuList` in a `VideoEncoder` subclass. + */ +function baseTypeNamesAround(file: string, line: number, context: ResolutionContext): Set { + const around = typesAround(file, line, context); + let memo = VB_BASE_NAMES.get(context); + if (!memo) VB_BASE_NAMES.set(context, (memo = new Map())); + const key = around.map((t) => t.id).join('|'); + const hit = memo.get(key); + if (hit) return hit; + const names = new Set(); + let frontier = around; + for (let depth = 0; depth < 5 && frontier.length > 0; depth++) { + const next: Node[] = []; + for (const t of frontier) { + if (names.has(t.name.toLowerCase())) continue; + names.add(t.name.toLowerCase()); + for (const sup of supertypesOf(t, context)) if (!sup.implemented) next.push(...projectTypesNamed(sup.name, context)); + } + frontier = next; + } + memo.set(key, names); + return names; +} + +/** + * Whether a type is one a name written with this qualifier can mean: its + * namespaces and outer types end with what is written before the name. + * Designer code's `New System.Drawing.Point(3, 3)` is not staxrip's nested + * `ButtonEx.SymbolDrawer.Point`; `New API.Base.UserDataBase(…)` is that one. + */ +export function isVbTypeQualifiedBy(n: Node, qualifier: string, file: string, context: ResolutionContext): boolean { + if (n.language !== 'vbnet' || !VB_TYPE_LIKE_KINDS.has(n.kind)) return true; + const written = readType(`${qualifier}.${n.name}`, 0); + if (!written?.qualifier) return true; + const t = unalias(written, file, context); + return !t.qualifier || endsWith(fullSegments(n, context).slice(0, -1), t.qualifier); +} + +/** + * The project types a type name written at a site means, as VB.NET looks it + * up: one its written qualifier names, nested in or declared in the namespace + * of a type around the site — the nearest first — else nested in a class + * those inherit, else in a namespace the file or project imports, else in + * the site's own project (a class's nested type only where it is in scope). + * SCrawler declares + * a `SiteSettings` in each site's namespace (`API.Pinterest`, `API.Bluesky`, + * …); a member typed `SiteSettings` in `API.Pinterest.UserData` is + * Pinterest's. `ambiguous` when what is left are different types. + */ +function typesNamedAt(written: VbType, context: ResolutionContext): { owners: Node[]; ambiguous: boolean } { + const t = written.file ? unalias(written, written.file, context) : written; + let candidates = projectTypesNamed(t.name, context); + // `System.Drawing.Color` is not the project's `Color`; `API.Base.UserDataBase` is that one. + if (t.qualifier) candidates = candidates.filter((c) => endsWith(fullSegments(c, context).slice(0, -1), t.qualifier!)); + if (candidates.length === 0 || !t.file) return { owners: candidates, ambiguous: false }; + const file = t.file; + const site = typesAround(file, t.line ?? 0, context)[0]; + const sitePath = site ? fullSegments(site, context) : projectInfo(file, context).root; + let tier: Node[] = []; + let best = -1; + for (const c of candidates) { + const container = fullSegments(c, context).slice(0, -1); + if (container.length > sitePath.length || container.some((s, i) => s !== sitePath[i])) continue; + if (container.length > best) { + best = container.length; + tier = [c]; + } else if (container.length === best) tier.push(c); + } + // A type nested in a class the site's types inherit. + if (tier.length === 0 && !t.qualifier) { + const bases = baseTypeNamesAround(file, t.line ?? 0, context); + tier = candidates.filter((c) => bases.has(enclosingType(c, context)?.name.toLowerCase() ?? '')); + } + if (tier.length === 0) { + const imports = importedNamespaces(file, context); + tier = candidates.filter((c) => imports.includes(fullSegments(c, context).slice(0, -1).join('.'))); + } + // Else any the project declares in a namespace (the root namespaces and + // project imports this cannot see) — but not a class's nested type, which + // is named bare only inside it or a class deriving from it. + if (tier.length === 0) tier = t.qualifier ? candidates : candidates.filter((c) => !isNestedInType(c, context)); + // A project's own declaration over another project's of the same name + // (staxrip's main app and its AutoCrop tool each declare a `ColorHSL`). + const own = tier.filter((c) => sameVbProject(c.filePath, file, context)); + if (own.length > 0) tier = own; + // One type's partial parts are one type. + const distinct = new Set(tier.map((c) => `${projectOf(c.filePath, context)}|${c.qualifiedName.toLowerCase()}`)); + return { owners: [...tier.filter((c) => c.filePath === file), ...tier.filter((c) => c.filePath !== file)], ambiguous: distinct.size > 1 }; +} + +/** The call site's own file first, then its project's, the rest after, each in its given order. */ +export function preferVbProject(nodes: Node[], ref: UnresolvedRef, context: ResolutionContext): Node[] { + if (nodes.length < 2) return nodes; + const file: Node[] = []; + const project: Node[] = []; + const rest: Node[] = []; + for (const n of nodes) { + if (n.filePath === ref.filePath) file.push(n); + else if (sameVbProject(n.filePath, ref.filePath, context)) project.push(n); + else rest.push(n); + } + return [...file, ...project, ...rest]; +} + +/** How many leading directories two files share. */ +function sharedDirs(a: string, b: string): number { + const da = a.split('/').slice(0, -1); + const db = b.split('/').slice(0, -1); + let i = 0; + while (i < da.length && i < db.length && da[i] === db[i]) i++; + return i; +} + +/** + * The one of several equally good guesses a call means: the one in its own + * file, else in its own project, else in the nearest directory — or null + * when none of these tells them apart. + */ +export function breakVbTie(tied: Node[], ref: UnresolvedRef, context: ResolutionContext): Node | null { + // One type's overloads (and partial parts) are one guess, not a tie: the first stands for them. + const seen = new Set(); + const distinct = tied.filter((n) => { + const key = `${projectOf(n.filePath, context)}|${n.qualifiedName.toLowerCase()}`; + if (seen.has(key)) return false; + seen.add(key); + return true; + }); + if (distinct.length === 1) return distinct[0]!; + let pool = distinct.filter((n) => n.filePath === ref.filePath); + if (pool.length === 0) { + const own = distinct.filter((n) => sameVbProject(n.filePath, ref.filePath, context)); + pool = own.length > 0 ? own : distinct; + } + if (pool.length === 1) return pool[0]!; + let best = -1; + let winners: Node[] = []; + for (const n of pool) { + const shared = sharedDirs(ref.filePath, n.filePath); + if (shared > best) { + best = shared; + winners = [n]; + } else if (shared === best) winners.push(n); + } + return winners.length === 1 ? winners[0]! : null; +} + +/** The type a field or property declares itself with, read from its declaration. */ +function memberDeclaredType(member: Node, context: ResolutionContext): VbType | null { + const code = codeLines(member.filePath, context).slice(member.startLine - 1, member.startLine + 2).join(' '); + const r = escapeRegex(member.name); + const property = new RegExp(`\\bProperty\\s+${r}\\b\\s*`, 'i').exec(code); + if (property) { + let at = property.index + property[0].length; + if (code[at] === '(') { + const close = closeParen(code, at); + if (close < 0) return null; + at = close + 1; + } + const as = /^\s*As\s+(New\s+)?/i.exec(code.slice(at)); + const type = as ? readType(code, at + as[0].length) : null; + return type && as?.[1] ? { ...type, array: false } : type; + } + const binding = bindingOn(code, member.name); + if (binding?.kind === 'type') return binding.type; + // `Private x = New Foo()`: a field's initializer is what it holds. + const init = new RegExp(`(? = new Map()): VbType | null { + const type = sited(member.kind === 'method' ? declaredReturnType(member, context) : memberDeclaredType(member, context), + member.filePath, member.startLine); + const given = type ? substitute(type, args) : null; + return given !== type ? given : resolveTypeParameter(type, member, context); +} + +/** + * The type of what a receiver-less `name` reads where a call is: a member of + * a class around the call or of one it inherits, else of a `Module`. Null when + * it declares no type the source shows, undefined when nothing has the name. + */ +function memberType(name: string, ref: UnresolvedRef, context: ResolutionContext): VbType | null | undefined { + const fits = (n: Node) => VB_VALUE_KINDS.has(n.kind) || n.kind === 'method'; + for (const around of typesAround(ref.filePath, ref.line, context)) { + for (const { node, args } of ancestry(around, context)) { + const member = membersNamed(node, name, context).find(fits); + if (member) return valueMemberType(member, context, args); + } + } + const global = moduleMembersNamed(name, ref, context).find(fits); + return global ? valueMemberType(global, context) : undefined; +} + +/** The type a method, function or property returns, read from its declaration; null for a `Sub` or none written. */ +function declaredReturnType(n: Node, context: ResolutionContext): VbType | null { + const code = codeLines(n.filePath, context).slice(n.startLine - 1, n.startLine + 6).join(' '); + const head = new RegExp(`\\b(Function|Property|Sub)\\s+${escapeRegex(n.name)}\\b`, 'i').exec(code); + if (!head || /^Sub$/i.test(head[1]!)) return null; + let at = head.index + head[0].length; + // `(Of T)`, then the parameter list. + for (let group = 0; group < 2; group++) { + const open = /^\s*\(/.exec(code.slice(at)); + if (!open) break; + const close = closeParen(code, at + open[0].length - 1); + if (close < 0) return null; + at = close + 1; + } + const as = /^\s*As\s+/i.exec(code.slice(at)); + return as ? readType(code, at + as[0].length) : null; +} + +const VB_TYPE_PARAMS = new WeakMap>>(); + +/** The type parameters a declaration's head names (`Class Repo(Of T As Entity)`), each with its constraint or null. */ +function typeParametersOf(n: Node, context: ResolutionContext): Map { + let memo = VB_TYPE_PARAMS.get(context); + if (!memo) VB_TYPE_PARAMS.set(context, (memo = new Map())); + const hit = memo.get(n.id); + if (hit) return hit; + const params = readTypeParameters(n, context); + memo.set(n.id, params); + return params; +} + +function readTypeParameters(n: Node, context: ResolutionContext): Map { + const params = new Map(); + const code = codeLines(n.filePath, context).slice(n.startLine - 1, n.startLine + 2).join(' '); + const head = new RegExp(`\\b(?:Class|Structure|Interface|Module|Function|Sub)\\s+${escapeRegex(n.name)}\\s*(?=\\(\\s*Of\\b)`, 'i').exec(code); + if (!head) return params; + for (const p of splitArgs(code, head.index + head[0].length) ?? []) { + const m = /^(?:Of\s+)?(?:In\s+|Out\s+)?([A-Za-z_]\w*)(?:\s+As\s+(.+))?$/i.exec(p); + if (!m) continue; + const constraints = (m[2] ?? '').replace(/^\{|\}$/g, '').split(',').map((c) => c.trim()) + .filter((c) => c !== '' && !/^(?:New|Class|Structure)$/i.test(c)); + params.set(m[1]!.toLowerCase(), constraints.length > 0 ? sited(readType(constraints[0]!, 0), n.filePath, n.startLine) : null); + } + return params; +} + +/** + * What a type means where it is written: a type parameter of the method + * (`owner`) or of a type around it stands for its constraint — null when it + * has none, as its value's type is then unknown — and any other name for itself. + */ +function resolveTypeParameter(t: VbType | null, owner: Node | null, context: ResolutionContext): VbType | null { + if (!t || t.array || isBuiltin(t) || t.qualifier || !t.file) return t; + const key = t.name.toLowerCase(); + for (const decl of [...(owner && owner.kind === 'method' ? [owner] : []), ...typesAround(t.file, t.line ?? 0, context)]) { + const params = typeParametersOf(decl, context); + if (params.has(key)) return params.get(key) ?? null; + } + return t; +} + +/** + * The project types `t` names where it is written, null when the name leaves + * different types (a guess between them would be no better than none), or + * none for an array or a built-in type. + */ +function ownersOf(t: VbType, context: ResolutionContext): Node[] | null { + if (t.array || isBuiltin(t)) return []; + const found = typesNamedAt(t, context); + return found.ambiguous ? null : found.owners; +} + +/** + * A method — or, `withValues`, a field or property — named `name` on one of + * `owners` (the type `typed` names) or a project type they inherit, with what + * the type parameters of the type declaring it stand for. + */ +function memberOn( + owners: Node[], + name: string, + ref: UnresolvedRef, + context: ResolutionContext, + withValues: boolean, + typed?: VbType, +): { node: Node; args: Map } | null { + const fits = (n: Node) => n.kind === 'method' || (withValues && VB_VALUE_KINDS.has(n.kind)); + for (const level of [0, 1]) { + for (const owner of preferVbProject(owners, ref, context)) { + // What the owner's own parameters are, as the receiver's type writes them (`Repo(Of Foo)`). + const given = new Map([...typeParametersOf(owner, context).keys()].flatMap((p, i): Array<[string, VbType]> => + (typed?.args?.[i] ? [[p, typed.args[i]!]] : []))); + const ancestors = ancestry(owner, context); + for (const { node, args } of level === 0 ? ancestors.slice(0, 1) : ancestors.slice(1)) { + const found = preferVbProject(membersNamed(node, name, context).filter(fits), ref, context)[0]; + if (found) return { node: found, args: level === 0 ? given : new Map([...args].map(([p, t]) => [p, substitute(t, given)])) }; + } + } + } + return null; +} + +/** The type of the value `Member(…)` / `obj.Member(…)` gives, from the member's declaration. */ +function callResultType( + call: { receiver: string | null; member: string }, + ref: UnresolvedRef, + context: ResolutionContext, + depth: number, +): VbType | null { + let member: { node: Node; args: Map } | null = null; + if (call.receiver === null || /^(?:Me|MyClass|MyBase)$/i.test(call.receiver)) { + const owners = typesAround(ref.filePath, ref.line, context).slice(0, 1); + member = owners.length > 0 ? memberOn(owners, call.member, ref, context, true) : null; + if (!member && call.receiver === null) { + const global = moduleMembersNamed(call.member, ref, context).find((n) => n.kind === 'method' || VB_VALUE_KINDS.has(n.kind)); + member = global ? { node: global, args: new Map() } : null; + } + } else { + const type = receiverType(call.receiver, ref, context, depth + 1); + // Not a variable: a type or module named for a shared call (`ObjectStorage.Load()`). + const owners = type === undefined ? ownersOf({ name: call.receiver, array: false, file: ref.filePath, line: ref.line }, context) + : type ? ownersOf(type, context) : null; + member = owners && owners.length > 0 ? memberOn(owners, call.member, ref, context, true, type ?? undefined) : null; + } + return member ? valueMemberType(member.node, context, member.args) : null; +} + +/** + * What a receiver named `name` is declared as where the call is: a local or + * parameter of its member, else a field or property of its class, of one the + * class inherits, or of a `Module`. Null when the receiver is bound but its + * type isn't known; undefined when nothing binds the name (a type or module). + */ +function receiverType(name: string, ref: UnresolvedRef, context: ResolutionContext, depth: number): VbType | null | undefined { + if (depth > 2) return null; + const local = localBinding(name, ref, context); + if (local) { + const { binding, line } = local; + const owner = context.getNodeById?.(ref.fromNodeId) ?? null; + if (binding.kind === 'type') return resolveTypeParameter(sited(binding.type, ref.filePath, line), owner, context); + if (binding.kind === 'call') return callResultType(binding, { ...ref, line }, context, depth); + if (binding.kind === 'each') { + const collection = receiverType(binding.collection, { ...ref, line }, context, depth + 1); + return collection ? elementType(collection) : null; + } + return null; + } + return memberType(name, ref, context); +} + +interface VbExtension { + node: Node; + /** The type it extends; null when that is one of its own type parameters (it extends anything). */ + param: VbType | null; +} + +const VB_EXTENSIONS = new WeakMap>(); + +/** A method's extension-method declaration (` Function F(s As String, …)`), or null when it isn't one. */ +function extensionOf(n: Node, context: ResolutionContext): VbExtension | null { + if (n.kind !== 'method' || n.language !== 'vbnet') return null; + let memo = VB_EXTENSIONS.get(context); + if (!memo) VB_EXTENSIONS.set(context, (memo = new Map())); + const hit = memo.get(n.id); + if (hit !== undefined) return hit; + const extension = readExtension(n, context); + memo.set(n.id, extension); + return extension; +} + +function readExtension(n: Node, context: ResolutionContext): VbExtension | null { + const code = codeLines(n.filePath, context).slice(Math.max(0, n.startLine - 2), n.startLine + 6).join(' '); + const decl = new RegExp(`\\b(?:Function|Sub)\\s+${escapeRegex(n.name)}\\b`, 'i').exec(code); + if (!decl) return null; + const attributes = /((?:<[^<>]*>\s*)+)(?:(?:Public|Friend|Private|Protected|Shared|Overloads|Async|Iterator)\s+)*$/i + .exec(code.slice(0, decl.index))?.[1]; + if (!attributes || !/\bExtension(?:Attribute)?\b/i.test(attributes)) return null; + let at = decl.index + decl[0].length; + const typeParams = new Set(); + if (/^\s*\(\s*Of\b/i.test(code.slice(at))) { + const open = code.indexOf('(', at); + for (const p of splitArgs(code, open) ?? []) { + const tp = /^(?:Of\s+)?(?:In\s+|Out\s+)?([A-Za-z_]\w*)/i.exec(p); + if (tp) typeParams.add(tp[1]!.toLowerCase()); + } + const close = closeParen(code, open); + if (close < 0) return null; + at = close + 1; + } + const open = /^\s*\(/.exec(code.slice(at)); + const first = open ? splitArgs(code, at + open[0].length - 1)?.[0]?.replace(/<[^<>]*>/g, '') : undefined; + const as = first ? /\bAs\s+/i.exec(first) : null; + const type = as ? readType(first!, as.index + as[0].length) : null; + if (!type) return null; + return { node: n, param: typeParams.has(type.name.toLowerCase()) && !type.array ? null : type }; +} + +/** `Object`'s instance methods, which every type has. */ +const OBJECT_METHODS = ['tostring', 'equals', 'gethashcode', 'gettype']; + +/** + * The instance methods of .NET's everyday types: a call one of them answers + * is the type's own, whatever extension of that name the project declares + * (`list.Sort()` on a `List(Of T)`), and a name one of them lacks is no + * instance method there (`list.Join(", ")` is an extension's). + */ +const BCL_INSTANCE_METHODS: ReadonlyMap> = new Map(Object.entries({ + string: ['clone', 'compareto', 'contains', 'copyto', 'endswith', 'getenumerator', 'gettypecode', 'indexof', 'indexofany', + 'insert', 'isnormalized', 'lastindexof', 'lastindexofany', 'normalize', 'padleft', 'padright', 'remove', 'replace', 'split', + 'startswith', 'substring', 'tochararray', 'tolower', 'tolowerinvariant', 'toupper', 'toupperinvariant', 'trim', 'trimend', + 'trimstart'], + stringbuilder: ['append', 'appendformat', 'appendjoin', 'appendline', 'clear', 'copyto', 'ensurecapacity', 'getchunks', 'insert', + 'remove', 'replace'], + list: ['add', 'addrange', 'asreadonly', 'binarysearch', 'clear', 'contains', 'convertall', 'copyto', 'exists', 'find', 'findall', + 'findindex', 'findlast', 'findlastindex', 'foreach', 'getenumerator', 'getrange', 'indexof', 'insert', 'insertrange', + 'lastindexof', 'remove', 'removeall', 'removeat', 'removerange', 'reverse', 'sort', 'toarray', 'trimexcess', 'trueforall'], + dictionary: ['add', 'clear', 'containskey', 'containsvalue', 'ensurecapacity', 'getenumerator', 'remove', 'trimexcess', 'tryadd', + 'trygetvalue'], + hashset: ['add', 'clear', 'contains', 'copyto', 'exceptwith', 'getenumerator', 'intersectwith', 'ispropersubsetof', + 'ispropersupersetof', 'issubsetof', 'issupersetof', 'overlaps', 'remove', 'removewhere', 'setequals', 'symmetricexceptwith', + 'trimexcess', 'trygetvalue', 'unionwith'], + '[]': ['clone', 'copyto', 'getenumerator', 'getlength', 'getlonglength', 'getlowerbound', 'getupperbound', 'getvalue', + 'initialize', 'setvalue'], +}).map(([type, methods]) => [type, new Set([...methods, ...OBJECT_METHODS])])); + +/** The instance methods of a .NET type the table above knows; undefined for any other. */ +function bclInstanceMethods(t: VbType): ReadonlySet | undefined { + return BCL_INSTANCE_METHODS.get(t.array ? '[]' : typeKey(t)); +} + +/** + * Whether a value of type `t` might reach an extension declared for `param` + * through a conversion nothing here can check: `Object`, an array's + * collection interfaces, an interface a built-in type implements, or any + * type an outside base type may lead to. + */ +function mayExtend(t: VbType, open: boolean, param: VbType): boolean { + if (typeKey(param) === 'object') return true; + if (param.array !== t.array) { + return t.array && /^(?:IEnumerable|IList|ICollection|IReadOnlyList|IReadOnlyCollection|Array)$/i.test(param.name); + } + if (isBuiltin(param)) return false; + if (isBuiltin(t)) return /^I[A-Z]/.test(param.name); + return open; +} + +/** + * The extension method a call on a value of type `t` reaches: one declared + * for `t` or for a type it inherits — else, when exactly one other extension + * of that name could apply and the name is no instance method of `t`'s + * (one of .NET's own names, for a type nothing here describes), that one. + */ +function extensionFor( + t: VbType, + owners: Node[], + method: string, + ref: UnresolvedRef, + context: ResolutionContext, + isStdMethod: (name: string) => boolean, +): ResolvedRef | null { + const instanceMethods = owners.length === 0 ? bclInstanceMethods(t) : undefined; + if (instanceMethods?.has(method.toLowerCase())) return null; + const extensions = context.getNodesByLowerName(method.toLowerCase()) + .map((n) => extensionOf(n, context)) + .filter((e): e is VbExtension => e !== null); + if (extensions.length === 0) return null; + // Every name the type goes by — its own, its base types', the interfaces + // they implement — and whether one of them is an outside type, whose own + // ancestry nothing here shows. + const names = new Set([typeKey({ name: t.name, array: false })]); + let open = owners.length === 0 && !isBuiltin(t) && !t.array; + for (const owner of owners) { + for (const type of hierarchy(owner, context)) { + names.add(type.name.toLowerCase()); + for (const sup of supertypesOf(type, context)) { + names.add(typeKey(sup)); + const found = typesNamedAt(sup, context).owners; + if (found.length === 0) open = true; + else if (sup.implemented) for (const i of found) for (const h of hierarchy(i, context)) names.add(h.name.toLowerCase()); + } + } + } + const exact = extensions.filter((e) => e.param !== null && + (typeKey(e.param) === typeKey(t) || (!t.array && !e.param.array && names.has(typeKey(e.param))))); + if (exact.length > 0) { + const target = preferVbProject(exact.map((e) => e.node), ref, context)[0]!; + return { original: ref, targetNodeId: target.id, confidence: 0.85, resolvedBy: 'instance-method' }; + } + const loose = extensions.filter((e) => e.param === null || mayExtend(t, open, e.param)); + if (loose.length === 1 && (instanceMethods !== undefined || !isStdMethod(method))) { + return { original: ref, targetNodeId: loose[0]!.node.id, confidence: 0.7, resolvedBy: 'instance-method' }; + } + return null; +} + +/** + * Resolve `receiver.method()` through the receiver's declared type: the + * type's own method or one it inherits, else an extension method for it, + * else null — a typed receiver is never a guess. A receiver naming one of the + * project's types or modules is a shared call on it. Undefined when the type + * isn't known, or is `Object` (a late-bound call), for the name strategies. + */ +export function matchVbTypedCall( + receiver: string, + method: string, + ref: UnresolvedRef, + context: ResolutionContext, + isStdMethod: (name: string) => boolean, +): ResolvedRef | null | undefined { + if (!/^[A-Za-z_]\w*$/.test(receiver)) return undefined; + const type = receiverType(receiver, ref, context, 0); + if (type === undefined) { + // A type or module named for a shared call — `M3U8.Download(…)` in SCrawler's + // `API.Reddit.UserData` is `API.Reddit.M3U8`'s, not another site's: its own + // member or one it inherits. + const named = typesNamedAt({ name: receiver, array: false, file: ref.filePath, line: ref.line }, context); + if (named.owners.length === 0) return undefined; + if (named.ambiguous) return null; + const shared = memberOn(named.owners, method, ref, context, false); + return shared ? { original: ref, targetNodeId: shared.node.id, confidence: 0.85, resolvedBy: 'qualified-name' } : null; + } + if (!type || typeKey(type) === 'object') return undefined; + const owners = ownersOf(type, context); + // A type name that leaves two of the project's types: no guess between them. + if (owners === null) return null; + if (owners.length > 0) { + const own = memberOn(owners, method, ref, context, false, type); + if (own) return { original: ref, targetNodeId: own.node.id, confidence: 0.9, resolvedBy: 'instance-method' }; + } + return extensionFor(type, owners, method, ref, context, isStdMethod); +} + +const VB_TYPE_IMPORTS = new WeakMap>>(); + +/** The last names of a file's `Imports` (a type imported this way lends its shared members; an alias lends none). */ +function importedNames(file: string, context: ResolutionContext): Set { + let memo = VB_TYPE_IMPORTS.get(context); + if (!memo) VB_TYPE_IMPORTS.set(context, (memo = new Map())); + const hit = memo.get(file); + if (hit) return hit; + const names = new Set(); + for (const line of headerLines(file, context)) { + const m = /^\s*Imports\s+([\w.]+)\s*$/i.exec(line); + if (m) names.add(m[1]!.split('.').pop()!.toLowerCase()); + } + memo.set(file, names); + return names; +} + +const VB_SCOPES = new WeakMap | null>>(); + +/** The lowercased names of the types around a call and of the project types they inherit; null outside any type. */ +function scopeOwners(ref: UnresolvedRef, context: ResolutionContext): Set | null { + let memo = VB_SCOPES.get(context); + if (!memo) VB_SCOPES.set(context, (memo = new WeakMap())); + const hit = memo.get(ref); + if (hit !== undefined) return hit; + const around = typesAround(ref.filePath, ref.line, context); + const owners = around.length === 0 ? null + : new Set(around.flatMap((t) => hierarchy(t, context)).map((t) => t.name.toLowerCase())); + memo.set(ref, owners); + return owners; +} + +/** + * Whether an unqualified VB.NET type name can mean `n`: a type nested in a + * class, structure or interface is named bare only inside it or a type + * deriving from it — designer code's `New Point(4, 285)` is not staxrip's + * nested `ButtonEx.SymbolDrawer.Point`. A `Module`'s types belong to its + * namespace; a file can import the outer type, or alias the nested one + * (`Imports UserMediaD = SCrawler.DownloadObjects.TDownloader.UserMediaD`). + * Not judged when no type is around the name. + */ +export function isVbNestedTypeInScope(n: Node, ref: UnresolvedRef, context: ResolutionContext): boolean { + if (n.language !== 'vbnet' || !VB_TYPE_LIKE_KINDS.has(n.kind)) return true; + const parent = enclosingType(n, context); + if (!parent) return true; + const owners = scopeOwners(ref, context); + if (owners === null || owners.has(parent.name.toLowerCase())) return true; + const alias = importAliases(ref.filePath, context).get(ref.referenceName.toLowerCase()); + if (alias && alias.name.toLowerCase() === n.name.toLowerCase() && + (!alias.qualifier || endsWith(fullSegments(n, context).slice(0, -1), alias.qualifier))) return true; + return importedNames(ref.filePath, context).has(parent.name.toLowerCase()); +} + +/** + * Whether a receiver-less VB.NET call (or one through `Me` / `MyClass` / + * `MyBase`) can mean member `n`: one of a type around the call or of a type + * those inherit, of a `Module`, or of a type the file imports. Not judged + * when no type is around the call, or `n` is not a member of a type. + */ +export function isVbMemberInScope(n: Node, ref: UnresolvedRef, context: ResolutionContext): boolean { + if (n.language !== 'vbnet' || !VB_MEMBER_KINDS.has(n.kind)) return true; + const cut = n.qualifiedName.lastIndexOf('::'); + if (cut < 0) return true; + const owners = scopeOwners(ref, context); + if (owners === null) return true; + const ownerQn = n.qualifiedName.slice(0, cut); + const owner = ownerQn.split(/::|\./).pop()!.toLowerCase(); + return owners.has(owner) || isModule(ownerQn, context) || importedNames(ref.filePath, context).has(owner); +} + +/** + * Drop this module's per-context memos — file lines and what was read from + * them — with the resolver's own caches (ReferenceResolver.clearCaches), so a + * sync never reads a changed file's old declarations. + */ +export function clearVbnetReceiverMemos(context: ResolutionContext): void { + for (const memo of [VB_CODE_LINES, VB_WORD_LINES, VB_BINDINGS, VB_FILE_TYPES, VB_MEMBERS, VB_MODULES, VB_SUPERS, VB_ANCESTRIES, + VB_PROJECTS, VB_PROJECT_INFO, VB_FILE_IMPORTS, VB_ALIASES, VB_BASE_NAMES, VB_TYPE_PARAMS, VB_EXTENSIONS, VB_TYPE_IMPORTS, + VB_SCOPES] as Array>) { + memo.delete(context); + } +} From 26e8488aedd632ff8102e2626808d71435a602a2 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 17:37:39 +0000 Subject: [PATCH 176/259] fix(windows): the kernel-parity sweep loads the engine as file:// URLs (#2352) scripts/kernel-parity.mjs handed import() a path from path.join(). On Windows that is `C:\...`, which Node's ESM loader rejects with ERR_UNSUPPORTED_ESM_URL_SCHEME, so the sweep could not run there at all and contributors kept patched scratch copies. The three engine imports now go through pathToFileURL(...).href, as the agent-eval probe scripts already do. On macOS and Linux an ordinary path resolves to the same file:// URL as before. Checked on Windows 11 after npm ci, npm run build and scripts/build-kernel.sh: the unmodified script threw at its first import; with the fix, the C# fixtures (3/3), src/search (4/4), all of src/ as TypeScript (261/263, 2 deferred to wasm) and every fixture language (42/42) run to the summary line and exit 0. Co-authored-by: Claude Opus 5.5 --- scripts/kernel-parity.mjs | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/scripts/kernel-parity.mjs b/scripts/kernel-parity.mjs index 5209369d8c..98c347eefd 100644 --- a/scripts/kernel-parity.mjs +++ b/scripts/kernel-parity.mjs @@ -25,7 +25,7 @@ import * as fs from 'node:fs'; import * as path from 'node:path'; -import { fileURLToPath } from 'node:url'; +import { fileURLToPath, pathToFileURL } from 'node:url'; const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..'); const dist = (p) => path.join(ROOT, 'dist', p); @@ -106,9 +106,11 @@ if (files.length === 0) { } // --- load the built engine --------------------------------------------------- -const { extractFromSource } = await import(dist('extraction/tree-sitter.js')); -const { initGrammars, loadGrammarsForLanguages, detectLanguage } = await import(dist('extraction/grammars.js')); -const kernel = await import(dist('extraction/kernel/index.js')); +// As file:// URLs: import() rejects a bare `C:\...` path on Windows +// (ERR_UNSUPPORTED_ESM_URL_SCHEME). +const { extractFromSource } = await import(pathToFileURL(dist('extraction/tree-sitter.js')).href); +const { initGrammars, loadGrammarsForLanguages, detectLanguage } = await import(pathToFileURL(dist('extraction/grammars.js')).href); +const kernel = await import(pathToFileURL(dist('extraction/kernel/index.js')).href); await initGrammars(); await loadGrammarsForLanguages([...KERNEL_LANGS]); From 69ee3b9a8597940b2a12ea5fe26c48ab0c9c0650 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 19:03:03 +0000 Subject: [PATCH 177/259] fix(ui): fall back past a port Windows refuses with EACCES, and say why (#2299) (#2353) On Windows, a port another program holds exclusively (SO_EXCLUSIVEADDRUSE) or one inside an excluded port range fails listen() with EACCES rather than EADDRINUSE. The `codegraph ui` port fallback only moved on after EADDRINUSE, so it stopped at the first such port. The error then blamed the POSIX privileged-port rule ("Ports below 1024 usually need elevated privileges") for ports like 4747 or 49912. Confirmed on Windows 11: a .NET listener on 0.0.0.0:P with ExclusiveAddressUse makes Node's listen(P, '127.0.0.1') fail with EACCES. Fix: - shouldTryNextPort moves past EADDRINUSE on every platform. It moves past EACCES at any port on Windows, which has no privileged ports, and from 1024 up elsewhere. - Below 1024 off Windows, EACCES is the privileged-port rule, and the next port falls under it too. The walk stops there with the message it always gave, rather than trying 20 privileged ports. - describeBindFailure takes the platform: - Windows names the exclusive hold or reserved range and how to list the ranges (netsh int ipv4 show excludedportrange protocol=tcp). - POSIX keeps the privileged-port text below 1024. Above it, it says the system refused permission. - A walk that ran out says "in use or reserved" (Windows) or "in use or not allowed" when any port was refused rather than in use. - Both helpers take the platform as a parameter, like browserOpenCommand, so the tests check every platform's answer on any host. This builds on PR #2319 by @sx4im, which advanced on EACCES, used the Windows wording and added the issue's listen-spy test. That PR advanced on EACCES on every platform and printed "Windows refused port N ... netsh" on Linux and macOS too. On POSIX it would also walk 20 privileged ports and end with "Ports 80-99 are all in use or reserved". This change restricts both to where they are true. The changelog entry goes in the viewer-launch file, because the viewer is not released yet. Verification: - __tests__/ui-server.test.ts gains three kinds of test: - The issue's end-to-end test: port N is taken and N+1 is refused through a listen spy, so the server must land past N+1 and serve 200. - A pinned-port message test. - Unit tests of the decision and the messages for win32, linux and darwin. - Results on __tests__/ui-server.test.ts: - Main source: 9 failed / 40 passed. The walk fails with "Not allowed to listen on port 57423. Ports below 1024 usually need elevated privileges". - With the fix: 49 passed / 2 skipped (existing POSIX-gated tests). - Built CLI on Windows, with 0.0.0.0: held exclusively: - `codegraph ui` while 4747 is held: main exits 1 with the privileged-port message. With the fix it serves on 4748 (GET / 200). - `codegraph ui --port `: prints the Windows message. - A 3-port walk over held ports reports "in use or reserved". - Re-listening on the same server after a real EACCES works on Node 22.22 and 24.16. - tsc --noEmit is clean and npm run build succeeds. The 20 viewer test files pass (439 passed, 5 skipped). - Full suite: 5732 passed, 10 failed, 319 skipped. The 10 failures are timeouts and EBUSY teardown under load in function-ref, git-index-currency, index-daemon-rebuild and rust-self-owner. All four files pass when run alone. Issue #2299 reported by @ijbranch. Co-authored-by: Saim Shafique Co-authored-by: Claude Opus 5.5 --- __tests__/ui-server.test.ts | 160 +++++++++++++++++++++++++++++++- docs/viewer-launch-changelog.md | 2 + src/ui-server/index.ts | 92 +++++++++++++++--- 3 files changed, 238 insertions(+), 16 deletions(-) diff --git a/__tests__/ui-server.test.ts b/__tests__/ui-server.test.ts index df25858a84..4b4e63fc8c 100644 --- a/__tests__/ui-server.test.ts +++ b/__tests__/ui-server.test.ts @@ -12,7 +12,7 @@ * name in undici, and forging it is the whole point of half these cases. */ -import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest'; import * as http from 'http'; import * as fs from 'fs'; import * as os from 'os'; @@ -21,12 +21,14 @@ import { browserOpenCommand, cacheControlFor, contentTypeFor, + describeBindFailure, isAllowedHost, isAllowedOrigin, isSafeRequestPath, PathRefusalError, resolveProjectFile, resolveStaticAsset, + shouldTryNextPort, startUiServer, type UiServerHandle, } from '../src/ui-server'; @@ -69,6 +71,31 @@ function request( }); } +/** A failed `listen()` as node reports it. */ +function bindError(code: string, port: number): NodeJS.ErrnoException { + return Object.assign(new Error(`listen ${code}: 127.0.0.1:${port}`), { code, syscall: 'listen' }); +} + +/** + * Make `listen()` on `refused` fail with EACCES, the way Windows refuses a port + * another program holds exclusively or one inside a reserved range, without + * needing either on the machine running the test (#2299). Every other port is + * a real bind. + */ +function refuseListenOn(...refused: number[]): { mockRestore(): void } { + const realListen = http.Server.prototype.listen; + return vi + .spyOn(http.Server.prototype, 'listen') + .mockImplementation(function (this: http.Server, ...args: unknown[]) { + const port = args[0]; + if (typeof port === 'number' && refused.includes(port)) { + process.nextTick(() => this.emit('error', bindError('EACCES', port))); + return this; + } + return (realListen as (...a: unknown[]) => http.Server).apply(this, args); + }); +} + describe('codegraph ui server', () => { let tempDir: string; let viewerDir: string; @@ -198,6 +225,50 @@ describe('codegraph ui server', () => { await new Promise((resolve) => blocker.close(() => resolve())); } }); + + /** + * #2299: on Windows a port another program holds exclusively, or one in a + * reserved range, fails with EACCES rather than EADDRINUSE, and the walk + * used to stop there. Port N is really taken, N+1 is refused, so the server + * has to land beyond both. N+1 is an ephemeral port, above 1023, so this + * holds on every platform. + */ + it('falls back past a port refused with EACCES', async () => { + const blocker = http.createServer(() => {}); + await new Promise((resolve) => blocker.listen(0, '127.0.0.1', resolve)); + const taken = (blocker.address() as { port: number }).port; + const refusal = refuseListenOn(taken + 1); + + try { + const second = await startUiServer({ projectRoot, viewerDir, port: taken }); + try { + expect(second.port).toBeGreaterThan(taken + 1); + // …and it actually works on the port it landed on. + const res = await request(second.port, '/'); + expect(res.status).toBe(200); + } finally { + await second.close(); + } + } finally { + refusal.mockRestore(); + await new Promise((resolve) => blocker.close(() => resolve())); + } + }); + + it('blames a refused pinned port on what refused it, not on privileged ports', async () => { + // The port from the report: nowhere near 1024. + const refusal = refuseListenOn(49912); + try { + const failure = startUiServer({ projectRoot, viewerDir, port: 49912, portFallback: false }); + await expect(failure).rejects.toThrow(/49912/); + const message = await failure.catch((err: Error) => err.message); + expect(message).not.toMatch(/below 1024|elevated privileges/); + expect(message).not.toMatch(/already in use/); + expect(message).toMatch(/refused/); + } finally { + refusal.mockRestore(); + } + }); }); describe('Host allowlist (DNS rebinding)', () => { @@ -538,6 +609,93 @@ describe('security helpers', () => { }); }); +/** + * What a bind failure means depends on the platform (#2299): Windows has no + * privileged ports and refuses a taken or reserved one with EACCES, while + * elsewhere EACCES below 1024 is the privileged-port rule. The platform is a + * parameter, so every host checks every platform's answer. + */ +describe('port fallback on each platform', () => { + const LIST_RESERVED = 'netsh int ipv4 show excludedportrange protocol=tcp'; + + it('moves past a port in use everywhere, and past EACCES on Windows at any port', () => { + for (const platform of ['win32', 'linux', 'darwin'] as const) { + expect(shouldTryNextPort('EADDRINUSE', 80, platform), platform).toBe(true); + expect(shouldTryNextPort('EADDRINUSE', 4747, platform), platform).toBe(true); + } + expect(shouldTryNextPort('EACCES', 49912, 'win32')).toBe(true); + // Windows has no privileged-port rule: a low port is held, not forbidden. + expect(shouldTryNextPort('EACCES', 80, 'win32')).toBe(true); + }); + + it('stops at EACCES below 1024 elsewhere, where the next port needs the same privilege', () => { + expect(shouldTryNextPort('EACCES', 80, 'linux')).toBe(false); + expect(shouldTryNextPort('EACCES', 1023, 'darwin')).toBe(false); + // From 1024 up the refusal is about that port, and the next may be allowed. + expect(shouldTryNextPort('EACCES', 1024, 'linux')).toBe(true); + expect(shouldTryNextPort('EACCES', 4747, 'darwin')).toBe(true); + }); + + it('never moves past any other failure', () => { + for (const platform of ['win32', 'linux'] as const) { + expect(shouldTryNextPort('EADDRNOTAVAIL', 4747, platform), platform).toBe(false); + expect(shouldTryNextPort('EPERM', 4747, platform), platform).toBe(false); + expect(shouldTryNextPort(undefined, 4747, platform), platform).toBe(false); + } + }); + + it('says why Windows refused a pinned port, without the POSIX privileged-port story', () => { + for (const port of [49912, 80]) { + const message = describeBindFailure(bindError('EACCES', port), port, { port, fallback: false }, 'win32').message; + expect(message).toMatch( + new RegExp(`^Windows refused port ${port}: another program holds it, or it is in a reserved range`) + ); + expect(message).toContain(LIST_RESERVED); + expect(message).toContain('omit --port'); + expect(message).not.toMatch(/below 1024|elevated privileges/); + } + }); + + it('keeps the privileged-port explanation off Windows, and only below 1024', () => { + for (const fallback of [false, true]) { + const message = describeBindFailure(bindError('EACCES', 80), 80, { port: 80, fallback }, 'linux').message; + expect(message).toBe( + 'Not allowed to listen on port 80. Ports below 1024 usually need elevated privileges — pick a higher one with --port.' + ); + } + const high = describeBindFailure(bindError('EACCES', 4747), 4747, { port: 4747, fallback: false }, 'darwin').message; + expect(high).toMatch(/^Not allowed to listen on port 4747: the system refused permission\./); + expect(high).not.toMatch(/below 1024|elevated privileges|Windows|netsh/); + }); + + it('says a walk that ran out met refusals, not only ports in use', () => { + const walk = { port: 4747, fallback: true }; + const inUse = bindError('EADDRINUSE', 4766); + const refused = bindError('EACCES', 4766); + + // Nothing refused: the message it always gave. + expect(describeBindFailure(inUse, 4766, walk, 'win32').message).toBe( + 'Ports 4747–4766 are all in use. Free one, or pick another with --port.' + ); + const windows = describeBindFailure(refused, 4766, walk, 'win32').message; + expect(windows).toMatch(/^Ports 4747–4766 are all in use or reserved/); + expect(windows).toContain(LIST_RESERVED); + // An earlier port was refused even though the last one was only in use. + expect(describeBindFailure(inUse, 4766, walk, 'win32', true).message).toBe(windows); + const posix = describeBindFailure(refused, 4766, walk, 'linux').message; + expect(posix).toMatch(/^Ports 4747–4766 are all in use or not allowed\./); + expect(posix).not.toMatch(/reserved|netsh/); + }); + + it('leaves the pinned in-use message and other failures as they were', () => { + expect( + describeBindFailure(bindError('EADDRINUSE', 8080), 8080, { port: 8080, fallback: false }, 'win32').message + ).toBe('Port 8080 is already in use. Pick another with --port, or omit --port to let CodeGraph find a free one.'); + const other = bindError('EADDRNOTAVAIL', 4747); + expect(describeBindFailure(other, 4747, { port: 4747, fallback: true }, 'linux')).toBe(other); + }); +}); + describe('browserOpenCommand', () => { it('uses the platform opener', () => { expect(browserOpenCommand('http://x', 'darwin')).toEqual({ command: 'open', args: ['http://x'] }); diff --git a/docs/viewer-launch-changelog.md b/docs/viewer-launch-changelog.md index 593ba632c0..84ed815051 100644 --- a/docs/viewer-launch-changelog.md +++ b/docs/viewer-launch-changelog.md @@ -205,6 +205,8 @@ These describe `codegraph ui` and its screens. They were taken out of `## [Unrel - A viewer trail now holds up to 64 hops everywhere — the trail bar, saved trails and "Read as flow". Thanks @inth3shadows for the report and @danusha2345. (#1976) +- **`codegraph ui` on Windows moves on from a port another program holds.** Windows refuses a port that another program keeps for itself, or one inside a range it reserves, with a different error than a port that is simply in use, so the viewer gave up at the first such port instead of trying the next one, and blamed ports below 1024 needing elevated privileges — a rule Windows doesn't have. It now moves on to the next free port, and when none is left, or the port you pinned with `--port` is refused, it says what Windows did and how to list the reserved ranges. Thanks @ijbranch for the report and @sx4im. (#2299) + ## Entries whose graph and `codegraph_explore` half already shipped The release notes carry a reworded version of each of these. Keep only the viewer half at launch. diff --git a/src/ui-server/index.ts b/src/ui-server/index.ts index 40c170f703..e5d0cced8a 100644 --- a/src/ui-server/index.ts +++ b/src/ui-server/index.ts @@ -397,9 +397,10 @@ function safeDecode(value: string): string { /** * Bind the first free port at or after `port`, on loopback only. * - * Only `EADDRINUSE` advances to the next port — a permission failure or a bad - * address will not get better one port over, and retrying twenty times would - * only bury the real error. + * Only a port that is taken advances to the next one (see + * {@link shouldTryNextPort}) — a permission failure every port shares, or a + * bad address, will not get better one port over, and retrying twenty times + * would only bury the real error. */ async function listenWithFallback( server: http.Server, @@ -407,6 +408,9 @@ async function listenWithFallback( ): Promise { // Port 0 means "any free port", so there is nothing to fall back from. const attempts = opts.port === 0 || !opts.fallback ? 1 : Math.max(1, opts.attempts); + const platform = process.platform; + // Whether any port was refused rather than in use — the message says so. + let refused = false; for (let i = 0; i < attempts; i++) { const candidate = opts.port === 0 ? 0 : opts.port + i; @@ -419,8 +423,9 @@ async function listenWithFallback( return address.port; } catch (err) { const code = (err as NodeJS.ErrnoException).code; - if (code !== 'EADDRINUSE' || i === attempts - 1) { - throw describeBindFailure(err, candidate, opts); + if (code === 'EACCES') refused = true; + if (!shouldTryNextPort(code, candidate, platform) || i === attempts - 1) { + throw describeBindFailure(err, candidate, opts, platform, refused); } } } @@ -434,7 +439,8 @@ async function listenWithFallback( * The same `http.Server` is reused across attempts: a `listen()` that failed * with EADDRINUSE never took a handle, so it can be listened on again directly * (verified on Node 20 and 22 — `server.listening` is still `false` afterwards, - * and `close()` on a never-listening server would itself throw). + * and `close()` on a never-listening server would itself throw). The same goes + * for a port Windows refused with EACCES (checked on Node 22 and 24). */ function listenOnce(server: http.Server, port: number): Promise { return new Promise((resolve, reject) => { @@ -452,22 +458,78 @@ function listenOnce(server: http.Server, port: number): Promise { }); } -/** Turn a bind failure into something a user can act on. */ -function describeBindFailure( +/** + * Whether a `listen()` on `port` that failed with `code` leaves the next port + * worth trying. + * + * `EADDRINUSE` does, and so does `EACCES` on Windows: a port another program + * holds exclusively (`SO_EXCLUSIVEADDRUSE`), or one inside a range the system + * reserves (Hyper-V and WinNAT reserve whole blocks of them), fails there with + * EACCES rather than EADDRINUSE. Elsewhere, EACCES below 1024 is the + * privileged-port rule, which the next port is under too, so walking on would + * only bury it. From 1024 up it is something refusing that one port — a + * security policy, say — and the next one may be allowed. + */ +export function shouldTryNextPort( + code: string | undefined, + port: number, + platform: NodeJS.Platform = process.platform +): boolean { + if (code === 'EADDRINUSE') return true; + return code === 'EACCES' && !isPrivilegedPort(port, platform); +} + +/** + * Whether `port` is one only an administrator may bind: below 1024, on every + * platform but Windows, which has no such rule. + */ +function isPrivilegedPort(port: number, platform: NodeJS.Platform): boolean { + return platform !== 'win32' && port > 0 && port < 1024; +} + +/** The command that lists the port ranges Windows reserves. */ +const LIST_RESERVED_PORTS = '`netsh int ipv4 show excludedportrange protocol=tcp`'; + +/** + * Turn a bind failure into something a user can act on, and that is true where + * it is read: Windows refuses ports for reasons POSIX doesn't have, and has no + * privileged ports. + * + * `refused` says whether any port a fallback walk tried was refused (EACCES) + * rather than in use; it only changes the message for a walk that ran out. + */ +export function describeBindFailure( err: unknown, port: number, - opts: { port: number; fallback: boolean; attempts: number } + opts: { port: number; fallback: boolean }, + platform: NodeJS.Platform = process.platform, + refused = false ): Error { const code = (err as NodeJS.ErrnoException).code; + const windows = platform === 'win32'; + if (opts.fallback && shouldTryNextPort(code, port, platform)) { + // A failure that moves on only ends the walk when there is nowhere left. + const taken = + !refused && code === 'EADDRINUSE' + ? 'in use' + : windows + ? `in use or reserved (${LIST_RESERVED_PORTS} lists the reserved ranges)` + : 'in use or not allowed'; + return new Error(`Ports ${opts.port}–${port} are all ${taken}. Free one, or pick another with --port.`); + } if (code === 'EADDRINUSE') { - return opts.fallback - ? new Error( - `Ports ${opts.port}–${port} are all in use. Free one, or pick another with --port.` - ) - : new Error(`Port ${port} is already in use. Pick another with --port, or omit --port to let CodeGraph find a free one.`); + return new Error(`Port ${port} is already in use. Pick another with --port, or omit --port to let CodeGraph find a free one.`); } if (code === 'EACCES') { - return new Error(`Not allowed to listen on port ${port}. Ports below 1024 usually need elevated privileges — pick a higher one with --port.`); + if (isPrivilegedPort(port, platform)) { + return new Error(`Not allowed to listen on port ${port}. Ports below 1024 usually need elevated privileges — pick a higher one with --port.`); + } + return new Error( + (windows + ? `Windows refused port ${port}: another program holds it, or it is in a reserved range (${LIST_RESERVED_PORTS} lists them).` + : `Not allowed to listen on port ${port}: the system refused permission.`) + + ' Pick another with --port, or omit --port to let CodeGraph find a free one.' + ); } return err instanceof Error ? err : new Error(String(err)); } From 6c61dd74ca672dd8aa4adc86718406641967e109 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 20:05:18 +0000 Subject: [PATCH 178/259] fix(vbnet): link Shared fields and properties read through their class (#2305) (#2355) VB.NET emitted a reference for calls only, so a value read or write through a type name -- `AppSession.SessionId`, `AppSession.CurrentUser = "demo"`, `Logger.Level` -- linked nothing, and `codegraph callers` on the field, the property and the class came back empty. The reporter's central session class showed 6 callers against ~460 referencing files. - Extraction (TS only; VB.NET is not kernel-routed): the static-member pass now covers VB.NET. A value read through a Capitalized name is sent as one `references` ref, `Name.Member`, the receiver kept as a call's is. Namespace roots (`System`, `Microsoft`, `My`, `Global`) and the built-in type keywords (`String.Empty`) are not sent. - Resolution (vbnet-receivers.ts matchVbMemberRead, ahead of the framework, import and name strategies): the receiver is looked up with #2351's VB.NET scoping, case-insensitively. A local, parameter, field, property or Module member of that name holds a value and links nothing; one typed as the type of its own name ("Color Color") reads that type. Otherwise the class, Module, Structure, Interface or Enum the namespaces, Imports and aliases around the read see is the type: the read links the member it declares or inherits (field, property, constant, event, Enum value, nested type; Shared before a same-named instance member, never the member the read is in) and, from outside the type, the type itself. A method named this way is a call made without parentheses (`calls`) unless AddressOf / NameOf only names it. - `AppSession.Items(0)`, an index into a Shared field that VB.NET writes as a call, now reads the field and the type instead of linking nothing. Contributor PR #2321 (@ChrisPrapas) had the right idea -- enable the static-member pass for VB.NET with a member-level ref -- and its test scenarios are adopted. Its resolution is replaced: it name-matched the receiver's bare name and checked a member only against the receiver's spelling. Ported onto main it passes its own tests, but binds a parameter `appSession As OtherSession` to AppSession's members, misses import aliases, and adds 8,652 / 13,554 edges on SCrawler / staxrip, about 2,000 per repo binding a receiver to another class's same-named member (a plugin form's local `Dim CONTAINER_MAIN` went to another form's control field). Verification: new vbnet-shared-member-refs.test.ts (7 tests) fails 6/7 on main and passes; the other VB.NET suites pass; tsc and build clean; the issue's repro lists Consumer.Run and Consumer2.Describe for SessionId, CurrentUser and AppSession. Validation, main vs this branch: nodes unchanged, 0 edges removed, no self-loops. SCrawler 14,893 -> 17,437 edges, staxrip 26,647 -> 32,568: Enum values 803 / 1,448, Shared fields and properties 363 / 1,459, the type read through 1,374 / 2,879, Shared functions called without parentheses 3 / 126, AddressOf / NameOf 1 / 9. Spot-checked against source: per-site SiteSettings, import aliases, escaped `[Date]` / `[New]`, nested enums, "Color Color" fields; designer `Point.Empty` and staxrip's AutoCrop duplicates stay unlinked. Index time within noise; value reads that link nothing are kept for re-resolution like failed calls (db 22.6 -> 28.9 MB, 28.3 -> 35.8 MB). Not covered: reads through Me / MyBase, unqualified reads, instance reads through a typed variable, Module variables as receivers, namespace-qualified receivers and With blocks. Co-authored-by: Claude Opus 5.5 Co-authored-by: ChrisPrapas <129202926+ChrisPrapas@users.noreply.github.com> --- CHANGELOG.md | 1 + __tests__/vbnet-shared-member-refs.test.ts | 218 +++++++++++++++++++++ src/extraction/tree-sitter.ts | 43 +++- src/resolution/index.ts | 8 +- src/resolution/vbnet-receivers.ts | 140 +++++++++++-- 5 files changed, 396 insertions(+), 14 deletions(-) create mode 100644 __tests__/vbnet-shared-member-refs.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index b5476b128f..9c5f79e4c4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - A file is no longer saved with no symbols when its language parser can't be loaded, which is what happened to every file a background server re-indexed after an upgrade removed its install: the file keeps what it had and is indexed again once the parser loads, and files an earlier version emptied this way are re-indexed by the next sync. A background server also exits on its own once its install is upgraded or removed, so the next session starts one from the current install. Thanks @lipchey for the report. (#2335) - `codegraph status` no longer says the index is up to date while indexed files are missing their symbols: it now names files the parser couldn't read and files stored without their symbols (which `codegraph sync` repairs), `status --json` counts both, and `codegraph files --json` lists each file's recorded errors. Thanks @lipchey for the report. (#2336) - Indexing large Python projects is much faster again and needs less memory: since 1.6.2, resolving Python references re-read source files over and over, so a project the size of CPython took several times as long to index. The graph it builds is unchanged. Thanks @bompus for the report. (#2332) +- In VB.NET, reading or setting a `Shared` field or property through its class or module name, like `AppSession.SessionId`, `AppSession.CurrentUser = "demo"` or `AppSession.Items(0)`, now counts as a use of that member and of the class, and so do reading an `Enum` value like `Mode.Fast` and calling a `Shared` function without parentheses. Before, only calls written with parentheses were linked, so `codegraph callers` on such a field, property or class came back empty and impact missed most of the code that depends on it. A local, parameter or field that only shares a class's name is not mistaken for the class. Thanks @serkanince for the report and @ChrisPrapas. (#2305) ## [1.6.2] - 2026-10-03 diff --git a/__tests__/vbnet-shared-member-refs.test.ts b/__tests__/vbnet-shared-member-refs.test.ts new file mode 100644 index 0000000000..bbcfa77ffe --- /dev/null +++ b/__tests__/vbnet-shared-member-refs.test.ts @@ -0,0 +1,218 @@ +/** + * VB.NET: a value read or written through a type or module name — + * `AppSession.SessionId`, `AppSession.CurrentUser = "demo"`, `Logger.Level`, + * `Mode.Fast` — is a use of the member and of the type (#2305). Before, only + * calls linked, so `codegraph callers` on a Shared field or property, and on + * the class itself, came back empty. + * + * The name is looked up as VB.NET does, without regard to case: a local, a + * parameter or a field of that name holds a value, and links nothing here (a + * member typed as its own name's type, `Property Settings As Settings`, reads + * that type's members either way); a type is the one the namespaces around + * the read, its `Imports` and aliases see — SCrawler declares a + * `SiteSettings` in every site's namespace. A method named without + * parentheses is called, unless `AddressOf` only names it, and an index into + * a Shared field (`AppSession.Items(0)`), which VB.NET writes as a call, reads + * the field. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +const files: Record = { + // The issue's three files. + 'AppSession.vb': `Public Class AppSession + Public Shared SessionId As Guid = Guid.NewGuid() + Public Shared Property CurrentUser As String + Public Shared Items As New List(Of String) + Public Shared Function GetGreeting() As String + Return "Hello " & CurrentUser + End Function + Public Shared Sub OnTick(ByVal sender As Object, ByVal e As EventArgs) + End Sub + Public Shared Function Describe() As String + Return AppSession.CurrentUser + End Function +End Class +`, + 'Consumer.vb': `Public Class Consumer + Public Sub Run() + Dim id As Guid = AppSession.SessionId + AppSession.CurrentUser = "demo" + Console.WriteLine(AppSession.GetGreeting()) + End Sub +End Class +`, + 'Consumer2.vb': `Public Class Consumer2 + Public Function Describe() As String + Return AppSession.CurrentUser & AppSession.SessionId.ToString() + End Function +End Class +`, + 'OtherSession.vb': `Public Class OtherSession + Public Shared SessionId As Guid = Guid.NewGuid() + Public Property Size As Integer +End Class +`, + 'Size.vb': `Public Class Size +End Class +`, + 'Logger.vb': `Public Module Logger + Public Level As Integer +End Module +`, + 'Mode.vb': `Public Enum Mode + Fast + Slow +End Enum +`, + 'Form1.vb': `Public Class Form1 + Private Panel1 As Panel + Public Sub Setup() + Logger.Level = 3 + Dim a = Panel1.Size + Me.Panel1.Size = New System.Drawing.Size(1, 2) + Dim speed = Mode.Fast + Dim greeting = AppSession.GetGreeting + AddHandler Timer1.Tick, AddressOf AppSession.OnTick + Dim first = AppSession.Items(0) + Dim empty = String.Empty + End Sub + Public Sub Shadowed(ByVal appSession As OtherSession) + Dim id = AppSession.SessionId + End Sub +End Class +`, + // A type declared in each site's namespace, and one read through an import alias. + 'Sites/Reddit/SiteSettings.vb': `Namespace API.Reddit + Friend Class SiteSettings + Friend Const Header As String = "r" + End Class + Friend Class UserData + Friend Function Key() As String + Return SiteSettings.Header + End Function + End Class +End Namespace +`, + 'Sites/Twitter/SiteSettings.vb': `Namespace API.Twitter + Friend Class SiteSettings + Friend Const Header As String = "t" + End Class + Friend Class UserData + Friend Function Key() As String + Return SiteSettings.Header + End Function + End Class +End Namespace +`, + 'Sites/Facebook/UserData.vb': `Imports RS = API.Reddit.SiteSettings +Namespace API.Facebook + Friend Class UserData + Friend Function Key() As String + Return RS.Header + End Function + End Class +End Namespace +`, + // A property typed as the type of its own name; a property initializer. + 'Settings.vb': `Public Class Settings + Public Property Theme As String +End Class +`, + 'Window.vb': `Public Class Window + Public Property Settings As Settings + Public Property Title As String = AppSession.CurrentUser + Public Function CurrentTheme() As String + Return Settings.Theme + End Function +End Class +`, + // CRLF line endings and multibyte text before the read. + 'Weird.vb': [ + 'Public Class Weird', + ' Public Sub Go()', + ' Dim s = "SessionId éééé 😀" & AppSession.SessionId', + ' End Sub', + 'End Class', + '', + ].join('\r\n'), +}; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-vb-shared-')); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +function nodeId(qualifiedName: string): string { + const found = cg.getNodesInFiles(Object.keys(files)).filter((n) => n.qualifiedName === qualifiedName); + if (found.length !== 1) throw new Error(`${found.length} nodes named ${qualifiedName}`); + return found[0]!.id; +} + +/** The qualified names of what links to `qualifiedName` with an edge of `kind`. */ +function linkedFrom(qualifiedName: string, kind: 'references' | 'calls'): string[] { + return cg + .getIncomingEdgesTo([nodeId(qualifiedName)], [kind]) + .map((e) => cg.getNode(e.source)!.qualifiedName) + .sort(); +} + +describe('VB.NET Shared members read through their type (#2305)', () => { + it('link a Shared field and property to the methods that read or write them', () => { + expect(linkedFrom('AppSession::SessionId', 'references')).toEqual(['Consumer2::Describe', 'Consumer::Run', 'Weird::Go']); + expect(linkedFrom('AppSession::CurrentUser', 'references')) + .toEqual(['AppSession::Describe', 'Consumer2::Describe', 'Consumer::Run', 'Window::Title']); + }); + + it('link the class to the code outside it that reads through it', () => { + expect(linkedFrom('AppSession', 'references')) + .toEqual(['Consumer2::Describe', 'Consumer2::Describe', 'Consumer::Run', 'Consumer::Run', 'Form1::Setup', 'Weird::Go', 'Window::Title']); + expect(cg.getCallers(nodeId('AppSession')).map((c) => c.node.qualifiedName).sort()) + .toEqual(['Consumer2::Describe', 'Consumer::Run', 'Form1::Setup', 'Weird::Go', 'Window::Title']); + }); + + it('keep the call to a Shared method, and call one named without parentheses', () => { + expect(linkedFrom('AppSession::GetGreeting', 'calls')).toEqual(['Consumer::Run', 'Form1::Setup']); + expect(linkedFrom('AppSession::OnTick', 'references')).toEqual(['Form1::Setup']); + expect(linkedFrom('AppSession::OnTick', 'calls')).toEqual([]); + }); + + it('read a Shared field indexed like a call, a Module variable and an Enum value', () => { + expect(linkedFrom('AppSession::Items', 'references')).toEqual(['Form1::Setup']); + expect(linkedFrom('Logger::Level', 'references')).toEqual(['Form1::Setup']); + expect(linkedFrom('Mode::Fast', 'references')).toEqual(['Form1::Setup']); + expect(linkedFrom('Mode', 'references')).toEqual(['Form1::Setup']); + }); + + it('link nothing through a local, parameter or field, whatever its case', () => { + expect(linkedFrom('OtherSession::SessionId', 'references')).toEqual([]); + expect(linkedFrom('OtherSession::Size', 'references')).toEqual([]); + expect(linkedFrom('Size', 'references')).toEqual([]); + expect(linkedFrom('OtherSession', 'references')).toEqual([]); + }); + + it('read the type the namespaces and imports around the read see', () => { + expect(linkedFrom('API.Reddit::SiteSettings::Header', 'references')) + .toEqual(['API.Facebook::UserData::Key', 'API.Reddit::UserData::Key']); + expect(linkedFrom('API.Twitter::SiteSettings::Header', 'references')).toEqual(['API.Twitter::UserData::Key']); + }); + + it('read through a member typed as the type of its own name', () => { + expect(linkedFrom('Settings::Theme', 'references')).toEqual(['Window::CurrentTheme']); + }); +}); diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index 222291f14c..17b0b02caf 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -361,7 +361,7 @@ const PHP_TYPE_NODES: ReadonlySet = new Set([ */ const MEMBER_ACCESS_TYPES: ReadonlySet = new Set([ 'field_access', // java (`Foo.BAR`) - 'member_access_expression', // c# (`Foo.Bar`) + 'member_access_expression', // c# / vb.net (`Foo.Bar`) 'navigation_expression', // kotlin / swift (`Foo.bar`) 'field_expression', // scala (`Foo.bar`) 'class_constant_access_expression', // php (`Foo::CONST`, `Foo::class`) @@ -380,11 +380,21 @@ const MEMBER_ACCESS_TYPES: ReadonlySet = new Set([ * static read is pure duplication) — while adding real graph noise (+1813 edges / * +2448 `references` on excalidraw, the retrieval-perf benchmark, all pointing at * already-covered types). Don't re-add `member_expression`/`attribute` here. + * VB.NET (#2305) sends the member with its receiver instead, and its resolver + * decides whether the receiver is a type (see extractVbMemberRead). */ const STATIC_MEMBER_LANGS: ReadonlySet = new Set([ - 'java', 'csharp', 'kotlin', 'swift', 'scala', 'dart', 'php', 'cpp', + 'java', 'csharp', 'kotlin', 'swift', 'scala', 'dart', 'php', 'cpp', 'vbnet', ]); +/** + * VB.NET receivers no project type can be named: the namespace roots + * (`System.IO.Path`, `My.Settings`, `Global.X`) and the built-in type keywords + * (`String.Empty`, `Integer.MaxValue`). A read through one is never sent. + */ +const VB_NON_TYPE_RECEIVERS = + /^(?:Global|System|Microsoft|My|Boolean|Byte|Char|Date|Decimal|Double|Integer|Long|Object|SByte|Short|Single|String|UInteger|ULong|UShort)$/i; + /** * Tree-sitter node kinds that represent constructor invocations * (`new Foo()` and friends). Used by extractInstantiation to emit @@ -5549,6 +5559,10 @@ export class TreeSitterExtractor { getChildByField(node, 'scope') ?? node.namedChild(0); if (!recv) return; + if (this.language === 'vbnet') { + this.extractVbMemberRead(node, recv, ownerId); + return; + } const t = recv.type; if ( t === 'identifier' || t === 'type_identifier' || t === 'simple_identifier' || @@ -5569,6 +5583,31 @@ export class TreeSitterExtractor { }); } + /** + * VB.NET: a value read or write through a name — `AppSession.SessionId`, + * `AppSession.CurrentUser = "demo"`, `Logger.Level`, `Mode.Fast` — is a use + * of the member as well as of what the name names (#2305). One `references` + * ref carries both, as `Name.Member` (the receiver kept, as a call's is); + * the resolver links the member and the type when the name means a project + * type or module there, and nothing when it holds a value (a local, a + * parameter, a field), which only it can tell in case-insensitive VB.NET + * (see vbnet-receivers' matchVbMemberRead). Its types are Capitalized all + * the same, so a lowercase receiver — a local, nearly always — is skipped. + */ + private extractVbMemberRead(node: SyntaxNode, recv: SyntaxNode, ownerId: string): void { + const member = getChildByField(node, 'member'); + if (recv.type !== 'identifier' || member?.type !== 'identifier') return; + const name = getNodeText(recv, this.source); + if (!/^[A-Z]\w*$/.test(name) || VB_NON_TYPE_RECEIVERS.test(name)) return; + this.unresolvedReferences.push({ + fromNodeId: ownerId, + referenceName: `${name}.${getNodeText(member, this.source)}`, + referenceKind: 'references', + line: node.startPosition.row + 1, + column: node.startPosition.column, + }); + } + /** * Find a `class_body` child of an `object_creation_expression` — the * marker for an anonymous class (`new T() { ... }`). Returns the body diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 4c9fb45f2c..fcd47b7720 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -25,7 +25,7 @@ import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, is import { isVisibleCppMacro, clearCppMacroVisibility } from './cpp-macro-visibility'; import { isCppConstructorRef, matchCppConstructor } from './cpp-constructor'; import { gateSwiftTypeTarget, clearSwiftTypeVisibility, swiftExtendedConformances } from './swift-type-visibility'; -import { clearVbnetReceiverMemos } from './vbnet-receivers'; +import { clearVbnetReceiverMemos, isVbMemberRead, matchVbMemberRead } from './vbnet-receivers'; import { gateTypeParameter, clearTypeParameterMemos } from './type-parameters'; import { resolveViaImport, resolvePhpImportedStaticCall, resolvePhpQualifiedClassRef, resolveJvmImport, extractImportMappings, extractReExports, loadCppIncludeDirs, isPhpIncludePathRef, isCobolCopybookRef, isNixPathImportRef, isJsPathImportRef, isBoundToOutOfRepoImport, clearImportResolverMemos, resolveImportPath, isExternalImport } from './import-resolver'; import { ResolverPool, minRefsForPool, shouldEngageAdaptively } from './resolver-pool'; @@ -1134,6 +1134,12 @@ export class ReferenceResolver { return this.gateLanguage(matchJsStoreBindingCall(ref, this.context), ref); } + // A VB.NET value read through a name (`AppSession.SessionId`, #2305) means + // what VB.NET's scoping says the name is — a project type, whose member + // and the type itself it links, or a value, which links nothing here — and + // no framework, import or name strategy guesses past that. + if (isVbMemberRead(ref)) return this.gateLanguage(matchVbMemberRead(ref, this.context), ref); + // Function-as-value refs (#756) get a dedicated, strictly-gated path: // import-based resolution first (an imported callback resolves through its // import, the most precise cross-file signal), then matchFunctionRef diff --git a/src/resolution/vbnet-receivers.ts b/src/resolution/vbnet-receivers.ts index 07732927e1..f252dfc979 100644 --- a/src/resolution/vbnet-receivers.ts +++ b/src/resolution/vbnet-receivers.ts @@ -432,9 +432,9 @@ function typesAround(file: string, line: number, context: ResolutionContext): No return types.filter((n) => n.startLine <= line && n.endLine >= line); } -/** The project's VB.NET types (classes, modules, structures, interfaces) named `name`, case aside. */ -function projectTypesNamed(name: string, context: ResolutionContext): Node[] { - return context.getNodesByLowerName(name.toLowerCase()).filter((n) => n.language === 'vbnet' && VB_TYPE_KINDS.has(n.kind)); +/** The project's VB.NET types (classes, modules, structures, interfaces — or `kinds`) named `name`, case aside. */ +function projectTypesNamed(name: string, context: ResolutionContext, kinds: ReadonlySet = VB_TYPE_KINDS): Node[] { + return context.getNodesByLowerName(name.toLowerCase()).filter((n) => n.language === 'vbnet' && kinds.has(n.kind)); } const VB_MEMBERS = new WeakMap>(); @@ -768,11 +768,16 @@ export function isVbTypeQualifiedBy(n: Node, qualifier: string, file: string, co * SCrawler declares * a `SiteSettings` in each site's namespace (`API.Pinterest`, `API.Bluesky`, * …); a member typed `SiteSettings` in `API.Pinterest.UserData` is - * Pinterest's. `ambiguous` when what is left are different types. + * Pinterest's. `ambiguous` when what is left are different types. `kinds` + * widens the types looked for (an `Enum` a value is read through). */ -function typesNamedAt(written: VbType, context: ResolutionContext): { owners: Node[]; ambiguous: boolean } { +function typesNamedAt( + written: VbType, + context: ResolutionContext, + kinds: ReadonlySet = VB_TYPE_KINDS, +): { owners: Node[]; ambiguous: boolean } { const t = written.file ? unalias(written, written.file, context) : written; - let candidates = projectTypesNamed(t.name, context); + let candidates = projectTypesNamed(t.name, context, kinds); // `System.Drawing.Color` is not the project's `Color`; `API.Base.UserDataBase` is that one. if (t.qualifier) candidates = candidates.filter((c) => endsWith(fullSegments(c, context).slice(0, -1), t.qualifier!)); if (candidates.length === 0 || !t.file) return { owners: candidates, ambiguous: false }; @@ -993,9 +998,9 @@ function ownersOf(t: VbType, context: ResolutionContext): Node[] | null { } /** - * A method — or, `withValues`, a field or property — named `name` on one of - * `owners` (the type `typed` names) or a project type they inherit, with what - * the type parameters of the type declaring it stand for. + * A method — or, `withValues`, a field or property; or what `fits` — named + * `name` on one of `owners` (the type `typed` names) or a project type they + * inherit, with what the type parameters of the type declaring it stand for. */ function memberOn( owners: Node[], @@ -1004,8 +1009,8 @@ function memberOn( context: ResolutionContext, withValues: boolean, typed?: VbType, + fits: (n: Node) => boolean = (n) => n.kind === 'method' || (withValues && VB_VALUE_KINDS.has(n.kind)), ): { node: Node; args: Map } | null { - const fits = (n: Node) => n.kind === 'method' || (withValues && VB_VALUE_KINDS.has(n.kind)); for (const level of [0, 1]) { for (const owner of preferVbProject(owners, ref, context)) { // What the owner's own parameters are, as the receiver's type writes them (`Repo(Of Foo)`). @@ -1238,7 +1243,11 @@ export function matchVbTypedCall( if (named.owners.length === 0) return undefined; if (named.ambiguous) return null; const shared = memberOn(named.owners, method, ref, context, false); - return shared ? { original: ref, targetNodeId: shared.node.id, confidence: 0.85, resolvedBy: 'qualified-name' } : null; + if (shared) return { original: ref, targetNodeId: shared.node.id, confidence: 0.85, resolvedBy: 'qualified-name' }; + // `AppSession.Items(0)`: an index into a shared field or property, which + // VB.NET writes as a call — a read of the member and of its type (#2305). + const indexed = memberOn(named.owners, method, ref, context, true); + return indexed ? readThrough(indexed.node, named.owners, ref, context, { edgeKind: 'references' }) : null; } if (!type || typeKey(type) === 'object') return undefined; const owners = ownersOf(type, context); @@ -1251,6 +1260,115 @@ export function matchVbTypedCall( return extensionFor(type, owners, method, ref, context, isStdMethod); } +/** The types a value is read through: those a shared call is made on, and an `Enum`. */ +const VB_READ_TYPE_KINDS: ReadonlySet = new Set([...VB_TYPE_KINDS, 'enum']); + +/** + * What a read through a type names: a value it holds or gives (a field, a + * property, a constant, an `Enum` case, an event), a method — which VB.NET + * runs when it is named without parentheses — or a type nested in it. + */ +const VB_READ_KINDS: ReadonlySet = new Set([...VB_VALUE_KINDS, 'enum_member', 'method', ...VB_TYPE_LIKE_KINDS]); + +/** `Name.Member`, as the extractor sends a VB.NET value read through a name (extractVbMemberRead). */ +const VB_MEMBER_READ = /^([A-Za-z_]\w*)\.(\[?[A-Za-z_]\w*\]?)$/; + +/** Whether a reference is a VB.NET value read through a name, which matchVbMemberRead alone resolves. */ +export function isVbMemberRead(ref: UnresolvedRef): boolean { + return ref.language === 'vbnet' && ref.referenceKind === 'references' && VB_MEMBER_READ.test(ref.referenceName); +} + +/** + * The project types `name` means where a value is read through it + * (`AppSession` in `AppSession.SessionId`): a class, module, structure, + * interface or enum. Null when the name holds a value there — a local, a + * parameter, a member of a type around the read or of a module — or names + * no project type, or two. A member typed as the type of its own name + * (`Public Property Settings As Settings`) reaches that type's members + * either way, as VB.NET's "Color Color" rule has it. + */ +function typesReadThrough(name: string, ref: UnresolvedRef, context: ResolutionContext): Node[] | null { + const bound = receiverType(name, ref, context, 0); + const written: VbType | null = bound === undefined ? { name, array: false, file: ref.filePath, line: ref.line } + : bound && !bound.array && bound.name.toLowerCase() === name.toLowerCase() ? bound : null; + if (!written) return null; + const found = typesNamedAt(written, context, VB_READ_TYPE_KINDS); + return found.ambiguous || found.owners.length === 0 ? null : found.owners; +} + +/** + * A read of `member` through one of `owners` (the parts of the type its + * receiver names): the member, and the type as a second target — unless the + * read is written inside that type, which doesn't depend on itself. + */ +function readThrough( + member: Node, + owners: Node[], + ref: UnresolvedRef, + context: ResolutionContext, + extra: Partial = {}, +): ResolvedRef | null { + if (member.id === ref.fromNodeId) return null; + const owner = owners.find((o) => o.filePath === member.filePath) ?? owners[0]!; + const from = context.getNodeById?.(ref.fromNodeId)?.qualifiedName.toLowerCase(); + const own = owner.qualifiedName.toLowerCase(); + const inside = owner.id === ref.fromNodeId || (from !== undefined && (from === own || from.startsWith(`${own}::`))); + return { + original: ref, + targetNodeId: member.id, + confidence: 0.85, + resolvedBy: 'qualified-name', + ...extra, + ...(inside ? {} : { alsoTargets: [{ targetNodeId: owner.id }] }), + }; +} + +/** Whether a method is named at a reference without being run: `AddressOf Type.Method`, `NameOf(Type.Method)`. */ +function namesWithoutRunning(ref: UnresolvedRef, context: ResolutionContext): boolean { + const line = context.getFileLines?.(ref.filePath)?.[ref.line - 1] ?? context.readFile(ref.filePath)?.split(/\r?\n/)[ref.line - 1] ?? ''; + return /\b(?:AddressOf\s+|NameOf\s*\(\s*)$/i.test(line.slice(0, Math.max(0, ref.column))); +} + +/** + * Resolve a VB.NET value read or write through a name (#2305) — + * `AppSession.SessionId`, `AppSession.CurrentUser = "demo"`, `Logger.Level`, + * `Mode.Fast` — to the member that the type the name means declares or + * inherits, and to the type. A method named this way is called (no type is + * linked, as for a call written with parentheses), unless `AddressOf` or + * `NameOf` only names it. Null when the name holds a value there (an + * instance's members are not read through its type), names no project type, + * or names one without that member: `Color.Red` is not a project `Color`'s. + */ +export function matchVbMemberRead(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + const m = VB_MEMBER_READ.exec(ref.referenceName); + if (!m) return null; + const owners = typesReadThrough(m[1]!, ref, context); + if (!owners) return null; + // Not the member the read is written in: staxrip's `Overrides ReadOnly + // Property Package` returns the class's `Shared ReadOnly Property Package`. + const fits = (n: Node) => VB_READ_KINDS.has(n.kind) && n.id !== ref.fromNodeId; + // `MySettings.Default` reads the property declared `[Default]`, and back. + const name = m[2]!.replace(/^\[(.*)\]$/, '$1'); + let member = (memberOn(owners, name, ref, context, true, undefined, fits) ?? + memberOn(owners, `[${name}]`, ref, context, true, undefined, fits))?.node; + if (!member) return null; + // Of a `Shared` member and an instance one of the same name, a type's name reads the `Shared` one. + if (!member.isStatic) { + const found = member; + member = context.getNodesByQualifiedName(found.qualifiedName).find((n) => n.isStatic && n.filePath === found.filePath && fits(n)) ?? found; + } + if (member.kind === 'method') { + return { + original: ref, + targetNodeId: member.id, + confidence: 0.85, + resolvedBy: 'qualified-name', + ...(namesWithoutRunning(ref, context) ? {} : { edgeKind: 'calls' as const }), + }; + } + return readThrough(member, owners, ref, context, { confidence: 0.9 }); +} + const VB_TYPE_IMPORTS = new WeakMap>>(); /** The last names of a file's `Imports` (a type imported this way lends its shared members; an alias lends none). */ From a192df40f96cd3fee5bc86100e5555eeac9316a7 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 20:06:42 +0000 Subject: [PATCH 179/259] fix(telemetry): count late index uploads, treat usage as live ingest, purge before catch-up (#2333) (#2356) Three bugs in the self-hosted telemetry services (telemetry-worker/ and telemetry-dashboard/), reported against 6560052a and all still on main. 1. An index run that uploads late never counted toward activation. The dashboard's funnel reads machine_first_seen.first_index_day, which only the nightly rollup wrote, and the rollup re-rolls just the last three days plus days with no daily_machines row. Ingest accepts timestamps up to 30 days old and the client keeps an event's original timestamp when it re-queues it, so an index event 4+ days late landed on a day nothing revisited and first_index_day stayed NULL. The ingest upsert of machine_first_seen now lowers first_index_day the way it lowers first_day: coalesce(min(old, new), old, new), because SQLite's min() is NULL if either side is. It is the same statement and row, with no new index and no migration (the column exists since 0002). The rollup still re-derives it from raw events, which covers events stored before this change. 2. ingest_stalled read only max(day) FROM events, but usage counters have lived in usage_daily since 0003, so a day with usage and no lifecycle events read as "The ingest worker ... is not storing anything". /api/meta now also reads max(day) FROM usage_daily (a primary-key lookup) and returns latest_usage_day and latest_ingest_day. Ingest counts as stalled only when there is no lifecycle event from yesterday or later and no usage counter from the day before yesterday or later, because clients upload a day's counters only after that day ends. The banner names latest_ingest_day. 3. runNightly rolled up the 3 regular days, then up to 31 missed days, and ran purgeOldEvents last. Each day first folds its legacy usage_rollup rows, which takes minutes on a heavy day, so a backlog could run past Cloudflare's 15-minute Cron Trigger limit and skip the purge and the summary line night after night. The purge now runs first: it is bounded and touches only days past the window, which no rollup reads. After that, rollups stop starting new days or fold chunks 10 minutes in (NIGHTLY_BUDGET_MS). A day stopped mid-fold gets no rollup at all, so it stays a missed day and the next night continues from its last committed chunk. The summary line counts those days as `deferred`, and `caught_up` now counts missed days actually rolled up. Verification: __tests__/telemetry-services.test.ts (new; the root vitest config picks it up) runs the worker's fetch/cron code and the dashboard API against the checked-in migrations in in-memory node:sqlite through a small D1-shaped adapter. With the source fix reversed, all 5 tests fail (activation 0, first_index_day null, ingest_stalled true, and "nightly rollup incomplete: 4 day(s) failed, purge failed" once statements pass the 15-minute mark). With the fix, all 5 pass. `npm run check` is clean in both packages. Against wrangler dev, run locally only: smoke:rollup 71/71, smoke:cutover 62/62, smoke:api 122/122 and render-check 87/87. smoke (ingest) is 46/47; the one failure is the 70 KB oversized-body curl argument hitting the Windows command-line limit, and main fails it the same way. The new smoke assertions fail on main. Takes effect only once both workers are redeployed; no migration. Co-authored-by: Claude Opus 5.5 --- __tests__/telemetry-services.test.ts | 438 ++++++++++++++++++++++ docs/design/telemetry.md | 6 +- telemetry-dashboard/README.md | 9 +- telemetry-dashboard/public/app.js | 3 +- telemetry-dashboard/scripts/fixture.sql | 1 + telemetry-dashboard/scripts/smoke-api.sh | 14 + telemetry-dashboard/src/api.ts | 51 ++- telemetry-worker/README.md | 32 +- telemetry-worker/scripts/smoke-cutover.sh | 11 +- telemetry-worker/scripts/smoke-ingest.sh | 8 + telemetry-worker/scripts/smoke-rollup.sh | 9 +- telemetry-worker/src/index.ts | 20 +- telemetry-worker/src/rollup.ts | 127 +++++-- 13 files changed, 662 insertions(+), 67 deletions(-) create mode 100644 __tests__/telemetry-services.test.ts diff --git a/__tests__/telemetry-services.test.ts b/__tests__/telemetry-services.test.ts new file mode 100644 index 0000000000..657681cc78 --- /dev/null +++ b/__tests__/telemetry-services.test.ts @@ -0,0 +1,438 @@ +/** + * The self-hosted telemetry services — `telemetry-worker/` (ingest + nightly rollup) + * and `telemetry-dashboard/` (the read API) — run against the checked-in D1 + * migrations in an in-memory node:sqlite database (#2333). + * + * D1 is SQLite, so every statement here is the SQL production runs. The one stand-in + * is a thin adapter giving node:sqlite D1's prepare/bind/first/all/run/batch shape, + * with `batch()` as one transaction the way D1 runs it. No wrangler, no workerd, no + * network: the end-to-end versions of these checks are the packages' own smoke + * suites (`npm run smoke`, `smoke:rollup`, `smoke:cutover`, `smoke:api`), which boot + * `wrangler dev`. + * + * npx vitest run __tests__/telemetry-services.test.ts + */ + +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import worker from '../telemetry-worker/src/index'; +import { runNightly } from '../telemetry-worker/src/rollup'; +import { handleApi } from '../telemetry-dashboard/src/api'; + +let nodeSqlite: typeof import('node:sqlite') | null = null; +try { + nodeSqlite = require('node:sqlite') as typeof import('node:sqlite'); +} catch { + /* Node < 22.5 — skipped below */ +} + +const MIGRATIONS = path.join(__dirname, '..', 'telemetry-worker', 'migrations'); +const MINUTE = 60_000; +/** Cloudflare ends a Cron Trigger invocation after 15 minutes of wall-clock time. */ +const CRON_LIMIT_MS = 15 * MINUTE; + +type Row = Record; + +// --------------------------------------------------------------------------- +// D1 over node:sqlite +// --------------------------------------------------------------------------- + +/** Binds integral numbers as INTEGER, as D1 does (node:sqlite would bind them as REAL). */ +const bindable = (v: unknown): unknown => (typeof v === 'number' && Number.isInteger(v) ? BigInt(v) : v); + +class SqliteD1Statement { + constructor( + private readonly d1: SqliteD1, + readonly sql: string, + private readonly params: unknown[] = [], + ) {} + + bind(...params: unknown[]): SqliteD1Statement { + return new SqliteD1Statement(this.d1, this.sql, params); + } + + execute(): { success: true; results: Row[]; meta: { changes: number } } { + this.d1.onStatement?.(this.sql); + const stmt = this.d1.db.prepare(this.sql); + const params = this.params.map(bindable) as never[]; + if (/^\s*(SELECT|WITH)\b/i.test(this.sql)) { + return { success: true, results: stmt.all(...params) as Row[], meta: { changes: 0 } }; + } + const info = stmt.run(...params); + return { success: true, results: [], meta: { changes: Number(info.changes) } }; + } + + async first(): Promise { + return (this.execute().results[0] as T | undefined) ?? null; + } + + async all(): Promise> { + return this.execute(); + } + + async run(): Promise> { + return this.execute(); + } +} + +class SqliteD1 { + /** Runs before every statement — the cron tests use it to model elapsed wall-clock time. */ + onStatement: ((sql: string) => void) | null = null; + + constructor(readonly db: import('node:sqlite').DatabaseSync) {} + + prepare(sql: string): SqliteD1Statement { + return new SqliteD1Statement(this, sql); + } + + async batch(statements: SqliteD1Statement[]): Promise[]> { + this.db.exec('BEGIN'); + try { + const results = statements.map((s) => s.execute()); + this.db.exec('COMMIT'); + return results; + } catch (err) { + this.db.exec('ROLLBACK'); + throw err; + } + } +} + +// --------------------------------------------------------------------------- +// Harness +// --------------------------------------------------------------------------- + +describe.skipIf(!nodeSqlite)('telemetry services against the D1 schema (#2333)', () => { + let db: import('node:sqlite').DatabaseSync; + let d1: SqliteD1; + let env: never; + let logged: Row[]; + + const at = (iso: string): void => { + vi.setSystemTime(new Date(iso)); + }; + + const one = (sql: string, ...params: unknown[]): Row | undefined => + db.prepare(sql).get(...(params.map(bindable) as never[])) as Row | undefined; + + /** POST /v1/events the way a client does, then wait for the off-response-path write. */ + async function post(machineId: string, events: Row[]): Promise { + const body = JSON.stringify({ + machine_id: machineId, + codegraph_version: '1.6.2', + os: 'linux', + arch: 'x64', + node_major: 22, + ci: false, + schema_version: 2, + events, + }); + const pending: Promise[] = []; + const request = new Request('https://telemetry.test/v1/events', { + method: 'POST', + headers: { 'content-type': 'application/json', 'content-length': String(Buffer.byteLength(body)) }, + body, + }); + const ctx = { waitUntil: (p: Promise) => void pending.push(p), passThroughOnException: () => {} }; + const response = await worker.fetch(request as never, env, ctx as never); + await Promise.all(pending); + return response.status; + } + + /** GET one dashboard endpoint, as the signed-in page does. */ + async function api(pathAndQuery: string): Promise { + const result = await handleApi(env, new URL(`https://stats.test${pathAndQuery}`)); + expect(result.status ?? 200).toBe(200); + return result.body as Row; + } + + /** + * One cron invocation at `iso`. `chunkMs` is the wall-clock time each legacy-usage + * fold chunk is charged; a statement that starts after Cloudflare's 15-minute limit + * throws, the way the runtime ends the invocation there. + */ + async function nightly(iso: string, chunkMs = 0): Promise<{ latestStartMs: number; summary: Row | undefined }> { + at(iso); + const started = Date.now(); + let latestStartMs = 0; + d1.onStatement = (sql) => { + const elapsed = Date.now() - started; + if (elapsed > CRON_LIMIT_MS) { + throw new Error(`statement started ${elapsed / MINUTE} min into the cron run — past the 15-minute limit`); + } + latestStartMs = Math.max(latestStartMs, elapsed); + if (/INSERT INTO usage_daily/.test(sql) && /FROM events/.test(sql)) vi.setSystemTime(Date.now() + chunkMs); + }; + const before = logged.length; + try { + await runNightly(env, Date.parse(iso)); + } finally { + d1.onStatement = null; + } + const summary = logged.slice(before).find((line) => line.msg === 'nightly rollup'); + return { latestStartMs, summary }; + } + + beforeEach(() => { + vi.useFakeTimers({ toFake: ['Date'] }); + db = new nodeSqlite!.DatabaseSync(':memory:'); + for (const file of fs.readdirSync(MIGRATIONS).filter((f) => f.endsWith('.sql')).sort()) { + db.exec(fs.readFileSync(path.join(MIGRATIONS, file), 'utf8')); + } + d1 = new SqliteD1(db); + env = { + DB: d1, + RETENTION_DAYS: 90, + MACHINE_RATE_LIMITER: { limit: async () => ({ success: true }) }, + ADMIN_RATE_LIMITER: { limit: async () => ({ success: true }) }, + } as never; + logged = []; + const capture = (line: unknown): void => { + try { + logged.push(JSON.parse(String(line)) as Row); + } catch { + /* not one of the worker's JSON lines */ + } + }; + vi.spyOn(console, 'log').mockImplementation(capture); + vi.spyOn(console, 'error').mockImplementation(capture); + }); + + afterEach(() => { + vi.restoreAllMocks(); + vi.useRealTimers(); + db.close(); + }); + + const M1 = '00000000-0000-4000-8000-000000000001'; + const M2 = '00000000-0000-4000-8000-000000000002'; + const M3 = '00000000-0000-4000-8000-000000000003'; + const install = (ts: string): Row => ({ event: 'install', ts, props: { scope: 'local', kind: 'fresh' } }); + const index = (ts: string): Row => ({ event: 'index', ts, props: { languages: ['typescript'] } }); + const usage = (day: string, count: number): Row => ({ + event: 'usage_rollup', + ts: `${day}T12:00:00.000Z`, + props: { kind: 'mcp_tool', name: 'codegraph_explore', count, client_name: 'Claude Code' }, + }); + + // ------------------------------------------------------------------------- + // 1. Activation counts an index run that uploads late + // ------------------------------------------------------------------------- + + describe('a late index upload', () => { + it('counts toward activation even when it arrives after the nightly run stopped revisiting its day', async () => { + // Sep 25: the machine installs. Its index run fails to upload and stays in the + // client's queue, keeping its original timestamp. + at('2026-09-25T15:00:00Z'); + expect(await post(M1, [install('2026-09-25T14:00:00Z')])).toBe(204); + await nightly('2026-09-26T00:30:00Z'); + + // Oct 3: the queued index run finally goes out — eight days late, inside the + // 30 days ingest accepts, but long past the three days each nightly run re-rolls. + at('2026-10-03T10:00:00Z'); + expect(await post(M1, [index('2026-09-25T14:05:00Z')])).toBe(204); + // Someone else's activity, so the rollup's coverage moves past the cohort's window. + expect(await post(M2, [install('2026-10-03T09:00:00Z')])).toBe(204); + await nightly('2026-10-04T00:30:00Z'); + + at('2026-10-04T08:00:00Z'); + const funnel = await api('/api/activation?from=2026-09-25&to=2026-09-25&window=7'); + expect(funnel).toMatchObject({ installs: 1, activated: 1, dropped: 0, covered_through: '2026-10-03' }); + expect(one('SELECT first_day, first_index_day FROM machine_first_seen WHERE machine_id = ?', M1)).toEqual({ + first_day: '2026-09-25', + first_index_day: '2026-09-25', + }); + }); + + it('is applied as it is stored, and only ever moves the first index day earlier', async () => { + const firstSeen = (machineId: string): Row | undefined => + one('SELECT first_day, first_index_day FROM machine_first_seen WHERE machine_id = ?', machineId); + + at('2026-10-03T10:00:00Z'); + // A batch without an index run says nothing about indexing. + expect(await post(M1, [install('2026-10-02T10:00:00Z')])).toBe(204); + expect(await post(M1, [usage('2026-10-01', 4)])).toBe(204); + expect(firstSeen(M1)).toEqual({ first_day: '2026-10-01', first_index_day: null }); + + // The earliest index run in a batch wins, with no rollup involved. + expect(await post(M1, [index('2026-10-03T09:00:00Z'), index('2026-10-02T11:00:00Z')])).toBe(204); + expect(firstSeen(M1)).toEqual({ first_day: '2026-10-01', first_index_day: '2026-10-02' }); + + // A later index run leaves it alone; a backdated one lowers it, and first_day with it. + expect(await post(M1, [index('2026-10-03T09:30:00Z')])).toBe(204); + expect(firstSeen(M1)).toEqual({ first_day: '2026-10-01', first_index_day: '2026-10-02' }); + expect(await post(M1, [index('2026-09-20T08:00:00Z')])).toBe(204); + expect(firstSeen(M1)).toEqual({ first_day: '2026-09-20', first_index_day: '2026-09-20' }); + + // A machine whose very first batch carries an index run. + expect(await post(M2, [install('2026-10-03T08:00:00Z'), index('2026-10-03T08:01:00Z')])).toBe(204); + expect(firstSeen(M2)).toEqual({ first_day: '2026-10-03', first_index_day: '2026-10-03' }); + + // The nightly rollup agrees with what ingest already wrote. + await nightly('2026-10-04T00:30:00Z'); + expect(firstSeen(M1)).toEqual({ first_day: '2026-09-20', first_index_day: '2026-09-20' }); + expect(firstSeen(M2)).toEqual({ first_day: '2026-10-03', first_index_day: '2026-10-03' }); + }); + }); + + // ------------------------------------------------------------------------- + // 2. Usage-only days are not a stalled ingest + // ------------------------------------------------------------------------- + + describe('the stalled-ingest flag', () => { + it('counts usage counters as ingest that is still storing', async () => { + // The last lifecycle event is Sep 25; after that only usage counters arrive. + at('2026-09-25T15:00:00Z'); + expect(await post(M1, [install('2026-09-25T14:00:00Z')])).toBe(204); + // Clients upload a day's counters only once it is over: Oct 3's arrive on Oct 4. + at('2026-10-04T06:00:00Z'); + expect(await post(M2, [usage('2026-10-03', 12)])).toBe(204); + + at('2026-10-04T08:00:00Z'); + const meta = await api('/api/meta'); + expect(meta).toMatchObject({ latest_raw_day: '2026-09-25', latest_ingest_day: '2026-10-03', ingest_stalled: false }); + + // Just past midnight nobody has sent Oct 4's counters yet; that is not a stall. + at('2026-10-05T00:10:00Z'); + expect(await api('/api/meta')).toMatchObject({ ingest_stalled: false }); + + // A whole day later with nothing stored at all, it is — and the banner date is + // the last day anything was stored, not the last lifecycle event. + at('2026-10-06T08:00:00Z'); + expect(await api('/api/meta')).toMatchObject({ latest_ingest_day: '2026-10-03', ingest_stalled: true }); + }); + + it('still reports a stall when nothing at all has arrived, and not on an empty database', async () => { + at('2026-10-04T08:00:00Z'); + expect(await api('/api/meta')).toMatchObject({ latest_ingest_day: null, ingest_stalled: false }); + + at('2026-09-25T15:00:00Z'); + expect(await post(M1, [install('2026-09-25T14:00:00Z')])).toBe(204); + at('2026-10-04T08:00:00Z'); + expect(await api('/api/meta')).toMatchObject({ latest_ingest_day: '2026-09-25', ingest_stalled: true }); + + // A lifecycle event from yesterday is fresh, as before. + expect(await post(M1, [index('2026-10-03T22:00:00Z')])).toBe(204); + expect(await api('/api/meta')).toMatchObject({ latest_ingest_day: '2026-10-03', ingest_stalled: false }); + }); + }); + + // ------------------------------------------------------------------------- + // 3. A long catch-up stays inside the cron's wall-clock limit, and the purge runs + // ------------------------------------------------------------------------- + + describe('the nightly run under a catch-up backlog', () => { + /** The rollup folds legacy usage rows 50,000 per transaction (rollup.ts, LEGACY_CHUNK_ROWS). */ + const CHUNK_ROWS = 50_000; + /** + * Each fold chunk is charged 4 minutes, so a backlog of legacy-heavy days costs + * several minutes apiece, as it did in production. The six missed days below hold + * seven chunks: 28 minutes of folding, against a 15-minute limit. + */ + const CHUNK_MS = 4 * MINUTE; + const NIGHT_1 = '2026-10-05T00:30:00Z'; + /** Missed days, newest first — the order catch-up takes them in. Sep 21 needs two chunks. */ + const MISSED = ['2026-09-23', '2026-09-22', '2026-09-21', '2026-09-20', '2026-09-19', '2026-09-18']; + const LEGACY_ROWS: Record = Object.fromEntries( + MISSED.map((day) => [day, day === '2026-09-21' ? CHUNK_ROWS + 3 : 3]), + ); + const PAST_WINDOW = '2026-06-01'; + + /** Legacy rows as they were stored before migrations/0003: one `events` row per upload, count 1. */ + function seedLegacyDay(day: string, rows: number): void { + db.prepare( + `WITH RECURSIVE n(i) AS (SELECT 1 UNION ALL SELECT i + 1 FROM n WHERE i < ?1) + INSERT INTO events (received_at, ts, day, event, machine_id, codegraph_version, os, arch, + node_major, ci, schema_version, props) + SELECT ?2 || 'T13:00:00.000Z', ?2 || 'T12:00:00.000Z', ?2, 'usage_rollup', + '00000000-0000-4000-8000-00000000010' || ((i - 1) % 5), '1.5.0', 'linux', 'x64', 22, 0, 1, + '{"kind":"mcp_tool","name":"codegraph_explore","count":1,"error_count":0,"client_name":"Claude Code"}' + FROM n`, + ).run(BigInt(rows), day); + for (let m = 0; m < Math.min(rows, 5); m++) { + const machine = `00000000-0000-4000-8000-00000000010${m}`; + db.prepare('INSERT INTO machine_days (machine_id, day, prod) VALUES (?, ?, 1)').run(machine, day); + db.prepare( + `INSERT INTO machine_first_seen (machine_id, first_day) VALUES (?, ?) + ON CONFLICT (machine_id) DO UPDATE SET first_day = min(first_day, excluded.first_day)`, + ).run(machine, day); + } + } + + /** Where each seeded usage count sits right now, and whether the day has been rolled up. */ + function ledger(day: string) { + const folded = Number(one('SELECT coalesce(sum(count), 0) AS n FROM usage_daily WHERE day = ?', day)?.n); + const unfolded = Number( + one( + `SELECT coalesce(sum(json_extract(props, '$.count')), 0) AS n FROM events + WHERE day = ? AND event = 'usage_rollup'`, + day, + )?.n, + ); + const rolledUsage = one(`SELECT count FROM daily_event_counts WHERE day = ? AND event = 'usage_rollup'`, day); + const rolledUp = one('SELECT machines FROM daily_machines WHERE day = ?', day) !== undefined; + return { folded, unfolded, rolledUsage: rolledUsage ? Number(rolledUsage.count) : null, rolledUp }; + } + + function expectConsistent(): void { + for (const day of MISSED) { + const l = ledger(day); + // Nothing lost and nothing counted twice, however far the fold got. + expect(l.folded + l.unfolded, day).toBe(LEGACY_ROWS[day]); + if (l.rolledUp) { + expect(l, day).toMatchObject({ unfolded: 0, rolledUsage: LEGACY_ROWS[day] }); + } else { + // A day whose fold did not finish gets no rollup at all, so it stays "missed". + expect(l.rolledUsage, day).toBeNull(); + } + } + } + + beforeEach(() => { + for (const day of MISSED) seedLegacyDay(day, LEGACY_ROWS[day]!); + // Past the 90-day window: the purge must take these. + db.prepare( + `INSERT INTO events (received_at, ts, day, event, machine_id, props) + VALUES ('2026-06-01T10:00:00.000Z', NULL, ?1, 'install', ?2, '{}')`, + ).run(PAST_WINDOW, M3); + db.prepare( + `INSERT INTO usage_daily (day, machine_id, kind, name, count) VALUES (?, ?, 'cli_command', 'index', 2)`, + ).run(PAST_WINDOW, M3); + }); + + it('purges on schedule, stops before the limit, and resumes the backlog the next nights', async () => { + // The three days the run re-rolls every night have ordinary, current traffic. + at('2026-10-04T12:00:00Z'); + expect(await post(M1, [install('2026-10-04T11:00:00Z'), usage('2026-10-03', 5)])).toBe(204); + expect(await post(M2, [index('2026-10-02T11:00:00Z')])).toBe(204); + + const first = await nightly(NIGHT_1, CHUNK_MS); + expect(first.latestStartMs).toBeLessThanOrEqual(CRON_LIMIT_MS); + // The purge ran, and the summary line was written. + expect(one('SELECT count(*) AS n FROM events WHERE day < ?', '2026-07-07')?.n).toBe(0); + expect(one('SELECT count(*) AS n FROM usage_daily WHERE day < ?', '2026-07-07')?.n).toBe(0); + expect(first.summary).toMatchObject({ purged: 1, usage_purged: 1, failed: 0 }); + // Tonight's regular days were rolled up first. + for (const day of ['2026-10-04', '2026-10-03', '2026-10-02']) { + expect(one('SELECT machines FROM daily_machines WHERE day = ?', day), day).toEqual({ machines: 1 }); + } + // The backlog did not fit: Sep 21 was left part-folded and un-rolled, the rest untouched. + expect(ledger('2026-09-21')).toMatchObject({ folded: CHUNK_ROWS, unfolded: 3, rolledUp: false }); + expect(MISSED.filter((day) => !ledger(day).rolledUp)).toEqual(MISSED.slice(2)); + expect(first.summary).toMatchObject({ caught_up: 2, deferred: 4 }); + expectConsistent(); + + // The next nights pick up exactly where this one stopped. + let nights = 1; + for (let day = 6; MISSED.some((d) => !ledger(d).rolledUp) && nights < 5; day++, nights++) { + const night = await nightly(`2026-10-0${day}T00:30:00Z`, CHUNK_MS); + expect(night.latestStartMs).toBeLessThanOrEqual(CRON_LIMIT_MS); + expect(night.summary).toMatchObject({ failed: 0 }); + expectConsistent(); + } + expect(nights).toBe(3); + for (const day of MISSED) expect(ledger(day), day).toMatchObject({ unfolded: 0, rolledUp: true }); + }); + }); +}); diff --git a/docs/design/telemetry.md b/docs/design/telemetry.md index 530600d93c..bf5deed9cc 100644 --- a/docs/design/telemetry.md +++ b/docs/design/telemetry.md @@ -232,9 +232,9 @@ Full documentation is [`telemetry-dashboard/README.md`](../../telemetry-dashboar `telemetry-worker/scripts/smoke-cutover.sh` exists to cover — a mismatch there is silent, showing up as a panel that reads zero forever rather than as an error. - **Reads rollups, not raw events**, so a chart stays correct for days whose raw rows have - been purged. `/api/activation` is the one exception — "did this machine ever run an index" - is not a daily aggregate — so it reads raw `events` and is bounded by the retention window, - which it reports as `raw_events_from`. + been purged. The activation funnel's "did this machine ever run an index" is not a daily + aggregate, so it reads `machine_first_seen.first_index_day` instead, which the ingest Worker + lowers as each index event is stored and the nightly cron re-derives from raw `events`. - **Auth is a shared password and a signed cookie**, sized for exactly two people: `ADMIN_PASSWORD` + `SESSION_SECRET` as Worker secrets, constant-time compare, HMAC-signed cookie with no session store, everything except `/login` and `robots.txt` gated. Rotating diff --git a/telemetry-dashboard/README.md b/telemetry-dashboard/README.md index 26feaac2df..10c1fb3fc2 100644 --- a/telemetry-dashboard/README.md +++ b/telemetry-dashboard/README.md @@ -40,7 +40,7 @@ renders. Bad input is a `400` with a message, never a guess. Chart data carries | Endpoint | Answers | |---|---| -| `/api/meta` | How current the data is: `today`, the rollup's last day (`latest_rollup_day`), the last day an event arrived, machines active yesterday, and two flags — `ingest_stalled` and `rollup_behind` — that put a warning above the panels when either writer stops. Cached 60 s. | +| `/api/meta` | How current the data is: `today`, the rollup's last day (`latest_rollup_day`), the last day ingest stored anything (`latest_ingest_day` — a lifecycle event or a usage counter), machines active yesterday, and two flags — `ingest_stalled` and `rollup_behind` — that put a warning above the panels when either writer stops. Cached 60 s. | | `/api/summary` | Big numbers: production users, active machines, new machines, installs, uninstalls, indexing runs, tool calls. | | `/api/timeseries?metric=` | `installs_uninstalls`, `new_installs`, `production_users`, `indexing_activity`, `tool_calls`, `duration_buckets`. One dense point per day — a day with nothing is a zero, not a gap. Days after the rollup's last day are `null` ("not counted yet") and `covered_through` says where that is; `new_installs` is written live and runs through today. | | `/api/breakdown?dim=` | `os`, `arch`, `codegraph_version`, `node_major`, `language`, `file_count_bucket`, `duration_bucket`, `target`, `scope`, `kind`, `name`, `client_name`, `name_error`. Optional `&event=`, `&metric=count\|machines`, `&limit=`. | @@ -53,7 +53,8 @@ kept forever, so a chart stays correct for days whose raw events have been purge panel reads raw `events`.** D1 runs one query at a time per database: the activation funnel used to join every cohort machine against `events`, which took ~55 s for one week of cohorts in production and failed every panel queued behind it. It now reads -`machine_first_seen.first_index_day`, which the nightly rollup maintains. +`machine_first_seen.first_index_day`, which the ingest worker lowers as each index event is +stored and the nightly rollup re-derives from raw events. The presets end on **today** (UTC — every event and rollup is keyed on the UTC day). Live numbers — production users, new machines, retention — run through today; rolled-up ones @@ -138,6 +139,10 @@ npm run smoke:render # the panels, in a browser (79 assertions) Each suite starts its own throwaway `wrangler dev` on its own port and cleans up after itself, so they can be run in any order (`DASH_PORT` overrides the port). +The repo's own test suite also runs `src/api.ts`, with the ingest worker feeding it, against +the writer's migrations in an in-memory SQLite database — no wrangler needed. From the repo +root (it is part of `npm test` there): `npx vitest run __tests__/telemetry-services.test.ts`. + **`smoke-auth.sh`** is the regression net for the gate: unauthenticated requests reach nothing (pages, API *and* static assets), the cookie is persistent and correctly flagged, flipped/truncated/forged cookies are all rejected, brute force is capped, and rotating the diff --git a/telemetry-dashboard/public/app.js b/telemetry-dashboard/public/app.js index b29506aada..abb2f46efa 100644 --- a/telemetry-dashboard/public/app.js +++ b/telemetry-dashboard/public/app.js @@ -408,7 +408,8 @@ function drawDataWarning() { if (meta?.ingest_stalled) { lines.push([ - `No new events since ${shortDay(meta.latest_raw_day)}.`, + // The last day anything was stored, lifecycle event or usage counter. + `No new events since ${shortDay(meta.latest_ingest_day ?? meta.latest_raw_day)}.`, ' The ingest worker at telemetry.getcodegraph.com is not storing anything; its logs and the D1 database are where to look.', ]); } diff --git a/telemetry-dashboard/scripts/fixture.sql b/telemetry-dashboard/scripts/fixture.sql index 9e2ddef020..e9ca7c151d 100644 --- a/telemetry-dashboard/scripts/fixture.sql +++ b/telemetry-dashboard/scripts/fixture.sql @@ -36,6 +36,7 @@ DELETE FROM daily_machines; DELETE FROM machine_days; DELETE FROM machine_first_seen; DELETE FROM events; +DELETE FROM usage_daily; -- --------------------------------------------------------------------------- -- install — 12, one per machine on its first day diff --git a/telemetry-dashboard/scripts/smoke-api.sh b/telemetry-dashboard/scripts/smoke-api.sh index 67008c47b4..8dda68403b 100755 --- a/telemetry-dashboard/scripts/smoke-api.sh +++ b/telemetry-dashboard/scripts/smoke-api.sh @@ -105,6 +105,7 @@ field "retention window" retention_days 14 "$META" # The fixture stops on 07-10, months ago: nothing has arrived since, but the # rollup did cover every day that saw activity, so it is not behind. field "ingest stalled (no events since 07-10)" ingest_stalled true "$META" +field "…named by the last day anything was stored" latest_ingest_day 2026-07-10 "$META" field "rollup not behind (it covered every active day)" rollup_behind false "$META" field "nobody active yesterday" machines_yesterday 0 "$META" @@ -307,6 +308,19 @@ field "…and yesterday's machine is counted" machines_yesterday 1 "$META" npx wrangler d1 execute codegraph-telemetry --local \ --command "DELETE FROM machine_days WHERE machine_id = '$STALE_ID'" >/dev/null 2>&1 +echo +echo "Usage counters alone are not a stalled ingest" +# Usage lands in usage_daily, not events, so a day of tool calls with no install or +# index run must not read as "not storing anything". Removed again straight after. +npx wrangler d1 execute codegraph-telemetry --local \ + --command "INSERT INTO usage_daily (day, machine_id, kind, name, count) VALUES ('$YESTERDAY', '$STALE_ID', 'mcp_tool', 'codegraph_explore', 3)" >/dev/null 2>&1 +META="$(get "/api/meta")" +field "not stalled while usage arrives" ingest_stalled false "$META" +field "…the usage day is the last day stored" latest_ingest_day "$YESTERDAY" "$META" +field "…though the last lifecycle event is 07-10" latest_raw_day 2026-07-10 "$META" +npx wrangler d1 execute codegraph-telemetry --local \ + --command "DELETE FROM usage_daily WHERE machine_id = '$STALE_ID'" >/dev/null 2>&1 + echo printf '%d passed, %d failed\n' "$PASS" "$FAIL" [[ "$FAIL" -eq 0 ]] diff --git a/telemetry-dashboard/src/api.ts b/telemetry-dashboard/src/api.ts index 63b77556c6..24deb33522 100644 --- a/telemetry-dashboard/src/api.ts +++ b/telemetry-dashboard/src/api.ts @@ -6,8 +6,8 @@ * - **Rollups only.** Every panel is answered from `daily_*`, `machine_days` and * `machine_first_seen`, which are kept forever. No panel reads raw `events`: * D1 runs one query at a time per database, so one slow scan there fails every - * panel queued behind it. (/api/meta reads the table's first and last day, one - * indexed lookup each.) + * panel queued behind it. (/api/meta reads the table's first and last day, and + * `usage_daily`'s last, one indexed lookup each.) * - **Today is in range; uncounted days are not zeros.** Rolled-up numbers stop * at the nightly rollup's last day and come back null after it, so a chart * ending today draws a gap where the count has not happened yet, not a cliff. @@ -214,6 +214,7 @@ interface MetaRow { earliest_active_day: string | null; earliest_raw_day: string | null; latest_raw_day: string | null; + latest_usage_day: string | null; machines_yesterday: number | null; } @@ -246,6 +247,7 @@ async function meta(env: Env): Promise { (SELECT min(day) FROM machine_days) AS earliest_active_day, (SELECT min(day) FROM events) AS earliest_raw_day, (SELECT max(day) FROM events) AS latest_raw_day, + (SELECT max(day) FROM usage_daily) AS latest_usage_day, (SELECT count(*) FROM machine_days WHERE day = ?) AS machines_yesterday`, ) .bind(yesterday) @@ -253,11 +255,22 @@ async function meta(env: Env): Promise { const latestRollup = row?.latest_rollup_day ?? null; const latestRaw = row?.latest_raw_day ?? null; + const latestUsage = row?.latest_usage_day ?? null; const latestActive = row?.latest_active_day ?? null; - - // Nothing at all since before yesterday. (Client clocks may run a few minutes - // ahead, so the latest day can be tomorrow — that is fresh, not stale.) - const ingestStalled = latestRaw !== null && latestRaw < yesterday; + // Ingest stores lifecycle events in `events` and usage counters in `usage_daily`, so + // either one arriving means it is working — a day of usage and no installs or index + // runs is quiet, not stalled. + const latestIngest = + latestUsage !== null && (latestRaw === null || latestUsage > latestRaw) ? latestUsage : latestRaw; + + // Stalled: nothing stored since before yesterday. (Client clocks may run a few + // minutes ahead, so the latest day can be tomorrow — that is fresh, not stale.) Usage + // counters get a day more, because a client uploads a day's counters only once that + // day is over: just after midnight UTC the newest one can be the day before yesterday. + const ingestStalled = + latestIngest !== null && + (latestRaw === null || latestRaw < yesterday) && + (latestUsage === null || latestUsage < addDays(today, -2)); // The 00:30 UTC run rolls up yesterday, so the day before that must always be in // by now. Only "behind" if there was activity after the last rolled-up day — a day // nobody used codegraph would be a silent rollup, not a missed one. @@ -274,7 +287,12 @@ async function meta(env: Env): Promise { latest_rollup_day: latestRollup, latest_active_day: latestActive, earliest_raw_day: row?.earliest_raw_day ?? null, + /** The last day of a stored lifecycle event (install, index, uninstall). */ latest_raw_day: latestRaw, + /** The last day of a stored usage counter. */ + latest_usage_day: latestUsage, + /** The later of the two: the last day ingest stored anything, which the stall banner names. */ + latest_ingest_day: latestIngest, machines_yesterday: row?.machines_yesterday ?? 0, rollup_behind: rollupBehind, ingest_stalled: ingestStalled, @@ -670,17 +688,18 @@ interface ActivationRow { * reinstalls does not re-enter the funnel, which is what makes this a * conversion rate rather than an install-event ratio. * - * "Ran an index" is `machine_first_seen.first_index_day`, which the nightly rollup - * keeps at the earliest day each machine indexed. That makes this a range read over - * one small table. It used to be a join against raw `events` — on production volume - * ~55 s per week of cohorts, which held D1's single query lane long enough to fail - * every other panel waiting behind it. A first index day is never before the first - * day (the ingest path keeps first_day at the machine's earliest event), so "within - * the window" is just `first_index_day <= first_day + window`. + * "Ran an index" is `machine_first_seen.first_index_day`, the earliest day each + * machine indexed. The ingest worker lowers it as each index event is stored — so a + * run that uploads days late still counts — and the nightly rollup re-derives it from + * raw events. That makes this a range read over one small table. It used to be a + * join against raw `events` — on production volume ~55 s per week of cohorts, which + * held D1's single query lane long enough to fail every other panel waiting behind + * it. A first index day is never before the first day (the ingest path keeps + * first_day at the machine's earliest event), so "within the window" is just + * `first_index_day <= first_day + window`. * - * Because first_index_day is rolled up, cohorts after the rollup's last day have no - * conversions counted yet. They are left out of the totals and drawn as gaps — - * counting them would show a drop in conversion that is only a lag. + * Cohorts after the rollup's last day are left out of the totals and drawn as gaps, + * like every other rolled-up number on the page. */ async function activation(env: Env, url: URL, range: Range): Promise { const rawWindow = url.searchParams.get('window'); diff --git a/telemetry-worker/README.md b/telemetry-worker/README.md index 296ec2251d..077d0812f0 100644 --- a/telemetry-worker/README.md +++ b/telemetry-worker/README.md @@ -80,7 +80,8 @@ nothing looks broken from the outside. That ran from 2026-08-11 to October 2026. ## Rollups & retention (nightly cron) -`src/rollup.ts` runs on a Cron Trigger at **00:30 UTC** and does two things. +`src/rollup.ts` runs on a Cron Trigger at **00:30 UTC** and does two things, the purge +first. **Rolls up** the day that just ended into `daily_machines`, `daily_event_counts`, `daily_dim_counts` and `machine_first_seen.first_index_day`, then re-runs the two days before @@ -88,7 +89,12 @@ it — offline clients ship completed-day rollups late, so a day keeps growing a It then **catches up** on any earlier day that saw activity but never got a rollup (a `machine_days` day with no `daily_machines` row — a night the run failed or the database refused writes), newest first, up to 31 a night. An outage heals on the first good night -instead of leaving a hole someone has to notice. The aggregation is one +instead of leaving a hole someone has to notice. Cloudflare ends a Cron Trigger after 15 +minutes of wall-clock time, and a day that still holds millions of legacy usage rows (below) +takes minutes, so the run starts no new day and no new fold chunk once 10 minutes have +passed (`NIGHTLY_BUDGET_MS`). A day it stops partway gets no rollup at all, so it is still a +missed day the next night, which carries on from the last folded chunk; the summary line +counts those days as `deferred`. The aggregation is one `INSERT … SELECT … ON CONFLICT DO UPDATE` per table or dimension, so it happens inside D1 and no event row crosses the wire. Every write overwrites the recomputed value rather than adding to it: **re-running a day is a no-op, never a double count.** Two things the SQL is careful @@ -100,7 +106,9 @@ rollup over old days is the whole migration. Adding a breakdown is a line in `RO never a migration — that is what the generic `(dim, value)` shape buys. **Purges** raw `events` and `usage_daily` rows older than `RETENTION_DAYS` (90, a var in -`wrangler.jsonc`) in bounded `DELETE` batches, and logs one line of counts. `machine_days` and `machine_first_seen` are +`wrangler.jsonc`) in bounded `DELETE` batches, before any rollup: behind a long catch-up it +would be the part the 15-minute limit cuts off, night after night, while the database +grows. The run ends with one line of counts. `machine_days` and `machine_first_seen` are never purged — retention cohorts need the full history and they are two orders of magnitude smaller. Rollups are kept forever, so shortening the window costs ad-hoc drill-back, never a chart. @@ -122,10 +130,13 @@ each day is a full scan of that day's events, and the request has a wall-clock b ### Backfilling first_index_day -`first_index_day` (migration `0002`) is what the dashboard's activation funnel reads, and only -the rollup writes it. Days rolled up before the migration left it NULL, and days stored before +`first_index_day` (migration `0002`) is what the dashboard's activation funnel reads. The +ingest worker lowers it as each index event is stored, so an index run that uploads days late +still counts, and the rollup re-derives it from the raw events of every day it rolls up. Days +rolled up before the migration left it NULL, an index event stored before the ingest worker +began setting it (#2333) counts only once its day is rolled up again, and days stored before `0003` still hold their usage as one `events` row per upload. Re-running the rollup over every -day that still has raw events fixes both — it folds that day's legacy usage rows into +day that still has raw events fixes all three — it folds that day's legacy usage rows into `usage_daily`, sets `first_index_day`, and recomputes the day's rollups. It is idempotent, so overlapping or repeating a range is harmless. A day with millions of legacy rows takes several minutes, so go a day at a time: @@ -300,6 +311,15 @@ To drive the cron body by hand, run `wrangler dev --test-scheduled` and hit `.dev.vars.example` to `.dev.vars` — without an `ADMIN_TOKEN` the route 404s, exactly as a deploy that never set the secret does. +The repo's own test suite also covers this worker and the dashboard API without wrangler: +it runs their source against these migrations in an in-memory SQLite database, including +late uploads and a nightly run whose catch-up outlasts the cron's time limit. From the repo +root (it is part of `npm test` there): + +```bash +npx vitest run __tests__/telemetry-services.test.ts +``` + ## Changing the schema The allowlist in `src/index.ts` mirrors `docs/design/telemetry.md` (and the user-facing diff --git a/telemetry-worker/scripts/smoke-cutover.sh b/telemetry-worker/scripts/smoke-cutover.sh index f0dff12443..6d55f8c79a 100755 --- a/telemetry-worker/scripts/smoke-cutover.sh +++ b/telemetry-worker/scripts/smoke-cutover.sh @@ -235,11 +235,14 @@ ts "tool calls (sums the prop)" tool_calls '[20]' '[2]' MET=$(api "meta") is "meta reports the rolled-up day" "$DAY" "$(jget "$MET" latest_day)" is "meta reports the rollup ran" "$DAY" "$(jget "$MET" latest_rollup_day)" +# Usage lands in usage_daily rather than events; the stalled-ingest check reads both. +is "meta reports the last day ingest stored" "$DAY" "$(jget "$MET" latest_ingest_day)" -# The funnel reads machine_first_seen.first_index_day, which only the nightly -# rollup writes (from raw `index` events). This is the seam that pins it: ingest -# writes the events, the rollup sets the column, the dashboard reads it. If the -# rollup stopped setting it, "activated" here would read 0. +# The funnel reads machine_first_seen.first_index_day. The ingest worker sets it as +# each index event is stored, and the nightly rollup re-derives it from raw `index` +# events (smoke-rollup.sh pins that half, on events it seeds past the ingest path). +# This is the seam: ingest writes the column, the dashboard reads it. If neither +# writer set it, "activated" here would read 0. # # Its denominator is FIRST-SEEN MACHINES, not `install` events (api.ts: "a machine # that reinstalls does not re-enter the funnel"). m3 is the discriminator: it never diff --git a/telemetry-worker/scripts/smoke-ingest.sh b/telemetry-worker/scripts/smoke-ingest.sh index 458123443a..9fd2b87fe4 100755 --- a/telemetry-worker/scripts/smoke-ingest.sh +++ b/telemetry-worker/scripts/smoke-ingest.sh @@ -214,6 +214,14 @@ is "machine_days: each backdated batch gets its own day" "$BACK_DAY,$RECENT_DAY" is "machine_first_seen recorded" "$RECENT_DAY" "$(q "select first_day from machine_first_seen where machine_id='$M_OK'")" is "machine_first_seen only moves earlier" "$BACK_DAY" \ "$(q "select first_day from machine_first_seen where machine_id='$M_BACK'")" +# The activation funnel's input is kept here too, not only by the nightly rollup: an +# index run that uploads after the cron stopped re-rolling its day must still count. +is "first_index_day recorded as the index run is stored" "$RECENT_DAY" \ + "$(q "select first_index_day from machine_first_seen where machine_id='$M_OK'")" +is "first_index_day only moves earlier, like first_day" "$BACK_DAY" \ + "$(q "select first_index_day from machine_first_seen where machine_id='$M_BACK'")" +is "a machine that never sent an index run has none" "null" \ + "$(q "select coalesce(first_index_day, 'null') from machine_first_seen where machine_id='$M_USE'")" echo echo "usage counters add up instead of piling up" diff --git a/telemetry-worker/scripts/smoke-rollup.sh b/telemetry-worker/scripts/smoke-rollup.sh index 294b4378f8..2befab4eb8 100755 --- a/telemetry-worker/scripts/smoke-rollup.sh +++ b/telemetry-worker/scripts/smoke-rollup.sh @@ -16,7 +16,8 @@ # * the purge deletes only rows past the window, and leaves machine_days / # machine_first_seen alone # * each machine's first_index_day is its earliest index, and survives the purge -# * the cron catches up on a day that saw activity but was never rolled up +# * the cron catches up on a day that saw activity but was never rolled up, and +# logs its one summary line # * usage comes from usage_daily; a legacy usage row still in `events` is folded # into it (counts added, row deleted) before the day is rolled up # * /admin/rollup does not exist without ADMIN_TOKEN, and rejects a wrong one @@ -196,6 +197,12 @@ is "rollup $DAY_RESET with reset → 200" 200 "$(roll "day=$DAY_RESET&reset=1")" # rolled up), and purges everything past the window. is "cron trigger → 200" 200 "$(curl -s -o /dev/null -w '%{http_code}' "$BASE/__scheduled?cron=30+0+*+*+*")" sleep 2 +# Its one summary line: written once the purge and every rollup are done, and with +# nothing this small left for the next night by the time budget. +is "the cron logs its summary, nothing failed or deferred" "0/0" \ + "$(grep -ao '{"msg":"nightly rollup".*}' "$LOG" | tail -1 | + node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{ + const r=JSON.parse(s||"{}");console.log(`${r.failed}/${r.deferred}`);})')" # Rolling a purged day with reset=1 must NOT blank the rollups it already has: past # the window the reset is ignored, so the delete-then-rebuild can't find zero events. diff --git a/telemetry-worker/src/index.ts b/telemetry-worker/src/index.ts index 9a6efa3394..3c4ac7cde3 100644 --- a/telemetry-worker/src/index.ts +++ b/telemetry-worker/src/index.ts @@ -200,9 +200,18 @@ const INSERT_EVENT = `INSERT INTO events ( const UPSERT_MACHINE_DAY = `INSERT INTO machine_days (machine_id, day, prod) VALUES (?, ?, ?) ON CONFLICT (machine_id, day) DO UPDATE SET prod = max(machine_days.prod, excluded.prod)`; -// A late-arriving offline buffer can move a machine's first day earlier, never later. -const UPSERT_FIRST_SEEN = `INSERT INTO machine_first_seen (machine_id, first_day) VALUES (?, ?) - ON CONFLICT (machine_id) DO UPDATE SET first_day = min(machine_first_seen.first_day, excluded.first_day)`; +// A late-arriving offline buffer can move a machine's first day earlier, never later — +// and its first index day (the activation funnel's input) the same way. That one is set +// here as well as by the nightly rollup because the rollup only re-rolls the last three +// days, while this path accepts events up to 30 days old: an index run that uploads a +// week late would otherwise never count. SQLite's multi-argument min() is NULL when +// either side is, hence the coalesce: a batch with no index run leaves the stored day +// alone, and the first index run on record sets it. +const UPSERT_FIRST_SEEN = `INSERT INTO machine_first_seen (machine_id, first_day, first_index_day) VALUES (?, ?, ?) + ON CONFLICT (machine_id) DO UPDATE SET + first_day = min(machine_first_seen.first_day, excluded.first_day), + first_index_day = coalesce(min(machine_first_seen.first_index_day, excluded.first_index_day), + machine_first_seen.first_index_day, excluded.first_index_day)`; // usage_rollup counters ADD into one row per machine × day × tool (migrations/0003): // clients upload the same counter many times over — once per process — so storing @@ -305,10 +314,13 @@ async function writeToD1( } } + // The earliest day this batch indexed on, if it did — see UPSERT_FIRST_SEEN. + let firstIndexDay: string | null = null; for (const e of batch) { if (e.event === 'usage_rollup') continue; const day = (e.ts ?? receivedAt).slice(0, 10); days.add(day); + if (e.event === 'index' && (firstIndexDay === null || day < firstIndexDay)) firstIndexDay = day; stmts.push( insertEvent.bind( receivedAt, @@ -328,7 +340,7 @@ async function writeToD1( const firstDay = [...days].sort()[0]; if (firstDay !== undefined) { - stmts.push(env.DB.prepare(UPSERT_FIRST_SEEN).bind(machineId, firstDay)); + stmts.push(env.DB.prepare(UPSERT_FIRST_SEEN).bind(machineId, firstDay, firstIndexDay)); } await env.DB.batch(stmts); diff --git a/telemetry-worker/src/rollup.ts b/telemetry-worker/src/rollup.ts index 87072a59a9..9502210984 100644 --- a/telemetry-worker/src/rollup.ts +++ b/telemetry-worker/src/rollup.ts @@ -9,12 +9,13 @@ * 1. ROLL UP the just-completed UTC day into `daily_machines`, `daily_event_counts`, * `daily_dim_counts` and `machine_first_seen.first_index_day` — plus the two days * before it, because clients buffer offline and ship completed-day rollups late, so - * a day keeps growing after it ends, plus any earlier day a failed run missed. Every - * write is an upsert that OVERWRITES the recomputed value rather than adding to it - * (or, for first_index_day, only ever lowers it), so re-running a day is a no-op - * and never double-counts. + * a day keeps growing after it ends, plus any earlier day a failed run missed, as + * many as fit in the run's time budget. Every write is an upsert that OVERWRITES + * the recomputed value rather than adding to it (or, for first_index_day, only ever + * lowers it), so re-running a day is a no-op and never double-counts. * - * 2. PURGE raw `events` past the retention window, in bounded batches. Rollups are + * 2. PURGE raw `events` past the retention window, in bounded batches — first, so a + * long catch-up can never push it past the cron's wall-clock limit. Rollups are * kept forever, so only ad-hoc drill-down has a horizon; `machine_days` and * `machine_first_seen` are never purged, because retention cohorts need the full * history and they are two orders of magnitude smaller than the raw rows. @@ -32,6 +33,15 @@ export const ROLLUP_LOOKBACK_DAYS = 3; export const MAX_MANUAL_DAYS = 31; /** Most missed days one nightly run catches up on, newest first; the next night takes the rest. */ export const MAX_CATCHUP_DAYS = 31; +/** + * How long into a nightly run it still starts rollup work. Cloudflare ends a Cron + * Trigger after 15 minutes of wall-clock time, and catching up a day that still holds + * millions of legacy usage rows takes minutes, so a backlog can run straight into that + * limit. Past this point the run starts no new day and no new fold chunk; the 5 + * minutes left cover the one statement already in flight and the summary line. What + * did not fit stays a missed day, so the next night carries on from there. + */ +export const NIGHTLY_BUDGET_MS = 10 * 60_000; /** Rows per purge DELETE — bounded so one statement stays well inside D1's limits. */ const PURGE_BATCH_ROWS = 5_000; @@ -148,6 +158,10 @@ const DAILY_MACHINES = `INSERT INTO daily_machines (day, machines, prod_machines * it straight off `machine_first_seen`, which keeps the funnel off raw `events` (a * cohort join there took most of a minute per week of cohorts) and past the purge. * + * The ingest path lowers it the same way as each index event is stored (src/index.ts), + * because a run that uploads late lands on a day this cron has stopped re-rolling. This + * statement is what fills it for events stored before that, and what a backfill re-runs. + * * Only rows that actually move are written, so re-running a day is free. `?1` is the * day — still the one bound parameter, used three times. */ @@ -221,6 +235,11 @@ export interface DayResult { rows: number; /** Day is past the retention window — a `reset` on it is ignored (see below). */ pastRetention: boolean; + /** + * The deadline passed before the day's legacy usage rows were all folded, so nothing + * was rolled up: the next run finishes the fold and rolls the day up then. + */ + deferred: boolean; } /** @@ -236,14 +255,20 @@ export interface DayResult { * exists would otherwise linger. It is IGNORED past the retention window, where it * would delete rows and then find no events to rebuild them from: silently blanking a * real day is the one irreversible thing this file could do. + * + * `deadline` (epoch ms) bounds the legacy fold that runs first. If it stops the fold + * partway, the day is not rolled up at all — usage counted from a half-folded day would + * be short, and its `daily_machines` row would stop the nightly run from ever coming + * back to it. The chunks already folded are committed, so the next call carries on. */ export async function rollupDay( env: Env, day: string, - opts: { cutoff: string; reset?: boolean }, + opts: { cutoff: string; reset?: boolean; deadline?: number }, ): Promise { const pastRetention = day < opts.cutoff; - await foldLegacyUsage(env, day); + const fold = await foldLegacyUsage(env, day, opts.deadline); + if (!fold.complete) return { day, rows: 0, pastRetention, deferred: true }; const statements: D1PreparedStatement[] = []; if (opts.reset && !pastRetention) { @@ -257,7 +282,7 @@ export async function rollupDay( const results = await env.DB.batch(statements); const rows = results.reduce((total, r) => total + (r.meta?.changes ?? 0), 0); - return { day, rows, pastRetention }; + return { day, rows, pastRetention, deferred: false }; } // --------------------------------------------------------------------------- @@ -298,15 +323,32 @@ const LEGACY_FOLD = `INSERT INTO usage_daily ( const LEGACY_DELETE = `DELETE FROM events WHERE day = ?1 AND event = 'usage_rollup' AND id < ?2`; -/** Folds any legacy usage rows for `day` into usage_daily. Returns how many rows it moved. */ -export async function foldLegacyUsage(env: Env, day: string): Promise { +export interface FoldResult { + /** Legacy rows moved into usage_daily by this call. */ + moved: number; + /** False when the deadline stopped it with rows still to fold; the next call carries on. */ + complete: boolean; +} + +/** + * Folds any legacy usage rows for `day` into usage_daily, a chunk per transaction. + * With a `deadline` (epoch ms) it starts no chunk once that has passed: each chunk + * commits on its own and always takes the oldest rows left, so stopping between two + * loses nothing and the next call simply continues. + */ +export async function foldLegacyUsage( + env: Env, + day: string, + deadline = Number.POSITIVE_INFINITY, +): Promise { const any = await env.DB.prepare(`SELECT 1 AS hit FROM events WHERE day = ? AND event = 'usage_rollup' LIMIT 1`) .bind(day) .first<{ hit: number }>(); - if (!any) return 0; + if (!any) return { moved: 0, complete: true }; let moved = 0; for (let chunk = 0; chunk < LEGACY_MAX_CHUNKS; chunk++) { + if (Date.now() >= deadline) return { moved, complete: false }; // Always the oldest remaining rows: everything below the next chunk's first id. // Rows folded by earlier chunks are gone, so there is no cursor to carry. const next = await env.DB.prepare( @@ -320,7 +362,7 @@ export async function foldLegacyUsage(env: Env, day: string): Promise { env.DB.prepare(LEGACY_DELETE).bind(day, below), ]); moved += deleted?.meta?.changes ?? 0; - if (!next) return moved; + if (!next) return { moved, complete: true }; } throw new Error(`legacy usage fold for ${day} did not finish within ${LEGACY_MAX_CHUNKS} chunks`); } @@ -399,56 +441,81 @@ async function missedDays(env: Env, cutoff: string, before: string): Promise { const started = Date.now(); + const deadline = started + NIGHTLY_BUDGET_MS; const keepDays = retentionDays(env); const cutoff = retentionCutoff(atMs, keepDays); - const days: string[] = []; - for (let back = 1; back <= ROLLUP_LOOKBACK_DAYS; back++) days.push(utcDay(atMs - back * DAY_MS)); + let purge: PurgeResult | null = null; + try { + purge = await purgeOldEvents(env, cutoff); + } catch (err) { + console.error(JSON.stringify({ msg: 'purge failed', cutoff, err: String(err) })); + } + + const regular: string[] = []; + for (let back = 1; back <= ROLLUP_LOOKBACK_DAYS; back++) regular.push(utcDay(atMs - back * DAY_MS)); // A failure to find the missed days must not cost tonight's regular rollup. - let caughtUp = 0; + let missed: string[] = []; try { - const missed = await missedDays(env, cutoff, days[days.length - 1] ?? utcDay(atMs)); - days.push(...missed); - caughtUp = missed.length; + missed = await missedDays(env, cutoff, regular[regular.length - 1] ?? utcDay(atMs)); } catch (err) { console.error(JSON.stringify({ msg: 'missed-day scan failed', err: String(err) })); } const rolled: string[] = []; const failed: string[] = []; + const deferred: string[] = []; + let caughtUp = 0; let rows = 0; - for (const day of days) { + for (const day of [...regular, ...missed]) { + if (Date.now() >= deadline) { + deferred.push(day); + continue; + } try { - rows += (await rollupDay(env, day, { cutoff })).rows; + const result = await rollupDay(env, day, { cutoff, deadline }); + if (result.deferred) { + deferred.push(day); + continue; + } + rows += result.rows; rolled.push(day); + if (!regular.includes(day)) caughtUp++; } catch (err) { failed.push(day); console.error(JSON.stringify({ msg: 'rollup day failed', day, err: String(err) })); } } - let purge: PurgeResult | null = null; - try { - purge = await purgeOldEvents(env, cutoff); - } catch (err) { - console.error(JSON.stringify({ msg: 'purge failed', cutoff, err: String(err) })); - } - console.log( JSON.stringify({ msg: 'nightly rollup', days: rolled, + /** Missed days rolled up tonight. */ caught_up: caughtUp, + /** Days the time budget left for the next run. */ + deferred: deferred.length, rows, failed: failed.length, retention_days: keepDays, From 8998697f1d87fc670771817a11943868d4821e1a Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 20:17:51 +0000 Subject: [PATCH 180/259] fix(vue,ts): link calls in Vue templates and in top-level destructuring (#2340) (#2357) `codegraph callers ` missed most of the components using it in a Nuxt app (Issue #2340), through two extraction gaps: - A destructuring declaration at module or ` +`; + const code = ending === 'CRLF' ? lf.replace(/\n/g, '\r\n') : lf; + const result = extractFromSource('Card.vue', code); + const component = result.nodes.find((n) => n.kind === 'component')!; + const calls = result.unresolvedReferences + .filter((r) => r.referenceKind === 'calls') + .map((r) => `${r.fromNodeId === component.id ? 'component' : r.fromNodeId}:${r.referenceName}@${r.line}`) + .sort(); + expect(calls).toEqual([ + 'component:label@3', + 'component:useBar@2', + 'component:useBar@2', + 'component:useFoo@7', + 'component:useFoo@8', + ]); + }); + it('should extract calls from Vue Options API object methods', () => { const code = `