Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

### Fixes

- **Indexing a project with a bundled minified JavaScript file is faster.** For each reference in a one-line file such as `jquery.min.js`, the resolver counted characters from the start of the line to find the reference's column, many times per reference. It now skips straight to the column when the text before it is ASCII. Alamofire, whose docs folder carries jQuery, resolves its references in a median 0.93 s instead of 1.25 s (profiling runs on a busy host, 5,614 nodes and 13,673 edges either way). Separately, the resolver now looks nodes up by query only when there are fewer than 15,000 pending references and fewer than 0.3 per node; at either limit it loads the node table. A full index of a small project such as gin (about 4.4 references per node) therefore loads the table, saving about 60 ms of its one-second index, while a one-file sync still queries.
- **Code written without a space before a name resolves like spaced code.** Where one token ends exactly where the next begins, the resolver read the syntax at that point as the earlier token: in `x=1;local function helper()` it saw the `;`, so the Lua local read as global and calls from other files linked to it, and in `if ($x instanceof Foo)$x->bar();` it saw the `)`, so the PHP guard did not type `$x`. It now reads the token that starts there, the same way the TypeScript side already did. On json, jq and CPython the graph is unchanged.
- **A Kotlin infix call on a parenthesized receiver now reaches the right class.** `(b) put 1`, `(b)put(1)` and `((b)) put (1)` linked to whichever class had a `put`, so with `b: A` and another class `B` that also declared `put`, the call landed on `B::put`. They now follow what `b` holds, as `b put 1` already did. An extension on a project class called through a typed receiver is still not linked, as before; the parenthesized form no longer guesses it by name.
- **A Vite plugin that spreads another plugin no longer turns off a React Router app's file routes.** Any `...` in a root `vite.config`, `react-router.config` or `remix.config` was taken for custom route configuration, so React Router's own RSC playground, which merges `...mdx({ … })` into a plugin entry, got no file routes. A spread of a call inside a `plugins` list is now ignored, since it only builds a plugin; a spread of anything else (`...shared`, `...makeConfig()` at the top level, `[...plugins]`, or inside a plugin call's arguments) still turns file routes off. The playground goes from 0 to 12 file routes.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,7 @@ The [benchmark](#benchmark-results) and [speed](#built-for-speed--the-rust-kerne

- **Larger database:** 11–21% bigger on six of the seven corpora above, and 37% on supabase, which has 1,978 Markdown files. It holds Markdown, binding rows and more nodes.
- **Fewer edges on some projects:** 3–5% fewer on pretix and CPython, because the fork declines links it cannot confirm. Some of those were correct links.
- **Slower full index on small projects:** on Node, gin and Alamofire take 0.14–0.43 s longer to index than on upstream. The cause has not been found ([open question](docs/design/open-questions.md#slower-full-index-on-small-projects)).
- **Slower full index on small projects:** at `34cc55e` (the measured build), on Node, gin and Alamofire took 0.14–0.43 s longer to index than on upstream. Two causes were found and fixed afterwards: the resolver walked a minified one-line JavaScript file's text from its start for every reference in it (Alamofire's bundled jQuery was 40% of its resolution time), and a project under about 15,000 references looked nodes up by query where loading the table is faster. In a profiling run, not the benchmark, Alamofire's resolution step fell from a median of 1.25 s to 0.93 s with the same graph. The benchmark has not been re-run.
- **On Bun:** slower concurrent explores and higher idle CPU, as above.

---
Expand Down
34 changes: 34 additions & 0 deletions __tests__/resolution-query-lookups.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
/**
* A kernel over the live db either loads the whole node table or answers its
* node lookups by query. The load costs per node and the queries per ref, so
* the choice depends on both: a small project's full index must load the table
* (querying cost gin about 60 ms of a 1 s index), while a one-file sync on a
* large project must still query (the load is about 150 ms on ktor).
*/
import { describe, it, expect } from 'vitest';
import { ReferenceResolver } from '../src/resolution/index';

function queriesFor(nodeCount: number): ReferenceResolver {
const resolver = Object.create(ReferenceResolver.prototype) as ReferenceResolver;
(resolver as unknown as { queries: { getNodeCount(): number } }).queries = { getNodeCount: () => nodeCount };
return resolver;
}

const shouldQuery = (nodeCount: number, refs: number): boolean =>
(queriesFor(nodeCount) as unknown as { shouldQueryLookups(n: number): boolean }).shouldQueryLookups(refs);

describe('kernel node lookups: table load or query', () => {
it('loads the table for a small project full index', () => {
expect(shouldQuery(3_089, 13_643)).toBe(false); // gin
expect(shouldQuery(5_614, 22_684)).toBe(false); // Alamofire
});

it('queries for a one-file sync on a small project', () => {
expect(shouldQuery(3_089, 60)).toBe(true);
});

it('queries below the crossover on a large project and loads above it', () => {
expect(shouldQuery(49_708, 4_900)).toBe(true); // ktor, 4.9k refs: query 302 ms vs table 375 ms
expect(shouldQuery(49_708, 36_015)).toBe(false); // above 15,000 refs
});
});
32 changes: 32 additions & 0 deletions codegraph-kernel/src/resolve/names.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1652,6 +1652,12 @@ pub(super) fn js_prefix(s: &str, end: usize) -> &str {

/// UTF-16 unit offset → byte offset (clamped to the next char boundary).
pub(super) fn js_unit_to_byte(s: &str, units: usize) -> usize {
// ASCII up to the offset: one byte per unit. A column far into a minified
// one-line file would otherwise walk every char before it, per lookup.
let reach = units.min(s.len());
if s.as_bytes()[..reach].is_ascii() {
return reach;
}
let mut seen = 0usize;
for (byte_idx, ch) in s.char_indices() {
if seen >= units {
Expand Down Expand Up @@ -1759,3 +1765,29 @@ fn imports_cgo(src: &SourceFile) -> bool {
!src.lines_containing("\"C\"").is_empty()
&& re!(r#"(?-u:\b)import\s+(?:\(\s*)?"C""#).is_match(&stripped_text(src))
}

#[cfg(test)]
mod js_unit_tests {
use super::js_unit_to_byte;

/// The char-by-char walk the ASCII fast path replaced.
fn walk(s: &str, units: usize) -> usize {
let mut seen = 0usize;
for (byte_idx, ch) in s.char_indices() {
if seen >= units {
return byte_idx;
}
seen += ch.len_utf16();
}
s.len()
}

#[test]
fn matches_the_char_walk_at_every_offset() {
for s in ["", "abc", "a\u{e9}b", "x😀y", "😀😀", "plain ascii line", "é", "ab\u{2028}cd"] {
for units in 0..s.len() + 4 {
assert_eq!(js_unit_to_byte(s, units), walk(s, units), "{s:?} at {units}");
}
}
}
}
11 changes: 0 additions & 11 deletions docs/design/open-questions.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,3 @@ one from the sentence that states it, and `__tests__/readme-sync.test.ts` fails
when a README sentence says a cause is unknown without linking an entry here, or
when an entry here is no longer linked from the README. Remove the entry in the
same change that finds the cause.

## Slower full index on small projects

On Node, a full index of gin and Alamofire takes 0.14–0.43 s longer than
upstream's. The runs are in
[fork-vs-upstream-node-bun-2026-10-03.md](../benchmarks/fork-vs-upstream-node-bun-2026-10-03.md).
The cause has not been found.

What would close it: a profile of the fixed per-index costs on a small project
(startup, schema setup, the maintenance step) for the fork and for upstream on
the same host, since the gap does not grow with project size.
6 changes: 6 additions & 0 deletions src/db/queries.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4488,6 +4488,12 @@ export class QueryBuilder {
return filesByLanguage;
}

/** Total node count: the whole graph's size, without the per-kind breakdown. */
getNodeCount(): number {
const row = this.db.prepare('SELECT COUNT(*) AS count FROM nodes').get() as { count: number };
return row.count;
}

/**
* Get graph statistics
*/
Expand Down
16 changes: 14 additions & 2 deletions src/resolution/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,11 @@ import { readDirEntries } from '../directory';
* loading the whole node table: the load is a fixed cost that per-name
* queries only overtake near 16k refs (measured on ktor, ledger §5.74). */
const SYNC_NODE_TABLE_MIN_REFS = 15_000;
/** The load costs per node and the queries per ref, so the crossover scales
* with the graph: ktor's 49.7k nodes cross near 16k refs (0.32 refs per
* node). A small project's full index (gin: 3k nodes, 13.6k refs) is far
* past it and loads the table. */
const SYNC_NODE_TABLE_REFS_PER_NODE = 0.3;
/** Refs per kernel `resolveChunk` call on the list paths. */
const KERNEL_CHUNK = 500;

Expand Down Expand Up @@ -922,13 +927,20 @@ export class ReferenceResolver {
this.kernelResolver = this.openKernelResolver(dbPath, generation, supertypesComplete, snapshot, queryLookups);
}

/** Whether a kernel over the live db answers node lookups by query rather
* than loading the node table, for `refCount` pending refs. */
private shouldQueryLookups(refCount: number): boolean {
return refCount < SYNC_NODE_TABLE_MIN_REFS
&& refCount < this.queries.getNodeCount() * SYNC_NODE_TABLE_REFS_PER_NODE;
}

/** The open kernel resolver, or the live db's opened on demand. */
private liveKernel(refCount: number): KernelResolverLike {
if (!this.kernelResolver) {
// The live db: prerequisites persist before the rest resolves, so every
// supertype edge the walks read is already written. A small list looks
// nodes up by query: loading the whole node table costs more than it.
this.initKernelResolver(undefined, undefined, true, false, refCount < SYNC_NODE_TABLE_MIN_REFS);
this.initKernelResolver(undefined, undefined, true, false, this.shouldQueryLookups(refCount));
}
return this.kernelResolver!;
}
Expand Down Expand Up @@ -1811,7 +1823,7 @@ export class ReferenceResolver {
if (kernel) return kernel;
await quiesceValveForKernel();
// A sync's few pending refs look nodes up by query, as in liveKernel.
this.initKernelResolver(parallel?.dbPath, undefined, true, false, total < SYNC_NODE_TABLE_MIN_REFS);
this.initKernelResolver(parallel?.dbPath, undefined, true, false, this.shouldQueryLookups(total));
kernel = this.kernelResolver!;
return kernel;
};
Expand Down
Loading