Repository navigation
fix(resolve): find the token that starts at a point, not the one ending there - #472
Merged
Merged
Conversation
…ng there The kernel's descendant_for_position took the first child whose span held the point with an inclusive end, so where one token ends exactly where the next begins it returned the earlier one. Callers that pass a token's start got the `;` in `x=1;local function helper()`, which made the Lua local read as global, and the `)` in `if ($x instanceof Foo)$x->bar();`, which hid the PHP guard. tree.ts got the same fix in #308. A child must now end after the point, as in tree.ts. Callers that passed column + 1 to land inside a token now pass the column itself; under the new rule that finds the same node, checked at all 2.37 million token starts of 1,866 real files in 11 languages. Without that change a one-character name or receiver (`a.f()`) would have found the token after it. Golden dumps are unchanged, and json, jq and CPython index to the same edges.
…t-lookup-token-boundary
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The kernel's point lookup returned the token that ends at a point instead of the one that starts there, so code written without a space between tokens read the wrong node: a Lua
local functionright after;, or a PHPinstanceofguard right before$x->bar(), resolved wrongly. Json, jq and CPython index to identical edges (CPython: 325,350 call edges, 0 lost, 0 gained).What changed
descendant_for_positioniniteration.rsnow requires a child to end after the point, matching the TypeScript twin fixed in fix(tsjs): type receivers from the code that declares them #308.)thenFooinif(ok)Foo f;findsFoo.a.f()would find the.), so they pass the column itself. A temporary test over every token start in 1,866 real files across 11 grammars found the same node under the old rule at column+1 and the new rule at the column at all 2,373,173 points.cpp_types.rs, five initeration.rs,lua_alias.rs,php_scope.rs,prefilter.rsanddart_fields.rs.tree.ts(a child must end after the point), not tree-sitter's own rule; this only matters for parser-inserted missing nodes.Evidence
Tests that fail on the base kernel and pass here: Rust
a_point_where_one_token_ends_and_the_next_begins_finds_the_next(if(ok)Foo f;,if(ok)target();,f(x)), and two TypeScript cases,reads a Lua local function written right after ';' as localandapplies a PHP instanceof guard to a call written right after ')'(each with a spaced control).Golden dumps unchanged;
eval:precisionidentical on json (11/11 absent held, 7/7 present held; jq has no cases);cargo test --lib68 passed; full suite 695 files, 8,888 passed, 39 skipped, no worker crash lines; clippy, the kernel-rules check andtscpass.Limits
The C++ namespace climb (
cpp_types.rs) and the Dart field lookup can in theory change too; I built no fixture for either and none of the three corpora moved. A Kotlin parenthesized infix receiver ((b)put(1)) now resolves by name like its spaced form(b) put 1already does on the base kernel; that older name fallback is not changed here.README rows checked, none changed: "Name resolution", "What the fork adds" and the "Measured results" timing table.
Summary by cubic
Fixes the kernel's point lookup so that at a token boundary it reads the token that starts there instead of the one ending there. Previously
x=1;local function helper()read the;and treated the Lua local function as global, andif ($x instanceof Foo)$x->bar();read the)and missed the PHPinstanceofguard; both now resolve like their spaced forms.Bug Fixes
descendant_for_positionnow requires a child to end after the point, matching the TypeScript side.finf(x)are not skipped.(b)put(1)) now resolves by name like its spaced form already did; the C++ namespace climb and Dart field lookup can in theory change too, but no fixture was built and none of the corpora moved.Written for commit 7fdae16. Summary will update on new commits.
Summary by CodeRabbit
instanceofcases, including calls written without whitespace at token boundaries.