diff --git a/CHANGELOG.md b/CHANGELOG.md index e20fcf2bd..74cbc2ffc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -462,6 +462,9 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In C++, a call on a variable, parameter or member declared through a `typedef` or `using` alias now links to the method of the type the alias names, with the alias looked up the way C++ does it: in the calling function, its class and the classes that class inherits from, then the namespaces around it. In google/leveldb, `table_.Insert(buf)` on a `Table table_;` declared next to `typedef SkipList Table;` now reaches `SkipList::Insert` instead of the unrelated `HandleTable::Insert`, and `Table::Iterator iter(&table_);` reaches `SkipList::Iterator`'s constructor. A class that only shares the alias's name is no longer taken for it, and when the aliased type has no such method, as with an alias of `std::vector` or of a template parameter, no method is guessed from the receiver's name. Re-index C++ projects after upgrading. - In C++, a call a method makes with no receiver, or through `this->`, now links to the method C++ itself would call: one of the method's own class, else one the class inherits from a base class, else a static member of a class it is nested in, which a nested class can name without an object. The same goes for a call in a lambda inside the method, and of a class's overloads the one the call's arguments fit is taken. Before, the call was matched by its name alone, so another class's method of that name could win because it was declared nearer or found first: in protocolbuffers/protobuf, every generated message's `Swap` and move assignment calling `InternalSwap(…)` reached `Any::InternalSwap`, and the `GetArena()` messages inherit from `MessageLite` reached an arena helper's `GetArena`, while googletest's `~linked_ptr() { depart(); }`, vendored in facebook/rocksdb, reached `linked_ptr_internal::depart`. Calls C++ doesn't look up in the class are matched as before: a free function's, a call to a parameter or local of that name, and a template's bare call into a base class that depends on its template parameters. Re-index C++ projects after upgrading. Thanks @danusha2345 for reporting the C++ case in #1952. +- In C++, a call on the variable of a range-based `for` loop over `auto` elements, like `cfd->Ref()` inside facebook/rocksdb's `for (auto cfd : cfds)`, now links to the method of the type the loop goes over. That type is read from what the loop iterates: a `std::vector`, `std::set`, `absl::Span` or similar container, protocolbuffers/protobuf's `RepeatedPtrField` or RocksDB's `autovector`, whether a variable, parameter or member holds it, an `auto` variable is set to it or a function or method returns it, and a project class you loop over through its `begin()`, like RocksDB's `ColumnFamilySet`. Before, the call was matched by the variable's name, or by an earlier variable of that name: in RocksDB, `blob_file.GetBlobFileNumber()` over a list of `BlobFileGarbage` reached `BlobFileAddition::GetBlobFileNumber`, and in protobuf's Objective-C generator, calls over a list of `std::unique_ptr` reached other generators' methods. A call on an element of a standard-library type, like `file.find(…)` over a `std::vector`, no longer links to a project method, and `->` on an element held in a `std::unique_ptr` or `std::shared_ptr` reaches the type it holds. Re-index C++ projects after upgrading. +- In C++, a call on a variable, parameter or member declared as one of your classes, like `iter->Valid()` on an `Iterator* iter` or `it.Valid()` on a `const Iterator& it`, now links to that method of the class C++ itself would find for the type where the call is written, or of a class it inherits from. Before, the method was picked by the type's last name, so a class of the same name nested elsewhere could win: in facebook/rocksdb, calls on an `Iterator` reached `MemTableRep::Iterator`, which declares `Valid` and `Next` itself while the `Iterator` the code means inherits them, google/leveldb's reached `SkipList::Iterator`, and in protocolbuffers/protobuf the Java generator's `context->options()` reached the hpb generator's `Context` and `text_parser.Parse(…)` on a `TextFormat::Parser` reached the `.proto` parser. When that class and the classes it derives from have no method of that name, the call no longer links to an unrelated class's method that shares it, as protobuf's `field->full_name()` on a `FieldDescriptor*` did. A type is also found through a `using` declaration the file writes, like protobuf's `using google::protobuf::Descriptor;`, before a class of the same name declared at global scope elsewhere in the repository; this also applies to base classes and to the type a smart pointer holds. Re-index C++ projects after upgrading. +- In this fork, the two C++ changes above run in the native resolver. A call on a variable whose type is a template parameter, like `T* iter`, stays unlinked rather than reaching a class that happens to share the parameter's name. A call stays linked as before when the class has no method of that name but one of its base classes comes from an outside library other than the standard library, such as googletest's `testing::Test`. - Framework support now switches on when an app's `package.json` sits three or more folders deep, as with an ASP.NET solution's Angular or React app in `src/Web/ClientApp/`, or a React app in a workspace folder like `web/ui/mantine-ui/`. Before, only the project root and the first two folder levels were read, so frameworks such as Angular, React Router, Vue Router, SvelteKit and TanStack Router could go undetected there: an Angular app's routes, its `routerLink` and `router.navigate(…)` navigation and its template's event bindings were missing, so a handler used only from a `(click)` binding had no callers, and a React app's `navigate('/login')` calls didn't reach their route. Re-index projects laid out this way after upgrading. - In Go, an interface that embeds another, like gin's `IRouter` embedding `IRoutes`, now links to it, so `codegraph_explore` and impact show the embedded interface as its base. Before, every embedded interface was dropped. An embedded field written through another package of your project, like `*refresh.Discovery` or `storage.Appender`, or with type arguments, like `List[int]`, now links too. One from the standard library or another module, like `io.Closer` or `sync.Mutex`, links to nothing rather than to a project type that shares its name, and a type-set constraint like `~int | ~float64` is not an embedding. An embedded type written without a package now always links to its own package's type: before, etcd's `Client`, which embeds its own `Lease` interface, was linked to a `Lease` struct in another package. Re-index Go projects after upgrading. @@ -518,6 +521,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - A file named the way Google names its tests, like `wire_format_unittest.cc` in protocolbuffers/protobuf, `logging_unittest.cc` in google/glog or a Python `foo_unittest.py`, now counts as a test, as `wire_format_test.cc` already did. Before, it was taken for production code: search and `codegraph_explore` ranked it alongside the code it tests, `codegraph affected` never listed it, and calls in production code could link to a class or type a unittest declares for itself, as google/breakpad's `stack_frame_entries_.size()` did to a unittest's `StackHelper`. Re-index projects that have such files after upgrading. - `codegraph sync` and the file watcher now link a file to a module it imports under a name of its own when the module is added after the file was indexed, comes back after being deleted, or gets the export in a later edit: a default import like `import tagsController from './tag/tag.controller'`, a namespace import like `import * as api from './api'`, or an aliased one like `import { Component as GroupsPageWrapper } from './groups_page'`, in JavaScript, TypeScript, Vue, Svelte and Astro. Before, the import and every use of the name, such as a call, `new`, a base class, a JSX or template tag or an Express `app.use` mount, stayed unlinked until the importing file changed or the project was re-indexed, so the module looked unused and `codegraph affected` missed the files that use it. Re-index after upgrading to pick up links an earlier version missed. +- `codegraph sync` and the file watcher now do the same for a Python import under a name of its own, like `from .models import User as UserModel`, `from . import helpers as h` or `from app.pkg import sub as subpkg`: when the module or package is added after the importing file was indexed, comes back after being deleted, or gets the imported name in a later edit, the import and every use of the alias, such as a call, a base class or the class passed as a value, now link to it. Before, they stayed unlinked until the importing file changed or the project was re-indexed. Re-index Python projects after upgrading to pick up links an earlier version missed. - In C++, a call on a variable, parameter or member declared with template arguments nested inside its template arguments, like `autovector>` or `Striped>`, now links to the method of the class it is declared as. Before, the nested arguments hid that class: in RocksDB, `files_marked_for_compaction_.clear()` on such an `autovector` linked to an unrelated `CompactionInputFiles::clear` that only shares words with the variable's name, and `mutex_.Get(key)` on a `Striped` linked to nothing. Re-index C++ projects after upgrading. @@ -551,7 +555,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - The call paths `codegraph context` prints now say what bridges each dynamic-dispatch hop, in the same words `codegraph_explore` uses: a call through an interface reads `interface → impl`, and a C++ override, a Redux thunk or a Gin middleware chain is named as one. Before, every hop of a kind it didn't know was labeled an "event", so a call through an interface looked like an event being emitted. -- In C++, a call on a range-based `for` loop's variable, like `suite->SetVerbose(verbose)` inside `for (ConformanceTestSuite *suite : suites)`, now links to the method of the type the loop declares. Before, the loop's declaration was never read, so the call was matched by the variable's name alone: in RocksDB, `log->file()` on a `log::Writer` went to `BlobLogWriter::file`, and a loop variable named like a member of its class was taken for that member. The variable counts only inside its loop, and loops over `auto` elements resolve as before. Re-index C++ projects after upgrading. +- In C++, a call on a range-based `for` loop's variable, like `suite->SetVerbose(verbose)` inside `for (ConformanceTestSuite *suite : suites)`, now links to the method of the type the loop declares. Before, the loop's declaration was never read, so the call was matched by the variable's name alone: in RocksDB, `log->file()` on a `log::Writer` went to `BlobLogWriter::file`, and a loop variable named like a member of its class was taken for that member. The variable counts only inside its loop. Re-index C++ projects after upgrading. - In C++, a call made with `->` on a variable, parameter or member declared as a `std::unique_ptr`, `std::shared_ptr` or `std::optional`, like `iter->Valid()` on a `std::unique_ptr iter`, now links to that method of the type it holds, or of a class that type inherits from, with the type looked up the way C++ does it, so a class of the same name nested elsewhere is not mistaken for it. Before, the call went to whichever project method shared its name: in facebook/rocksdb, `iter->Valid()` reached `ArenaWrappedDBIter::Valid` and `fs->GetFileSize(…)` on a `std::shared_ptr` reached a test's stand-in file system, and calls like `it->Next()` were often not linked at all. When what it holds is a template parameter or a standard-library type, like `std::unique_ptr>`, the call no longer links to a project method. Re-index C++ projects after upgrading. diff --git a/README.md b/README.md index 6877cb29a..e06421e28 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,7 @@ Follow [@getcodegraph](https://x.com/getcodegraph) on X for updates. ## About this fork -This is **bompus/codegraph**, a fork of [colbymchenry/codegraph](https://github.com/colbymchenry/codegraph). Its default branch, `fork/consolidated`, contains upstream `main` through [`8a5ba3d0`](https://github.com/colbymchenry/codegraph/commit/8a5ba3d0) (after v1.6.2) plus the fork's own work, and it takes upstream changes as they land. Changes that suit upstream are also offered there as pull requests. +This is **bompus/codegraph**, a fork of [colbymchenry/codegraph](https://github.com/colbymchenry/codegraph). Its default branch, `fork/consolidated`, contains upstream `main` through [`5171a6aa`](https://github.com/colbymchenry/codegraph/commit/5171a6aa) (after v1.6.2) plus the fork's own work, and it takes upstream changes as they land. Changes that suit upstream are also offered there as pull requests. The fork publishes no releases. The install scripts, npm package, badges and `codegraph upgrade` further down this page install **upstream's** releases. To run the fork, build it from source (below). diff --git a/__tests__/cpp-auto-range-for-receiver.test.ts b/__tests__/cpp-auto-range-for-receiver.test.ts new file mode 100644 index 000000000..973b75743 --- /dev/null +++ b/__tests__/cpp-auto-range-for-receiver.test.ts @@ -0,0 +1,658 @@ +/** + * A C++ range-based for over `auto` elements has the element type of the + * range it iterates. + * + * #2439 read the type a loop declares, `for (ConformanceTestSuite *suite : + * suites)`. A loop over `auto` elements declares none, so the scan went on + * as for an `auto` local whose initializer it can't read: it reached an + * earlier declaration of the same name, often another variable's, or the + * call fell through to a guess by the receiver's name. In facebook/rocksdb, + * `for (auto cfd : cfds)` iterates a `const autovector&` + * parameter. The element type is read from the range where that can be done + * reliably: + * - a name the calling function declares, or a data member of its class: + * a container's first template argument, aliases followed with their + * template arguments; + * - a call whose declaration writes its return type, `*x`, `x.m`, `x->m`; + * - one of the project's classes, through what its `begin()` returns: + * rocksdb's `ColumnFamilySet`, whose iterator's `operator*` returns a + * `ColumnFamilyData*`. + * A guess can still land on the right method by a shared word, so these tests + * check the typed path's confidence, not only the target. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { Node } from '../src/types'; + +const roots: string[] = []; +afterEach(() => { + for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true }); +}); + +async function indexed(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cpp-auto-range-for-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); +} + +function callable(cg: CodeGraph, qualifiedName: string): Node { + const node = [...cg.getNodesByKind('method'), ...cg.getNodesByKind('function')].find((n) => n.qualifiedName === qualifiedName); + if (!node) throw new Error(`no function or method ${qualifiedName}`); + return node; +} + +/** `calls` callees of a function or method, as `qualifiedName @confidence`, one per call site. */ +function calls(cg: CodeGraph, caller: string): string[] { + return cg + .getCallees(callable(cg, caller).id) + .filter((r) => r.edge.kind === 'calls') + .map((r) => `${r.node.qualifiedName} @${(r.edge.metadata as { confidence?: number } | undefined)?.confidence}`) + .sort(); +} + +const lines = (...text: string[]): string => [...text, ''].join('\n'); + +/** + * facebook/rocksdb's column families, trimmed. `ColumnFamilyHandle` has + * methods of the same names; `DBImpl` is declared in db_impl.h and its + * methods are defined in db_impl_write.cc, so the scan's same-stem header + * never reads `versions_`. + */ +const ROCKSDB = { + 'util/autovector.h': lines( + '#include ', + 'namespace rocksdb {', + 'template ', + 'class autovector {', + ' public:', + ' size_t size() const { return 0; }', + ' bool empty() const { return true; }', + '};', + '} // namespace rocksdb', + ), + 'db/column_family.h': lines( + '#include ', + 'namespace rocksdb {', + 'class ColumnFamilyData {', + ' public:', + ' bool IsDropped() const;', + ' void Ref();', + ' uint32_t GetID() const;', + '};', + 'class ColumnFamilySet {', + ' public:', + ' class iterator {', + ' public:', + ' explicit iterator(ColumnFamilyData* cfd) : current_(cfd) {}', + ' iterator& operator++() { current_ = nullptr; return *this; }', + ' bool operator!=(const iterator& other) const { return current_ != other.current_; }', + ' ColumnFamilyData* operator*() { return current_; }', + ' private:', + ' ColumnFamilyData* current_;', + ' };', + ' iterator begin() { return iterator(head_); }', + ' iterator end() { return iterator(nullptr); }', + ' private:', + ' ColumnFamilyData* head_ = nullptr;', + '};', + 'class ColumnFamilyHandle {', + ' public:', + ' bool IsDropped() const;', + ' void Ref();', + ' uint32_t GetID() const;', + '};', + '} // namespace rocksdb', + ), + 'db/column_family.cc': lines( + '#include "db/column_family.h"', + 'namespace rocksdb {', + 'bool ColumnFamilyData::IsDropped() const { return false; }', + 'void ColumnFamilyData::Ref() {}', + 'uint32_t ColumnFamilyData::GetID() const { return 0; }', + 'bool ColumnFamilyHandle::IsDropped() const { return false; }', + 'void ColumnFamilyHandle::Ref() {}', + 'uint32_t ColumnFamilyHandle::GetID() const { return 0; }', + '} // namespace rocksdb', + ), + 'db/version_set.h': lines( + '#include ', + '#include "db/column_family.h"', + 'namespace rocksdb {', + 'class VersionSet {', + ' public:', + ' ColumnFamilySet* GetColumnFamilySet() { return column_family_set_.get(); }', + ' private:', + ' std::unique_ptr column_family_set_;', + '};', + '} // namespace rocksdb', + ), + 'db/db_impl.h': lines( + '#include ', + '#include "util/autovector.h"', + '#include "db/version_set.h"', + 'namespace rocksdb {', + 'class DBImpl {', + ' public:', + ' void AssignAtomicFlushSeq(const autovector& cfds);', + ' void ScheduleFlushes();', + ' private:', + ' std::unique_ptr versions_;', + '};', + '} // namespace rocksdb', + ), + 'db/db_impl_write.cc': lines( + '#include "db/db_impl.h"', + 'namespace rocksdb {', + 'void UseHandle(ColumnFamilyHandle* cfd) {', + ' cfd->Ref();', + '}', + 'void DBImpl::AssignAtomicFlushSeq(const autovector& cfds) {', + ' for (auto cfd : cfds) {', + ' cfd->GetID();', + ' }', + '}', + 'void DBImpl::ScheduleFlushes() {', + ' for (auto cfd : *versions_->GetColumnFamilySet()) {', + ' if (!cfd->IsDropped()) cfd->Ref();', + ' }', + '}', + '} // namespace rocksdb', + ), +}; + +/** protocolbuffers/protobuf's descriptors, trimmed; `upb` has a `FieldDefPtr` with the same accessors. */ +const PROTOBUF = { + 'src/google/protobuf/descriptor.h': lines( + '#include ', + 'namespace google {', + 'namespace protobuf {', + 'class FieldDescriptor {', + ' public:', + ' int number() const;', + ' const std::string& full_name() const;', + '};', + 'class Descriptor {', + ' public:', + ' int field_count() const;', + '};', + '} // namespace protobuf', + '} // namespace google', + ), + 'src/google/protobuf/descriptor.cc': lines( + '#include "google/protobuf/descriptor.h"', + 'namespace google {', + 'namespace protobuf {', + 'int FieldDescriptor::number() const { return 0; }', + 'const std::string& FieldDescriptor::full_name() const { static std::string s; return s; }', + 'int Descriptor::field_count() const { return 0; }', + '} // namespace protobuf', + '} // namespace google', + ), + 'upb/reflection/def.hpp': lines( + 'namespace upb {', + 'class FieldDefPtr {', + ' public:', + ' int number() const { return 0; }', + ' const char* full_name() const { return ""; }', + '};', + '} // namespace upb', + ), + 'src/google/protobuf/compiler/cpp/field_layout.h': lines( + '#include ', + '#include "absl/types/span.h"', + '#include "google/protobuf/descriptor.h"', + 'namespace google {', + 'namespace protobuf {', + 'namespace compiler {', + 'namespace cpp {', + 'std::vector FieldNumberOrder(const Descriptor* descriptor);', + 'class FieldLayout {', + ' public:', + ' absl::Span optimized_order() const { return {}; }', + '};', + '} // namespace cpp', + '} // namespace compiler', + '} // namespace protobuf', + '} // namespace google', + ), + 'src/google/protobuf/compiler/cpp/message.cc': lines( + '#include "google/protobuf/compiler/cpp/field_layout.h"', + 'namespace google {', + 'namespace protobuf {', + 'namespace compiler {', + 'namespace cpp {', + 'std::vector FieldNumberOrder(const Descriptor* descriptor) {', + ' return {};', + '}', + 'int Numbers(const Descriptor* descriptor) {', + ' int sum = 0;', + ' for (const auto* field : FieldNumberOrder(descriptor)) {', + ' sum += field->number();', + ' }', + ' return sum;', + '}', + 'void Names(const FieldLayout& layout) {', + ' for (const auto* field : layout.optimized_order()) {', + ' field->full_name();', + ' }', + '}', + '} // namespace cpp', + '} // namespace compiler', + '} // namespace protobuf', + '} // namespace google', + ), +}; + +describe('an `auto` range-for variable has the element type of its range', () => { + it('reproduction: rocksdb\'s `for (auto cfd : cfds)` over an autovector parameter', async () => { + const cg = await indexed(ROCKSDB); + try { + // The scan went past the loop to UseHandle's `ColumnFamilyHandle* cfd`. + expect(calls(cg, 'rocksdb::DBImpl::AssignAtomicFlushSeq')).toEqual(['rocksdb::ColumnFamilyData::GetID @0.9']); + expect(calls(cg, 'rocksdb::UseHandle')).toEqual(['rocksdb::ColumnFamilyHandle::Ref @0.9']); + } finally { + cg.close(); + } + }); + + it('a project class iterated through its begin(), reached from a member another header declares', async () => { + const cg = await indexed(ROCKSDB); + try { + // `versions_` is DBImpl's `std::unique_ptr`; GetColumnFamilySet() + // returns a `ColumnFamilySet*`, whose iterator's `operator*` returns a + // `ColumnFamilyData*`. (The range's own call is left out here: no + // receiver scan reads db_impl.h from db_impl_write.cc.) + expect(calls(cg, 'rocksdb::DBImpl::ScheduleFlushes').filter((c) => !c.startsWith('rocksdb::VersionSet::'))).toEqual([ + 'rocksdb::ColumnFamilyData::IsDropped @0.9', + 'rocksdb::ColumnFamilyData::Ref @0.9', + ]); + } finally { + cg.close(); + } + }); + + it('a call whose declaration writes its return type, and a nullability annotation in an element', async () => { + const cg = await indexed(PROTOBUF); + try { + expect(calls(cg, 'google::protobuf::compiler::cpp::Numbers')).toEqual([ + 'google::protobuf::FieldDescriptor::number @0.9', + 'google::protobuf::compiler::cpp::FieldNumberOrder @0.9', + ]); + expect(calls(cg, 'google::protobuf::compiler::cpp::Names')).toEqual([ + 'google::protobuf::FieldDescriptor::full_name @0.9', + 'google::protobuf::compiler::cpp::FieldLayout::optimized_order @0.9', + ]); + } finally { + cg.close(); + } + }); + + it('an `auto` local\'s initializer, an alias with template arguments, and a smart pointer element', async () => { + const cg = await indexed({ + 'db/blob_file_meta.h': lines( + '#include ', + 'namespace rocksdb {', + 'class BlobFileMetaData {', + ' public:', + ' uint64_t GetBlobFileNumber() const { return 0; }', + '};', + 'class BlobFileAddition {', + ' public:', + ' uint64_t GetBlobFileNumber() const { return 0; }', + '};', + '} // namespace rocksdb', + ), + 'db/version_storage.h': lines( + '#include ', + '#include ', + '#include "db/blob_file_meta.h"', + 'namespace rocksdb {', + 'class VersionStorageInfo {', + ' public:', + ' using BlobFiles = std::vector>;', + ' const BlobFiles& GetBlobFiles() const { return blob_files_; }', + ' private:', + ' BlobFiles blob_files_;', + '};', + '} // namespace rocksdb', + ), + 'db/db_filesnapshot.cc': lines( + '#include "db/version_storage.h"', + 'namespace rocksdb {', + 'uint64_t Largest(const VersionStorageInfo* vstorage) {', + ' uint64_t largest = 0;', + ' const auto& blob_files = vstorage->GetBlobFiles();', + ' for (const auto& meta : blob_files) {', + ' largest = meta->GetBlobFileNumber();', + ' }', + ' return largest;', + '}', + '} // namespace rocksdb', + ), + }); + try { + // `->` on the `std::shared_ptr` element reaches the type it holds. + expect(calls(cg, 'rocksdb::Largest')).toEqual([ + 'rocksdb::BlobFileMetaData::GetBlobFileNumber @0.9', + 'rocksdb::VersionStorageInfo::GetBlobFiles @0.9', + ]); + } finally { + cg.close(); + } + }); + + it('a data member of another object, and loops inside loops', async () => { + const cg = await indexed({ + 'db/compaction/subcompaction_state.h': lines( + 'namespace rocksdb {', + 'class Output {', + ' public:', + ' void Finish() {}', + '};', + 'class SubcompactionState {', + ' public:', + ' void RemoveLastEmptyOutput() {}', + ' Output outputs[2];', + '};', + 'class FlushJob {', + ' public:', + ' void RemoveLastEmptyOutput() {}', + '};', + '} // namespace rocksdb', + ), + 'db/compaction/compaction_state.h': lines( + '#include ', + '#include "db/compaction/subcompaction_state.h"', + 'namespace rocksdb {', + 'class CompactionState {', + ' public:', + ' std::vector sub_compact_states;', + '};', + '} // namespace rocksdb', + ), + 'db/compaction/compaction_job.h': lines( + '#include "db/compaction/compaction_state.h"', + 'namespace rocksdb {', + 'class CompactionJob {', + ' public:', + ' void RemoveEmptyOutputs();', + ' private:', + ' CompactionState* compact_;', + '};', + '} // namespace rocksdb', + ), + 'db/compaction/compaction_job.cc': lines( + '#include "db/compaction/compaction_job.h"', + 'namespace rocksdb {', + 'void CompactionJob::RemoveEmptyOutputs() {', + ' for (auto& state : compact_->sub_compact_states) {', + ' state.RemoveLastEmptyOutput();', + ' }', + '}', + 'void Finish(const std::vector& compactions) {', + ' for (const auto* compaction : compactions) {', + ' for (auto& state : compaction->sub_compact_states) {', + ' state.RemoveLastEmptyOutput();', + ' }', + ' }', + '}', + '} // namespace rocksdb', + ), + }); + try { + expect(calls(cg, 'rocksdb::CompactionJob::RemoveEmptyOutputs')).toEqual(['rocksdb::SubcompactionState::RemoveLastEmptyOutput @0.9']); + expect(calls(cg, 'rocksdb::Finish')).toEqual(['rocksdb::SubcompactionState::RemoveLastEmptyOutput @0.9']); + } finally { + cg.close(); + } + }); +}); + +describe('what an `auto` element calls when it is not the project\'s', () => { + const SLICE = { + 'util/slice.h': lines( + '#include ', + 'namespace rocksdb {', + 'class Slice {', + ' public:', + ' size_t size() const { return 0; }', + ' const char* c_str() const { return ""; }', + '};', + 'class TableReader {', + ' public:', + ' bool Valid() const { return true; }', + ' void reset() {}', + '};', + 'class TruncatedRangeDelIterator {', + ' public:', + ' bool Valid() const { return true; }', + '};', + '} // namespace rocksdb', + ), + }; + + it('a standard-library element\'s own member links to nothing', async () => { + const cg = await indexed({ + ...SLICE, + 'db/names.cc': lines( + '#include ', + '#include ', + '#include ', + '#include "util/slice.h"', + 'namespace rocksdb {', + 'size_t Total(const std::vector& files) {', + ' size_t total = 0;', + ' for (const auto& file : files) {', + ' total += file.size();', + ' }', + ' return total;', + '}', + '} // namespace rocksdb', + ), + }); + try { + // The guess took `Slice::size` by the method's name. + expect(calls(cg, 'rocksdb::Total')).toEqual([]); + } finally { + cg.close(); + } + }); + + it('a smart pointer element: `->` reaches the type it holds, `.` the holder\'s own member', async () => { + const cg = await indexed({ + ...SLICE, + 'db/range_del_aggregator.h': lines( + '#include ', + '#include ', + '#include "util/slice.h"', + 'namespace rocksdb {', + 'class RangeDelAggregator {', + ' public:', + ' bool IsRangeOverlapped();', + ' void Reset();', + ' private:', + ' std::vector> iters_;', + '};', + '} // namespace rocksdb', + ), + 'db/range_del_aggregator.cc': lines( + '#include "db/range_del_aggregator.h"', + 'namespace rocksdb {', + 'bool RangeDelAggregator::IsRangeOverlapped() {', + ' for (auto& iter : iters_) {', + ' if (iter->Valid()) return true;', + ' }', + ' return false;', + '}', + 'void RangeDelAggregator::Reset() {', + ' for (auto& iter : iters_) iter.reset();', + '}', + '} // namespace rocksdb', + ), + }); + try { + expect(calls(cg, 'rocksdb::RangeDelAggregator::IsRangeOverlapped')).toEqual(['rocksdb::TruncatedRangeDelIterator::Valid @0.9']); + // `std::unique_ptr::reset`, not TableReader's. + expect(calls(cg, 'rocksdb::RangeDelAggregator::Reset')).toEqual([]); + } finally { + cg.close(); + } + }); + + it('an element a template argument decides is not read', async () => { + const cg = await indexed({ + ...SLICE, + 'db/each.h': lines( + '#include ', + '#include "util/slice.h"', + 'namespace rocksdb {', + 'template ', + 'bool AllValid(const std::vector& items) {', + ' for (auto* item : items) {', + ' if (!item->Valid()) return false;', + ' }', + ' return true;', + '}', + '} // namespace rocksdb', + ), + }); + try { + // T is whatever the caller iterates: no typed edge, as before. + expect(calls(cg, 'rocksdb::AllValid').filter((c) => c.endsWith('@0.9'))).toEqual([]); + } finally { + cg.close(); + } + }); +}); + +describe('an `auto` element that points at a class template', () => { + it('`->` reaches the template\'s member', async () => { + // rocksdb's `for (const auto mems : mems_list) mems->empty();` over a + // `const autovector*>&`. + const cg = await indexed({ + ...ROCKSDB, + 'db/memtable_list.cc': lines( + '#include "util/autovector.h"', + '#include "db/column_family.h"', + 'namespace rocksdb {', + 'bool AnyEmpty(const autovector*>& mems_list) {', + ' for (const auto mems : mems_list) {', + ' if (mems->empty()) return true;', + ' }', + ' return false;', + '}', + '} // namespace rocksdb', + ), + }); + try { + expect(calls(cg, 'rocksdb::AnyEmpty')).toEqual(['rocksdb::autovector::empty @0.9']); + } finally { + cg.close(); + } + }); +}); + +describe('an `auto` element read through a lookup that has to pick one declaration', () => { + it('a type written `::Registry` is the global class, not one nested in the caller\'s class', async () => { + const cg = await indexed({ + 'ui/screen.h': lines( + '#include ', + 'struct Widget { void Draw() {} };', + 'struct Registry { std::vector items() { return {}; } };', + 'namespace app {', + 'struct Gadget { void Draw() {} };', + 'class Screen {', + ' public:', + ' struct Registry { std::vector items() { return {}; } };', + ' void DrawAll();', + ' ::Registry registry_;', + '};', + '} // namespace app', + ), + 'ui/screen.cc': lines( + '#include "ui/screen.h"', + 'namespace app {', + 'void Screen::DrawAll() {', + ' for (auto w : registry_.items()) {', + ' w->Draw();', + ' }', + '}', + '} // namespace app', + ), + }); + try { + const drawn = calls(cg, 'app::Screen::DrawAll').filter((c) => c.includes('Draw @')); + expect(drawn).toEqual(['Widget::Draw @0.9']); + } finally { + cg.close(); + } + }); + + it('overloads that return different types give no element type', async () => { + const cg = await indexed({ + 'ui/registry.h': lines( + '#include ', + 'namespace app {', + 'struct Widget { void Draw() {} };', + 'struct Gadget { void Draw() {} };', + 'class Registry {', + ' public:', + ' const std::vector& items(int kind) const { return gadgets_; }', + ' const std::vector& items() const { return widgets_; }', + ' std::vector gadgets_;', + ' std::vector widgets_;', + '};', + '} // namespace app', + ), + 'ui/draw.cc': lines( + '#include "ui/registry.h"', + 'namespace app {', + 'void DrawAll(const Registry& registry) {', + ' for (auto w : registry.items()) {', + ' w->Draw();', + ' }', + '}', + '} // namespace app', + ), + }); + try { + // Which overload the call picks is not read, so neither return type is taken for it. + expect(calls(cg, 'app::DrawAll')).toEqual(['app::Registry::items @0.9']); + } finally { + cg.close(); + } + }); + + it('a name the loop body declares again hides the loop\'s variable', async () => { + const cg = await indexed({ + 'ui/draw.h': lines( + '#include ', + 'namespace app {', + 'struct Gadget { Gadget* Self() { return this; } void Draw() {} };', + 'struct Widget { Widget* Self() { return this; } void Draw() {} };', + '} // namespace app', + ), + 'ui/draw.cc': lines( + '#include "ui/draw.h"', + 'namespace app {', + 'void DrawAll(const std::vector& widgets, Gadget* g) {', + ' for (auto w : widgets) { { Gadget* w = g; w->Self()->Draw(); } }', + '}', + '} // namespace app', + ), + }); + try { + // The inner `w` is a `Gadget*`: the chained call is no `Widget`'s. + expect(calls(cg, 'app::DrawAll').filter((c) => c.startsWith('app::Widget::'))).toEqual([]); + } finally { + cg.close(); + } + }); +}); diff --git a/__tests__/cpp-declared-type-receiver.test.ts b/__tests__/cpp-declared-type-receiver.test.ts new file mode 100644 index 000000000..953cf2f6e --- /dev/null +++ b/__tests__/cpp-declared-type-receiver.test.ts @@ -0,0 +1,470 @@ +/** + * A C++ call on a variable, parameter or member the calling function or its + * class declares — `Iterator* iter; iter->Valid()`, `const Iterator& it; + * it.Valid()` — calls a member of the class C++ name lookup finds for the + * declared type where the call is written. + * + * Receiver typing used to look the method up by the type's last name, and + * took any class of that name that declares it. facebook/rocksdb has 22 + * classes named `Iterator`; the `Iterator` the code means + * (`ROCKSDB_NAMESPACE::Iterator`) inherits `Valid`, `Next` and `key` from + * `IteratorBase`, so a nested namesake that declares them — + * `MemTableRep::Iterator` — took about 1,100 calls. google/leveldb's + * `Iterator*` calls went to `SkipList::Iterator` the same way. + * + * The type is now looked up as for a smart pointer's held type (#2440): from + * the calling function's scopes outwards, then the file's `using`s, then the + * global scope, with aliases followed. The method is the class's own, or + * one a class it derives from declares. A class with no such method gets no + * edge: protocolbuffers/protobuf's descriptor accessors aren't indexed, and + * `field->full_name()` was guessed onto unrelated classes. When the lookup + * can't place the type — a template parameter, a class only another source + * file defines — or the declaration may be another variable's, the call + * resolves as before. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { Node } from '../src/types'; + +const roots: string[] = []; +afterEach(() => { + for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true }); +}); + +async function indexed(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cpp-declared-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); +} + +function callable(cg: CodeGraph, qualifiedName: string): Node { + const node = [...cg.getNodesByKind('method'), ...cg.getNodesByKind('function')].find((n) => n.qualifiedName === qualifiedName); + if (!node) throw new Error(`no function or method ${qualifiedName}`); + return node; +} + +/** `calls` callees of a function or method, as `qualifiedName`. */ +function calls(cg: CodeGraph, caller: string): string[] { + return cg + .getCallees(callable(cg, caller).id) + .filter((r) => r.edge.kind === 'calls') + .map((r) => r.node.qualifiedName) + .sort(); +} + +const lines = (...l: string[]): string => [...l, ''].join('\n'); + +/** rocksdb's iterator family, trimmed: the public Iterator, which inherits its cursor methods, and a nested namesake that declares them. */ +const ROCKSDB = { + 'include/rocksdb/iterator_base.h': lines( + 'namespace rocksdb {', + 'class IteratorBase {', + ' public:', + ' virtual ~IteratorBase() {}', + ' virtual bool Valid() const = 0;', + ' virtual void Next() = 0;', + '};', + '} // namespace rocksdb', + ), + 'include/rocksdb/iterator.h': lines( + '#include "include/rocksdb/iterator_base.h"', + 'namespace rocksdb {', + 'class Iterator : public IteratorBase {', + ' public:', + ' virtual int value() const = 0;', + '};', + '} // namespace rocksdb', + ), + 'include/rocksdb/memtablerep.h': lines( + 'namespace rocksdb {', + 'class MemTableRep {', + ' public:', + ' class Iterator {', + ' public:', + ' virtual bool Valid() const = 0;', + ' virtual void Next() = 0;', + ' };', + '};', + '} // namespace rocksdb', + ), + 'include/rocksdb/db.h': lines( + '#include "include/rocksdb/iterator.h"', + 'namespace rocksdb {', + 'class DB {', + ' public:', + ' virtual Iterator* NewIterator() = 0;', + '};', + '} // namespace rocksdb', + ), +}; + +describe('a call on a receiver the caller declares reaches the class its type names', () => { + it('reproduction: `Iterator* iter; iter->Valid()` is the Iterator the scope sees, not a nested namesake', async () => { + const cg = await indexed({ + ...ROCKSDB, + 'db/db_impl.cc': lines( + '#include "include/rocksdb/db.h"', + '#include "include/rocksdb/memtablerep.h"', + 'namespace rocksdb {', + 'int CountKeys(DB* db) {', + ' Iterator* iter = db->NewIterator();', + ' int n = 0;', + ' for (; iter->Valid(); iter->Next()) n += iter->value();', + ' delete iter;', + ' return n;', + '}', + '} // namespace rocksdb', + ), + }); + try { + // `Valid` and `Next` are IteratorBase's, which Iterator derives from; not MemTableRep::Iterator's. + expect(calls(cg, 'rocksdb::CountKeys')).toEqual([ + 'rocksdb::DB::NewIterator', + 'rocksdb::Iterator::value', + 'rocksdb::IteratorBase::Next', + 'rocksdb::IteratorBase::Valid', + ]); + } finally { + cg.close(); + } + }); + + it('a reference parameter, a value, and a member the class declares in its header', async () => { + const cg = await indexed({ + ...ROCKSDB, + 'util/bit_fields.h': lines( + 'namespace rocksdb {', + 'class BitFields {', + ' public:', + ' int Get() const { return 0; }', + '};', + 'class ConcurrentTest {', + ' public:', + ' struct State {', + ' int Get() const { return 1; }', + ' };', + '};', + '} // namespace rocksdb', + ), + 'db/compaction_job.h': lines( + '#include "include/rocksdb/iterator.h"', + 'namespace rocksdb {', + 'class CompactionJob {', + ' public:', + ' bool Advance();', + ' private:', + ' Iterator* input_;', + '};', + '} // namespace rocksdb', + ), + 'db/compaction_job.cc': lines( + '#include "db/compaction_job.h"', + '#include "include/rocksdb/memtablerep.h"', + '#include "util/bit_fields.h"', + 'namespace rocksdb {', + 'struct State : public BitFields {};', + 'bool IsValid(const Iterator& it) {', + ' return it.Valid();', + '}', + 'int Slot() {', + ' State state;', + ' return state.Get();', + '}', + 'bool CompactionJob::Advance() {', + ' input_->Next();', + ' return input_->Valid();', + '}', + '} // namespace rocksdb', + ), + }); + try { + expect(calls(cg, 'rocksdb::IsValid')).toEqual(['rocksdb::IteratorBase::Valid']); + // Not ConcurrentTest::State::Get, the one class named State that declares Get. + expect(calls(cg, 'rocksdb::Slot')).toEqual(['rocksdb::BitFields::Get']); + expect(calls(cg, 'rocksdb::CompactionJob::Advance')).toEqual(['rocksdb::IteratorBase::Next', 'rocksdb::IteratorBase::Valid']); + } finally { + cg.close(); + } + }); + + it('a nested class is found first from inside a class that inherits it', async () => { + const cg = await indexed({ + ...ROCKSDB, + 'memtable/skiplistrep.cc': lines( + '#include "include/rocksdb/iterator.h"', + '#include "include/rocksdb/memtablerep.h"', + 'namespace rocksdb {', + 'class SkipListRep : public MemTableRep {', + ' public:', + ' bool Check();', + ' private:', + ' Iterator* iter_;', + '};', + 'bool SkipListRep::Check() { return iter_->Valid(); }', + '} // namespace rocksdb', + ), + }); + try { + // MemTableRep's own `Iterator`, which SkipListRep inherits, hides rocksdb::Iterator. + expect(calls(cg, 'rocksdb::SkipListRep::Check')).toEqual(['rocksdb::MemTableRep::Iterator::Valid']); + } finally { + cg.close(); + } + }); + + it('a `using` in the namespace around the call comes before a global class of that name', async () => { + // protocolbuffers/protobuf's upb generator writes `using + // google::protobuf::Descriptor;` inside `namespace upb::generator`, and its + // PHP extension declares a C struct `Descriptor` in a header. + const cg = await indexed({ + 'php/ext/google/protobuf/def.h': lines( + 'typedef struct Descriptor {', + ' int index;', + '} Descriptor;', + ), + 'src/google/protobuf/descriptor.h': lines( + 'namespace google {', + 'namespace protobuf {', + 'class Descriptor {', + ' public:', + ' int FindFieldByNumber(int number) const { return number; }', + '};', + '} // namespace protobuf', + '} // namespace google', + ), + 'upb_generator/common/cpp_to_upb_def.cc': lines( + '#include "src/google/protobuf/descriptor.h"', + 'namespace upb {', + 'namespace generator {', + 'using google::protobuf::Descriptor;', + 'int FindField(const Descriptor* message, int number) {', + ' return message->FindFieldByNumber(number);', + '}', + '} // namespace generator', + '} // namespace upb', + ), + }); + try { + expect(calls(cg, 'upb::generator::FindField')).toEqual(['google::protobuf::Descriptor::FindFieldByNumber']); + } finally { + cg.close(); + } + }); +}); + +describe('a class with no such method gets no guess', () => { + it('a method the class only inherits from a library type', async () => { + // rocksdb's `MultiScanException& ex; ex.what()` reached the Java bindings' `KVException::what`. + const cg = await indexed({ + 'java/rocksjni/kv_helper.h': lines( + 'namespace rocksdb {', + 'class KVException {', + ' public:', + ' const char* what() const { return ""; }', + '};', + '} // namespace rocksdb', + ), + 'include/rocksdb/multi_scan.h': lines( + '#include ', + 'namespace rocksdb {', + 'class MultiScanException : public std::runtime_error {', + ' public:', + ' explicit MultiScanException(const char* message) : std::runtime_error(message) {}', + '};', + '} // namespace rocksdb', + ), + 'tools/db_bench_tool.cc': lines( + '#include "include/rocksdb/multi_scan.h"', + 'namespace rocksdb {', + 'const char* Describe(const MultiScanException& ex) {', + ' return ex.what();', + '}', + '} // namespace rocksdb', + ), + }); + try { + expect(calls(cg, 'rocksdb::Describe')).toEqual([]); + } finally { + cg.close(); + } + }); +}); + +describe('what keeps the call as it was', () => { + it('a template parameter, and a class another source file defines', async () => { + // rocksdb's multi-CF iterator takes its heap type as a template parameter + // named like the class it is used with; version_set.cc defines its file + // pickers, then includes the header whose code uses them. + const cg = await indexed({ + 'util/heap.h': lines( + 'namespace rocksdb {', + 'class BinaryHeap {', + ' public:', + ' void push(int v) {}', + '};', + '} // namespace rocksdb', + ), + 'db/multi_cf_iterator_impl.h': lines( + '#include "util/heap.h"', + 'namespace rocksdb {', + 'template ', + 'class MultiCfIteratorImpl {', + ' public:', + ' void Reset(BinaryHeap& heap) { heap.push(1); }', + '};', + '} // namespace rocksdb', + ), + 'db/version_set.cc': lines( + 'namespace rocksdb {', + 'class FilePicker {', + ' public:', + ' int GetNextFile() { return 0; }', + '};', + '} // namespace rocksdb', + '#include "db/version_set_sync_and_async.h"', + ), + 'db/version_set_sync_and_async.h': lines( + 'namespace rocksdb {', + 'int NextFile() {', + ' FilePicker fp;', + ' return fp.GetNextFile();', + '}', + '} // namespace rocksdb', + ), + }); + try { + // The fork's resolver names no class for a template parameter, whatever the + // project declares (cpp-base-class-lookup), so it guesses no namesake either. + expect(calls(cg, 'rocksdb::MultiCfIteratorImpl::Reset')).toEqual([]); + expect(calls(cg, 'rocksdb::NextFile')).toEqual(['rocksdb::FilePicker::GetNextFile']); + } finally { + cg.close(); + } + }); + + it('a class with a base the index does not link, which may declare the method', async () => { + // protobuf's FlatAllocator derives from a `decltype(…)` that names FlatAllocatorImpl. + const cg = await indexed({ + 'src/google/protobuf/descriptor.cc': lines( + 'namespace google {', + 'namespace protobuf {', + 'namespace internal {', + 'template ', + 'class FlatAllocatorImpl {', + ' public:', + ' void PlanArray(int array_size) {}', + '};', + 'template ', + 'FlatAllocatorImpl ApplyTypeList(int);', + 'class FlatAllocator : public decltype(ApplyTypeList(0)) {};', + '} // namespace internal', + 'void PlanAllocationSize(internal::FlatAllocator& alloc) {', + ' alloc.PlanArray(1);', + '}', + '} // namespace protobuf', + '} // namespace google', + ), + }); + try { + // The declared-type lookup leaves this call alone (the base is unlinked), and the + // fork's fallback guesses no method by its name for a receiver whose class it + // knows but which lacks the method, so the call stays unresolved. + expect(calls(cg, 'google::protobuf::PlanAllocationSize')).toEqual([]); + } finally { + cg.close(); + } + }); + + it('a declaration read from outside the calling function and its class', async () => { + // Reading back from the call reaches an earlier function's local of the + // same name, not the global the call uses, so its type is no evidence. + const cg = await indexed({ + ...ROCKSDB, + 'db/table_cursor.h': lines( + '#include "include/rocksdb/memtablerep.h"', + 'namespace rocksdb {', + 'extern MemTableRep::Iterator* iter;', + '} // namespace rocksdb', + ), + 'db/db_iter.cc': lines( + '#include "include/rocksdb/iterator.h"', + '#include "db/table_cursor.h"', + 'namespace rocksdb {', + 'void Prepare() {', + ' Iterator* iter = nullptr;', + '}', + 'bool Step() {', + ' return iter->Valid();', + '}', + '} // namespace rocksdb', + ), + }); + try { + // The declared-type lookup ignores Prepare's local. The fork's older text scan + // still reads it, though, and links the `Valid` that local's type, Iterator, + // inherits: a known limitation of that scan, not of the lookup this file covers. + expect(calls(cg, 'rocksdb::Step')).toEqual(['rocksdb::IteratorBase::Valid']); + } finally { + cg.close(); + } + }); +}); + +describe('a method the class has gets no refusal', () => { + it('one a base declares in a header and defines in another source file', async () => { + // rocksdb's `DBTest* self; ASSERT_EQ(self->NumTableFilesAtLevel(0), …)`: the + // overloads are indexed only in db_test_util.cc, and every base is linked + // (rocksdb vendors gtest), so a lookup miss must not read as "no such method". + const cg = await indexed({ + 'include/rocksdb/rocksdb_namespace.h': lines('#ifndef ROCKSDB_NAMESPACE', '#define ROCKSDB_NAMESPACE rocksdb', '#endif'), + 'third-party/gtest/gtest.h': lines( + 'namespace testing {', + 'class Test {', + ' public:', + ' virtual ~Test() {}', + '};', + '} // namespace testing', + ), + 'db/db_test_util.h': lines( + '#include "include/rocksdb/rocksdb_namespace.h"', + '#include "third-party/gtest/gtest.h"', + 'namespace ROCKSDB_NAMESPACE {', + 'class ColumnFamilyHandle;', + 'class DBTestBase : public testing::Test {', + ' public:', + ' int NumTableFilesAtLevel(int level, int cf = 0);', + ' int NumTableFilesAtLevel(int level, ColumnFamilyHandle* column_family, int db);', + '};', + '} // namespace ROCKSDB_NAMESPACE', + ), + 'db/db_test_util.cc': lines( + '#include "db/db_test_util.h"', + 'namespace ROCKSDB_NAMESPACE {', + 'int DBTestBase::NumTableFilesAtLevel(int level, int cf) { return level; }', + 'int DBTestBase::NumTableFilesAtLevel(int level, ColumnFamilyHandle* cfh, int db) { return level; }', + '} // namespace ROCKSDB_NAMESPACE', + ), + 'db/db_test.cc': lines( + '#include "db/db_test_util.h"', + 'namespace ROCKSDB_NAMESPACE {', + 'class DBTest : public DBTestBase {};', + 'void MinLevelHelper(DBTest* self) {', + ' ASSERT_EQ(self->NumTableFilesAtLevel(0), 0);', + '}', + '} // namespace ROCKSDB_NAMESPACE', + ), + }); + try { + expect(calls(cg, 'ROCKSDB_NAMESPACE::MinLevelHelper')).toEqual(['ROCKSDB_NAMESPACE::DBTestBase::NumTableFilesAtLevel']); + } finally { + cg.close(); + } + }); +}); diff --git a/__tests__/sync-python-import-binding-retry.test.ts b/__tests__/sync-python-import-binding-retry.test.ts new file mode 100644 index 000000000..870d022d4 --- /dev/null +++ b/__tests__/sync-python-import-binding-retry.test.ts @@ -0,0 +1,289 @@ +/** + * A sync links a reference through a Python import alias once the module appears. + * + * `from .models import User as UserModel` and `from . import helpers as h` + * bind a name their module never declares: it has `User` and `helpers`. Every + * reference through that name — the import itself, a call, a base class, a + * class or function passed as a value — resolves only through the module. Sync + * retries a parked failed ref by its tail (#1240), and these were parked under + * the alias, which neither the module's file nor its symbols carry. So a module + * added after its importers were indexed, one restored after a delete, or one + * that gained the name in a later edit stayed unlinked from them until each + * importer changed or the project was indexed again. #2450 parked the ES-module + * bindings under their module; a Python import names its module by a dotted + * path, relative to the importing file's package when it starts with dots. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import CodeGraph from '../src/index'; +import { DatabaseConnection, getDatabasePath } from '../src/db'; +import { moduleReferenceKeys } from '../src/db/reference-tail'; + +type Files = Record; + +/** The files that reach a module through a name the module declares under another. */ +const IMPORTERS: Files = { + 'app/__init__.py': '', + 'app/sub/__init__.py': '', + 'app/pkg/__init__.py': '', + // A package that re-exports a name from a module beside it. + 'app/store/__init__.py': 'from .user import Account\n', + 'app/main.py': [ + 'from .models import User as UserModel', + 'from . import helpers as h', + 'from app.models import Group as GroupModel', + 'from .pkg import sub as subpkg', + 'from app.pkg import other as other_mod', + 'from .svc import make_thing as mk', + 'from .store import Account as Acct', + '', + 'class Admin(UserModel):', + ' pass', + '', + 'def run(u):', + ' x = mk()', + ' if isinstance(u, UserModel):', + ' return GroupModel(), Acct(), x', + ' return mk', + '', + ].join('\n'), + 'app/sub/deep.py': [ + 'from .. import helpers as hh', + 'from ..models import User as U2', + 'from ..pkg import sub as sp2', + '', + 'def use():', + ' return U2()', + '', + ].join('\n'), + // A module at the root, imported by its absolute name. + 'main.py': 'from settings import Config as Cfg\n\n\ndef load():\n return Cfg()\n', +}; + +const MODULES: Files = { + 'app/models.py': 'class User:\n pass\n\n\nclass Group:\n pass\n', + 'app/helpers.py': 'def x():\n return 1\n', + // A subpackage: what `from .pkg import sub` imports is its `__init__.py`. + 'app/pkg/sub/__init__.py': 'VALUE = 1\n', + 'app/pkg/other.py': 'def f():\n return 2\n', + 'app/svc.py': 'def make_thing():\n return 3\n', + 'app/store/user.py': 'class Account:\n pass\n', + 'settings.py': 'class Config:\n pass\n', +}; + +/** What a full index links through the aliases: `: -> ::`. */ +const LINKS = [ + 'Acct: imports file app/main.py -> class app/store/user.py::Account', + 'Acct: instantiates function app/main.py -> class app/store/user.py::Account', + 'Cfg: imports file main.py -> class settings.py::Config', + 'Cfg: instantiates function main.py -> class settings.py::Config', + 'GroupModel: imports file app/main.py -> class app/models.py::Group', + 'GroupModel: instantiates function app/main.py -> class app/models.py::Group', + 'U2: imports file app/sub/deep.py -> class app/models.py::User', + 'U2: instantiates function app/sub/deep.py -> class app/models.py::User', + 'UserModel: extends class app/main.py -> class app/models.py::User', + 'UserModel: imports file app/main.py -> class app/models.py::User', + 'UserModel: references function app/main.py -> class app/models.py::User', + 'h: imports file app/main.py -> file app/helpers.py::helpers.py', + 'hh: imports file app/sub/deep.py -> file app/helpers.py::helpers.py', + 'mk: calls function app/main.py -> function app/svc.py::make_thing', + 'mk: imports file app/main.py -> function app/svc.py::make_thing', + 'mk: references function app/main.py -> function app/svc.py::make_thing', + 'other_mod: imports file app/main.py -> file app/pkg/other.py::other.py', + 'sp2: imports file app/sub/deep.py -> file app/pkg/sub/__init__.py::__init__.py', + 'subpkg: imports file app/main.py -> file app/pkg/sub/__init__.py::__init__.py', +]; + +/** An alias of a name its module doesn't have, which a full index links to a namesake. */ +const BY_NAME: Files = { + 'app/__init__.py': '', + 'app/lib.py': 'unrelated = 1\n', + 'app/form.py': 'from .lib import FormSchemaBase as FormSchema\n\n\nclass Form(FormSchema):\n pass\n', +}; +const BY_NAME_ADDED: Files = { 'app/schemas.py': 'class FormSchema:\n pass\n' }; + +let roots: string[] = []; +let graphs: CodeGraph[] = []; + +afterEach(() => { + for (const graph of graphs) graph.close(); + graphs = []; + for (const root of roots) fs.rmSync(root, { recursive: true, force: true }); + roots = []; +}); + +function tempRoot(): string { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-sync-pybinding-')); + roots.push(root); + return root; +} + +function write(root: string, files: Files): void { + for (const [file, text] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, file)), { recursive: true }); + fs.writeFileSync(path.join(root, file), text); + } +} + +async function index(root: string): Promise { + const graph = await CodeGraph.init(root, { index: true }); + graphs.push(graph); + return graph; +} + +/** Every edge, by its ends' natural keys, with where it is written and how it resolved. */ +function edgesOf(graph: CodeGraph): string[] { + const keys = new Map( + graph.getFiles() + .flatMap((file) => graph.getNodesInFile(file.path)) + .map((n) => [n.id, `${n.kind} ${n.filePath}:${n.qualifiedName}:${n.startLine}`] as const) + ); + return graph.getOutgoingEdgesFrom([...keys.keys()]) + .map((e) => `${e.kind} ${keys.get(e.source)} -> ${keys.get(e.target) ?? e.target} @${e.line}:${e.column} ${JSON.stringify(e.metadata ?? {})}`) + .sort(); +} + +/** The {@link LINKS} the graph has. */ +function linksOf(graph: CodeGraph): string[] { + const names = new Set(LINKS.map((link) => link.slice(0, link.indexOf(':')))); + const ids = graph.getFiles().flatMap((file) => graph.getNodesInFile(file.path)).map((n) => n.id); + const links: string[] = []; + for (const edge of graph.getOutgoingEdgesFrom(ids)) { + const ref = edge.metadata?.refName; + if (typeof ref !== 'string' || !names.has(ref)) continue; + const source = graph.getNode(edge.source)!; + const target = graph.getNode(edge.target)!; + links.push(`${ref}: ${edge.kind} ${source.kind} ${source.filePath} -> ${target.kind} ${target.filePath}::${target.name}`); + } + return [...new Set(links)].sort(); +} + +/** The final tree, indexed from scratch in a folder of its own: what the sync has to match. */ +async function fullIndexOf(files: Files): Promise { + const root = tempRoot(); + write(root, files); + return edgesOf(await index(root)); +} + +describe('sync links references through a Python import alias to their module', () => { + it('when the module is added after its importers', async () => { + const root = tempRoot(); + write(root, IMPORTERS); + const graph = await index(root); + expect(linksOf(graph)).toEqual([]); + + write(root, MODULES); + const result = await graph.sync(); + // The importers did not change: only the retry of their parked refs links them. + expect(result.filesAdded).toBe(Object.keys(MODULES).length); + expect(result.filesModified).toBe(0); + expect(linksOf(graph)).toEqual(LINKS); + expect(edgesOf(graph)).toEqual(await fullIndexOf({ ...IMPORTERS, ...MODULES })); + }, 120_000); + + it('when the module comes back after it was deleted', async () => { + const root = tempRoot(); + write(root, { ...IMPORTERS, ...MODULES }); + const graph = await index(root); + expect(linksOf(graph)).toEqual(LINKS); + + for (const file of Object.keys(MODULES)) fs.rmSync(path.join(root, file)); + expect((await graph.sync()).filesRemoved).toBe(Object.keys(MODULES).length); + expect(linksOf(graph)).toEqual([]); + + write(root, MODULES); + expect((await graph.sync()).filesAdded).toBe(Object.keys(MODULES).length); + expect(linksOf(graph)).toEqual(LINKS); + expect(edgesOf(graph)).toEqual(await fullIndexOf({ ...IMPORTERS, ...MODULES })); + }, 120_000); + + it('when an edit gives the module the name its importers import', async () => { + const root = tempRoot(); + write(root, IMPORTERS); + write(root, Object.fromEntries(Object.keys(MODULES).map((file) => [file, 'PLACEHOLDER = 0\n']))); + const graph = await index(root); + + write(root, MODULES); + const result = await graph.sync(); + expect(result.filesModified).toBe(Object.keys(MODULES).length); + expect(result.filesAdded).toBe(0); + expect(linksOf(graph)).toEqual(LINKS); + expect(edgesOf(graph)).toEqual(await fullIndexOf({ ...IMPORTERS, ...MODULES })); + }, 120_000); + + it('and by its own name when its module never declares it', async () => { + // As vben's case for an ES module: the module has no such name, so a full + // index links the alias by its own name to the class that carries it. + const root = tempRoot(); + write(root, BY_NAME); + const graph = await index(root); + + write(root, BY_NAME_ADDED); + expect((await graph.sync()).filesAdded).toBe(1); + const formSchema = edgesOf(graph).filter((edge) => edge.includes('"refName":"FormSchema"')); + expect(formSchema).toHaveLength(2); + expect(formSchema.every((edge) => edge.includes('-> class app/schemas.py:'))).toBe(true); + expect(edgesOf(graph)).toEqual(await fullIndexOf({ ...BY_NAME, ...BY_NAME_ADDED })); + }, 120_000); +}); + +describe('a failed reference through a Python import alias', () => { + it('waits for its module, and every other reference keeps its own tail', async () => { + const root = tempRoot(); + write(root, { + ...IMPORTERS, + ...BY_NAME, + // An unaliased import, a standard-library alias and a member read + // through an alias wait for the name they use. + 'app/other.py': [ + 'from .models import Group', + 'from collections import OrderedDict as OD', + 'from .models import User as UserModel', + '', + 'def build():', + ' return Group(), OD(), UserModel.create()', + '', + ].join('\n'), + }); + (await CodeGraph.init(root, { index: true })).close(); + + const db = DatabaseConnection.open(getDatabasePath(root)); + try { + const rows = db.getDb() + .prepare("SELECT reference_name AS name, reference_kind AS kind, file_path AS file, name_tail AS tail FROM unresolved_refs WHERE status = 'failed'") + .all() as Array<{ name: string; kind: string; file: string; tail: string }>; + const tailOf = (file: string, name: string, kind: string): string | undefined => + rows.find((r) => r.file === file && r.name === name && r.kind === kind)?.tail; + + // A module named relative to the importing file's package, or absolutely. + expect(tailOf('app/main.py', 'UserModel', 'imports')).toBe('module:models'); + expect(tailOf('app/main.py', 'UserModel', 'extends')).toBe('module:models'); + expect(tailOf('app/main.py', 'GroupModel', 'calls')).toBe('module:models'); + expect(tailOf('app/sub/deep.py', 'U2', 'calls')).toBe('module:models'); + expect(tailOf('app/main.py', 'mk', 'calls')).toBe('module:svc'); + // `from . import helpers` imports from the file's own package, `..` from the one above. + expect(tailOf('app/main.py', 'h', 'imports')).toBe('module:app'); + expect(tailOf('app/sub/deep.py', 'hh', 'imports')).toBe('module:app'); + expect(tailOf('app/main.py', 'subpkg', 'imports')).toBe('module:pkg'); + expect(tailOf('app/main.py', 'other_mod', 'imports')).toBe('module:pkg'); + expect(tailOf('app/main.py', 'Acct', 'calls')).toBe('module:store'); + expect(tailOf('main.py', 'Cfg', 'calls')).toBe('module:settings'); + // Still found by its own name too: sync's lookup by whole name. + expect(tailOf('app/form.py', 'FormSchema', 'extends')).toBe('module:lib'); + + expect(tailOf('app/other.py', 'Group', 'calls')).toBe('Group'); + expect(tailOf('app/other.py', 'OD', 'calls')).toBe('OD'); + expect(tailOf('app/other.py', 'UserModel.create', 'calls')).toBe('create'); + + // A submodule's keys hold its package's: a module file's folder, and + // the folder above a subpackage's `__init__.py`. + expect(moduleReferenceKeys('app/helpers.py')).toContain('module:app'); + expect(moduleReferenceKeys('app/pkg/sub/__init__.py')).toEqual(expect.arrayContaining(['module:sub', 'module:pkg'])); + expect(moduleReferenceKeys('pkg/__init__.py')).toEqual(['module:pkg', 'module:__init__.py', 'module:__init__']); + } finally { + db.close(); + } + }, 60_000); +}); diff --git a/codegraph-kernel/src/resolve/cpp_aliases.rs b/codegraph-kernel/src/resolve/cpp_aliases.rs index 76ce08b47..7046a0aa7 100644 --- a/codegraph-kernel/src/resolve/cpp_aliases.rs +++ b/codegraph-kernel/src/resolve/cpp_aliases.rs @@ -119,6 +119,27 @@ impl KernelResolver { pub(super) fn cpp_alias_expansion(&mut self, raw: &str, r: &ResolveRefIn, depth: u32, included: &HashSet) -> Res> { if depth > 4 { return Ok(Some(CppAlias { raw: None, site: r.clone(), pointer: false, reference: false })); } let Some(parts) = type_segments(raw) else { return Ok(None); }; + let Some((rhs, declaration, consumed)) = self.cpp_alias_declaration(raw, &parts, r, depth, included)? else { return Ok(None); }; + let mut site = r.clone().at(&declaration); site.from_node_id = declaration.id.clone(); site.column = declaration.start_column; + let pointer = pointer_type(&rhs); + let reference = reference_type(&rhs); + let target = type_segments(&rhs); + let dependent = match &target { Some(target) => !rhs.trim_start().starts_with("::") && self.cpp_template_parameter(&declaration, &target[0], r)?, None => false }; + if rhs.contains("typename") || dependent { + return Ok(Some(CppAlias { raw: None, site, pointer, reference })); + } + let Some(mut target) = target else { return Ok(Some(CppAlias { raw: None, site, pointer, reference })); }; + target.extend_from_slice(&parts[consumed..]); + let target = format!("{}{}", if rhs.trim_start().starts_with("::") { "::" } else { "" }, target.join("::")); + if let Some(mut next) = self.cpp_alias_expansion(&target, &site, depth + 1, included)? { next.pointer |= pointer; next.reference |= reference; return Ok(Some(next)); } + Ok(Some(CppAlias { raw: Some(target), site, pointer, reference })) + } + + /// The alias `raw` (split into `parts`) names, looked up from `r` as C++ looks a type name up: + /// what it is declared as, as written (template arguments and all), the node declaring it (the + /// caller, for a typedef local to it) and how many of `parts` name the alias. None when `raw` + /// names no alias. + pub(super) fn cpp_alias_declaration(&mut self, raw: &str, parts: &[String], r: &ResolveRefIn, depth: u32, included: &HashSet) -> Res, usize)>> { let caller = self.node_by_id(&r.from_node_id)?; let mut local = None; if !raw.trim().starts_with("::") { @@ -172,20 +193,7 @@ impl KernelResolver { alias = Some((rhs, found, consumed)); break; } } - let Some((rhs, declaration, consumed)) = alias else { return Ok(None); }; - let mut site = r.clone().at(&declaration); site.from_node_id = declaration.id.clone(); site.column = declaration.start_column; - let pointer = pointer_type(&rhs); - let reference = reference_type(&rhs); - let target = type_segments(&rhs); - let dependent = match &target { Some(target) => !rhs.trim_start().starts_with("::") && self.cpp_template_parameter(&declaration, &target[0], r)?, None => false }; - if rhs.contains("typename") || dependent { - return Ok(Some(CppAlias { raw: None, site, pointer, reference })); - } - let Some(mut target) = target else { return Ok(Some(CppAlias { raw: None, site, pointer, reference })); }; - target.extend_from_slice(&parts[consumed..]); - let target = format!("{}{}", if rhs.trim_start().starts_with("::") { "::" } else { "" }, target.join("::")); - if let Some(mut next) = self.cpp_alias_expansion(&target, &site, depth + 1, included)? { next.pointer |= pointer; next.reference |= reference; return Ok(Some(next)); } - Ok(Some(CppAlias { raw: Some(target), site, pointer, reference })) + Ok(alias) } } diff --git a/codegraph-kernel/src/resolve/cpp_implicit_this.rs b/codegraph-kernel/src/resolve/cpp_implicit_this.rs index f8c31547a..62a9087b1 100644 --- a/codegraph-kernel/src/resolve/cpp_implicit_this.rs +++ b/codegraph-kernel/src/resolve/cpp_implicit_this.rs @@ -166,7 +166,7 @@ impl KernelResolver { /// class's own, for code in its body. A function node counts only when its definition is /// written `Class::name(…)`. None for a free function, and for one defined through a namespace /// (`void detail::helper() {}` is a method node of no class). - fn cpp_member_owner(&mut self, caller: &KNode) -> Res> { + pub(super) fn cpp_member_owner(&mut self, caller: &KNode) -> Res> { if class_kind(&caller.kind) { return Ok(Some(caller.qualified_name.clone())); } if caller.kind != "method" && caller.kind != "function" { return Ok(None); } let Some(cut) = caller.qualified_name.rfind("::").filter(|c| *c > 0) else { return Ok(None) }; @@ -299,7 +299,7 @@ impl KernelResolver { /// C++ class `cls`'s method `name`: its own, else the nearest one of a class it derives from, /// through the base edges every declaration of `cls` has. - fn cpp_method_of(&mut self, cls: &KNode, name: &str, r: &ResolveRefIn, depth: usize, seen: &mut HashSet) -> Res>> { + pub(super) fn cpp_method_of(&mut self, cls: &KNode, name: &str, r: &ResolveRefIn, depth: usize, seen: &mut HashSet) -> Res>> { let own: Vec<_> = self.nodes_by_qualified_name(&format!("{}::{name}", cls.qualified_name))? .iter().filter(|n| n.kind == "method" && c_family(&n.language)).cloned().collect(); if !own.is_empty() { return Ok(prefer_call_site_file(own, &r.file_path).into_iter().next()); } @@ -359,7 +359,7 @@ impl KernelResolver { /// The base specifiers a C or C++ class declaration writes (`public Base`, `private /// Mixin`), read from its own head; none for a forward declaration. - fn cpp_base_specifiers(&mut self, decl: &KNode) -> Vec { + pub(super) fn cpp_base_specifiers(&mut self, decl: &KNode) -> Vec { let Some(source) = self.read_file(&decl.file_path) else { return Vec::new() }; let lines = source.cpp_code_lines(); let first = (decl.start_line - 1).max(0) as usize; diff --git a/codegraph-kernel/src/resolve/cpp_range_for.rs b/codegraph-kernel/src/resolve/cpp_range_for.rs new file mode 100644 index 000000000..1a7dc5767 --- /dev/null +++ b/codegraph-kernel/src/resolve/cpp_range_for.rs @@ -0,0 +1,480 @@ +//! The element type of what a C++ range-based `for` over `auto` iterates, so the loop's variable +//! has a type: `for (auto cfd : cfds)` over a `const autovector&` parameter +//! calls `ColumnFamilyData`'s methods, not those of an earlier variable named `cfd`. +//! +//! Port of upstream's `cppAutoLoopElement` and the helpers it reads with (name-matcher.ts at +//! 01ab45d0, #2492); read the original with `git show 01ab45d0:src/resolution/name-matcher.ts`. +//! The range is read from the syntax tree rather than from lines of text: +//! - a parameter or local of the calling function (an `auto` local through its initializer), an +//! enclosing `auto` loop's variable, or a data member of the caller's class; +//! - `*x`, `x.m`, `x->m`, `this`, and a call of a function or method whose declaration writes its +//! return type (`FieldNumberOrder(desc)`, `vstorage->GetBlobFiles()`); +//! +//! with aliases followed, template arguments and all, from where each is declared. Its element is a +//! standard or abseil container's first template argument (protobuf's repeated fields' and rocksdb's +//! `autovector`'s too), or what a project class's `begin()` returns, dereferenced. Every type is +//! looked up where it is written. A type a template parameter decides, or one that can't be read, +//! gives no element, and the call resolves as it did before. +use super::cpp_aliases::type_segments; +use super::cpp_receivers::{cpp_ast_declared_type, cpp_binding_type, cpp_declarator_name, cpp_held_type}; +use super::iteration::{descendant_for_position, named_children}; +use super::*; + +type TsNode<'t> = tree_sitter::Node<'t>; + +/// A C++ type as written, and the site it is written at, which its names are looked up from. +#[derive(Clone)] +pub(super) struct CppWritten { + pub text: String, + pub site: ResolveRefIn, +} + +/// The standard library's and abseil's containers whose elements are their first template +/// argument. A map's elements are key and value pairs, which no call this resolves is made on. +const LIBRARY_SEQUENCES: &[&str] = &[ + "vector", "list", "forward_list", "deque", "set", "multiset", "unordered_set", "unordered_multiset", "array", "span", + "initializer_list", "Span", "InlinedVector", "FixedArray", "flat_hash_set", "node_hash_set", "btree_set", "btree_multiset", +]; +/// Widely used project containers of the same kind: protobuf's repeated fields, rocksdb's `autovector`. +const PROJECT_SEQUENCES: &[&str] = &["RepeatedField", "RepeatedPtrField", "autovector"]; +const MAX_DEPTH: u32 = 4; + +/// A written type without nullability annotations (`const FieldDescriptor* absl_nonnull`) or +/// cv-qualifiers, which are no part of the name it is looked up by. +pub(super) fn cpp_plain_type(text: &str) -> String { + let text = re!(r"\b(?:absl_nonnull|absl_nullable|absl_nullability_unknown|_Nonnull|_Nullable|_Null_unspecified|const|volatile)\b").replace_all(text, " "); + text.split_whitespace().collect::>().join(" ") +} + +/// A type written `auto`, `auto&`, `const auto*`. +pub(super) fn is_cpp_auto(raw: &str) -> bool { + type_segments(raw).is_some_and(|parts| parts.len() == 1 && parts[0] == "auto") +} + +/// Whether `text` is a pointer: a `*` outside its template arguments. +fn top_level_pointer(text: &str) -> bool { + let mut depth = 0i32; + text.chars().any(|c| { + match c { '<' => depth += 1, '>' => depth -= 1, _ => {} } + c == '*' && depth == 0 + }) +} + +/// The template arguments of `text`'s first template, split at its top-level commas. +fn template_arguments(text: &str) -> Option> { + let open = text.find('<')?; + let mut depth = 0i32; + let mut arguments = Vec::new(); + let mut current = String::new(); + for c in text[open..].chars() { + match c { + '<' | '(' => { + depth += 1; + if depth == 1 { continue; } + } + '>' | ')' => { + depth -= 1; + if depth == 0 { + arguments.push(current.trim().to_string()); + return Some(arguments); + } + } + ',' if depth == 1 => { + arguments.push(current.trim().to_string()); + current.clear(); + continue; + } + _ => {} + } + current.push(c); + } + None +} + +/// The type `*x` has when `x` has the type `text`: what a pointer points at, or what a standard +/// smart pointer or optional holds. None for any other type. +pub(super) fn cpp_pointee(text: &str) -> Option { + let text = text.trim().trim_end_matches('&').trim(); + let mut depth = 0i32; + for (i, c) in text.char_indices().rev() { + match c { + '>' => depth += 1, + '<' => depth -= 1, + '*' if depth == 0 => return Some(text[..i].trim().to_string()), + _ => {} + } + } + let plain = cpp_plain_type(text); + let held = re!(r"^(?:::)?std::(?:unique_ptr|shared_ptr|optional)\s*<(.*)>$").captures(&plain)?; + cpp_held_type(&held[1]) +} + +/// The function declarator of a declarator that declares a function (`f()`, `*f()`, `&f()`), not +/// a variable. +fn declared_function(mut declarator: TsNode) -> Option { + loop { + match declarator.kind() { + "function_declarator" => return Some(declarator), + "pointer_declarator" | "reference_declarator" | "parenthesized_declarator" => { + declarator = declarator.child_by_field_name("declarator").or_else(|| declarator.named_child(0))?; + } + _ => return None, + } + } +} + +fn without_spaces(text: &str) -> String { text.chars().filter(|c| !c.is_whitespace()).collect() } + +/// The initializer of `name`'s declaration in the block `parent`, before `at`: `vstorage-> +/// GetBlobFiles()` in `const auto& blob_files = vstorage->GetBlobFiles();`. The same declarations +/// `cpp_binding_type` reads. +fn binding_initializer<'t>(parent: TsNode<'t>, at: TsNode<'t>, name: &str, source: &SourceFile) -> Option> { + if !matches!(parent.kind(), "compound_statement" | "declaration_list" | "translation_unit") { return None; } + for declaration in named_children(parent).into_iter().rev().filter(|n| n.kind() == "declaration" && n.end_byte() <= at.start_byte()) { + let mut cursor = declaration.walk(); + let declarators: Vec<_> = declaration.children_by_field_name("declarator", &mut cursor).collect(); + if let Some(declarator) = declarators.into_iter().find(|d| cpp_declarator_name(*d, source) == name) { + if declarator.kind() != "init_declarator" { return None; } + return declarator.child_by_field_name("value").filter(|v| !matches!(v.kind(), "initializer_list" | "argument_list")); + } + } + None +} + +fn site_at(r: &ResolveRefIn, node: &KNode) -> ResolveRefIn { + let mut site = r.clone().at(node); + site.from_node_id = node.id.clone(); + site.column = node.start_column; + site +} + +fn cpp_class_node(node: &KNode) -> bool { + matches!(node.language.as_str(), "c" | "cpp") && matches!(node.kind.as_str(), "class" | "struct" | "union") +} + +impl KernelResolver { + /// The element type of the range the `auto` range-for `range_for` (in `r`'s file) iterates, as + /// written where the range's type is declared: what the loop's variable is. None when it can't be + /// read reliably, or a template parameter decides it. + pub(super) fn cpp_range_for_element(&mut self, range_for: TsNode, source: &Rc, r: &ResolveRefIn, depth: u32) -> Res> { + if depth > MAX_DEPTH { return Ok(None); } + let Some(right) = range_for.child_by_field_name("right") else { return Ok(None) }; + let Some(range) = self.cpp_expression_written(right, source, r, depth + 1)? else { return Ok(None) }; + let Some(element) = self.cpp_range_element_of(range, r, 0)? else { return Ok(None) }; + let Some(element) = self.cpp_follow_written_aliases(element, r)? else { return Ok(None) }; + if self.cpp_written_template_parameter(&element)? { return Ok(None); } + Ok(Some(element)) + } + + /// The project class of the variable `receiver` an `auto` range-for opening on line `line` + /// (0-based) declares, when the call at `r` is in that loop's body: what its range's element is, + /// or points at. None for any other loop, an element that is no project class (a smart pointer + /// or a standard type), or one that can't be read. + pub(super) fn cpp_enclosing_range_element(&mut self, receiver: &str, line: usize, r: &ResolveRefIn) -> Res>> { + if r.language != "cpp" { return Ok(None); } + let Some(source) = self.read_file(&r.file_path) else { return Ok(None) }; + let Some(tree) = self.parsed_tree(&source, r) else { return Ok(None) }; + let mut at = descendant_for_position(tree.root_node(), source.text(), ((r.line - 1).max(0) as usize, r.column.max(0) as usize + 1)); + while let Some(parent) = at.parent() { + // A nearer declaration of the name, `{ Gadget* w = g; w->Self(); }` in the body, hides the loop's. + if cpp_binding_type(parent, at, receiver, &source).is_some() { return Ok(None); } + if parent.kind() == "for_range_loop" && parent.start_position().row == line + && parent.child_by_field_name("body").is_some_and(|body| body.id() == at.id()) + && parent.child_by_field_name("declarator").is_some_and(|d| cpp_declarator_name(d, &source) == receiver) { + let (Some(ty), Some(declarator)) = (parent.child_by_field_name("type"), parent.child_by_field_name("declarator")) else { return Ok(None) }; + if !is_cpp_auto(&cpp_ast_declared_type(ty, declarator, &source)) { return Ok(None); } + let Some(element) = self.cpp_range_for_element(parent, &source, r, 0)? else { return Ok(None) }; + let element = if top_level_pointer(&element.text) { + match cpp_pointee(&element.text) { Some(text) if !top_level_pointer(&text) => CppWritten { text, site: element.site }, _ => return Ok(None) } + } else { element }; + if re!(r"^(?:::)?(?:std|absl)::").is_match(element.text.trim()) { return Ok(None); } + return self.cpp_written_class(&element, r); + } + at = parent; + } + Ok(None) + } + + /// Whether the type `written` names is a template parameter's, which a template argument decides. + fn cpp_written_template_parameter(&mut self, written: &CppWritten) -> Res { + let Some(parts) = type_segments(&written.text) else { return Ok(false) }; + let Some(node) = self.node_by_id(&written.site.from_node_id)? else { return Ok(false) }; + self.cpp_template_parameter(&node, &parts[0], &written.site) + } + + /// `written` once the aliases it is written with are followed, each from its own declaration: + /// rocksdb's `const BlobFiles&` is `std::vector>`. None when an + /// alias on the way can't be read: a member of an aliased type, `typename`, `decltype`. + pub(super) fn cpp_follow_written_aliases(&mut self, written: CppWritten, r: &ResolveRefIn) -> Res> { + let included = self.namespace_visible_files(&r.file_path, "cpp")?; + let mut current = written; + for _ in 0..4 { + let Some(parts) = type_segments(¤t.text) else { return Ok(Some(current)) }; + if parts[0] == "std" { return Ok(Some(current)); } + let Some((rhs, declaration, consumed)) = self.cpp_alias_declaration(¤t.text, &parts, ¤t.site, 0, &included)? else { return Ok(Some(current)) }; + if consumed < parts.len() || rhs.trim().is_empty() || re!(r"\b(?:typename|decltype)\b").is_match(&rhs) { return Ok(None); } + let text = if top_level_pointer(¤t.text) { format!("{}*", rhs.trim()) } else { rhs.trim().to_string() }; + let site = if declaration.id == current.site.from_node_id { current.site.clone() } else { site_at(¤t.site, &declaration) }; + current = CppWritten { text, site }; + } + Ok(None) + } + + /// The project class `written` names, looked up as C++ looks it up where it is written: the + /// classes around that place and their bases first, then the namespaces. + pub(super) fn cpp_written_class(&mut self, written: &CppWritten, r: &ResolveRefIn) -> Res>> { + let Some(written) = self.cpp_follow_written_aliases(written.clone(), r)? else { return Ok(None) }; + if top_level_pointer(&written.text) || self.cpp_written_template_parameter(&written)? { return Ok(None); } + let Some(parts) = type_segments(&written.text) else { return Ok(None) }; + let plain = cpp_plain_type(&written.text); + let primary = plain.split('<').next().unwrap_or(&plain).trim_end_matches('&').trim(); + // `::Registry` is the global one, never one nested in the classes around it. + let mut owner = if parts.len() == 1 && !primary.starts_with("::") { self.cpp_class_scope_type(&parts[0], &written.site)? } else { None }; + if owner.is_none() { + let included = self.namespace_visible_files(&r.file_path, "cpp")?; + owner = self.cpp_type_owner_visible(primary, &written.site, 0, false, &included)?; + } + Ok(owner.filter(|n| cpp_class_node(n))) + } + + /// The C++ class `cls`'s methods named `name`: its own declarations and definitions, else the + /// nearest one a class it derives from has. + pub(super) fn cpp_methods_named(&mut self, cls: &KNode, name: &str, r: &ResolveRefIn) -> Res>> { + let own: Vec<_> = self.nodes_by_qualified_name(&format!("{}::{name}", cls.qualified_name))?.iter() + .filter(|n| matches!(n.kind.as_str(), "method" | "function") && matches!(n.language.as_str(), "c" | "cpp")).cloned().collect(); + if !own.is_empty() { return Ok(own); } + let mut seen = HashSet::new(); + Ok(self.cpp_method_of(cls, name, r, 0, &mut seen)?.into_iter().collect()) + } + + /// What one of `functions` (overloads, a declaration and its definition) returns, as written + /// where it is declared. None when overloads taking different parameters return different + /// types, since which one a call picks is not read; a `const` and a non-`const` overload with + /// the same parameters (`items()`, `items() const`) give the first one's. + fn cpp_return_type_of(&mut self, functions: &[Arc], r: &ResolveRefIn) -> Res> { + let mut first: Option<(CppWritten, String)> = None; + for function in functions { + let Some((text, parameters)) = self.cpp_written_return_type(function, r)? else { continue }; + match &first { + None => first = Some((CppWritten { text, site: site_at(r, function) }, parameters)), + Some((seen, seen_parameters)) => if without_spaces(&seen.text) != without_spaces(&text) && *seen_parameters != parameters { return Ok(None); }, + } + } + Ok(first.map(|(written, _)| written)) + } + + /// The return type of a C or C++ function as its declaration writes it: `std::vector` (the index keeps only `vector`), and its parameter list without spaces. + /// None for a constructor, `void`, `auto` or `decltype(…)`. + fn cpp_written_return_type(&mut self, function: &Arc, r: &ResolveRefIn) -> Res> { + let Some(source) = self.read_file(&function.file_path) else { return Ok(None) }; + let site = site_at(r, function); + let Some(tree) = self.parsed_tree(&source, &site) else { return Ok(None) }; + let mut node = descendant_for_position(tree.root_node(), source.text(), ((function.start_line - 1).max(0) as usize, function.start_column.max(0) as usize + 1)); + let declaration = loop { + match node.kind() { + "function_definition" | "field_declaration" | "declaration" => break node, + "template_declaration" => match named_children(node).into_iter().find(|n| matches!(n.kind(), "function_definition" | "field_declaration" | "declaration")) { + Some(inner) => break inner, + None => return Ok(None), + }, + _ => {} + } + let Some(parent) = node.parent() else { return Ok(None) }; + node = parent; + }; + let (Some(ty), Some(declarator)) = (declaration.child_by_field_name("type"), declaration.child_by_field_name("declarator")) else { return Ok(None) }; + let Some(function_declarator) = declared_function(declarator) else { return Ok(None) }; + let raw = cpp_plain_type(&cpp_ast_declared_type(ty, declarator, &source)); + if raw.is_empty() || raw == "void" || is_cpp_auto(&raw) || raw.contains("decltype") { return Ok(None); } + let parameters = function_declarator.child_by_field_name("parameters").map(|p| without_spaces(&source.text()[p.byte_range()])).unwrap_or_default(); + Ok(Some((raw, parameters))) + } + + /// The type the C++ class `cls` declares its data member `field` with, as written in the class. + fn cpp_data_member_written(&mut self, cls: &str, field: &str, r: &ResolveRefIn) -> Res> { + let visible = self.namespace_visible_files(&r.file_path, "cpp")?; + let decls: Vec<_> = self.nodes_by_qualified_name(cls)?.iter().filter(|n| cpp_class_node(n) && visible.contains(&n.file_path)).cloned().collect(); + for decl in decls { + if let Some(text) = self.cpp_field_type(&decl, field, r)? { + return Ok(Some(CppWritten { text, site: site_at(r, &decl) })); + } + } + Ok(None) + } + + /// The type of a C++ expression in `r`'s caller, as written where it is declared. None for + /// anything else, or a name declared only outside the caller and its class. + fn cpp_expression_written(&mut self, node: TsNode, source: &Rc, r: &ResolveRefIn, depth: u32) -> Res> { + if depth > MAX_DEPTH { return Ok(None); } + match node.kind() { + "parenthesized_expression" => match named_children(node).as_slice() { + [only] => self.cpp_expression_written(*only, source, r, depth + 1), + _ => Ok(None), + }, + "this" => { + let Some(caller) = self.node_by_id(&r.from_node_id)? else { return Ok(None) }; + Ok(self.cpp_member_owner(&caller)?.map(|owner| CppWritten { text: format!("::{owner}*"), site: r.clone() })) + } + "identifier" => self.cpp_name_written(node, source, r, depth), + "pointer_expression" => { + if node.child_by_field_name("operator").map(|o| o.kind()) != Some("*") { return Ok(None); } + let Some(argument) = node.child_by_field_name("argument") else { return Ok(None) }; + let Some(pointer) = self.cpp_expression_written(argument, source, r, depth + 1)? else { return Ok(None) }; + let Some(pointer) = self.cpp_follow_written_aliases(pointer, r)? else { return Ok(None) }; + Ok(cpp_pointee(&pointer.text).map(|text| CppWritten { text, site: pointer.site })) + } + // `compact_->sub_compact_states`: a data member of the object's class. + "field_expression" => { + let Some((cls, field)) = self.cpp_member_of_expression(node, source, r, depth)? else { return Ok(None) }; + self.cpp_data_member_written(&cls.qualified_name, &field, r) + } + "call_expression" => { + let Some(function) = node.child_by_field_name("function") else { return Ok(None) }; + match function.kind() { + // `recv.method(…)`, `recv->method(…)`: the method of the receiver's class. + "field_expression" => { + let Some((cls, method)) = self.cpp_member_of_expression(function, source, r, depth)? else { return Ok(None) }; + let methods = self.cpp_methods_named(&cls, &method, r)?; + self.cpp_return_type_of(&methods, r) + } + // `f(…)`, `ns::f(…)`, `Class::f(…)`: looked up from the caller's scope outwards. + "identifier" | "qualified_identifier" => { + let spelled: String = source.text()[function.byte_range()].split_whitespace().collect(); + if !re!(r"^(?:[A-Za-z_]\w*::)*[A-Za-z_]\w*$").is_match(&spelled) { return Ok(None); } + let Some(caller) = self.node_by_id(&r.from_node_id)? else { return Ok(None) }; + let scopes: Vec = caller.qualified_name.split("::").map(str::to_string).collect(); + for i in (0..scopes.len()).rev() { + let qualified = if i == 0 { spelled.clone() } else { format!("{}::{spelled}", scopes[..i].join("::")) }; + let functions: Vec<_> = self.nodes_by_qualified_name(&qualified)?.iter() + .filter(|n| matches!(n.kind.as_str(), "method" | "function") && matches!(n.language.as_str(), "c" | "cpp")).cloned().collect(); + if !functions.is_empty() { return self.cpp_return_type_of(&functions, r); } + } + Ok(None) + } + _ => Ok(None), + } + } + _ => Ok(None), + } + } + + /// The class of the object a member expression `x.m` or `x->m` names a member of, and `m`. + fn cpp_member_of_expression(&mut self, member: TsNode, source: &Rc, r: &ResolveRefIn, depth: u32) -> Res, String)>> { + let (Some(argument), Some(field)) = (member.child_by_field_name("argument"), member.child_by_field_name("field")) else { return Ok(None) }; + let name = source.text()[field.byte_range()].to_string(); + if !re!(r"^[A-Za-z_]\w*$").is_match(&name) { return Ok(None); } + let operator = super::awaited::strip_ts_comments(&source.text()[argument.end_byte()..field.start_byte()]); + let Some(object) = self.cpp_expression_written(argument, source, r, depth + 1)? else { return Ok(None) }; + let Some(object) = self.cpp_follow_written_aliases(object, r)? else { return Ok(None) }; + let object = match operator.trim() { + "->" => match cpp_pointee(&object.text) { + Some(text) => CppWritten { text, site: object.site }, + None => return Ok(None), + }, + "." if !top_level_pointer(&object.text) => object, + _ => return Ok(None), + }; + Ok(self.cpp_written_class(&object, r)?.map(|cls| (cls, name))) + } + + /// The declared type of the name `node` in `r`'s caller: a parameter or local (an `auto` local + /// through its initializer), an enclosing loop's variable, else a data member of its class. + fn cpp_name_written(&mut self, node: TsNode, source: &Rc, r: &ResolveRefIn, depth: u32) -> Res> { + let name = &source.text()[node.byte_range()]; + let mut at = node; + while let Some(parent) = at.parent() { + if let Some(raw) = cpp_binding_type(parent, at, name, source) { + if raw.is_empty() || raw.contains('[') { return Ok(None); } + if !is_cpp_auto(&raw) { return Ok(Some(CppWritten { text: raw, site: r.clone() })); } + // `const auto& blob_files = vstorage->GetBlobFiles();`: what its initializer is. + let Some(value) = binding_initializer(parent, at, name, source) else { return Ok(None) }; + return self.cpp_expression_written(value, source, r, depth + 1); + } + if parent.kind() == "for_range_loop" && parent.child_by_field_name("body").is_some_and(|body| body.id() == at.id()) + && parent.child_by_field_name("declarator").is_some_and(|d| cpp_declarator_name(d, source) == name) { + let (Some(ty), Some(declarator)) = (parent.child_by_field_name("type"), parent.child_by_field_name("declarator")) else { return Ok(None) }; + let raw = cpp_ast_declared_type(ty, declarator, source); + if !is_cpp_auto(&raw) { return Ok(Some(CppWritten { text: raw, site: r.clone() })); } + return self.cpp_range_for_element(parent, source, r, depth + 1); + } + at = parent; + } + // Declared nowhere in the calling function (`versions_` in db_impl_write.cc, whose class is + // declared in db_impl.h): a data member of its class. + let Some(caller) = self.node_by_id(&r.from_node_id)? else { return Ok(None) }; + let Some(owner) = self.cpp_member_owner(&caller)? else { return Ok(None) }; + self.cpp_data_member_written(&owner, name, r) + } + + /// The element type of a range of the type `range`, as written: a container's first template + /// argument, or, for one of the project's classes, what its `begin()` returns, dereferenced. + fn cpp_range_element_of(&mut self, range: CppWritten, r: &ResolveRefIn, hops: u32) -> Res> { + if hops > 2 { return Ok(None); } + let Some(range) = self.cpp_follow_written_aliases(range, r)? else { return Ok(None) }; + if top_level_pointer(&range.text) { return Ok(None); } + let Some(parts) = type_segments(&range.text) else { return Ok(None) }; + let last = parts.last().map(String::as_str).unwrap_or(""); + let library = LIBRARY_SEQUENCES.contains(&last); + if library || PROJECT_SEQUENCES.contains(&last) { + // A project type of a standard container's name, written without `std::` or `absl::`, + // is that type, not the container. + if library && parts[0] != "std" && parts[0] != "absl" && self.nodes_by_name(last)?.iter().any(|n| matches!(n.language.as_str(), "c" | "cpp") + && matches!(n.kind.as_str(), "class" | "struct" | "union" | "type_alias")) { return Ok(None); } + let Some(first) = template_arguments(&range.text).and_then(|a| a.into_iter().next()) else { return Ok(None) }; + return Ok(Some(CppWritten { text: cpp_plain_type(&first), site: range.site })); + } + let Some(cls) = self.cpp_written_class(&range, r)? else { return Ok(None) }; + let begin = self.cpp_methods_named(&cls, "begin", r)?; + let Some(begin) = self.cpp_return_type_of(&begin, r)? else { return Ok(None) }; + self.cpp_iterator_element(begin, r, hops) + } + + /// What dereferencing an iterator of the type `iterator` gives, as written. + fn cpp_iterator_element(&mut self, iterator: CppWritten, r: &ResolveRefIn, hops: u32) -> Res> { + let Some(iterator) = self.cpp_follow_written_aliases(iterator, r)? else { return Ok(None) }; + if top_level_pointer(&iterator.text) { return Ok(cpp_pointee(&iterator.text).map(|text| CppWritten { text, site: iterator.site })); } + // `std::vector::const_iterator`: the container's element. + let bare = re!(r"\b(?:typename|const)\b").replace_all(&iterator.text, " ").replace('&', " "); + if let Some(nested) = re!(r"^(.*>)\s*::\s*(?:const_)?(?:reverse_)?iterator$").captures(bare.trim()) { + return self.cpp_range_element_of(CppWritten { text: nested[1].trim().to_string(), site: iterator.site.clone() }, r, hops + 1); + } + let Some(cls) = self.cpp_written_class(&iterator, r)? else { return Ok(None) }; + let star = self.cpp_methods_named(&cls, "operator*", r)?; + Ok(self.cpp_return_type_of(&star, r)?.map(|t| CppWritten { text: t.text.trim_end_matches('&').trim().to_string(), site: t.site })) + } + + /// Whether every base the C++ class `cls` writes, and every base of those, is one a base edge + /// links to a class, or a standard-library type, which declares no project method. A base the + /// index doesn't link may declare any method: protobuf's `FlatAllocator` derives from a + /// `decltype(…)` that names `FlatAllocatorImpl`. + pub(super) fn cpp_bases_known(&mut self, cls: &KNode, depth: usize, seen: &mut HashSet) -> Res { + if !seen.insert(cls.qualified_name.clone()) { return Ok(true); } + let decls: Vec<_> = self.nodes_by_qualified_name(&cls.qualified_name)?.iter().filter(|n| cpp_class_node(n)).cloned().collect(); + for decl in decls { + let bases = self.supertype_nodes(&decl.id)?; + let written = self.cpp_base_specifiers(&decl).into_iter().filter(|s| !re!(r"^(?:::)?std::").is_match(s.trim())).count(); + // (`cpp_method_of` looks no deeper than this.) + if if depth >= 4 { bases.len() + written > 0 } else { written > bases.len() } { return Ok(false); } + for base in bases { + if !cpp_class_node(&base) || !self.cpp_bases_known(&base, depth + 1, seen)? { return Ok(false); } + } + } + Ok(true) + } +} + +#[cfg(test)] +mod tests { + use super::{cpp_pointee, template_arguments, cpp_plain_type}; + + #[test] + fn written_type_pieces() { + assert_eq!(template_arguments("std::vector>&").unwrap(), vec!["std::shared_ptr"]); + assert_eq!(template_arguments("autovector").unwrap(), vec!["ColumnFamilyData*", "8"]); + assert_eq!(cpp_plain_type("const FieldDescriptor* const absl_nonnull"), "FieldDescriptor*"); + assert_eq!(cpp_pointee("ColumnFamilySet*").as_deref(), Some("ColumnFamilySet")); + assert_eq!(cpp_pointee("std::unique_ptr&").as_deref(), Some("VersionSet")); + assert_eq!(cpp_pointee("std::vector"), None); + } +} diff --git a/codegraph-kernel/src/resolve/cpp_receivers.rs b/codegraph-kernel/src/resolve/cpp_receivers.rs index ddf1ca2ef..ced94f9b0 100644 --- a/codegraph-kernel/src/resolve/cpp_receivers.rs +++ b/codegraph-kernel/src/resolve/cpp_receivers.rs @@ -27,11 +27,23 @@ impl KernelResolver { if receiver.kind() != "identifier" { return Ok(None); } let name = &source.text()[receiver.start_byte()..receiver.end_byte()]; let mut raw = None; + // Where the declared type is written, which its names are looked up from: the call, or for + // an `auto` loop's element, where its range's type is declared. + let mut site = r.clone(); + let mut element = false; let mut scope = receiver; while let Some(parent) = scope.parent() { if let Some(ty) = cpp_binding_type(parent, scope, name, &source) { raw = Some(ty); break; } if parent.kind() == "for_range_loop" && parent.child_by_field_name("declarator").is_some_and(|n| cpp_declarator_name(n, &source) == name) { raw = parent.child_by_field_name("type").zip(parent.child_by_field_name("declarator")).map(|(ty, declarator)| cpp_ast_declared_type(ty, declarator, &source)); + // A loop over `auto` elements has the element type of its range. + if raw.as_deref().is_some_and(super::cpp_range_for::is_cpp_auto) { + if let Some(written) = self.cpp_range_for_element(parent, &source, r, 0)? { + raw = Some(super::cpp_range_for::cpp_plain_type(&written.text)); + site = written.site; + element = true; + } + } break; } scope = parent; @@ -47,10 +59,10 @@ impl KernelResolver { } } let Some(raw) = raw else { return Ok(None) }; - if raw.trim_end_matches('&').trim() == "auto" { return Ok(None); } + if super::cpp_range_for::is_cpp_auto(&raw) { return Ok(None); } if raw.contains('[') { return Ok(Some(None)); } let included = self.namespace_visible_files(&r.file_path, "cpp")?; - if self.cpp_alias_expansion(&raw, r, 0, &included)?.is_some_and(|alias| alias.pointer || (operator.trim() == "->" && alias.raw.as_deref().is_some_and(|ty| ty.ends_with("::iterator") || ty.ends_with("::const_iterator")))) { return Ok(None); } + if self.cpp_alias_expansion(&raw, &site, 0, &included)?.is_some_and(|alias| alias.pointer || (operator.trim() == "->" && alias.raw.as_deref().is_some_and(|ty| ty.ends_with("::iterator") || ty.ends_with("::const_iterator")))) { return Ok(None); } let mut depth = 0usize; let mut pointers = 0usize; for ch in raw.chars() { @@ -59,6 +71,8 @@ impl KernelResolver { let ty = match operator.trim() { "." if pointers == 0 => raw.trim_end_matches('&').trim().to_string(), "->" if pointers == 1 && !raw.contains('<') => raw.replace(['*', '&'], "").trim().to_string(), + // An element that points at a class template (`const autovector*`). + "->" if pointers == 1 && element => super::cpp_range_for::cpp_pointee(&raw).unwrap_or_default(), // A standard smart pointer or optional: `->` reaches a member of the type it holds. "->" if pointers == 0 => { let Some(hit) = re!(r"^(?:const\s+)?(?:::)?std::(?:unique_ptr|shared_ptr|optional)\s*<(.*)>\s*&?$").captures(&raw) else { return Ok(Some(None)) }; @@ -69,37 +83,42 @@ impl KernelResolver { _ => return Ok(Some(None)), }; let primary = ty.split('<').next().unwrap_or(&ty).trim(); - if let Some(caller) = self.node_by_id(&r.from_node_id)? { if self.cpp_template_parameter(&caller, primary, r)? { return Ok(Some(None)); } } - if self.cpp_alias_expansion(primary, r, 0, &included)?.is_some_and(|alias| alias.pointer) { return Ok(None); } + // A standard type's own member (`iter.reset()` on a `std::unique_ptr` element) is no project method. + if element && re!(r"^(?:::)?(?:std|absl)::").is_match(primary) { return Ok(Some(None)); } + if let Some(caller) = self.node_by_id(&site.from_node_id)? { if self.cpp_template_parameter(&caller, primary, &site)? { return Ok(Some(None)); } } + if self.cpp_alias_expansion(primary, &site, 0, &included)?.is_some_and(|alias| alias.pointer) { return Ok(None); } let held = operator.trim() == "->" && pointers == 0; - let mut owner = if held { self.cpp_class_scope_type(primary, r)? } else { None }; - if owner.is_none() { owner = self.cpp_type_owner(primary, r, 0, false)?; } + // C++ looks a type name up in the classes around the code and their bases before the + // namespaces: `Iterator` in a class deriving `MemTableRep` is `MemTableRep::Iterator`. + let mut owner = self.cpp_class_scope_type(primary, &site)?; + if owner.is_none() { owner = self.cpp_type_owner_visible(primary, &site, 0, false, &included)?; } if owner.is_none() && held { if let Some(parts) = super::cpp_aliases::type_segments(primary) { // A class template and its specializations share a qualified name, and so their members. - let family = self.cpp_scoped_types(primary, &parts.join("::"), r, &included)?; + let family = self.cpp_scoped_types(primary, &parts.join("::"), &site, &included)?; if let Some(first) = family.first().filter(|f| family.len() > 1 && family.iter().all(|n| n.qualified_name == f.qualified_name && matches!(n.kind.as_str(), "class" | "struct"))) { owner = Some(first.clone()); } - if owner.is_none() { owner = self.cpp_type_via_using(primary, &parts, r, &included)?; } + if owner.is_none() { owner = self.cpp_type_via_using(primary, &parts, &site, &included)?; } } } let Some(owner) = owner else { // A held type that is an alias of a library type holds no project class. - let aliased_library = held && self.cpp_alias_expansion(primary, r, 0, &included)? + let aliased_library = held && self.cpp_alias_expansion(primary, &site, 0, &included)? .is_some_and(|alias| alias.raw.as_deref().is_some_and(|ty| ty.trim_start_matches("::").starts_with("std::"))); return Ok(aliased_library.then_some(None)); }; // Preserve the existing guarded alias and specialization fallback on a miss. let member = self.match_bound_type_member(&owner.qualified_name, method, r)?; - // A class that lacks the method, and whose bases lack it too, is called nothing - // (but a class template's specialization may declare it). - if member.is_none() && held && self.supertypes_complete && !self.cpp_alias_template_owner(&owner, r)? { return Ok(Some(None)); } + // A class that lacks the method, and whose bases lack it too, is called nothing (but a + // class template's specialization may declare it, and so may a base the index doesn't link). + if member.is_none() && self.supertypes_complete && !self.cpp_alias_template_owner(&owner, r)? + && (held || (self.cpp_methods_named(&owner, method, r)?.is_empty() && self.cpp_bases_known(&owner, 0, &mut HashSet::new())?)) { return Ok(Some(None)); } Ok(member.map(Some)) } /// A class `ty` names from inside a class that declares or inherits it, which /// C++ looks up before the namespaces around: the scopes of the caller, innermost /// first, each with the classes it derives from. - fn cpp_class_scope_type(&mut self, ty: &str, r: &ResolveRefIn) -> Res>> { + pub(super) fn cpp_class_scope_type(&mut self, ty: &str, r: &ResolveRefIn) -> Res>> { if ty.contains("::") { return Ok(None); } let Some(caller) = self.node_by_id(&r.from_node_id)? else { return Ok(None) }; let scopes: Vec<_> = caller.qualified_name.split("::").map(str::to_string).collect(); @@ -256,7 +275,7 @@ impl KernelResolver { self.cpp_type_owner_visible(primary, &site, 0, false, &included) } - fn cpp_field_type(&mut self, owner: &Arc, field: &str, r: &ResolveRefIn) -> Res> { + pub(super) fn cpp_field_type(&mut self, owner: &Arc, field: &str, r: &ResolveRefIn) -> Res> { let Some(source) = self.read_file(&owner.file_path) else { return Ok(None) }; let mut site = r.clone().at(owner); site.from_node_id = owner.id.clone(); site.column = owner.start_column; let Some(tree) = self.cpp_field_tree(&source, &site) else { return Ok(None) }; @@ -325,7 +344,7 @@ fn cpp_element_type(raw: &str) -> Option { /// The first template argument of a standard smart pointer or optional, or None when it /// is a pointer: `Foo`, `ns::Foo`, `const Foo` for `Foo, Deleter`. -fn cpp_held_type(arguments: &str) -> Option { +pub(super) fn cpp_held_type(arguments: &str) -> Option { let mut depth = 0usize; let mut end = arguments.len(); for (i, ch) in arguments.char_indices() { @@ -345,14 +364,14 @@ fn cpp_held_type(arguments: &str) -> Option { (!held.is_empty()).then(|| held.to_string()) } -fn cpp_declarator_name<'a>(mut node: tree_sitter::Node, source: &'a SourceFile) -> &'a str { +pub(super) fn cpp_declarator_name<'a>(mut node: tree_sitter::Node, source: &'a SourceFile) -> &'a str { while matches!(node.kind(), "reference_declarator" | "pointer_declarator" | "parenthesized_declarator" | "init_declarator" | "array_declarator") { let Some(inner) = node.child_by_field_name("declarator").or_else(|| node.named_child(0)) else { return "" }; node = inner; } if matches!(node.kind(), "identifier" | "field_identifier") { &source.text()[node.start_byte()..node.end_byte()] } else { "" } } -fn cpp_binding_type(parent: tree_sitter::Node, at: tree_sitter::Node, name: &str, source: &SourceFile) -> Option { +pub(super) fn cpp_binding_type(parent: tree_sitter::Node, at: tree_sitter::Node, name: &str, source: &SourceFile) -> Option { if matches!(parent.kind(), "compound_statement" | "declaration_list" | "translation_unit") { for declaration in named_children(parent).into_iter().rev().filter(|n| n.kind() == "declaration" && n.end_byte() <= at.start_byte()) { // C++ parses a parenthesized initializer as a function declarator. @@ -403,7 +422,7 @@ fn cpp_binding_type(parent: tree_sitter::Node, at: tree_sitter::Node, name: &str None } -fn cpp_ast_declared_type(ty: tree_sitter::Node, mut declarator: tree_sitter::Node, source: &SourceFile) -> String { +pub(super) fn cpp_ast_declared_type(ty: tree_sitter::Node, mut declarator: tree_sitter::Node, source: &SourceFile) -> String { let type_node = if matches!(ty.kind(), "class_specifier" | "struct_specifier" | "union_specifier" | "enum_specifier") { ty.child_by_field_name("name").unwrap_or_else(|| { let first = ty.parent().and_then(|p| p.child_by_field_name("declarator")).unwrap_or(declarator); diff --git a/codegraph-kernel/src/resolve/mod.rs b/codegraph-kernel/src/resolve/mod.rs index 5afc92888..a6ad805ce 100644 --- a/codegraph-kernel/src/resolve/mod.rs +++ b/codegraph-kernel/src/resolve/mod.rs @@ -571,6 +571,7 @@ mod cpp_types; mod cpp_aliases; mod shopify; mod cpp_receivers; +mod cpp_range_for; mod cpp_implicit_this; mod cpp_namespace; mod inherited_method; diff --git a/codegraph-kernel/src/resolve/receivers.rs b/codegraph-kernel/src/resolve/receivers.rs index 3de1ce244..9067883f6 100644 --- a/codegraph-kernel/src/resolve/receivers.rs +++ b/codegraph-kernel/src/resolve/receivers.rs @@ -690,6 +690,13 @@ impl KernelResolver { if line.is_empty() || !has_word(line, receiver) { continue; } + // `for (auto cfd : cfds)` around the call: the element type of its range, when + // that can be read; when it can't, the scan goes on as before. + if line.contains("for") { + if let Some(class) = self.cpp_enclosing_range_element(receiver, i, r)? { + return self.normalize_cpp_type_name(&class.qualified_name, preserve); + } + } if let Some(decl) = self.cpp_declarator_match(line, &escaped)? { match self.normalize_cpp_type_name(&decl, preserve)? { Some(t) if t == "auto" || t.ends_with("::auto") => { diff --git a/src/db/reference-tail.ts b/src/db/reference-tail.ts index a0b954c59..2bdd6646a 100644 --- a/src/db/reference-tail.ts +++ b/src/db/reference-tail.ts @@ -127,5 +127,12 @@ export function importPathKeys(filePath: string): string[] { * name (`importBindingTail`), which is parked under the same keys. */ export function moduleReferenceKeys(filePath: string): string[] { - return importPathKeys(filePath).map((key) => MODULE_KEY + key); + const keys = importPathKeys(filePath); + // A Python package's `__init__.py` is a submodule of the package above it, + // as a module file beside it is through its folder: `app/pkg/sub/__init__.py` + // is what `from app.pkg import sub as s`, parked as 'module:pkg', imports. + const parts = filePath.split('/'); + const above = parts.length >= 3 && parts[parts.length - 1] === '__init__.py' ? parts[parts.length - 3]! : ''; + if (above && !keys.includes(above)) keys.push(above); + return keys.map((key) => MODULE_KEY + key); } diff --git a/src/resolution/import-resolver.ts b/src/resolution/import-resolver.ts index ef22ecd6d..847eef2ee 100644 --- a/src/resolution/import-resolver.ts +++ b/src/resolution/import-resolver.ts @@ -1054,7 +1054,8 @@ export function isBoundToOutOfRepoImport( /** Park an unresolved renamed/default import under the module a later sync can introduce. */ export function importBindingTail(ref: UnresolvedRef, context: ResolutionContext): string | undefined { - if (!ESM_IMPORT_LANGUAGES.has(ref.language) || !/^[A-Za-z_$][\w$]*$/.test(ref.referenceName)) return undefined; + if (ref.language === 'python') return pythonImportBindingTail(ref, context); + if (!ESM_IMPORT_LANGUAGES.has(ref.language) || !IDENTIFIER.test(ref.referenceName)) return undefined; const binding = context.getImportMappings(ref.filePath, ref.language).find(m => m.localName === ref.referenceName); if (!binding || (!binding.isDefault && !binding.isNamespace && binding.exportedName === binding.localName)) return undefined; if (isExternalImport(binding.source, ref.language, context, ref.filePath)) return undefined; @@ -1062,3 +1063,45 @@ export function importBindingTail(ref: UnresolvedRef, context: ResolutionContext ? path.posix.join(path.posix.dirname(ref.filePath), binding.source) : binding.source; return moduleTail(modulePath) || undefined; } + +/** + * {@link importBindingTail} for Python: `from .models import User as + * UserModel` and `from . import helpers as h` bind a name their module has as + * `User` and `helpers`, `import app.tools as tl` one it has as `tools`. The + * module is the one the import names by its dotted path, counted from the + * importing file's own package when it starts with dots (`.` is that package, + * each further dot the one above): `.models` in `app/main.py` is `app/models`, + * parked as 'module:models', and `.` is `app`, 'module:app'. The name may + * import a submodule of it (`helpers` is `app/helpers.py`), whose keys hold + * the package's name too (moduleReferenceKeys). + * + * An unaliased import keeps its tail: the module, or the one it re-exports the + * name from, declares that name. So does an import of the standard library. An + * installed package's import can't be told from a module the project doesn't + * have yet (`from settings import Config as Cfg`), so it waits for its module + * too, and a sync still finds it by its own name. + */ +function pythonImportBindingTail(ref: UnresolvedRef, context: ResolutionContext): string | undefined { + if (!IDENTIFIER.test(ref.referenceName)) return undefined; + const binding = context.getImportMappings(ref.filePath, 'python').find((m) => m.localName === ref.referenceName); + if (!binding) return undefined; + const declared = binding.isNamespace ? binding.source.slice(binding.source.lastIndexOf('.') + 1) : binding.exportedName; + if (declared === binding.localName || isExternalImport(binding.source, 'python', context, ref.filePath)) return undefined; + const modulePath = pythonModulePath(binding.source, ref.filePath); + return modulePath ? moduleTail(modulePath) || undefined : undefined; +} + +/** + * A Python import's module as a project path: `app.models` → 'app/models'; + * from `app/sub/views.py`, `.models` → 'app/sub/models', `..` → 'app' and + * `..forms` → 'app/forms'. Null when its dots climb above the project. + */ +function pythonModulePath(source: string, fromFile: string): string | null { + const name = source.replace(/^\.+/, ''); + const dots = source.length - name.length; + if (dots === 0) return name.replace(/\./g, '/'); + const modulePath = path.posix.join(path.posix.dirname(fromFile), '../'.repeat(dots - 1), name.replace(/\./g, '/')); + return modulePath === '..' || modulePath.startsWith('../') ? null : modulePath; +} + +const IDENTIFIER = /^[A-Za-z_$][\w$]*$/;