From 935ad02df36e74c11044d55471c42bfae4d501da Mon Sep 17 00:00:00 2001 From: Aaron Queen Date: Sun, 11 Oct 2026 03:04:15 -0600 Subject: [PATCH 1/4] docs: record that the test-linkage gap is the framework entry point, and what generalises --- docs/design/competitive-landscape-adoption.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/design/competitive-landscape-adoption.md b/docs/design/competitive-landscape-adoption.md index 7a515cc57..7b77a3125 100644 --- a/docs/design/competitive-landscape-adoption.md +++ b/docs/design/competitive-landscape-adoption.md @@ -202,3 +202,14 @@ Octocode found one of six needed files where the two CodeGraph builds found none | 5 (default) | 421 | 0.65 | 1.00 | Running every test has precision 0.62, so the default depth narrows almost nothing on a tightly coupled package. Explore's per-symbol test list was scored on every sixth executed function (57). Twenty-seven were not in its short blast-radius list, so they could not be scored. Of the other 30, twelve list direct tests (17 names, all correct, recall 0.31 where the list was not truncated), six list tests reached through callers (10 names, 7 correct) and twelve print "no tests found" although about nine test modules run each of them (for example `handle_http_exception`, `finalize_request`, `preprocess_request` and `from_mapping`). This is one repository and one language, and line coverage says a test ran the code, not that it would fail if the code broke. The definition behind repowise's 97.5% was not found, so the figures are not comparable with it. + +The twelve Flask symbols with no tests fall into three groups. Three (`handle_http_exception`, `finalize_request`, `preprocess_request`) sit behind `Flask.__call__`, which nothing in the project calls, because tests reach it through the test client and werkzeug. Eight are hooks that library code calls or methods applied as decorators. One (`stream_with_context`) has a path to a test past explore's 3-hop limit. No group is a resolver mis-wire; the graph has no edge for a test driving the app through its framework entry. + +The same trace on two other frameworks, again with no coverage oracle (functions chosen because any request test runs them), shows the entry-point gap is general and the rest is not: + +| Framework | Entry point | Rest of the request path | +|---|---|---| +| FastAPI `f5c6e9b` | `FastAPI.__call__` and the per-request closure have no test | Route registration in tests links `get_request_handler` (1 hop), `solve_dependencies`, `serialize_response` and `run_endpoint_function` (3 hops); `get_dependant` is linked at 4 hops, past the limit | +| Express `a419aa7` | `app.handle` and `app.init` have no test, and no test file mentions them | `res.send` and `res.json` are linked (1 and 2 hops); `app.render`, `res.render` and `res.sendFile` are not, although 1 to 2 test files call each, through untyped `res` and `app` parameters | + +Linking every test that uses a test client to the app entry would not narrow anything: each pipeline function would then list every client-using test, which is what `affected` already does at depth 5 (precision 0.65 against 0.62 for running everything). A useful link would match the request path to the route handler. Nothing is built; whether it saves an agent reads belongs to the diff-impact task class in the shortlist. About eight symbols per framework, so the figures are indicative. From a3a8fedfdd99dc4db61fc62dae46ce5c4e267b2e Mon Sep 17 00:00:00 2001 From: Aaron Queen Date: Sun, 11 Oct 2026 03:04:23 -0600 Subject: [PATCH 2/4] docs: mark the Express receiver explanation as unverified --- docs/design/competitive-landscape-adoption.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/competitive-landscape-adoption.md b/docs/design/competitive-landscape-adoption.md index 7b77a3125..d4e0d5d3e 100644 --- a/docs/design/competitive-landscape-adoption.md +++ b/docs/design/competitive-landscape-adoption.md @@ -210,6 +210,6 @@ The same trace on two other frameworks, again with no coverage oracle (functions | Framework | Entry point | Rest of the request path | |---|---|---| | FastAPI `f5c6e9b` | `FastAPI.__call__` and the per-request closure have no test | Route registration in tests links `get_request_handler` (1 hop), `solve_dependencies`, `serialize_response` and `run_endpoint_function` (3 hops); `get_dependant` is linked at 4 hops, past the limit | -| Express `a419aa7` | `app.handle` and `app.init` have no test, and no test file mentions them | `res.send` and `res.json` are linked (1 and 2 hops); `app.render`, `res.render` and `res.sendFile` are not, although 1 to 2 test files call each, through untyped `res` and `app` parameters | +| Express `a419aa7` | `app.handle` and `app.init` have no test, and no test file mentions them | `res.send` and `res.json` are linked (1 and 2 hops); `app.render`, `res.render` and `res.sendFile` are not, although 1 to 2 test files call each (probably because the receivers are untyped handler parameters; not verified) | Linking every test that uses a test client to the app entry would not narrow anything: each pipeline function would then list every client-using test, which is what `affected` already does at depth 5 (precision 0.65 against 0.62 for running everything). A useful link would match the request path to the route handler. Nothing is built; whether it saves an agent reads belongs to the diff-impact task class in the shortlist. About eight symbols per framework, so the figures are indicative. From 88bb8f26aae1334764a743666661e6dbb8be0112 Mon Sep 17 00:00:00 2001 From: Aaron Queen Date: Sun, 11 Oct 2026 03:09:48 -0600 Subject: [PATCH 3/4] docs: reclassify the two test-called Flask symbols and limit the cross-framework claim --- docs/design/competitive-landscape-adoption.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/design/competitive-landscape-adoption.md b/docs/design/competitive-landscape-adoption.md index d4e0d5d3e..3305b4138 100644 --- a/docs/design/competitive-landscape-adoption.md +++ b/docs/design/competitive-landscape-adoption.md @@ -203,9 +203,9 @@ Octocode found one of six needed files where the two CodeGraph builds found none Running every test has precision 0.62, so the default depth narrows almost nothing on a tightly coupled package. Explore's per-symbol test list was scored on every sixth executed function (57). Twenty-seven were not in its short blast-radius list, so they could not be scored. Of the other 30, twelve list direct tests (17 names, all correct, recall 0.31 where the list was not truncated), six list tests reached through callers (10 names, 7 correct) and twelve print "no tests found" although about nine test modules run each of them (for example `handle_http_exception`, `finalize_request`, `preprocess_request` and `from_mapping`). This is one repository and one language, and line coverage says a test ran the code, not that it would fail if the code broke. The definition behind repowise's 97.5% was not found, so the figures are not comparable with it. -The twelve Flask symbols with no tests fall into three groups. Three (`handle_http_exception`, `finalize_request`, `preprocess_request`) sit behind `Flask.__call__`, which nothing in the project calls, because tests reach it through the test client and werkzeug. Eight are hooks that library code calls or methods applied as decorators. One (`stream_with_context`) has a path to a test past explore's 3-hop limit. No group is a resolver mis-wire; the graph has no edge for a test driving the app through its framework entry. +The twelve Flask symbols with no tests fall into four groups. Three (`handle_http_exception`, `finalize_request`, `preprocess_request`) sit behind `Flask.__call__`, which nothing in the project calls, because tests reach it through the test client and werkzeug. Two (`from_mapping`, `add_app_template_test`) are called directly by tests (`app.config.from_mapping(...)`, `bp.add_app_template_test(...)`) but have no resolved call edge: the index holds unresolved references to `from_mapping`, and why the receiver is not resolved was not traced. Six are internal hooks or decorator-applied methods, one of which (`create_global_jinja_loader`) a test only overrides. One (`stream_with_context`) has a path to a test past explore's 3-hop limit. None is a wrong edge; the first group is a missing edge class (a test driving the app through its framework entry) and the second a missing edge. -The same trace on two other frameworks, again with no coverage oracle (functions chosen because any request test runs them), shows the entry-point gap is general and the rest is not: +The same trace on two other frameworks, again with no coverage oracle (functions chosen because any request test runs them), shows an entry-point gap in these two repositories as well; two repositories do not establish that it is general: | Framework | Entry point | Rest of the request path | |---|---|---| From f61b97ec1d19defe4ce048fb91b4197429b65d72 Mon Sep 17 00:00:00 2001 From: Aaron Queen Date: Sun, 11 Oct 2026 03:15:38 -0600 Subject: [PATCH 4/4] docs: say the test overrides rather than calls create_global_jinja_loader --- docs/design/competitive-landscape-adoption.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/competitive-landscape-adoption.md b/docs/design/competitive-landscape-adoption.md index 3305b4138..9b665c698 100644 --- a/docs/design/competitive-landscape-adoption.md +++ b/docs/design/competitive-landscape-adoption.md @@ -203,7 +203,7 @@ Octocode found one of six needed files where the two CodeGraph builds found none Running every test has precision 0.62, so the default depth narrows almost nothing on a tightly coupled package. Explore's per-symbol test list was scored on every sixth executed function (57). Twenty-seven were not in its short blast-radius list, so they could not be scored. Of the other 30, twelve list direct tests (17 names, all correct, recall 0.31 where the list was not truncated), six list tests reached through callers (10 names, 7 correct) and twelve print "no tests found" although about nine test modules run each of them (for example `handle_http_exception`, `finalize_request`, `preprocess_request` and `from_mapping`). This is one repository and one language, and line coverage says a test ran the code, not that it would fail if the code broke. The definition behind repowise's 97.5% was not found, so the figures are not comparable with it. -The twelve Flask symbols with no tests fall into four groups. Three (`handle_http_exception`, `finalize_request`, `preprocess_request`) sit behind `Flask.__call__`, which nothing in the project calls, because tests reach it through the test client and werkzeug. Two (`from_mapping`, `add_app_template_test`) are called directly by tests (`app.config.from_mapping(...)`, `bp.add_app_template_test(...)`) but have no resolved call edge: the index holds unresolved references to `from_mapping`, and why the receiver is not resolved was not traced. Six are internal hooks or decorator-applied methods, one of which (`create_global_jinja_loader`) a test only overrides. One (`stream_with_context`) has a path to a test past explore's 3-hop limit. None is a wrong edge; the first group is a missing edge class (a test driving the app through its framework entry) and the second a missing edge. +The twelve Flask symbols with no tests fall into four groups. Three (`handle_http_exception`, `finalize_request`, `preprocess_request`) sit behind `Flask.__call__`, which nothing in the project calls, because tests reach it through the test client and werkzeug. Two (`from_mapping`, `add_app_template_test`) are called directly by tests (`app.config.from_mapping(...)`, `bp.add_app_template_test(...)`) but have no resolved call edge: the index holds unresolved references to `from_mapping`, and why the receiver is not resolved was not traced. Six are internal hooks or decorator-applied methods, one of which (`create_global_jinja_loader`) a test overrides rather than calls. One (`stream_with_context`) has a path to a test past explore's 3-hop limit. None is a wrong edge; the first group is a missing edge class (a test driving the app through its framework entry) and the second a missing edge. The same trace on two other frameworks, again with no coverage oracle (functions chosen because any request test runs them), shows an entry-point gap in these two repositories as well; two repositories do not establish that it is general: