Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions docs/design/competitive-landscape-adoption.md
Original file line number Diff line number Diff line change
Expand Up @@ -202,3 +202,14 @@ Octocode found one of six needed files where the two CodeGraph builds found none
| 5 (default) | 421 | 0.65 | 1.00 |

Running every test has precision 0.62, so the default depth narrows almost nothing on a tightly coupled package. Explore's per-symbol test list was scored on every sixth executed function (57). Twenty-seven were not in its short blast-radius list, so they could not be scored. Of the other 30, twelve list direct tests (17 names, all correct, recall 0.31 where the list was not truncated), six list tests reached through callers (10 names, 7 correct) and twelve print "no tests found" although about nine test modules run each of them (for example `handle_http_exception`, `finalize_request`, `preprocess_request` and `from_mapping`). This is one repository and one language, and line coverage says a test ran the code, not that it would fail if the code broke. The definition behind repowise's 97.5% was not found, so the figures are not comparable with it.

The twelve Flask symbols with no tests fall into four groups. Three (`handle_http_exception`, `finalize_request`, `preprocess_request`) sit behind `Flask.__call__`, which nothing in the project calls, because tests reach it through the test client and werkzeug. Two (`from_mapping`, `add_app_template_test`) are called directly by tests (`app.config.from_mapping(...)`, `bp.add_app_template_test(...)`) but have no resolved call edge: the index holds unresolved references to `from_mapping`, and why the receiver is not resolved was not traced. Six are internal hooks or decorator-applied methods, one of which (`create_global_jinja_loader`) a test overrides rather than calls. One (`stream_with_context`) has a path to a test past explore's 3-hop limit. None is a wrong edge; the first group is a missing edge class (a test driving the app through its framework entry) and the second a missing edge.

The same trace on two other frameworks, again with no coverage oracle (functions chosen because any request test runs them), shows an entry-point gap in these two repositories as well; two repositories do not establish that it is general:

| Framework | Entry point | Rest of the request path |
|---|---|---|
| FastAPI `f5c6e9b` | `FastAPI.__call__` and the per-request closure have no test | Route registration in tests links `get_request_handler` (1 hop), `solve_dependencies`, `serialize_response` and `run_endpoint_function` (3 hops); `get_dependant` is linked at 4 hops, past the limit |
| Express `a419aa7` | `app.handle` and `app.init` have no test, and no test file mentions them | `res.send` and `res.json` are linked (1 and 2 hops); `app.render`, `res.render` and `res.sendFile` are not, although 1 to 2 test files call each (probably because the receivers are untyped handler parameters; not verified) |

Linking every test that uses a test client to the app entry would not narrow anything: each pipeline function would then list every client-using test, which is what `affected` already does at depth 5 (precision 0.65 against 0.62 for running everything). A useful link would match the request path to the route handler. Nothing is built; whether it saves an agent reads belongs to the diff-impact task class in the shortlist. About eight symbols per framework, so the figures are indicative.
Comment thread
coderabbitai[bot] marked this conversation as resolved.