Benchmark progress: socket-patch scan

[Mikola Lysenko (mikolalysenko)]

[agent] Bench: progress log for the socket-patch scan benchmark suite (crates/socket-patch-bench, .github/workflows/bench.yml). The daily benchmark steward rewrites this body on every run and posts one comment per run. Machine-readable history is in history.json on the bench/ledger branch. The first run (2026-10-02) was logged in #580 and backfilled there.

Last run

• When: 2026-10-07 ~08:00 UTC
• main: 9c43dfc9 (Cache the macOS vexctl build in the test job, Cache the macOS vexctl build in the CI test job #874). Unchanged since the 10-06 run, so there was no daily A/B.
• Suite source: main, plus hatch/* from Bench: cover Gradle and Hatch hosted modes #925, measured with the Bench: cover Gradle and Hatch hosted modes #925 bench binary on the same main CLI. gradle/* (also in Bench: cover Gradle and Hatch hosted modes #925) was not re-measured today.
• Runner: 4 vCPU, Intel(R) Xeon(R) Processor @ 2.80GHz, cloud sandbox. It ran about 1.3x slower than 10-06's runner on absolute medians. Verdicts come only from same-machine interleaved compare.
• Validation: 41/41 main scenarios are valid on today's main, plus 2/2 new hatch/*. Request counts match 10-06 in every scenario.
• Daily A/B: skipped (LAST_SHA == HEAD).
• Weekly A/B (2463257a v5 prerelease: scan → vex → vendor workflow, hosted by default #277 → 9c43dfc9; the 7-day SHA f6b7fb9e predates the v5 output):
  • New and confirmed: npm drift of +15.0% to +19.6% on npm/hosted and +12.6% to +15.8% on npm/rescan, over two rounds. Filed as Perf regression: npm/hosted wall +15% (2463257a..9c43dfc9) #993. npm/dry-run (+15.1%) and npm/public-proxy (+10.1%) move the same way. There is no single culprit: about half lands before 045d7ec7 and half after, and neither half is significant alone. Callgrind shows +9.4% instructions, from new npm lock passes in VEX discovery (Fix npm rewiring git/URL/file lock entries (#326) #345, Fix npm overrides of git deps being refused (#490) #491, Fix npm VEX attesting a patch the twin lock lacks (#798) #799) and the shared lock cache (Full Gradle support in agent, hosted and vendored modes #646).
  • Known Fix Bun/vlt bundled copies left unpatched (#469, #471) #472 regressions: bun/hosted +99.2%, bun/rescan +98.7% (Perf regression: bun/hosted wall +110% (1169ae68, #472) #578, data commented); vlt/hosted +127.6%, vlt/rescan +142.8% (accepted, Perf regression: vlt/hosted wall +127% (1169ae68, #472) #579).
  • pip/rescan was flagged at +12.7%, but the confirmation round gave +8.9% [+2.3, +15.8], under the gate, so it is not confirmed. Watch it.
  • nuget/hosted is still 19.5% faster (Unify hosted NuGet routing and XML splice anchors #597).
• A/A sanity check: clean. npm/hosted -2.3%, uv/hosted +2.2%, poetry/hosted +2.8%, pip/rescan +3.5%; hatch/hosted -2.1%, hatch/rescan +2.5%.

Scoreboard

Day: no daily A/B today (main unchanged). Week: wall ratio vs 2463257a (#277). The median ms/pkg across */hosted is 0.1553; hatch is excluded until it lands. npm/hosted's 439.7 ms median comes from a noisy 3-sample run (min 331.6 ms). The compare medians of 317–322 ms are the better figure.

scenario	median wall (ms)	ms/pkg	day	week	status
npm/hosted	439.7	0.1466	—	1.150	regressed (#993)
npm/rescan	258.5	0.0862	—	1.158	regressed (#993)
pnpm/hosted	216.5	0.0722	—	1.042	OK
pnpm/rescan	213.6	0.0712	—	0.998	OK
yarn-classic/hosted	303.1	0.1010	—	1.058	OK
yarn-classic/rescan	246.4	0.0821	—	1.055	OK
yarn-berry/hosted	412.8	0.1376	—	—	OK
yarn-berry/rescan	283.7	0.0946	—	—	OK
bun/hosted	396.5	0.1322	—	1.992	regressed (#578)
bun/rescan	367.7	0.1226	—	1.987	regressed (#578)
bun-isolated/hosted	465.8	0.1553	—	—	OK
bun-isolated/rescan	467.1	0.1557	—	—	OK
vlt/hosted	1246.6	0.8311	—	2.276	slow; #472 cost accepted (#579)
vlt/rescan	1301.7	0.8678	—	2.428	slow; #472 cost accepted (#579)
pip/hosted	126.3	0.1263	—	1.060	OK
pip/rescan	126.3	0.1263	—	1.127	OK
uv/hosted	220.3	0.5507	—	1.033	slow (#836)
uv/rescan	213.1	0.5327	—	0.956	slow (#836)
pylock/hosted	194.8	0.4869	—	1.027	slow (#836)
pylock/rescan	183.0	0.4575	—	1.041	slow (#836)
poetry/hosted	369.5	0.9237	—	0.988	slow (#760)
poetry/rescan	294.2	0.7354	—	1.048	slow (#760)
pipenv/hosted	133.8	0.3345	—	1.021	slow (none)
pipenv/rescan	151.8	0.3795	—	0.981	slow (none)
pdm/hosted	309.5	0.7738	—	1.052	slow (#762)
pdm/rescan	277.0	0.6924	—	1.032	slow (#762)
bundler/hosted	204.7	0.2559	—	1.078	OK
bundler/rescan	265.4	0.3317	—	1.099	OK
composer/hosted	175.7	0.2197	—	1.026	OK
composer/rescan	217.3	0.2717	—	1.111	OK
cargo/hosted	332.4	0.5540	—	1.011	slow (#761)
cargo/rescan	331.9	0.5532	—	1.010	slow (#761)
golang/hosted	95.1	0.0792	—	1.052	OK
golang/rescan	100.6	0.0838	—	1.082	OK
nuget/hosted	98.5	0.1095	—	0.805	OK (faster, week 0.805)
nuget/rescan	90.4	0.1004	—	1.042	OK
maven/hosted	98.8	0.0988	—	1.016	OK
maven/rescan	103.5	0.1035	—	1.079	OK
npm/dry-run	387.9	0.1293	—	1.151	regressed, week (#993)
npm/public-proxy	398.4	0.1328	—	1.101	regressed, week (#993)
npm/latency	756.4	0.2521	—	1.038	OK
hatch/hosted (new, in #925)	86.1	0.0861	—	—	OK
hatch/rescan (new, in #925)	77.0	0.0770	—	—	OK

Trends

Median <pm>/hosted wall time per run, oldest → newest (ms). Runners differ between runs (10-05 and 10-07 were about 1.3-1.4x slower everywhere), so read these as trends only.

pm	10-02	10-03	10-04	10-05	10-06	10-07
npm	198.6	204.0	198.3	296.1	233.5	439.7
pnpm	153.6	180.5	162.1	234.8	197.4	216.5
yarn-classic	179.9	195.4	166.9	227.5	181.9	303.1
yarn-berry	219.5	228.8	208.6	293.5	242.0	412.8
bun	289.7	303.6	321.5	394.6	297.0	396.5
vlt	843.8	894.2	764.0	1109.7	840.0	1246.6
pip	102.2	97.0	81.8	125.8	136.1	126.3
uv	136.0	143.9	127.1	244.0	156.6	220.3
pylock	113.6	142.1	126.2	179.8	122.1	194.8
poetry	187.6	221.4	201.6	342.9	261.7	369.5
pipenv	106.2	96.5	86.5	143.9	100.2	133.8
pdm	180.4	180.5	162.4	260.3	193.6	309.5
bundler	136.0	180.4	141.5	205.9	165.6	204.7
composer	122.2	123.0	115.4	180.8	122.9	175.7
cargo	228.4	262.5	258.5	364.1	255.1	332.4
golang	68.5	71.4	64.3	110.3	80.5	95.1
nuget	90.9	90.1	87.2	133.3	91.3	98.5
maven	79.3	81.6	62.0	87.4	73.5	98.8
bun-isolated	—	—	336.6	392.4	342.4	465.8
gradle	—	—	—	—	170.5	—
hatch	—	—	—	—	—	86.1

Open issues and PR

• Perf regression: npm/hosted wall +15% (2463257a..9c43dfc9) #993 (new): Perf regression: npm/hosted wall +15% (2463257..9c43dfc). Weekly drift with no single culprit. Candidates: Fix npm VEX attesting a patch the twin lock lacks (#798) #799, Full Gradle support in agent, hosted and vendored modes #646, Fix npm rewiring git/URL/file lock entries (#326) #345, Fix npm overrides of git deps being refused (#490) #491.
• Perf regression: bun/hosted wall +110% (1169ae68, #472) #578: Perf regression: bun/hosted wall +110% (1169ae6, Fix Bun/vlt bundled copies left unpatched (#469, #471) #472). Still regressed (weekly +99%); today's data is commented. Accepting it the way Perf regression: vlt/hosted wall +127% (1169ae68, #472) #579 was accepted is a human call.
• Perf regression: vlt/hosted wall +127% (1169ae68, #472) #579 (closed 2026-10-03 by a maintainer): the vlt/* Fix Bun/vlt bundled copies left unpatched (#469, #471) #472 cost is accepted. Tracked as slow only.
• Slow scan: poetry hosted 4.5x median ms/pkg (per-patch poetry.lock re-parse) #760: Slow scan: poetry. Draft fix in Fix per-patch Poetry/PDM lock re-parse (#760, #762) #877.
• Slow scan: cargo hosted 3.8x median ms/pkg (per-patch Cargo.lock re-parse) #761: Slow scan: cargo.
• Slow scan: pdm hosted 3.6x median ms/pkg (per-patch pdm.lock re-parse) #762: Slow scan: pdm. Draft fix in Fix per-patch Poetry/PDM lock re-parse (#760, #762) #877.
• Slow scan: uv/pylock hosted 4.1x/3.0x median ms/pkg (per-patch lock re-serialize) #836: Slow scan: uv/pylock (per-patch lock re-serialize).
• Bench: cover Gradle and Hatch hosted modes #925 (bench/refresh): Bench: cover Gradle and Hatch hosted modes. Today it gained hatch/hosted and hatch/rescan (5b9ae8ea). state: ready, with a fresh CI run on the new head.

Standing slow-systems list

A package manager is slow when its hosted ms/pkg is more than 2x the median across */hosted (0.1553 today), or when any of its scenarios takes over 500 ms (npm/latency is excluded). An issue opens after 3 consecutive runs, capped at 3 new issues per run.

pm	ms/pkg today (x median)	consecutive runs	issue	hot spot
poetry	0.924 (5.9x)	6	#760 (fix draft #877)	utils::poetry_lock::rewrite_poetry_lock_in, 77% (whole lock re-parsed per patch). Callgrind 10-04
vlt	0.831 (5.4x)	6	none (#579 accepted)	#472 bundled-copy walk, plus DepId re-splitting in redirect::vlt (callgrind 10-02)
pdm	0.774 (5.0x)	6	#762 (fix draft #877)	utils::pdm_lock::rewrite_pdm_lock_in, 77% (same per-patch re-parse as Poetry). Callgrind 10-04
cargo	0.554 (3.6x)	6	#761	formats::cargo::CargoLock::parse inside rewrite_cargo, 79%. Callgrind 10-04
uv	0.551 (3.5x)	6	#836	PythonLockSession::rewrite 39%, mostly DocumentMut Display (whole lock re-serialized per patched dep). Callgrind 10-05
pylock	0.487 (3.1x)	6	#836	Same as uv: PythonLockSession::rewrite 38%; vex pypi_locks::extract 31%. Callgrind 10-05
pipenv	0.335 (2.15x)	1	none	patch::redirect::pipenv::entries 69% inclusive (pipenv::properties 45%): Pipfile.lock re-tokenized per patch. Callgrind 10-07. It flips around 2x (10-05 2.4x, 10-06 1.8x)

No scenario's peak RSS is above 2x the median (34.2 MiB). Scenarios over 500 ms: vlt/hosted, vlt/rescan (npm/latency is excluded by design). The new hatch/hosted runs at 0.086 ms/pkg (0.55x), so it is not slow.

Coverage-gap backlog

main hasn't changed since 10-06.

• gradle (Full Gradle support in agent, hosted and vendored modes #646): covered by gradle/* in Bench: cover Gradle and Hatch hosted modes #925.
• hatch (Fix uv/Hatch hosted re-pin to a newer patch (#742, #650) #743/Fix Hatch takeover leaving direct-ref permission (#674) #680): covered by hatch/* in Bench: cover Gradle and Hatch hosted modes #925 (lockless hatch.toml env pins).
• Hatch variants: [tool.hatch.envs] in pyproject only (no hatch.toml), a hatch project with a requirements.txt (confirmation via requirements), and the uv-installer refusal path.
• Gradle variants: Kotlin DSL, multi-project builds, version catalogs, gradle/verification-metadata.xml rewrite and mavenLocal() m2 gating are all uncovered (Full Gradle support in agent, hosted and vendored modes #646).
• Yarn 4 pnpm linker (nodeLinker: pnpm, Fix npm crawler missing Bun, Deno and Yarn 4 stores (#366, #373, #405, #495) #496): uncovered.
• Yarn berry re-pin of old npm:…::__archiveUrl pins (Fix yarn berry hosted pin leaking npm auth (#404) #465): uncovered.
• bun.lockb (binary lockfile): the bun_binary rewriter and codec (Fix Bun/vlt bundled copies left unpatched (#469, #471) #472) are unbenchmarked.
• Bundled copies: no fixture hits the redirect_*_bundled_instance_skipped paths (Fix Bun/vlt bundled copies left unpatched (#469, #471) #472, Fix in-run hosted VEX attesting npm bundled copies (#325) #669).
• pnpm global store / modulesDir store (Fix pnpm global-store transitive deps passing silently (#362) #829, Fix pnpm modulesDir store being skipped (#661, #696) #698): uncovered.
• vlt 1.3 brotli lock nodes (Fix vlt 1.3 brotli lock nodes being refused (#372) #820): uncovered.
• Python env discovery: UV_PROJECT_ENVIRONMENT / PDM venv locations (Fix Python env discovery ignoring uv/PDM env (#502, #525, #528) #540), .egg-info (Fix Python crawler missing .egg-info installs (#447) #452), Poetry envs.toml (Fix Poetry venv selection ignoring envs.toml (#476, #526) #527) and lock-only requirements.txt (Fix lock-only requirements.txt discovery (#412, #523) #530) are uncovered.
• pnpm-workspace.yaml append (Fix pnpm-workspace.yaml key detection and appends (#400, #402) #414): only the fresh-file case is covered.
• Gemfile multi-layer .bundle/config (Fix Bundler settings resolution order (#483, #507) #532, Gem settings resolution skips Bundler's global config (~/.bundle/config / BUNDLE_USER_CONFIG), so a global cache_path or gemfile gets no warning or refusal and VEX attests an unpatched install #577), gems.locked (Fix gem lock readers ignoring gems.locked (#736) #750) and Bundler 4 standalone (Fix gem crawler missing Bundler 4 standalone bundle (#796) #797): uncovered.
• deno: no hosted rewrite, so no scenario is needed.

Stale / redundant scenarios

None yet. Merging hosted/rescan pairs needs 14+ days of history; there are 6 days so far. Candidate: bun/rescan overlaps bun-isolated/rescan on the lockfile path.

---

Generated by Claude Code

[github-actions[bot]]

[agent] Bench: mirror test. The bench/ledger workflow posted this comment. Daily runs add their summaries after it.

---

Generated by Claude Code

[github-actions[bot]]

[agent] Bench: run 2026-10-03, main 045d7ec7, suite from main (#485 landed).

• All 39 scenarios validated on today's main.
• Daily A/B vs 1169ae68 (the previous run): no regressions across 37 comparable scenarios. yarn-berry/* had no comparable base because of Fix yarn berry hosted pin leaking npm auth (#404) #465's intentional package.json pin. A/A was clean.
• Weekly A/B vs v5 prerelease: scan → vex → vendor workflow, hosted by default #277 (2463257a; the 7-day SHA predates v5): only the known Fix Bun/vlt bundled copies left unpatched (#469, #471) #472 regressions remain, still unfixed. bun/* +117–125% (Perf regression: bun/hosted wall +110% (1169ae68, #472) #578) and vlt/* +139–153% (Perf regression: vlt/hosted wall +127% (1169ae68, #472) #579), with fresh data commented on both.
• Main's bench gate failed at 203e092b and 0ac5b91a: Fix yarn berry hosted pin leaking npm auth (#404) #465 changed yarn-berry's rewritten files before Stream ZIP member hashes during vendored verification #587 updated the fixture. It is green again at 045d7ec7.
• Slow list (run 2 of 3): vlt 4.7x, poetry 4.3x, pdm 3.5x, cargo 3.4x, uv 2.8x, pylock 2.8x. pipenv dropped off (1.9x).
• Suite change: Bench: cover Bun isolated .bun store; fix rescan restore #667 adds bun-isolated/{hosted,rescan}, which covers Bun 1.3's default .bun store walked since Fix npm crawler missing Bun, Deno and Yarn 4 stores (#366, #373, #405, #495) #496. A pre-Fix npm crawler missing Bun, Deno and Yarn 4 stores (#366, #373, #405, #495) #496 binary fails it.
• New issues: none.

---

Generated by Claude Code

[github-actions[bot]]

[agent] Janitor: state drift. The body still lists #579 as an open regression (vlt/hosted and vlt/rescan rows, the weekly A/B line, and the open-issues list), but a maintainer closed #579 at 2026-10-03 12:24Z as accepted: the tool is slower because it now scans code it used to skip. #578 (bun/*) is still open.

---

Generated by Claude Code

[github-actions[bot]]

[agent] Bench: run 2026-10-04, main 045d7ec7 (unchanged since 10-03), suite from main + #667.

• All 41 scenarios validated on today's main.
• Daily A/B: skipped, because main didn't move. Weekly A/B vs v5 prerelease: scan → vex → vendor workflow, hosted by default #277 (2463257a): only the known Fix Bun/vlt bundled copies left unpatched (#469, #471) #472 regressions. bun/* +117% / +127% (Perf regression: bun/hosted wall +110% (1169ae68, #472) #578, data commented). vlt/* +112% / +116%, now accepted: a maintainer closed Perf regression: vlt/hosted wall +127% (1169ae68, #472) #579. A/A clean.
• Slow list, run 3 of 3: vlt 4.5x, poetry 4.5x, cargo 3.8x, pdm 3.6x, uv 2.8x, pylock 2.8x.
• New issues (callgrind hot spots): Slow scan: poetry hosted 4.5x median ms/pkg (per-patch poetry.lock re-parse) #760 poetry, Slow scan: cargo hosted 3.8x median ms/pkg (per-patch Cargo.lock re-parse) #761 cargo, Slow scan: pdm hosted 3.6x median ms/pkg (per-patch pdm.lock re-parse) #762 pdm. Each one re-parses the whole lockfile per patched dep. uv and pylock overflow the 3-issue cap and get filed next run. vlt gets no slow issue because its cost was accepted on Perf regression: vlt/hosted wall +127% (1169ae68, #472) #579.
• Suite change: Bench: cover Bun isolated .bun store; fix rescan restore #667 gains 93b1c929. A rescan whose prep scan failed validation no longer leaves the fixture dirty, which had made compare report the other binary INVALID (seen today on bun-isolated/rescan).

---

Generated by Claude Code

[github-actions[bot]]

[agent] Bench: run 2026-10-05, main 045d7ec7 (unchanged since 10-02), suite from main + #667.

• All 41 scenarios validated on today's main.
• Daily A/B: skipped, because main didn't move. Weekly A/B vs v5 prerelease: scan → vex → vendor workflow, hosted by default #277 (2463257a): only the known Fix Bun/vlt bundled copies left unpatched (#469, #471) #472 regressions, bun/hosted +107%, bun/rescan +92% (Perf regression: bun/hosted wall +110% (1169ae68, #472) #578, data commented) and vlt/hosted +123% (accepted, Perf regression: vlt/hosted wall +127% (1169ae68, #472) #579). The other 33 comparable scenarios are unchanged. A/A clean.
• Today's runner was about 1.4x slower on every scenario. The weekly A/B is flat, so absolute medians jumped for machine reasons only.
• New issue Slow scan: uv/pylock hosted 4.1x/3.0x median ms/pkg (per-patch lock re-serialize) #836: uv and pylock, slow 4 runs in a row. PythonLockSession::rewrite re-serializes the whole lock per patched dep (callgrind: 39% / 38%, mostly toml_edit Display).
• Slow list: poetry 5.8x, vlt 5.0x, pdm 4.4x, uv 4.1x, cargo 4.1x, pylock 3.0x. pipenv is newly borderline at 2.4x (run 1, noisy).
• Suite: no changes needed. Bench: cover Bun isolated .bun store; fix rescan restore #667 is unchanged and still ready for review.
• Gap: vlt/rescan's weekly confirm round was cut short by a session timeout (its cost is already accepted).

---

Generated by Claude Code

[github-actions[bot]]

[agent] Janitor: state drift. The coverage-gaps note for hatch still says "Open PRs #700/#743/#680 are reworking Hatch". #743 (0475695f) and #680 (cd75b38e) merged on 2026-10-05. Only #700 is still open.

---

Generated by Claude Code

[github-actions[bot]]

[agent] Bench: run 2026-10-06, main 9c43dfc9 (74 commits since 045d7ec7), suite from main, plus gradle/* from #925.

• All 41 scenarios validated on today's main. The 2 new gradle/* scenarios validate too.
• Daily A/B (045d7ec7 → 9c43dfc9): no regressions and no request-count changes. nuget/hosted is 26% faster (0.739 [0.686, 0.901]), likely from Unify hosted NuGet routing and XML splice anchors #597.
• Weekly A/B: the base is 2463257a (v5 prerelease: scan → vex → vendor workflow, hosted by default #277), because the 7-day SHA f6b7fb9e predates the v5 output. Only the known Fix Bun/vlt bundled copies left unpatched (#469, #471) #472 regressions remain: bun/hosted +112% and bun/rescan +123% (Perf regression: bun/hosted wall +110% (1169ae68, #472) #578, data commented), vlt +105%/+133% (accepted). A/A is clean.
• New PR Bench: cover Gradle and Hatch hosted modes #925: adds gradle/hosted and gradle/rescan, covering Gradle hosted mode (Full Gradle support in agent, hosted and vendored modes #646). Measured at 0.17 ms/pkg; A/A is clean.
• Slow list: poetry 4.8x, vlt 4.1x, pdm 3.6x, cargo 3.1x, uv 2.9x, pylock 2.2x, all already tracked. pipenv is back under 2x (1.8x). No new issues.
• Next gap: a hatch scenario. Fix uv/Hatch hosted re-pin to a newer patch (#742, #650) #743/Fix Hatch takeover leaving direct-ref permission (#674) #680 have landed.

---

Generated by Claude Code

[github-actions[bot]]

[agent] Bench: run 2026-10-07, main 9c43dfc9 (unchanged since 10-06, so there was no daily A/B), suite from main, plus hatch/* from #925.

• All 41 scenarios validated on today's main, and the 2 new hatch/* scenarios validate too. Request counts are unchanged everywhere.
• New: confirmed weekly npm drift. npm/hosted is +15.0% / +19.6% and npm/rescan +15.8% / +12.6% vs 2463257a (two rounds; A/A clean). Filed as Perf regression: npm/hosted wall +15% (2463257a..9c43dfc9) #993. No single culprit: about half lands before 045d7ec7 and half after. Callgrind shows +9.4% instructions, from new npm lock passes (Fix npm rewiring git/URL/file lock entries (#326) #345, Fix npm overrides of git deps being refused (#490) #491, Fix npm VEX attesting a patch the twin lock lacks (#798) #799, Full Gradle support in agent, hosted and vendored modes #646).
• Known Fix Bun/vlt bundled copies left unpatched (#469, #471) #472 regressions are unchanged: bun +99% (Perf regression: bun/hosted wall +110% (1169ae68, #472) #578, data commented), vlt +128% / +143% (accepted). pip/rescan +12.7% didn't hold up in the confirmation round (+8.9%). nuget/hosted is still 19.5% faster.
• Bench: cover Gradle and Hatch hosted modes #925 updated: adds hatch/hosted and hatch/rescan (a lockless hatch.toml env rewrite, 1000 pkgs / 25 patched, ~80 ms). A/A is clean.
• Slow list: poetry 5.9x, vlt 5.4x, pdm 5.0x, cargo 3.6x, uv 3.5x, pylock 3.1x, all already tracked. pipenv is back at 2.15x. Its hot spot is redirect::pipenv::entries, 69%. No slow-system issue yet.

---

Generated by Claude Code