Repository navigation
gh-162282: Fix quadratic cost in urllib.robotparser._add_entry - #162283
Open
VadimToptunov wants to merge 1 commit into
Open
VadimToptunov wants to merge 1 commit into
VadimToptunov wants to merge 1 commit into
Conversation
Parsing a robots.txt with many identical User-agent: lines in a single group ran in O(N^2) time: _add_entry iterated entry.useragents without deduplication, and from the second duplicate on merge_entries was called once per iteration; merge_entries is itself O(N). Track the agents already registered during the current call and skip duplicates. For every input shape other than 'same agent repeated within one entry' the behavior is identical. On 10,000 duplicate User-agent lines the parse time drops from ~2.5 s to ~0.003 s. A regression test parses 50,000 duplicate User-agent lines and requires completion in under 5 seconds.
manoelcalixto
added a commit
to Electivus/pstack-claude
that referenced
this pull request
Nov 19, 2026
…ntext has reported (#41) ## What and why - `watch-pr` must not call a pending check advisory while a required check has not reported yet. #38 built the required set only from checks already on the head commit. python/cpython requires "All required checks pass", a job GitHub creates only after the build and test jobs finish. Until then every pending check was called advisory, and a PR held as blocked stopped the watcher with a merge-gate blocker and exit 6 instead of a wait. - The lookup in the port-only `watch-pr/advisory.ts` now adds the status checks that the base branch's rulesets and classic branch protection require. It names an advisory set only when every required check has reported, and on the `gh pr checks` path only when each one is in the list it labels. - Classic protection comes from `GET repos/{owner}/{repo}/branches/{base}`, not from the protection endpoint. The protection endpoint answers 404 to a reader without admin rights, as it did on python/cpython, so reading 404 as "no protection" would keep the bug. - If either read fails, every pending check counts as required, as before #38. #38's intent stays: when the required checks are green and only advisory ones are pending, the watcher says so and does not report `READY`. - Released as 0.9.148 (0.9.147 went to #39). No upstream-owned file changed, so `tools/forks.json` needs no update. ## Linked issue Related to #10. Fixes the regression from #38. ## Test run ```text scripts: bun install --frozen-lockfile: ok scripts: bun run typecheck: ok, no errors scripts: bun test watch-pr check-plan.test.ts: 249 pass, 0 fail scripts: bunx prettier@3.6.2 --check .: all files use Prettier code style red: watch-pr/advisory.cli.test.ts on the test-only commit: 1 pass, 3 fail (exit 6 merge-blocked, "advisory": true) root: bun install --frozen-lockfile; bun install --frozen-lockfile --cwd plugins: ok root: bun tools/generate.mjs --check: ok root: bun test tests/: 2205 pass, 161 skip, 0 fail root: bun tools/sync.mjs --check-rules: ok, 99 rules match root: bun tools/ci-local.mjs: all CI lint jobs pass root: bun tools/check-release.mjs origin/main: ok, 0.9.147 to 0.9.148 live, before (0.9.146), python/cpython#162283 mid-CI: --status-only: "required green, 25 advisory pending" beside "blocked" --timeout 45: BLOCKER: merge-blocked, exit 6 live, after, same PR, "All required checks pass" not yet created (final head): --timeout 45: WAITING: frontier=#162283; 7 checks pending / TIMEOUT: checks still pending, exit 5 live: --status-only on Electivus/pstack-claude #39 and #40: rows print, CI green live: ghRequiredNames called directly: #40: the 9 ruleset checks python/cpython#162279 (finished): the 7 classic-protection checks python/cpython#162283 (mid-CI): null ``` Not run live: the advisory verdict itself, on a PR whose required checks have all reported while advisory checks are still pending. The direct lookup calls above show the required set is read on both kinds of protection. The verdict has unit and spawn tests only. ## Checklist - [x] I ran `bun install --frozen-lockfile`, `bun tools/generate.mjs`, and `bun test tests/`, and committed the generator's output. - [x] `bun tools/generate.mjs --check` exits 0. - [x] I changed model defaults only in `plugins/pstack/models.json`, and every Pi model ID is in Pi's catalog for that provider. (No model defaults changed.) - [x] If `bun tools/check-release.mjs origin/main` reports a shipped path, I ran `bun tools/release.mjs` and committed `VERSION`, `CHANGES.md`, and the manifests it wrote. - [x] If this touches `plugins/pstack/pi/`, `pi-tools.md`, `plugins/pstack/hooks/`, `copilot-tools.md`, or `setup-pstack`, I ran the runtime checks that [CONTRIBUTING.md](https://github.com/Electivus/pstack-claude/blob/main/CONTRIBUTING.md#before-you-open-a-pr) lists for those paths. (None touched.) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parsing a
robots.txtwith many identicalUser-agent:lines in one group previously ran in O(N²)._add_entryiteratedentry.useragentswithout deduplication and calledmerge_entriesfor every duplicate from the second on;merge_entriesis O(N) itself. On Python 3.14.6 / M-series, 10 000 lines took 2.5 s and 20 000 took 10.5 s.The fix tracks the agents already registered in the current call and skips repeats. For every input shape other than "same agent repeated within one entry" the loop behaves identically; for that shape it removes the self-merges responsible for the quadratic.
The 111 existing
test_robotparsertests all pass against the patched module. A representative diff against the stock parser on inputs covering single group, multiple groups, agent-shared-across-entries, duplicate agents in one entry, wildcards, sitemaps, and crawl-delay / request-rate shows zero behavioral difference incan_fetch,crawl_delay,request_rate, orsite_maps.On the pathological shape, 10 000 lines go from 2.5 s to 0.003 s.
A regression test
RepeatedUserAgentPerformanceTestis added: it parses 50 000 duplicateUser-agent:lines and requires completion in under 5 seconds. On the stock parser this bound is exceeded by an order of magnitude.Fixes: gh-162282