Skip to content

gh-162282: Fix quadratic cost in urllib.robotparser._add_entry - #162283

Open
VadimToptunov wants to merge 1 commit into
python:mainfrom
VadimToptunov:fix-robotparser-quadratic-user-agent
Open

VadimToptunov wants to merge 1 commit into
python:mainfrom
VadimToptunov:fix-robotparser-quadratic-user-agent

Conversation

@VadimToptunov

Copy link
Copy Markdown

Parsing a robots.txt with many identical User-agent: lines in one group previously ran in O(N²). _add_entry iterated entry.useragents without deduplication and called merge_entries for every duplicate from the second on; merge_entries is O(N) itself. On Python 3.14.6 / M-series, 10 000 lines took 2.5 s and 20 000 took 10.5 s.

The fix tracks the agents already registered in the current call and skips repeats. For every input shape other than "same agent repeated within one entry" the loop behaves identically; for that shape it removes the self-merges responsible for the quadratic.

The 111 existing test_robotparser tests all pass against the patched module. A representative diff against the stock parser on inputs covering single group, multiple groups, agent-shared-across-entries, duplicate agents in one entry, wildcards, sitemaps, and crawl-delay / request-rate shows zero behavioral difference in can_fetch, crawl_delay, request_rate, or site_maps.

On the pathological shape, 10 000 lines go from 2.5 s to 0.003 s.

A regression test RepeatedUserAgentPerformanceTest is added: it parses 50 000 duplicate User-agent: lines and requires completion in under 5 seconds. On the stock parser this bound is exceeded by an order of magnitude.

Fixes: gh-162282

Parsing a robots.txt with many identical User-agent: lines in a single
group ran in O(N^2) time: _add_entry iterated entry.useragents without
deduplication, and from the second duplicate on merge_entries was
called once per iteration; merge_entries is itself O(N).

Track the agents already registered during the current call and skip
duplicates. For every input shape other than 'same agent repeated
within one entry' the behavior is identical. On 10,000 duplicate
User-agent lines the parse time drops from ~2.5 s to ~0.003 s.

A regression test parses 50,000 duplicate User-agent lines and
requires completion in under 5 seconds.
@python-cla-bot

python-cla-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

manoelcalixto added a commit to Electivus/pstack-claude that referenced this pull request Nov 19, 2026
…ntext has reported (#41)

## What and why

- `watch-pr` must not call a pending check advisory while a required
check has not reported yet.
#38 built the required
set only from checks already on the head commit. python/cpython requires
"All required checks pass", a job GitHub creates only after the build
and test jobs finish. Until then every pending check was called
advisory, and a PR held as blocked stopped the watcher with a merge-gate
blocker and exit 6 instead of a wait.
- The lookup in the port-only `watch-pr/advisory.ts` now adds the status
checks that the base branch's rulesets and classic branch protection
require. It names an advisory set only when every required check has
reported, and on the `gh pr checks` path only when each one is in the
list it labels.
- Classic protection comes from `GET
repos/{owner}/{repo}/branches/{base}`, not from the protection endpoint.
The protection endpoint answers 404 to a reader without admin rights, as
it did on python/cpython, so reading 404 as "no protection" would keep
the bug.
- If either read fails, every pending check counts as required, as
before #38. #38's intent stays: when the required checks are green and
only advisory ones are pending, the watcher says so and does not report
`READY`.
- Released as 0.9.148 (0.9.147 went to #39). No upstream-owned file
changed, so `tools/forks.json` needs no update.

## Linked issue

Related to #10. Fixes the regression from
#38.

## Test run

```text
scripts: bun install --frozen-lockfile: ok
scripts: bun run typecheck: ok, no errors
scripts: bun test watch-pr check-plan.test.ts: 249 pass, 0 fail
scripts: bunx prettier@3.6.2 --check .: all files use Prettier code style
red: watch-pr/advisory.cli.test.ts on the test-only commit: 1 pass, 3 fail (exit 6 merge-blocked, "advisory": true)
root: bun install --frozen-lockfile; bun install --frozen-lockfile --cwd plugins: ok
root: bun tools/generate.mjs --check: ok
root: bun test tests/: 2205 pass, 161 skip, 0 fail
root: bun tools/sync.mjs --check-rules: ok, 99 rules match
root: bun tools/ci-local.mjs: all CI lint jobs pass
root: bun tools/check-release.mjs origin/main: ok, 0.9.147 to 0.9.148
live, before (0.9.146), python/cpython#162283 mid-CI:
  --status-only: "required green, 25 advisory pending" beside "blocked"
  --timeout 45: BLOCKER: merge-blocked, exit 6
live, after, same PR, "All required checks pass" not yet created (final head):
  --timeout 45: WAITING: frontier=#162283; 7 checks pending / TIMEOUT: checks still pending, exit 5
live: --status-only on Electivus/pstack-claude #39 and #40: rows print, CI green
live: ghRequiredNames called directly:
  #40: the 9 ruleset checks
  python/cpython#162279 (finished): the 7 classic-protection checks
  python/cpython#162283 (mid-CI): null
```

Not run live: the advisory verdict itself, on a PR whose required checks
have all reported while advisory checks are still pending. The direct
lookup calls above show the required set is read on both kinds of
protection. The verdict has unit and spawn tests only.

## Checklist

- [x] I ran `bun install --frozen-lockfile`, `bun tools/generate.mjs`,
and `bun test tests/`, and committed the generator's output.
- [x] `bun tools/generate.mjs --check` exits 0.
- [x] I changed model defaults only in `plugins/pstack/models.json`, and
every Pi model ID is in Pi's catalog for that provider. (No model
defaults changed.)
- [x] If `bun tools/check-release.mjs origin/main` reports a shipped
path, I ran `bun tools/release.mjs` and committed `VERSION`,
`CHANGES.md`, and the manifests it wrote.
- [x] If this touches `plugins/pstack/pi/`, `pi-tools.md`,
`plugins/pstack/hooks/`, `copilot-tools.md`, or `setup-pstack`, I ran
the runtime checks that
[CONTRIBUTING.md](https://github.com/Electivus/pstack-claude/blob/main/CONTRIBUTING.md#before-you-open-a-pr)
lists for those paths. (None touched.)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

urllib.robotparser.RobotFileParser.parse() is quadratic in repeated identical "User-agent:" lines

1 participant