Skip to content

gh-162263: Fix robotparser ending a group at Crawl-delay and Request-rate - #162286

Open
Wali05 wants to merge 3 commits into
python:mainfrom
Wali05:gh-162263-robotparser-grouping
Open

Wali05 wants to merge 3 commits into
python:mainfrom
Wali05:gh-162263-robotparser-grouping

Conversation

@Wali05

@Wali05 Wali05 commented Nov 19, 2026 •

Copy link
Copy Markdown

Crawl-delay and Request-rate lines currently end a group, so in

User-agent: ExampleBot
Crawl-delay: 10
User-agent: OtherBot
Disallow: /

the Disallow rule only applies to OtherBot. RFC 9309, section 2.2.4, says records like these must not end a group, which is how Sitemap is already handled.

Just leaving the state unchanged fixes the rules, but it also merges the delays. Many sites give each bot its own delay in blocks like the one above (eff.org, moodle.org and vimeo.com, for example), and the last Crawl-delay in the merged group would then override the others. On eff.org, CCBot's delay would drop from 60 to 30. So instead, a User-agent line after Crawl-delay or Request-rate starts a new entry in the same group. The rules are shared by the whole group, and each delay and rate stays with the user agents listed directly above it, as before.

How I checked it:

  • The new tests fail on main and pass with the change.
  • crawl_delay() and request_rate() return the same values as before, and can_fetch() matches the old parser on the same file with the Crawl-delay and Request-rate lines removed. I compared all 271,452 files of up to 5 lines built from user-agent, allow, disallow, crawl-delay, request-rate, sitemap and unknown lines, plus 100,000 random longer files. The only differences were 36 cases where the user agent matches a group only as a substring (like Googlebot-Image and a Googlebot group). In each of them the new result comes from that agent's merged group, which the old code missed.
  • On the robots.txt files of the top 2,000 sites in the Tranco list (1,100 of them parsed), can_fetch() changes for some agents in 34 files, which is the fix. No crawl delay or request rate changes.

@Wali05
Wali05 requested a review from orsenthil as a code owner November 19, 2026 19:54
@python-cla-bot

python-cla-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant