You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
urllib.robotparser: Crawl-delay between two User-agent lines splits the group #162263
Parsing of other records MUST NOT interfere with the parsing of explicitly defined records in Section 2. For example, a "Sitemaps" record MUST NOT terminate a group.
Crawl-delay is not one of the protocol's records. In google/robotstxt#51 one of the RFC's authors says lines such as Crawl-delay and Sitemap "should indeed not affect parsing of other lines".
The same file with a Sitemap: line in that place gives False, as expected, because the sitemap branch of RobotFileParser.parse() leaves state alone. The crawl-delay and request-rate branches set state = 2, so the next User-agent line starts a new entry. main has the same code.
Activity
pythongh-162263: Fix robotparser ending a group at Crawl-delay and Re…