Skip to content

gh-162263: Fix robotparser ending a group at Crawl-delay and Request-rate #220125

gh-162263: Fix robotparser ending a group at Crawl-delay and Request-rate

gh-162263: Fix robotparser ending a group at Crawl-delay and Request-rate #220125

Job Run time
14s
30s
10m 44s
28m 8s
2m 49s
3m 3s
2m 31s
3m 6s
9m 49s
6m 13s
2m 46s
4m 38s
5m 21s
17m 17s
36m 0s
12m 8s
13m 45s
12m 32s
5m 0s
10m 41s
11m 30s
10m 12s
20m 40s
6m 15s
13s
18m 54s
16m 30s
19m 54s
13m 32s
21m 38s
24m 23s
24m 1s
19m 4s
22m 17s
10m 44s
36m 48s
36m 53s
19m 29s
19m 14s
15m 29s
0s
6m 57s
2m 3s
2m 24s
1m 48s
4s
9h 28m 11s