fix(throttling-manager): Defer throttled 429s without spending retries and give up on persistently rate-limited domains

[Mantisus]

Description

A 429 from a domain throttled by ThrottlingRequestManager no longer counts as a failure. The request goes back to its queue and waits out the domain's backoff. The retry doesn't count towards max_request_retries or max_session_rotations, the session isn't marked as bad, and the 429 doesn't reach error_handler or failed_request_handler. Setting ignore_http_error_status_codes={429} no longer passes such a response to the request handler.

A domain that keeps rate-limiting every request no longer holds the crawl forever. The new max_domain_stall option on ThrottlingRequestManager (15 minutes by default) sets how long that may last. The crawler still finishes every other domain first, then crawler.run() raises PersistentRateLimitError. The stalled domain's requests stay queued, so a later run can resume them.

New errors (parity with Crawlee TS):

• RequestThrottledError: the crawler raises it for a throttled 429 and defers the request.
• PersistentRateLimitError: raised from crawler.run() when a domain has stalled for longer than max_domain_stall.

Issues

• Closes: ThrottlingRequestManager does not stop a 429 from retiring the session #2143

Testing

• Added new unit tests

[vdusek]

LGTM, thanks! I just pushed a few minor follow-up commits.

Since @janbuchar will be away for a while, I'm going to go ahead and merge this.