You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ThrottlingRequestManager computes the wrong backoff for concurrent 429s and for Retry-After: 0 #2125
consecutive_429_count advances once per response, not per rate-limit event. A burst of N in-flight requests all come back 429 and jump the exponent N steps at once — 8 concurrent requests take the delay from 2s to past max_delay in one go. The TS port added a guard for this in feat: implement per-domain request throttling (ThrottlingRequestManager) crawlee#3741, though it broke crawl-delay doing it, so that one isn't worth copying verbatim.
Retry-After: 0 parses to timedelta(0) and takes priority over the exponential backoff, so the domain ends up with no delay at all and the backoff never engages. test_parse_retry_after_zero_seconds currently asserts the broken behaviour. Treating a zero delay as "no usable header" and falling through to the backoff seems right.
Two things from the TS port that belong with the first bullet:
The suppression is only safe with the crawl-delay on its own clock. A crawl-delay is armed on every
dispatch, so with a single throttled_until the domain always looks like it is already backing off
and every 429 gets swallowed, Retry-After included. feat: implement per-domain request throttling (ThrottlingRequestManager) crawlee#3741 ended up with separate backoff_until and crawl_delay_until — that is the version worth copying, rather than the
intermediate one this issue warns about.
record_success is a third way the exponent goes wrong. mark_request_as_handled calls it
unconditionally and BasicCrawler marks permanently failed requests as handled too, so a request
that died to 429s resets consecutive_429_count. TS dropped the success signal entirely and decays
the counter on a timer instead.
Activity
janbuchar commentedon Aug 9, 2026
Two things from the TS port that belong with the first bullet:
dispatch, so with a single
throttled_untilthe domain always looks like it is already backing offand every 429 gets swallowed,
Retry-Afterincluded. feat: implement per-domain request throttling (ThrottlingRequestManager) crawlee#3741 ended up with separatebackoff_untilandcrawl_delay_until— that is the version worth copying, rather than theintermediate one this issue warns about.
record_successis a third way the exponent goes wrong.mark_request_as_handledcalls itunconditionally and
BasicCrawlermarks permanently failed requests as handled too, so a requestthat died to 429s resets
consecutive_429_count. TS dropped the success signal entirely and decaysthe counter on a timer instead.
fix(throttling-manager): Correct the backoff for concurrent 429s and …