Skip to content

ThrottlingRequestManager computes the wrong backoff for concurrent 429s and for Retry-After: 0 #2125

Closed

Description

@janbuchar

Two bugs in record_domain_delay, both from #1762:

  • consecutive_429_count advances once per response, not per rate-limit event. A burst of N in-flight requests all come back 429 and jump the exponent N steps at once — 8 concurrent requests take the delay from 2s to past max_delay in one go. The TS port added a guard for this in feat: implement per-domain request throttling (ThrottlingRequestManager) crawlee#3741, though it broke crawl-delay doing it, so that one isn't worth copying verbatim.
  • Retry-After: 0 parses to timedelta(0) and takes priority over the exponential backoff, so the domain ends up with no delay at all and the backoff never engages. test_parse_retry_after_zero_seconds currently asserts the broken behaviour. Treating a zero delay as "no usable header" and falling through to the backoff seems right.

Activity

  1. added
    bugSomething isn't working.
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Aug 6, 2026
  2. modified the milestone: 2.0 on Aug 7, 2026
  3. janbuchar commented on Aug 9, 2026

    @janbuchar
    CollaboratorAuthor

    Two things from the TS port that belong with the first bullet:

    • The suppression is only safe with the crawl-delay on its own clock. A crawl-delay is armed on every
      dispatch, so with a single throttled_until the domain always looks like it is already backing off
      and every 429 gets swallowed, Retry-After included. feat: implement per-domain request throttling (ThrottlingRequestManager) crawlee#3741 ended up with separate
      backoff_until and crawl_delay_until — that is the version worth copying, rather than the
      intermediate one this issue warns about.
    • record_success is a third way the exponent goes wrong. mark_request_as_handled calls it
      unconditionally and BasicCrawler marks permanently failed requests as handled too, so a request
      that died to 429s resets consecutive_429_count. TS dropped the success signal entirely and decays
      the counter on a timer instead.
  4. self-assigned this
    on Aug 10, 2026
  5. added a commit that references this issue on Aug 31, 2026
    142ec58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions