Skip to content

Crawler can get softlocked when consuming the RequestQueue #1598

Closed

Description

@Pijukatel

Example scenario:

There is only 1 request in RQ
The request is processed for a long time - it fails twice, and it will pass only on the third attempt.
After each failure, the request is reclaimed.

The problem is that AutoscalePool can go completely rouge and can start burning resources up to the point where it blocks itself due to the system being overloaded.

The root cause is that __is_task_ready_function (which decides whether a new task should be scheduled) is checking if the not await request_manager.is_empty(). Since there is one request in progress and RQ is not empty, this will return True and the AutoscalePool will schedule a new task. The problem is that RQ has no available requests as the only one is already in progress. This will create a situation where new tasks are spawned as fast as possible, only to get no request from RQ.

Preferably, in such scenario __is_task_ready_function should return False, because the RQ should be aware of the fact that the only request available is already being handled. So __is_task_ready_function should return True if there are some available requests and return False if the only requests available are already being processed - but there is no such interface on RQ for this.

Bottom line not RequestManager.is_empty does not imply __is_task_ready, but our code has this expectation.

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Dec 4, 2025
  2. self-assigned this
    on Dec 5, 2025
  3. Pijukatel commented on Dec 5, 2025

    @Pijukatel
    CollaboratorAuthor

    Exposed when trying to add MacOS executors to CI: #1334

  4. janbuchar commented on Dec 5, 2025

    @janbuchar
    Collaborator

    I see two problems here:

    1. the apify.storages.ApifyRequestQueueClient.is_empty method returns True even when there are requests being handled
      • this is probably because is_empty and is_finished got merged into a single method at some point, which is a bit of a pickle
      • crawlee.storages.RequestQueue provides a general implementation of is_finished, but this does not take in-progress requests into account
    2. the fact that not is_empty is expected to imply __is_task_ready is neither documented nor enforced, as you mentioned
  5. Pijukatel commented on Dec 5, 2025

    @Pijukatel
    CollaboratorAuthor

    There might be something OS specific that makes this situation bad up to the softlock point, as it is fairly easy to reproduce only macOS.

  6. Pijukatel commented on Dec 8, 2025

    @Pijukatel
    CollaboratorAuthor

    It is probably mainly a theoretical problem. The real issue when reproducing it on the MacOS executor in GitHub was the executor itself - which consumed 100% CPU in certain cases and was blocking the AutoscaledPool regardless of the scenario apify/workflows#236

  7. janbuchar commented on Dec 8, 2025

    @janbuchar
    Collaborator

    It is probably mainly a theoretical problem.

    I don't think so, the fact that we don't distinguish is_empty from is_finished in the request queue client sounds like something that could bite us later.

  8. added a commit that references this issue on Jun 26, 2026
    0ef1078
  9. added a commit that references this issue on Jun 30, 2026
    497716a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions