You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Crawler can get softlocked when consuming the RequestQueue #1598
There is only 1 request in RQ
The request is processed for a long time - it fails twice, and it will pass only on the third attempt.
After each failure, the request is reclaimed.
The problem is that AutoscalePool can go completely rouge and can start burning resources up to the point where it blocks itself due to the system being overloaded.
The root cause is that __is_task_ready_function (which decides whether a new task should be scheduled) is checking if the not await request_manager.is_empty(). Since there is one request in progress and RQ is not empty, this will return True and the AutoscalePool will schedule a new task. The problem is that RQ has no available requests as the only one is already in progress. This will create a situation where new tasks are spawned as fast as possible, only to get no request from RQ.
Preferably, in such scenario __is_task_ready_function should return False, because the RQ should be aware of the fact that the only request available is already being handled. So __is_task_ready_function should return True if there are some available requests and return False if the only requests available are already being processed - but there is no such interface on RQ for this.
Bottom line not RequestManager.is_empty does not imply __is_task_ready, but our code has this expectation.
It is probably mainly a theoretical problem. The real issue when reproducing it on the MacOS executor in GitHub was the executor itself - which consumed 100% CPU in certain cases and was blocking the AutoscaledPool regardless of the scenario apify/workflows#236
I don't think so, the fact that we don't distinguish is_empty from is_finished in the request queue client sounds like something that could bite us later.
Activity
Pijukatel commentedon Dec 5, 2025
Exposed when trying to add MacOS executors to CI: #1334
janbuchar commentedon Dec 5, 2025
I see two problems here:
apify.storages.ApifyRequestQueueClient.is_emptymethod returnsTrueeven when there are requests being handledis_emptyandis_finishedgot merged into a single method at some point, which is a bit of a picklecrawlee.storages.RequestQueueprovides a general implementation ofis_finished, but this does not take in-progress requests into accountnot is_emptyis expected to imply__is_task_readyis neither documented nor enforced, as you mentionedPijukatel commentedon Dec 5, 2025
There might be something OS specific that makes this situation bad up to the softlock point, as it is fairly easy to reproduce only macOS.
Pijukatel commentedon Dec 8, 2025
It is probably mainly a theoretical problem. The real issue when reproducing it on the MacOS executor in GitHub was the executor itself - which consumed 100% CPU in certain cases and was blocking the
AutoscaledPoolregardless of the scenario apify/workflows#236janbuchar commentedon Dec 8, 2025
I don't think so, the fact that we don't distinguish
is_emptyfromis_finishedin the request queue client sounds like something that could bite us later.fix: Add `is_finished` and correct `is_empty` semantics in `RequestQu…
fix: Add `is_finished` and correct `is_empty` semantics in `RequestQu…