You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Investigate potential autoscaledpool deadlock when client_info=1 #1207
client_info=1 of autoscaled pool can completely stop the crawler, without any recovery.
Investigate and fix if not desired.
Context:
A benchmark test was running for Parsel crawler. Crawler was set up in a way that it was scraping one site hosted locally by server. This eliminated any network delays during scraping as all the server responses were nearly instant. Requests to API were kept as it is, so for example request queue requests were still traveling over network to the real Apify API.
This is caused by Autoscaled pool not taking into consideration only new client errors,but all of them. So once we get some, the Autoscaled pool will report client_info = 1.0 forever and thus it will be blocked forever.
Activity
Pijukatel commentedon Jun 2, 2025
This is caused by Autoscaled pool not taking into consideration only new client errors,but all of them. So once we get some, the Autoscaled pool will report
client_info = 1.0forever and thus it will be blocked forever.fix: Fix `ClientSnapshot` overload calculation (#1228)