Skip to content

Investigate potential autoscaledpool deadlock when client_info=1 #1207

Closed

Description

@Pijukatel

Problem:

client_info=1 of autoscaled pool can completely stop the crawler, without any recovery.
Investigate and fix if not desired.

Context:

A benchmark test was running for Parsel crawler. Crawler was set up in a way that it was scraping one site hosted locally by server. This eliminated any network delays during scraping as all the server responses were nearly instant. Requests to API were kept as it is, so for example request queue requests were still traveling over network to the real Apify API.

https://console.apify.com/actors/UdJ0NQp2j8140G9db/runs/ZPlTj1VgMLQA7Yuxl#log

After a while the crawler got stopped forever by client_info = 1.0

...
2025-05-21T08:44:09.332Z [ParselCrawler] INFO Processing http://127.0.0.1:60059/6583 ...
2025-05-21T08:44:09.333Z [ParselCrawler] INFO Processing http://127.0.0.1:60059/6581 ...
2025-05-21T08:44:09.339Z [ParselCrawler] INFO Processing http://127.0.0.1:60059/6582 ...
2025-05-21T08:44:57.719Z [ParselCrawler] INFO Current request statistics:
2025-05-21T08:44:57.720Z ┌───────────────────────────────┬─────────────┐
2025-05-21T08:44:57.721Z │ requests_finished │ 3736 │
2025-05-21T08:44:57.722Z │ requests_failed │ 0 │
2025-05-21T08:44:57.723Z │ retry_histogram │ [3736] │
2025-05-21T08:44:57.724Z │ request_avg_failed_duration │ None │
2025-05-21T08:44:57.725Z │ request_avg_finished_duration │ 0.269101 │
2025-05-21T08:44:57.725Z │ requests_finished_per_minute │ 1245 │
2025-05-21T08:44:57.726Z │ requests_failed_per_minute │ 0 │
2025-05-21T08:44:57.727Z │ request_total_duration │ 1005.360811 │
2025-05-21T08:44:57.728Z │ requests_total │ 3736 │
2025-05-21T08:44:57.728Z │ crawler_runtime │ 180.038534 │
2025-05-21T08:44:57.729Z └───────────────────────────────┴─────────────┘
2025-05-21T08:44:57.730Z [crawlee._autoscaling.autoscaled_pool] INFO current_concurrency = 0; desired_concurrency = 14; cpu = 0.0; mem = 0.0; event_loop = 0.0; client_info = 1.0
...

Activity

  1. self-assigned this
    on May 21, 2025
  2. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on May 21, 2025
  3. Pijukatel commented on Jun 2, 2025

    @Pijukatel
    CollaboratorAuthor

    This is caused by Autoscaled pool not taking into consideration only new client errors,but all of them. So once we get some, the Autoscaled pool will report client_info = 1.0 forever and thus it will be blocked forever.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions