Skip to content

Issue with concurrency  #786

Closed as not planned
Closed as not planned

Description

@omar6995

Hi guys,

I've an issue with concurrency.

It doesnt use the desired concurrency , neither the min concurrency parameter.

Here is the setup of the crawler :

def _setup_crawler(self):
    proxy_configuration = ProxyConfiguration(
    tiered_proxy_urls=[
        # lower tier, cheaper, preferred as long as they work
        [GLOBAL_ROTATING_PROXY, USA_ROTATING_PROXY],
        # higher tier, more expensive, used as a fallback
        [RESIDENTIAL_ROTATING_PROXY]
        ]
    )
    http_client = CurlImpersonateHttpClient(
        # Optional additional keyword arguments for `curl_cffi.requests.AsyncSession`.
        timeout=10,
        impersonate='chrome',
        headers={'Accept-Encoding': 'gzip'}
    )
    concurrency_settings = ConcurrencySettings(
        min_concurrency=self.max_workers *0.8,
        max_concurrency=self.max_workers,
        desired_concurrency=self.max_workers
    )
    self.crawler = HttpCrawler(
        #http_client=http_client,
        concurrency_settings=concurrency_settings,
        proxy_configuration=proxy_configuration,
        max_request_retries=3,
        retry_on_blocked=True,
        max_crawl_depth=0
    )

    @self.crawler.router.default_handler
    async def request_handler(context: HttpCrawlingContext) -> None:
       my request handler logic

And here is the output i'm getting : I don't understand why i'm at 22 concurrent request while I asked for 500?


[crawlee._autoscaling.snapshotter] INFO  Setting max_memory_size of this run to 3.89 GB.
[crawlee.http_crawler._http_crawler] INFO  Current request statistics:
┌───────────────────────────────┬──────────┐
│ requests_finished             │ 0        │
│ requests_failed               │ 0        │
│ retry_histogram               │ [0]      │
│ request_avg_failed_duration   │ None     │
│ request_avg_finished_duration │ None     │
│ requests_finished_per_minute  │ 0        │
│ requests_failed_per_minute    │ 0        │
│ request_total_duration        │ 0.0      │
│ requests_total                │ 0        │
│ crawler_runtime               │ 0.028202 │
└───────────────────────────────┴──────────┘
[crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 0; desired_concurrency = 500; cpu = 0; mem = 0; event_loop = 0.0; client_info = 0.0
[crawlee.http_crawler._http_crawler] INFO  Current request statistics:
┌───────────────────────────────┬─────────────┐
│ requests_finished             │ 1390        │
│ requests_failed               │ 0           │
│ retry_histogram               │ [1390]      │
│ request_avg_failed_duration   │ None        │
│ request_avg_finished_duration │ 0.761927    │
│ requests_finished_per_minute  │ 1389        │
│ requests_failed_per_minute    │ 0           │
│ request_total_duration        │ 1059.078615 │
│ requests_total                │ 1390        │
│ crawler_runtime               │ 60.032452   │
└───────────────────────────────┴─────────────┘
[crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 22; desired_concurrency = 500; cpu = 0.0; mem = 0.0; event_loop = 0.062; client_info = 0.0
[crawlee._autoscaling.autoscaled_pool] INFO  Waiting for remaining tasks to finish
[crawlee.http_crawler._http_crawler] INFO  Final request statistics:
┌───────────────────────────────┬─────────────┐
│ requests_finished             │ 2000        │
│ requests_failed               │ 0           │
│ retry_histogram               │ [2000]      │
│ request_avg_failed_duration   │ None        │
│ request_avg_finished_duration │ 0.720098    │
│ requests_finished_per_minute  │ 1152        │
│ requests_failed_per_minute    │ 0           │
│ request_total_duration        │ 1440.196803 │
│ requests_total                │ 2000        │
│ crawler_runtime               │ 104.179595  │
└───────────────────────────────┴─────────────┘

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Dec 5, 2024
  2. omar6995 commented on Dec 6, 2024

    @omar6995
    Author

    I have tried some new parameters to try to understand what is happening .

    With the following concurrent settings :
    -min concurrency : 200
    -Desired concurrency : 300
    -Max concurrency : 500

    I'm getting the following logs :

    I don't understand why the autoloadpool decide to scale down the concurrency below the min concurrency setting and with cpu, mem and event_loop parameters being almost at 0 .

    Maybe it is related to #759 ?

    [crawlee.http_crawler._http_crawler] INFO  Current request statistics:
    ┌───────────────────────────────┬──────────┐
    │ requests_finished             │ 0        │
    │ requests_failed               │ 0        │
    │ retry_histogram               │ [0]      │
    │ request_avg_failed_duration   │ None     │
    │ request_avg_finished_duration │ None     │
    │ requests_finished_per_minute  │ 0        │
    │ requests_failed_per_minute    │ 0        │
    │ request_total_duration        │ 0.0      │
    │ requests_total                │ 0        │
    │ crawler_runtime               │ 0.020802 │
    └───────────────────────────────┴──────────┘
    [crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 0; desired_concurrency = 315; cpu = 0; mem = 0; event_loop = 0.0; client_info = 0.0
    [crawlee.http_crawler._http_crawler] INFO  Current request statistics:
    ┌───────────────────────────────┬─────────────┐
    │ requests_finished             │ 1176        │
    │ requests_failed               │ 0           │
    │ retry_histogram               │ [1176]      │
    │ request_avg_failed_duration   │ None        │
    │ request_avg_finished_duration │ 0.882241    │
    │ requests_finished_per_minute  │ 1175        │
    │ requests_failed_per_minute    │ 0           │
    │ request_total_duration        │ 1037.515893 │
    │ requests_total                │ 1176        │
    │ crawler_runtime               │ 60.061702   │
    └───────────────────────────────┴─────────────┘
    [crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 405; desired_concurrency = 405; cpu = 0.0; mem = 0.0; event_loop = 0.042; client_info = 0.0
    [crawlee.http_crawler._http_crawler] INFO  Current request statistics:
    ┌───────────────────────────────┬─────────────┐
    │ requests_finished             │ 2289        │
    │ requests_failed               │ 0           │
    │ retry_histogram               │ [2265, 24]  │
    │ request_avg_failed_duration   │ None        │
    │ request_avg_finished_duration │ 0.988027    │
    │ requests_finished_per_minute  │ 1143        │
    │ requests_failed_per_minute    │ 0           │
    │ request_total_duration        │ 2261.593424 │
    │ requests_total                │ 2289        │
    │ crawler_runtime               │ 120.177076  │
    └───────────────────────────────┴─────────────┘
    [crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 500; desired_concurrency = 500; cpu = 0.0; mem = 0.0; event_loop = 0.085; client_info = 0.0
    [crawlee.http_crawler._http_crawler] INFO  Current request statistics:
    ┌───────────────────────────────┬─────────────┐
    │ requests_finished             │ 3541        │
    │ requests_failed               │ 0           │
    │ retry_histogram               │ [3517, 24]  │
    │ request_avg_failed_duration   │ None        │
    │ request_avg_finished_duration │ 0.919243    │
    │ requests_finished_per_minute  │ 1179        │
    │ requests_failed_per_minute    │ 0           │
    │ request_total_duration        │ 3255.039468 │
    │ requests_total                │ 3541        │
    │ crawler_runtime               │ 180.18069   │
    └───────────────────────────────┴─────────────┘
    [crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 9; desired_concurrency = 500; cpu = 0.0; mem = 0.0; event_loop = 0.09; client_info = 0.0
    
  3. janbuchar commented on Dec 6, 2024

    @janbuchar
    Collaborator

    Hi @omar6995 and thanks for your interest in Crawlee! We just released version 0.4.5 which might fix your problem (#780) - could you update and let us know if it helped?

  4. omar6995 commented on Dec 6, 2024

    @omar6995
    Author

    Hi, thanks for the release! I still have the same problem. The autoloadscaled pool still sometimes scale down the pool below the min concurrency even though it seems it happens less frequently.

    [crawlee.http_crawler._http_crawler] INFO  Current request statistics:
    ┌───────────────────────────────┬──────────┐
    │ requests_finished             │ 0        │
    │ requests_failed               │ 0        │
    │ retry_histogram               │ [0]      │
    │ request_avg_failed_duration   │ None     │
    │ request_avg_finished_duration │ None     │
    │ requests_finished_per_minute  │ 0        │
    │ requests_failed_per_minute    │ 0        │
    │ request_total_duration        │ 0.0      │
    │ requests_total                │ 0        │
    │ crawler_runtime               │ 0.000362 │
    └───────────────────────────────┴──────────┘
    [crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 0; desired_concurrency = 475; cpu = 0.0; mem = 0.0; event_loop = 1.0; client_info = 0.0
    [crawlee.http_crawler._http_crawler] INFO  Current request statistics:
    ┌───────────────────────────────┬─────────────┐
    │ requests_finished             │ 1244        │
    │ requests_failed               │ 0           │
    │ retry_histogram               │ [1244]      │
    │ request_avg_failed_duration   │ None        │
    │ request_avg_finished_duration │ 0.882175    │
    │ requests_finished_per_minute  │ 1243        │
    │ requests_failed_per_minute    │ 0           │
    │ request_total_duration        │ 1097.425944 │
    │ requests_total                │ 1244        │
    │ crawler_runtime               │ 60.049814   │
    └───────────────────────────────┴─────────────┘
    [crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 500; desired_concurrency = 500; cpu = 0.0; mem = 0.0; event_loop = 0.042; client_info = 0.0
    [crawlee.http_crawler._http_crawler] INFO  Current request statistics:
    ┌───────────────────────────────┬─────────────┐
    │ requests_finished             │ 2232        │
    │ requests_failed               │ 0           │
    │ retry_histogram               │ [2232]      │
    │ request_avg_failed_duration   │ None        │
    │ request_avg_finished_duration │ 0.842759    │
    │ requests_finished_per_minute  │ 1115        │
    │ requests_failed_per_minute    │ 0           │
    │ request_total_duration        │ 1881.038413 │
    │ requests_total                │ 2232        │
    │ crawler_runtime               │ 120.054858  │
    └───────────────────────────────┴─────────────┘
    [crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 500; desired_concurrency = 500; cpu = 0.0; mem = 0.0; event_loop = 0.113; client_info = 0.0
    [crawlee.http_crawler._http_crawler] INFO  Current request statistics:
    ┌───────────────────────────────┬─────────────┐
    │ requests_finished             │ 2959        │
    │ requests_failed               │ 0           │
    │ retry_histogram               │ [2959]      │
    │ request_avg_failed_duration   │ None        │
    │ request_avg_finished_duration │ 0.799269    │
    │ requests_finished_per_minute  │ 985         │
    │ requests_failed_per_minute    │ 0           │
    │ request_total_duration        │ 2365.035641 │
    │ requests_total                │ 2959        │
    │ crawler_runtime               │ 180.314569  │
    └───────────────────────────────┴─────────────┘
    [crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 500; desired_concurrency = 500; cpu = 0.0; mem = 0.0; event_loop = 0.232; client_info = 0.0
    [crawlee.http_crawler._http_crawler] INFO  Current request statistics:
    ┌───────────────────────────────┬─────────────┐
    │ requests_finished             │ 3698        │
    │ requests_failed               │ 0           │
    │ retry_histogram               │ [3698]      │
    │ request_avg_failed_duration   │ None        │
    │ request_avg_finished_duration │ 0.76877     │
    │ requests_finished_per_minute  │ 923         │
    │ requests_failed_per_minute    │ 0           │
    │ request_total_duration        │ 2842.910111 │
    │ requests_total                │ 3698        │
    │ crawler_runtime               │ 240.34619   │
    └───────────────────────────────┴─────────────┘
    [crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 500; desired_concurrency = 500; cpu = 0.0; mem = 0.0; event_loop = 0.182; client_info = 0.0
    [crawlee.http_crawler._http_crawler] INFO  Current request statistics:
    ┌───────────────────────────────┬─────────────┐
    │ requests_finished             │ 4556        │
    │ requests_failed               │ 0           │
    │ retry_histogram               │ [4556]      │
    │ request_avg_failed_duration   │ None        │
    │ request_avg_finished_duration │ 0.751872    │
    │ requests_finished_per_minute  │ 910         │
    │ requests_failed_per_minute    │ 0           │
    │ request_total_duration        │ 3425.530428 │
    │ requests_total                │ 4556        │
    │ crawler_runtime               │ 300.457215  │
    └───────────────────────────────┴─────────────┘
    [crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 500; desired_concurrency = 500; cpu = 0.0; mem = 0.0; event_loop = 0.251; client_info = 0.0
    [crawlee.http_crawler._http_crawler] INFO  Current request statistics:
    ┌───────────────────────────────┬─────────────┐
    │ requests_finished             │ 5312        │
    │ requests_failed               │ 0           │
    │ retry_histogram               │ [5312]      │
    │ request_avg_failed_duration   │ None        │
    │ request_avg_finished_duration │ 0.743034    │
    │ requests_finished_per_minute  │ 884         │
    │ requests_failed_per_minute    │ 0           │
    │ request_total_duration        │ 3946.994408 │
    │ requests_total                │ 5312        │
    │ crawler_runtime               │ 360.460791  │
    └───────────────────────────────┴─────────────┘
    [crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 13; desired_concurrency = 500; cpu = 0.0; mem = 0.0; event_loop = 0.102; client_info = 0.0 
    *********** Issue here , why it went below the min concurrency ? *********** 
    [crawlee.http_crawler._http_crawler] INFO  Current request statistics:
    ┌───────────────────────────────┬─────────────┐
    │ requests_finished             │ 6143        │
    │ requests_failed               │ 0           │
    │ retry_histogram               │ [6143]      │
    │ request_avg_failed_duration   │ None        │
    │ request_avg_finished_duration │ 0.757151    │
    │ requests_finished_per_minute  │ 876         │
    │ requests_failed_per_minute    │ 0           │
    │ request_total_duration        │ 4651.181663 │
    │ requests_total                │ 6143        │
    │ crawler_runtime               │ 420.831057  │
    
    

    Also, i was wondering why the crawler was slow even with 500 concurrent requests.

    i noticed that even if it displays 500 concurrent requests on my proxy dashboard i'm never having more than 11-20 active thread.

  5. Pijukatel commented on Dec 9, 2024

    @Pijukatel
    Collaborator

    Hi @omar6995 , I will try to explain the observed behavior.

    There are some things to understand first.
    autoscaled_pool.current_concurency is not directly regulated by autoscaled pool. It just shows how many tasks are currently running. It is the autoscaled_pool.desired_concurency that is the regulated value. That value can be between min_concurrency and max_concurrency. This makes it possible for current_concurency to be below min_concurrency in several common cases. One of them is for example if no new requests are ready. In such case system is ready to process new requests, but none are available and thus current_concurency, can be below minimal_concurrency.

    autoscaled_pool will try to start new task and thus increase autoscaled_pool.current_concurency if it is possible based on several conditions. Please see: new tasks starts
    Pay extra attention to the debug messages that summarize, why the task was not scheduled. Only one of them is 'Not scheduling new tasks - already running at desired concurrency'.
    There are three other reasons though:
    'Not scheduling new tasks - system is overloaded'
    'Not scheduling new tasks - the autoscaled pool is paused'
    'Not scheduling new task - no task is ready'

    So I think we would need to see DEBUG level logs, to see what really happened and if that is something wrong. On this level of logs I can't tell as it can be still behavior as expected, but it can also be some hidden bug.

  6. Pijukatel commented on Jan 23, 2025

    @Pijukatel
    Collaborator

    Not enough information to investigate further.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions