Repository navigation
Closed as not planned
Labels
t-toolingIssues with this label are in the ownership of the tooling team.Issues with this label are in the ownership of the tooling team.
Description
Hi guys,
I've an issue with concurrency.
It doesnt use the desired concurrency , neither the min concurrency parameter.
Here is the setup of the crawler :
def _setup_crawler(self):
proxy_configuration = ProxyConfiguration(
tiered_proxy_urls=[
# lower tier, cheaper, preferred as long as they work
[GLOBAL_ROTATING_PROXY, USA_ROTATING_PROXY],
# higher tier, more expensive, used as a fallback
[RESIDENTIAL_ROTATING_PROXY]
]
)
http_client = CurlImpersonateHttpClient(
# Optional additional keyword arguments for `curl_cffi.requests.AsyncSession`.
timeout=10,
impersonate='chrome',
headers={'Accept-Encoding': 'gzip'}
)
concurrency_settings = ConcurrencySettings(
min_concurrency=self.max_workers *0.8,
max_concurrency=self.max_workers,
desired_concurrency=self.max_workers
)
self.crawler = HttpCrawler(
#http_client=http_client,
concurrency_settings=concurrency_settings,
proxy_configuration=proxy_configuration,
max_request_retries=3,
retry_on_blocked=True,
max_crawl_depth=0
)
@self.crawler.router.default_handler
async def request_handler(context: HttpCrawlingContext) -> None:
my request handler logic
And here is the output i'm getting : I don't understand why i'm at 22 concurrent request while I asked for 500?
[crawlee._autoscaling.snapshotter] INFO Setting max_memory_size of this run to 3.89 GB.
[crawlee.http_crawler._http_crawler] INFO Current request statistics:
┌───────────────────────────────┬──────────┐
│ requests_finished │ 0 │
│ requests_failed │ 0 │
│ retry_histogram │ [0] │
│ request_avg_failed_duration │ None │
│ request_avg_finished_duration │ None │
│ requests_finished_per_minute │ 0 │
│ requests_failed_per_minute │ 0 │
│ request_total_duration │ 0.0 │
│ requests_total │ 0 │
│ crawler_runtime │ 0.028202 │
└───────────────────────────────┴──────────┘
[crawlee._autoscaling.autoscaled_pool] INFO current_concurrency = 0; desired_concurrency = 500; cpu = 0; mem = 0; event_loop = 0.0; client_info = 0.0
[crawlee.http_crawler._http_crawler] INFO Current request statistics:
┌───────────────────────────────┬─────────────┐
│ requests_finished │ 1390 │
│ requests_failed │ 0 │
│ retry_histogram │ [1390] │
│ request_avg_failed_duration │ None │
│ request_avg_finished_duration │ 0.761927 │
│ requests_finished_per_minute │ 1389 │
│ requests_failed_per_minute │ 0 │
│ request_total_duration │ 1059.078615 │
│ requests_total │ 1390 │
│ crawler_runtime │ 60.032452 │
└───────────────────────────────┴─────────────┘
[crawlee._autoscaling.autoscaled_pool] INFO current_concurrency = 22; desired_concurrency = 500; cpu = 0.0; mem = 0.0; event_loop = 0.062; client_info = 0.0
[crawlee._autoscaling.autoscaled_pool] INFO Waiting for remaining tasks to finish
[crawlee.http_crawler._http_crawler] INFO Final request statistics:
┌───────────────────────────────┬─────────────┐
│ requests_finished │ 2000 │
│ requests_failed │ 0 │
│ retry_histogram │ [2000] │
│ request_avg_failed_duration │ None │
│ request_avg_finished_duration │ 0.720098 │
│ requests_finished_per_minute │ 1152 │
│ requests_failed_per_minute │ 0 │
│ request_total_duration │ 1440.196803 │
│ requests_total │ 2000 │
│ crawler_runtime │ 104.179595 │
└───────────────────────────────┴─────────────┘
Metadata
Metadata
Assignees
Labels
t-toolingIssues with this label are in the ownership of the tooling team.Issues with this label are in the ownership of the tooling team.
Activity
omar6995 commentedon Dec 6, 2024
I have tried some new parameters to try to understand what is happening .
With the following concurrent settings :
-min concurrency : 200
-Desired concurrency : 300
-Max concurrency : 500
I'm getting the following logs :
I don't understand why the autoloadpool decide to scale down the concurrency below the min concurrency setting and with cpu, mem and event_loop parameters being almost at 0 .
Maybe it is related to #759 ?
janbuchar commentedon Dec 6, 2024
Hi @omar6995 and thanks for your interest in Crawlee! We just released version 0.4.5 which might fix your problem (#780) - could you update and let us know if it helped?
omar6995 commentedon Dec 6, 2024
Hi, thanks for the release! I still have the same problem. The autoloadscaled pool still sometimes scale down the pool below the min concurrency even though it seems it happens less frequently.
Also, i was wondering why the crawler was slow even with 500 concurrent requests.
i noticed that even if it displays 500 concurrent requests on my proxy dashboard i'm never having more than 11-20 active thread.
Pijukatel commentedon Dec 9, 2024
Hi @omar6995 , I will try to explain the observed behavior.
There are some things to understand first.
autoscaled_pool.current_concurency is not directly regulated by autoscaled pool. It just shows how many tasks are currently running. It is the autoscaled_pool.desired_concurency that is the regulated value. That value can be between min_concurrency and max_concurrency. This makes it possible for current_concurency to be below min_concurrency in several common cases. One of them is for example if no new requests are ready. In such case system is ready to process new requests, but none are available and thus current_concurency, can be below minimal_concurrency.
autoscaled_pool will try to start new task and thus increase autoscaled_pool.current_concurency if it is possible based on several conditions. Please see: new tasks starts
Pay extra attention to the debug messages that summarize, why the task was not scheduled. Only one of them is 'Not scheduling new tasks - already running at desired concurrency'.
There are three other reasons though:
'Not scheduling new tasks - system is overloaded'
'Not scheduling new tasks - the autoscaled pool is paused'
'Not scheduling new task - no task is ready'
So I think we would need to see DEBUG level logs, to see what really happened and if that is something wrong. On this level of logs I can't tell as it can be still behavior as expected, but it can also be some hidden bug.
Pijukatel commentedon Jan 23, 2025
Not enough information to investigate further.