Description
It seems to me that autoscaled_pool.desired_concurrency_ratio is currently completely useless.
It is in init here:
https://github.com/apify/crawlee-python/blob/master/src/crawlee/_autoscaling/autoscaled_pool.py#L57
It is checked for bounds 0> desired_concurrency_ratio >=1
https://github.com/apify/crawlee-python/blob/master/src/crawlee/_autoscaling/autoscaled_pool.py#L97
It is used in only one place:
https://github.com/apify/crawlee-python/blob/master/src/crawlee/_autoscaling/autoscaled_pool.py#L198
And from there in condition here:
https://github.com/apify/crawlee-python/blob/master/src/crawlee/_autoscaling/autoscaled_pool.py#L202
That condition will always be true for currently runtime enforced values 0> desired_concurrency_ratio >=1:
self.current_concurrency >= math.floor(self._desired_concurrency_ratio * self.current_concurrency)
Did I miss something?
Activity
janbuchar commentedon Nov 29, 2024
Before we look deeper into this, did you cross-check this against the JS version? Does it do the same thing there?
Pijukatel commentedon Nov 29, 2024
I think it is exactly the same
Same bounds:
https://github.com/apify/crawlee/blob/master/packages/core/src/autoscaling/autoscaled_pool.ts#L226
Same usage:
https://github.com/apify/crawlee/blob/30d0bf807f110b2bacf74d8099e9565edb509df7/packages/core/src/autoscaling/autoscaled_pool.ts#L598
vikyw89 commentedon Dec 2, 2024
In js concurrency setting works, in here it doesn't
Pijukatel commentedon Dec 3, 2024
Yes, in JS it is different. I read it incorrectly earlier. So it seems like there was a mistake when porting the code to Python.
JS:
const minCurrentConcurrency = Math.floor(this._desiredConcurrency * this.desiredConcurrencyRatio);
Python:
min_current_concurrency = math.floor(self._desired_concurrency_ratio * self.current_concurrency)
janbuchar commentedon Dec 3, 2024
Hm, then I might have made a mistake 🙂 Seems like an easy fix, right?
Pijukatel commentedon Dec 3, 2024
Yes, I will align the Python version with the JS version.
fix: Fix incorrect use of desired concurrency ratio (#780)
fix: Fix incorrect use of desired concurrency ratio (apify#780)