Skip to content

autoscaled_pool.desired_concurrency_ratio is useless #759

Closed

Description

@Pijukatel

It seems to me that autoscaled_pool.desired_concurrency_ratio is currently completely useless.

It is in init here:
https://github.com/apify/crawlee-python/blob/master/src/crawlee/_autoscaling/autoscaled_pool.py#L57

It is checked for bounds 0> desired_concurrency_ratio >=1
https://github.com/apify/crawlee-python/blob/master/src/crawlee/_autoscaling/autoscaled_pool.py#L97

It is used in only one place:
https://github.com/apify/crawlee-python/blob/master/src/crawlee/_autoscaling/autoscaled_pool.py#L198

And from there in condition here:
https://github.com/apify/crawlee-python/blob/master/src/crawlee/_autoscaling/autoscaled_pool.py#L202

That condition will always be true for currently runtime enforced values 0> desired_concurrency_ratio >=1:

self.current_concurrency >= math.floor(self._desired_concurrency_ratio * self.current_concurrency)

Did I miss something?

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Nov 28, 2024
  2. janbuchar commented on Nov 29, 2024

    @janbuchar
    Collaborator

    Before we look deeper into this, did you cross-check this against the JS version? Does it do the same thing there?

  3. vikyw89 commented on Dec 2, 2024

    @vikyw89

    In js concurrency setting works, in here it doesn't

  4. Pijukatel commented on Dec 3, 2024

    @Pijukatel
    CollaboratorAuthor

    Yes, in JS it is different. I read it incorrectly earlier. So it seems like there was a mistake when porting the code to Python.

    JS:
    const minCurrentConcurrency = Math.floor(this._desiredConcurrency * this.desiredConcurrencyRatio);

    Python:
    min_current_concurrency = math.floor(self._desired_concurrency_ratio * self.current_concurrency)

  5. janbuchar commented on Dec 3, 2024

    @janbuchar
    Collaborator

    Hm, then I might have made a mistake 🙂 Seems like an easy fix, right?

  6. self-assigned this
    on Dec 3, 2024
  7. Pijukatel commented on Dec 3, 2024

    @Pijukatel
    CollaboratorAuthor

    Yes, I will align the Python version with the JS version.

  8. added a commit that references this issue on Dec 5, 2024
    d1f8bfb
  9. added a commit that references this issue on Dec 10, 2024
    8ea12d3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions