Skip to content

Autoscaling failsafe to prevent using up all memory when running locally #1232

Closed

Description

@janbuchar

When running locally, Crawlee thinks "memory is overloaded" means "stuff that we own takes up more memory than some configured part of the total system memory" - if our calculation of the used memory fails to take something into account, we could easily smother the system.

We should have some safety mechanism that would say the system is overloaded if the system memory has something like >95% utilization (by any processes, not just current crawlee process), regardless of CRAWLEE_AVAILABLE_MEMORY_RATIO? shouldn't we?

Originally posted by @janbuchar in #1210 (comment)

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Jun 7, 2025
  2. added a commit that references this issue on Jul 17, 2025
    30de3bd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions