Skip to content

feat: Prevent overloading system memory when running locally - #1270

Merged
janbuchar merged 5 commits into
masterfrom
autoscaling-memory-failsafe
Jul 17, 2025
Merged

janbuchar merged 5 commits into
masterfrom
autoscaling-memory-failsafe

Conversation

@janbuchar

Copy link
Copy Markdown
Collaborator

@janbuchar janbuchar added the t-tooling Issues with this label are in the ownership of the tooling team. label Jun 25, 2025
@janbuchar
janbuchar requested review from Pijukatel and vdusek June 25, 2025 15:40
@github-actions github-actions Bot added this to the 117th sprint - Tooling team milestone Jun 25, 2025
@github-actions github-actions Bot added the tested Temporary label used only programatically for some analytics. label Jun 25, 2025
Comment thread src/crawlee/_autoscaling/_types.py Resolved
Comment thread src/crawlee/_autoscaling/snapshotter.py Resolved
current_size: ByteSize
"""Memory usage of the current Python process and its children."""

system_wide_used_size: ByteSize | None

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe these two new values can have default None, I saw it is anyway initialized with None both in code and test

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But I already updated all the call sites... and I'd need to reorder the fields in the dataclass ☹️ I'd prefer to leave it this way.

Comment thread src/crawlee/_autoscaling/_types.py Resolved

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@janbuchar
janbuchar requested a review from Pijukatel July 17, 2025 09:25
@janbuchar
janbuchar merged commit 30de3bd into master Jul 17, 2025
@janbuchar
janbuchar deleted the autoscaling-memory-failsafe branch July 17, 2025 09:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Autoscaling failsafe to prevent using up all memory when running locally

3 participants