You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
High RAM consumption by any HttpCrawlers when working with large proxy pools #895
When working with large proxy pools (e.g., Apify RESIDENTIAL), I observe significant RAM usage growth. Memory consumption increases by more than a gigabyte within a few minutes during active scraping.
AFAIK this works just fine in the JS version, it feels like a memory leak in the python version somewhere to me. We should investigate why this happens, I don't see why 1000 sessions should be a problem.
Either way, a complete reproduction for this is needed unless you plan to deal with this on your own.
creating an HTTP session for each proxy
There should be a session per proxy. Or better say, there should be a proxy per session, its the other way around, we create sessions, and they are represented by a proxy.
I think there is a difference between Python and JS at the level of HTTP clients being used, as memory testing for 1000 httpx.AsyncClient instances shows consumption of about 1.7 gigabytes.
I think there is a difference between Python and JS at the level of HTTP clients being used, as memory testing for 1000 httpx.AsyncClient instances shows consumption of about 1.7 gigabytes.
Right, we are supposed to make a separate http client per proxy URL. But... this look like we just make one per request, huh?
Right, we are supposed to make a separate http client per proxy URL...
Yes, I understand why we're doing this.
this look like we just make one per request, huh?
Only at the start of the crawler's run. Under ideal conditions (when proxies aren't being blocked), the number of httpx.AsyncClient instances will tend toward the size of the SessionPool. With a large proxy pool, each session will have its own proxy URL. So when the crawler starts working with a SessionPool with max_pool_size of 1000, sessions will rotate with each request, and we create a httpx.AsyncClient for each request, which leads to high memory consumption at the start of crawler run. Once it reaches max_pool_sizehttpx.AsyncClient instances, the memory consumption stabilizes. The actual memory consumption for 1000 httpx.AsyncClient instances will be higher than in the provided code example since it's not making requests - it will be more than 2 gigabytes. This can be considered the crawler's baseline memory consumption, not counting other loads (Response size, data parsing, queue, caches, etc.)
My main concern is that for an HTTP crawler with default configuration, this memory consumption is very high. For users, this will really look like there's a terrible memory leak somewhere, as Martin suggested.
The situation when working with low-quality proxies will look even worse.
How costly is creating the HTTP client instance? Maybe we should dispose them the same way as we dispose stale browsers. IMO this is not a good reason to lower the session pool size.
Activity
B4nan commentedon Jan 10, 2025
AFAIK this works just fine in the JS version, it feels like a memory leak in the python version somewhere to me. We should investigate why this happens, I don't see why 1000 sessions should be a problem.
Either way, a complete reproduction for this is needed unless you plan to deal with this on your own.
There should be a session per proxy. Or better say, there should be a proxy per session, its the other way around, we create sessions, and they are represented by a proxy.
Mantisus commentedon Jan 13, 2025
I think there is a difference between Python and JS at the level of HTTP clients being used, as memory testing for 1000 httpx.AsyncClient instances shows consumption of about 1.7 gigabytes.
The same thing happens in the
get_clientmethod - https://github.com/apify/crawlee-python/blob/master/src/crawlee/http_clients/_httpx.py#L217janbuchar commentedon Jan 14, 2025
Right, we are supposed to make a separate http client per proxy URL. But... this look like we just make one per request, huh?
Mantisus commentedon Jan 14, 2025
Yes, I understand why we're doing this.
Only at the start of the crawler's run. Under ideal conditions (when proxies aren't being blocked), the number of
httpx.AsyncClientinstances will tend toward the size of theSessionPool. With a large proxy pool, each session will have its own proxy URL. So when the crawler starts working with aSessionPoolwithmax_pool_sizeof 1000, sessions will rotate with each request, and we create ahttpx.AsyncClientfor each request, which leads to high memory consumption at the start of crawler run. Once it reachesmax_pool_sizehttpx.AsyncClientinstances, the memory consumption stabilizes. The actual memory consumption for 1000httpx.AsyncClientinstances will be higher than in the provided code example since it's not making requests - it will be more than 2 gigabytes. This can be considered the crawler's baseline memory consumption, not counting other loads (Response size, data parsing, queue, caches, etc.)My main concern is that for an HTTP crawler with default configuration, this memory consumption is very high. For users, this will really look like there's a terrible memory leak somewhere, as Martin suggested.
The situation when working with low-quality proxies will look even worse.
B4nan commentedon Jan 14, 2025
How costly is creating the HTTP client instance? Maybe we should dispose them the same way as we dispose stale browsers. IMO this is not a good reason to lower the session pool size.
Mantisus commentedon Jan 14, 2025
About 2 MB per http client instance.
Is large session pool size needed for faster and more efficient scraping?
B4nan commentedon Jan 14, 2025
I meant how costly it is to create the instance.
fix: Optimize memory consumption for `HttpxHttpClient`, fix proxy han…