Skip to content

High RAM consumption by any HttpCrawlers when working with large proxy pools #895

Closed

Description

@Mantisus

When working with large proxy pools (e.g., Apify RESIDENTIAL), I observe significant RAM usage growth. Memory consumption increases by more than a gigabyte within a few minutes during active scraping.

My version:
The issue appears to be related to creating an HTTP session for each proxy https://github.com/apify/crawlee-python/blob/master/src/crawlee/http_clients/_httpx.py#L132 combined with the high default maximum SessionPool size of 1000 sessions. This results in the crawler creating a new HTTP session for almost every new request during its initial run.

The absence of cleanup logic for created HTTP sessions will likely worsen the situation when the proxy pool contains a large number of "bad" proxies.

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Jan 10, 2025
  2. B4nan commented on Jan 10, 2025

    @B4nan
    Member

    AFAIK this works just fine in the JS version, it feels like a memory leak in the python version somewhere to me. We should investigate why this happens, I don't see why 1000 sessions should be a problem.

    Either way, a complete reproduction for this is needed unless you plan to deal with this on your own.

    creating an HTTP session for each proxy

    There should be a session per proxy. Or better say, there should be a proxy per session, its the other way around, we create sessions, and they are represented by a proxy.

  3. self-assigned this
    on Jan 13, 2025
  4. Mantisus commented on Jan 13, 2025

    @Mantisus
    CollaboratorAuthor

    I think there is a difference between Python and JS at the level of HTTP clients being used, as memory testing for 1000 httpx.AsyncClient instances shows consumption of about 1.7 gigabytes.

    import asyncio
    import resource
    from httpx import AsyncClient, AsyncHTTPTransport
    
    
    def log_memory_usage():
        usage = resource.getrusage(resource.RUSAGE_SELF)
        memory_mb = usage.ru_maxrss / 1024  
        print(f"Memory usage: {memory_mb:.2f} MB")
    
    async def create_httpx_sessions():
        sessions = {
            i: AsyncClient(
                http1=True,
                http2=True,
                proxy=f"http://{i}:{i}",
                transport=(AsyncHTTPTransport(
                                        proxy=f"http://{i}:{i}", 
                                        http1=True, 
                                        http2=True))
            ) for i in range(1000)
        }
        return sessions
    
    async def run():
        print("Starting test...")
        log_memory_usage()
        
        sessions = await create_httpx_sessions()
    
        print("Sessions created")
        log_memory_usage()
        
    def main() -> None:
        asyncio.run(run())
    Starting test...
    Memory usage: 26.57 MB
    Sessions created
    Memory usage: 1710.93 MB

    The same thing happens in the get_client method - https://github.com/apify/crawlee-python/blob/master/src/crawlee/http_clients/_httpx.py#L217

  5. janbuchar commented on Jan 14, 2025

    @janbuchar
    Collaborator

    I think there is a difference between Python and JS at the level of HTTP clients being used, as memory testing for 1000 httpx.AsyncClient instances shows consumption of about 1.7 gigabytes.

    Right, we are supposed to make a separate http client per proxy URL. But... this look like we just make one per request, huh?

  6. Mantisus commented on Jan 14, 2025

    @Mantisus
    CollaboratorAuthor

    Right, we are supposed to make a separate http client per proxy URL...

    Yes, I understand why we're doing this.

    this look like we just make one per request, huh?

    Only at the start of the crawler's run. Under ideal conditions (when proxies aren't being blocked), the number of httpx.AsyncClient instances will tend toward the size of the SessionPool. With a large proxy pool, each session will have its own proxy URL. So when the crawler starts working with a SessionPool with max_pool_size of 1000, sessions will rotate with each request, and we create a httpx.AsyncClient for each request, which leads to high memory consumption at the start of crawler run. Once it reaches max_pool_size httpx.AsyncClient instances, the memory consumption stabilizes. The actual memory consumption for 1000 httpx.AsyncClient instances will be higher than in the provided code example since it's not making requests - it will be more than 2 gigabytes. This can be considered the crawler's baseline memory consumption, not counting other loads (Response size, data parsing, queue, caches, etc.)

    My main concern is that for an HTTP crawler with default configuration, this memory consumption is very high. For users, this will really look like there's a terrible memory leak somewhere, as Martin suggested.

    The situation when working with low-quality proxies will look even worse.

  7. B4nan commented on Jan 14, 2025

    @B4nan
    Member

    How costly is creating the HTTP client instance? Maybe we should dispose them the same way as we dispose stale browsers. IMO this is not a good reason to lower the session pool size.

  8. Mantisus commented on Jan 14, 2025

    @Mantisus
    CollaboratorAuthor

    About 2 MB per http client instance.

    Is large session pool size needed for faster and more efficient scraping?

  9. B4nan commented on Jan 14, 2025

    @B4nan
    Member

    I meant how costly it is to create the instance.

  10. assigned and unassigned on Jan 15, 2025
  11. added a commit that references this issue on Jan 22, 2025
    d7ad480
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions