Skip to content

Request handler should be able to timeout blocking sync code in user defined handler #908

Open

Description

@Pijukatel

Current purely async implementation of request handler is not capable of triggering timeout for blocking sync code. Such code is created by users and we have no control over it. So we can't expect only async blocking code.

Following simple user defined handler can't currently trigger timeout even if it should.

@crawler.router.default_handler
async def handler(context: BasicCrawlingContext) -> None:
    time.sleep(some_big_time)

Activity

  1. added
    bugSomething isn't working.
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Jan 15, 2025
  2. janbuchar commented on Jan 20, 2025

    @janbuchar
    Collaborator

    It is almost impossible to handle this cleanly AFAIK - we cannot interrupt sync code that blocks the event loop. Some options I see:

    1. Run request handler in a separate event loop - might break things like playwright in unusual ways
    2. Detect when something blocks the event loop for too long and notify the user
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working.t-toolingIssues with this label are in the ownership of the tooling team.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions