Skip to content

Default crawlee[cli] template project broken on Windows #1204

Closed

Description

@solenum

This could just be a misunderstanding on my part, first time using crawlee. I followed the guide on the homepage and used pipx to create a project

pipx run 'crawlee[cli]' create my-crawler

This produces a project that fails to run after I activate the venv in powershell and attempt to run the project

cd my-crawler
.\.venv\Scripts\activate
python -m my-crawler
[asyncio] ERROR Task exception was never retrieved
future: <Task finished name='Task-12' coro=<Connection.run() done, defined at C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_connection.py:272> exception=NotImplementedError()>
      Traceback (most recent call last):
        File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_connection.py", line 279, in run
          await self._transport.connect()
        File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_transport.py", line 133, in connect
          raise exc
        File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_transport.py", line 120, in connect
          self._proc = await asyncio.create_subprocess_exec(
        File "C:\Python310\lib\asyncio\subprocess.py", line 218, in create_subprocess_exec
          transport, protocol = await loop.subprocess_exec(
        File "C:\Python310\lib\asyncio\base_events.py", line 1667, in subprocess_exec
          transport = await self._make_subprocess_transport(
        File "C:\Python310\lib\asyncio\base_events.py", line 498, in _make_subprocess_transport
          raise NotImplementedError
      NotImplementedError

Annoyingly this took me ages to figure out. It seems to be related to this line in the _main_.py that gets generated, as commenting it out resolves the problem:

    if platform.system() == 'Windows':
        # This mitigates a warning raised by curl-cffi. If you do not need to use curl-impersonate, you may remove this.
        asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
        # __comment out ^ to fix__

Is this just an issue with my configuration? I've updated pip and all packages to the latest versions, I guess my problems resolved with the above but I couldn't find a relevant issue anywhere and wasted a lot of my workday trying to figure this out so I figured I'd create this issue to hopefully prevent others from getting stuck on the same problem. (I'm not a Windows person - entirely possible this is something I'm just not understanding)

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on May 20, 2025
  2. solenum commented on May 20, 2025

    @solenum
    Author

    Full powershell terminal output

    (.venv) PS C:\Users\redacted\Documents\Git\Work\redacted\redacted> python -m redacted
    [crawlee.crawlers._playwright._playwright_crawler] INFO  Current request statistics:
    ┌───────────────────────────────┬──────────┐
    │ requests_finished             │ 0        │
    │ requests_failed               │ 0        │
    │ retry_histogram               │ [0]      │
    │ request_avg_failed_duration   │ None     │
    │ request_avg_finished_duration │ None     │
    │ requests_finished_per_minute  │ 0        │
    │ requests_failed_per_minute    │ 0        │
    │ request_total_duration        │ 0.0      │
    │ requests_total                │ 0        │
    │ crawler_runtime               │ 0.017117 │
    └───────────────────────────────┴──────────┘
    [asyncio] ERROR Task exception was never retrieved
    future: <Task finished name='Task-12' coro=<Connection.run() done, defined at C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_connection.py:272> exception=NotImplementedError()>
          Traceback (most recent call last):
            File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_connection.py", line 279, in run
              await self._transport.connect()
            File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_transport.py", line 133, in connect
              raise exc
            File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_transport.py", line 120, in connect
              self._proc = await asyncio.create_subprocess_exec(
            File "C:\Python310\lib\asyncio\subprocess.py", line 218, in create_subprocess_exec
              transport, protocol = await loop.subprocess_exec(
            File "C:\Python310\lib\asyncio\base_events.py", line 1667, in subprocess_exec
              transport = await self._make_subprocess_transport(
            File "C:\Python310\lib\asyncio\base_events.py", line 498, in _make_subprocess_transport
              raise NotImplementedError
          NotImplementedError
    Traceback (most recent call last):
      File "C:\Python310\lib\runpy.py", line 196, in _run_module_as_main
        return _run_code(code, main_globals, None,
      File "C:\Python310\lib\runpy.py", line 86, in _run_code
        exec(code, run_globals)
      File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\redacted\__main__.py", line 12, in <module>
        asyncio.run(main())
      File "C:\Python310\lib\asyncio\runners.py", line 44, in run
        return loop.run_until_complete(main)
      File "C:\Python310\lib\asyncio\base_events.py", line 646, in run_until_complete
        return future.result()
      File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\redacted\main.py", line 22, in main
        await crawler.run(
      File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\crawlee\crawlers\_basic\_basic_crawler.py", line 606, in run
        await run_task
      File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\crawlee\crawlers\_basic\_basic_crawler.py", line 657, in _run_crawler
        await exit_stack.enter_async_context(context)  # type: ignore[arg-type]
      File "C:\Python310\lib\contextlib.py", line 619, in enter_async_context
        result = await _cm_type.__aenter__(cm)
      File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\crawlee\browsers\_browser_pool.py", line 202, in __aenter__
        await asyncio.wait_for(plugin.__aenter__(), timeout)
      File "C:\Python310\lib\asyncio\tasks.py", line 445, in wait_for
        return fut.result()
      File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\crawlee\browsers\_playwright_browser_plugin.py", line 141, in __aenter__
        self._playwright = await self._playwright_context_manager.__aenter__()
      File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\async_api\_context_manager.py", line 46, in __aenter__
        playwright = AsyncPlaywright(next(iter(done)).result())
      File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_connection.py", line 279, in run
        await self._transport.connect()
      File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_transport.py", line 133, in connect
        raise exc
      File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_transport.py", line 120, in connect
        self._proc = await asyncio.create_subprocess_exec(
      File "C:\Python310\lib\asyncio\subprocess.py", line 218, in create_subprocess_exec
        transport, protocol = await loop.subprocess_exec(
      File "C:\Python310\lib\asyncio\base_events.py", line 1667, in subprocess_exec
        transport = await self._make_subprocess_transport(
      File "C:\Python310\lib\asyncio\base_events.py", line 498, in _make_subprocess_transport
        raise NotImplementedError
    NotImplementedError
    
  3. janbuchar commented on May 20, 2025

    @janbuchar
    Collaborator

    Hi @solenum, I'm not a Windows person, either 🙂 Can you share more details about your Windows installation (version, architecture)? Also, you're running this on windows directly, not using WSL or anything like that, right?

  4. solenum commented on May 20, 2025

    @solenum
    Author

    Yeah directly on windows (via a powershell terminal specifically), not using WSL or anything like that here. Windows 11 Pro 23H2, OS build 22631.5335

    Python 3.10.5
    pip 20.0.4
    pipx 1.7.1

  5. Mantisus commented on May 21, 2025

    @Mantisus
    Collaborator
  6. janbuchar commented on May 21, 2025

    @janbuchar
    Collaborator

    Thanks @Mantisus, makes sense. What do we make of this? Do we disable the code that sets WindowsSelectorEventLoopPolicy when the playwright crawler is selected?

  7. Mantisus commented on May 21, 2025

    @Mantisus
    Collaborator

    Yeah, that's the best option for us.

    Judging by the comment, it's set for curl-cffi. Perhaps you remember the problem with curl-cffi when using ProactorEventLoop?

  8. janbuchar commented on May 21, 2025

    @janbuchar
    Collaborator

    Yeah, that's the best option for us.

    Judging by the comment, it's set for curl-cffi. Perhaps you remember the problem with curl-cffi when using ProactorEventLoop?

    Yeah, it printed out some random warning 🤷 Maybe it's not a problem anymore. But I wouldn't give it too much attention. The Playwright+curl-cffi combo should be pretty rare now that we have a playwright http client.

  9. self-assigned this
    on May 21, 2025
  10. added a commit that references this issue on May 21, 2025
    f3b839f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions