Repository navigation
Description
This could just be a misunderstanding on my part, first time using crawlee. I followed the guide on the homepage and used pipx to create a project
pipx run 'crawlee[cli]' create my-crawlerThis produces a project that fails to run after I activate the venv in powershell and attempt to run the project
cd my-crawler
.\.venv\Scripts\activate
python -m my-crawler[asyncio] ERROR Task exception was never retrieved
future: <Task finished name='Task-12' coro=<Connection.run() done, defined at C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_connection.py:272> exception=NotImplementedError()>
Traceback (most recent call last):
File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_connection.py", line 279, in run
await self._transport.connect()
File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_transport.py", line 133, in connect
raise exc
File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_transport.py", line 120, in connect
self._proc = await asyncio.create_subprocess_exec(
File "C:\Python310\lib\asyncio\subprocess.py", line 218, in create_subprocess_exec
transport, protocol = await loop.subprocess_exec(
File "C:\Python310\lib\asyncio\base_events.py", line 1667, in subprocess_exec
transport = await self._make_subprocess_transport(
File "C:\Python310\lib\asyncio\base_events.py", line 498, in _make_subprocess_transport
raise NotImplementedError
NotImplementedErrorAnnoyingly this took me ages to figure out. It seems to be related to this line in the _main_.py that gets generated, as commenting it out resolves the problem:
if platform.system() == 'Windows':
# This mitigates a warning raised by curl-cffi. If you do not need to use curl-impersonate, you may remove this.
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
# __comment out ^ to fix__Is this just an issue with my configuration? I've updated pip and all packages to the latest versions, I guess my problems resolved with the above but I couldn't find a relevant issue anywhere and wasted a lot of my workday trying to figure this out so I figured I'd create this issue to hopefully prevent others from getting stuck on the same problem. (I'm not a Windows person - entirely possible this is something I'm just not understanding)
Activity
solenum commentedon May 20, 2025
Full powershell terminal output
janbuchar commentedon May 20, 2025
Hi @solenum, I'm not a Windows person, either 🙂 Can you share more details about your Windows installation (version, architecture)? Also, you're running this on windows directly, not using WSL or anything like that, right?
solenum commentedon May 20, 2025
Yeah directly on windows (via a powershell terminal specifically), not using WSL or anything like that here. Windows 11 Pro 23H2, OS build 22631.5335
Python 3.10.5
pip 20.0.4
pipx 1.7.1
Mantisus commentedon May 21, 2025
Playwright does not support working with
WindowsSelectorEventLoopPolicy- https://playwright.dev/python/docs/library#incompatible-with-selectoreventloop-of-asyncio-on-windowsThanks for catching that.
janbuchar commentedon May 21, 2025
Thanks @Mantisus, makes sense. What do we make of this? Do we disable the code that sets
WindowsSelectorEventLoopPolicywhen the playwright crawler is selected?Mantisus commentedon May 21, 2025
Yeah, that's the best option for us.
Judging by the comment, it's set for curl-cffi. Perhaps you remember the problem with
curl-cffiwhen usingProactorEventLoop?janbuchar commentedon May 21, 2025
Yeah, it printed out some random warning 🤷 Maybe it's not a problem anymore. But I wouldn't give it too much attention. The Playwright+curl-cffi combo should be pretty rare now that we have a playwright http client.
fix: set `WindowsSelectorEventLoopPolicy` only for curl-impersonate t…