You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ValueError("URL should be absolute") when crawling https://crawlee.dev/js/api/core/changelog and respecting robots.txt #1499
�[90m[crawlee.crawlers._basic._basic_crawler]�[0m �[33mWARN �[0m Retrying request to https://crawlee.dev/js/api/core/changelog due to: URL should be absoluteFile "python3.12/site-packages/yarl/_url.py", line 628, in _origin, raise ValueError("URL should be absolute")
This only happens when I set respect_robots_txt_file=True, I tried putting it to false and it doesn't fail. This is my crawler config in case it helps:
I am not planning to crawl your page ;) , I was using it just as an example but it looks like there is some error when checking robots.txt with a relative path maybe?
Activity
Mantisus commentedon Oct 17, 2025
Hey. Thanks for catching that!
The problem is related to URIs that are not HTTP links and are filtered out at a later stage.
fix: exclude incorrect links before checking `robots.txt` (apify#1502)