You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
URL Validation edge case - Protocol/Scheme relative URLs #417
It is possible that one encounters URLs that are protocol relative in HTML, like an a tag with href="//example.com". This is technically a valid URL, and essentially means it takes on whatever protocol that is used by the page that is in.
The is_url_absolute function returns True for this case and while that may be correct, Pydantic doesn't know what to do with it and even the AnyUrl type requires a scheme (any scheme) to be present. So it throws a validation error.
A potential fix is to use the urllib.parse.urljoin function regardless of whether the raw url is absolute or not. According to the urllib docs, "if url is an absolute URL (that is, it starts with // or scheme://), the url’s hostname and/or scheme will be present in the result". So it was a full absolute url with its own different protocol and hostname, it stays that way. And if it is one of this protocol-relative kind of absolute url 😂 , it can take on the protocol of the base url it is being joined to.
I actually came across this because of a different but related issue - not sure if I should be creating a new one though. But essentially this URL validation fail shouldn't really cause a crawl to "crash", you'd expect just the one URL to not get queued in. But that is what I was experiencing. And it also stops the whole crawl job if it encounters a tel: link as well (also a Pydantic validation error). BUT curiously, if I forego using the context.enqueue_links() function and do the page parsing and request list building myself and then add it with context.add_requests(), it works! I mean the validation error still happens, but the crawl continues on with the other links. I basically copied the code from the enqueue_links implementation so I'm not sure why it would work here in the request handler function but not there, I'm guessing there's some difference in the error handling or something - but take this last theorizing piece as purely anecdotal, I haven't really investigated.
Anyway, sorry for the rambling - thanks for your work on this awesome project!
Hey @aka-Ani I am experience the same issues, i have tried resolving but it seems they only did thier implementations perfect of JS. please can you sure the code where you used the add_requests()
Hi, and thanks for your interest in Crawlee! We appreciate the effort you put into investigating this. Could you provide us with a bunch of URL pairs (page URL + link), the expected behavior of add_requests and the actual results?
Using that, it will be much easier for us to troubleshoot.
Hello Jan,
Thank you for reaching out.
When I try to crawl this website to crawl all the links on the website
https://obeezi.com/
If I use the enqueue_links it fails throws an error because it detected an
absolute path.
I have to manually use the add_request feature to Handle this with a custom
function to detect all the links and fix the issue with absolute paths.
Hey I did a bit of further digging and think i found a representative example of the underlying issue.
The Pydantic validation error is one thing, but but the crawl crashing i think is the "actual" issue. Here is an example (using context.enqueue_links()). If I crawl https://github.com/ (I used the Playwright crawler here) the crawl kicks off and you can see it has some Pydantic errors when it encounters mailto links like on the /copilot page or the /enterprise/startup page. But this validation error doesn't break anything. You get an error on that one page, and the crawl processes continue.
Bear in mind that startups page does have a ton of other regular links, so you would expect only that one link to not get properly added to the queue but the rest do. But what it seems like is happening, is that if the very first page has any invalid link, the crawl job dies, but if by some chance the first page is ok, then invalid links in subsequent pages seem to throw errors without killing the entire crawl.
It could be because of the way the asyncio part works? like if all tasks have died the crawl dies, but if the first page "works", then there's enough time to spin up 2 tasks and then a new task is spun up every time one dies or something. Pure speculation though haha 😅
[crawlee.playwright_crawler.playwright_crawler] INFO Processing https://www.wikipedia.org/ ...
1 validation error for BaseRequestData
url
Input should be a valid URL, relative URL without a base [type=url_parsing, input_value='//en.wikipedia.org/', input_type=str]
For further information visit https://errors.pydantic.dev/2.7/v/url_parsing
[crawlee.autoscaling.autoscaled_pool] INFO Waiting for remaining tasks to finish
It's not really my place to say where "best" to solve this issue. After all, a mailto or tel link is indeed not an AnyHttpUrl and therefore should raise a ValidationError when attempting to validate a new BaseRequestData model. But I think it probably makes sense to not error out of the whole task when parsing html and encountering an invalid URL.
The simplest change I can think of is just catching the ValidationError exception in the enqueue_links() implementation and skipping over that URL onto the next one (without exiting the process).
try:
request = BaseRequestData.from_url(url, user_data=link_user_data)
except ValidationError as e:
context.log.error(f"Failed to create request for URL {url}\n{e}")
continue
And that's enough to not have the crawl process crash even if it encounters an invalid URL on the first page of the first spun up task, like in this example where I start at one of the problem pages from my last example (https://github.com/features/copilot) - only this time it just logs the error and continues on:
That's great! Yes I agree it is simple enough, but I think some consideration could still be made as to how "best" to place the fix in the library.
There is a _utils/urls.py file where some helper functions exist that leverage the urllib library. This is also where the validate_http_url function lives, so I think it makes sense to put checks and transforms somewhere in here. This function uses the Pydantic type AnyHttpUrl to check whether the scheme is http or https, so doing it with string methods in each type of request handler feels unnecessary.
The urllib.parse.urljoin function handles both cases where where the url is relative /wiki/something as well as where url is technically absolute but protocol relative //en.wikipedia.org/
So I think it makes sense to call this function without trying to run a condition check as the rules for what makes a URL absolute is kind of tough to pin down, especially with regards to deciding which urls we want to enqueue and which we don't. This in addition to handling the ValidationError when performing the pydantic validation would be a neater way in my opinion.
Thank you guys for all the effort you invested into this. I reviewed & tested the PR #431 and it seems to be solving this issue. I provided some suggestions and hopefully, we can merge & release it soon.
Activity
mattyjr007 commentedon Aug 10, 2024
Hey @aka-Ani I am experience the same issues, i have tried resolving but it seems they only did thier implementations perfect of JS. please can you sure the code where you used the add_requests()
janbuchar commentedon Aug 12, 2024
Hi, and thanks for your interest in Crawlee! We appreciate the effort you put into investigating this. Could you provide us with a bunch of URL pairs (page URL + link), the expected behavior of add_requests and the actual results?
Using that, it will be much easier for us to troubleshoot.
mattyjr007 commentedon Aug 12, 2024
aka-Ani commentedon Aug 12, 2024
Hey I did a bit of further digging and think i found a representative example of the underlying issue.
The Pydantic validation error is one thing, but but the crawl crashing i think is the "actual" issue. Here is an example (using

context.enqueue_links()). If I crawl https://github.com/ (I used the Playwright crawler here) the crawl kicks off and you can see it has some Pydantic errors when it encountersmailtolinks like on the/copilotpage or the/enterprise/startuppage. But this validation error doesn't break anything. You get an error on that one page, and the crawl processes continue.Now we can contrast that with a new crawl job, but here the crawl starts at https://github.com/enterprise/startups. Now the very first page encountered has a problematic mailto link which triggers the Pydantic validation error, but crucially, this time the entire crawl job dies.

Bear in mind that startups page does have a ton of other regular links, so you would expect only that one link to not get properly added to the queue but the rest do. But what it seems like is happening, is that if the very first page has any invalid link, the crawl job dies, but if by some chance the first page is ok, then invalid links in subsequent pages seem to throw errors without killing the entire crawl.
aka-Ani commentedon Aug 12, 2024
It could be because of the way the asyncio part works? like if all tasks have died the crawl dies, but if the first page "works", then there's enough time to spin up 2 tasks and then a new task is spun up every time one dies or something. Pure speculation though haha 😅
black7375 commentedon Aug 13, 2024
https://www.wikipedia.org/ is another sample.
Log:
aka-Ani commentedon Aug 13, 2024
It's not really my place to say where "best" to solve this issue. After all, a
mailtoortellink is indeed not anAnyHttpUrland therefore should raise a ValidationError when attempting to validate a newBaseRequestDatamodel. But I think it probably makes sense to not error out of the whole task when parsing html and encountering an invalid URL.The simplest change I can think of is just catching the ValidationError exception in the
enqueue_links()implementation and skipping over that URL onto the next one (without exiting the process).So basically changing this line:
crawlee-python/src/crawlee/playwright_crawler/playwright_crawler.py
Line 144 in 9b35cfa
to something like:
And that's enough to not have the crawl process crash even if it encounters an invalid URL on the first page of the first spun up task, like in this example where I start at one of the problem pages from my last example (https://github.com/features/copilot) - only this time it just logs the error and continues on:

fix: Relative URLS supports & Allow only http apify#417
black7375 commentedon Aug 14, 2024
It seemed simple enough, so I created a PR.
It works well for me.
aka-Ani commentedon Aug 15, 2024
That's great! Yes I agree it is simple enough, but I think some consideration could still be made as to how "best" to place the fix in the library.
There is a
_utils/urls.pyfile where some helper functions exist that leverage the urllib library. This is also where thevalidate_http_urlfunction lives, so I think it makes sense to put checks and transforms somewhere in here. This function uses the Pydantic typeAnyHttpUrlto check whether the scheme is http or https, so doing it with string methods in each type of request handler feels unnecessary.The urllib.parse.urljoin function handles both cases where where the url is relative
/wiki/somethingas well as where url is technically absolute but protocol relative//en.wikipedia.org/So I think it makes sense to call this function without trying to run a condition check as the rules for what makes a URL absolute is kind of tough to pin down, especially with regards to deciding which urls we want to enqueue and which we don't. This in addition to handling the ValidationError when performing the pydantic validation would be a neater way in my opinion.
black7375 commentedon Aug 15, 2024
Then I'll change the commit to always call
urljoin. Thanks for the suggestion.It has been reflected.I got a unit test error, so I reverted to the previous version first.
fix: Relative URLS supports & Allow only http apify#417
vdusek commentedon Aug 19, 2024
Thank you guys for all the effort you invested into this. I reviewed & tested the PR #431 and it seems to be solving this issue. I provided some suggestions and hopefully, we can merge & release it soon.
black7375 commentedon Aug 20, 2024
Thank you for your review. I don't have time today, so I'll reflect on it tomorrow.
fix: Relative URLS supports & Allow only http apify#417
fix: Relative URLS supports & Allow only http apify#417
fix: Relative URLS supports & If not a URL, pass apify#417
fix: Relative URLS supports & If not a URL, pass apify#417
fix: Relative URLS supports & If not a URL, pass #417 (#431)