You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
PlaywrightCrawler extract_links does not respect strategy #1212
I was crawling a test site that I have hosted locally (localhost).
My PlaywrightCrawler subclass must add additional user_data to each request, so I extract and form each request manually.
new_requests = await context.extract_links(strategy='same-origin')
for new_request in new_requests:
print(f"[LINK] Extracted link: {new_request.url}")
I start my crawl pointed at http://localhost, yet the crawler ends up crawling YouTube since there is a link to YouTube on my site.
Apparently, there was an error in splitting the original enqueue_links function into extract_links and add_requests. Since the strategy is applied in add_requests. And is completely ignored in extract_links now
Activity
Mantisus commentedon May 23, 2025
Hey @phughesion-h3
Thanks for catching that.
Apparently, there was an error in splitting the original
enqueue_linksfunction intoextract_linksandadd_requests. Since the strategy is applied inadd_requests. And is completely ignored inextract_linksnowfix: respect `EnqueueLinksKwargs` for `extract_links` function (#1213)