Skip to content

PlaywrightCrawler extract_links does not respect strategy #1212

Closed

Description

@phughesion-h3

I was crawling a test site that I have hosted locally (localhost).

My PlaywrightCrawler subclass must add additional user_data to each request, so I extract and form each request manually.

new_requests = await context.extract_links(strategy='same-origin')
for new_request in new_requests:
        print(f"[LINK] Extracted link: {new_request.url}")

I start my crawl pointed at http://localhost, yet the crawler ends up crawling YouTube since there is a link to YouTube on my site.

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on May 22, 2025
  2. Mantisus commented on May 23, 2025

    @Mantisus
    Collaborator

    Hey @phughesion-h3

    Thanks for catching that.

    Apparently, there was an error in splitting the original enqueue_links function into extract_links and add_requests. Since the strategy is applied in add_requests. And is completely ignored in extract_links now

    new_requests = await context.extract_links()
    await context.add_requests(new_requests, strategy='same-origin')
  3. self-assigned this
    on May 23, 2025
  4. added a commit that references this issue on May 27, 2025
    c9907d6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions