Skip to content

SAME_HOSTNAME not working on non www URLs #955

Closed

Description

@ROYOSTI

When using the EnqueueStrategy.SAME_HOSTNAME I noticed it does not work properly on non www urls.

In the debugger I noticed it passes origin to the _check_enqueue_strategy but it uses the context.request.loaded_url if available.
So every URL that is checked will mismatch because of the difference in hostname

Image

I tested this with multiple urls with & without www prefix and got the same behaviour.

Image

Changing the line to origin = context.request.url fix this issue, but I have no idea what implications this would have on the other code.

I use the PlaywrightCrawler in my code with context.enqueue_links

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Feb 3, 2025
  2. self-assigned this
    on Feb 3, 2025
  3. vdusek commented on Feb 3, 2025

    @vdusek
    Collaborator

    Hi @ROYOSTI, thanks for reporting this. It does seem like a bug and we'll try to look into it soon.

  4. added a commit that references this issue on Feb 4, 2025
    694102e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions