Skip to content

Include whitelisted HTTP headers in the extended_unique_key computation #548

Closed

Description

@vdusek
  • Modify the extended_unique_key computation to include a set of predefined HTTP headers, alongside the existing normalized URL and payload.
  • Only include headers from the whitelist.
  • Identify which headers should be included, such as Accept, Accept-Language, Authorization, and others that may affect request outcomes.
  • See the issue Improve the deduplication of requests #178 for further context.

Activity

  1. added
    enhancementNew feature or request.
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Sep 27, 2024
  2. ravi-hash commented on Oct 2, 2024

    @ravi-hash

    Hey will you assign this issue to me

  3. vdusek commented on Oct 3, 2024

    @vdusek
    CollaboratorAuthor

    Hi @ravi-hash, sure, thanks for your interest in Crawlee.

  4. added a commit that references this issue on Oct 23, 2024
    6c4746f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request.hacktoberfestt-toolingIssues with this label are in the ownership of the tooling team.

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions