Skip to content

Improve the deduplication of requests #178

Closed

Description

@vdusek

Context

A while ago, Honza Javorek raised some good points regarding the deduplication process in the request queue (#190).

The first one:

Is it possible that Apify's request queue dedupes the requests only based on the URL? Because the POSTs all have the same URL, just different payload. Which should be very common - by definition of what POST is, or even in practical terms with all the GraphQL APIs around.

In response, we improved the unique key generation logic in the Python SDK (PR #193) to align with the TS Crawlee. This logic was lates copied to crawlee-python and can be found in crawlee/_utils/requests.py.

The second one:

Also wondering whether two identical requests with one different HTTP header should be considered same or different. Even with a simple GET request, I could make one with Accept-Language: cs, another with Accept-Language: en, and I can get two wildly different responses from the same server.

Currently, HTTP headers are not considered in the computation of unique keys. Additionally, we do not offer an option to explicitly bypass request deduplication, unlike the dont_filter option in Scrapy (docs).

Questions

  • Should we include HTTP headers in the unique_key and extended_unique_key computation?
    • Yes.
  • Should we implement a dont_filter feature?
    • It will be just a syntax sugar appending some random string to a unique key.
    • Also come up with a better name (e.g. always_enqueue)?
  • Should use_extended_unique_key be set as the default behavior?
    • Probably not now.

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    enhancementNew feature or request.
    on Jun 10, 2024
  2. added this to the milestone on Jun 10, 2024
  3. Mantisus commented on Jun 24, 2024

    @Mantisus
    Collaborator

    Should we include HTTP headers in the unique_key (extended_unique_key) computation?

    I think yes, you should do that.
    In addition to the Accept-Language mentioned. You can consider the situation when within the crawler work requests are executed from different authorized users. The only difference is the header containing the authorization token.
    There are other special cases when the header has a significant impact on the content of the response.

    Should we implement a dont_filter feature?

    Yes. For example, for cases where the server returns a 200 response status. But the response body contains data that an error occurred and this request should be executed again.
    If I see the current implementation correctly, this will not be possible without this option.

  4. removed this from the milestone on Jul 15, 2024
  5. B4nan commented on Sep 23, 2024

    @B4nan
    Member

    Let's try to find a better name than dont_filter, we want a new option that will put a random value into unique_key.

  6. added
    solutioningThe issue is not being implemented but only analyzed and planned.
    and removed
    enhancementNew feature or request.
    on Sep 27, 2024
  7. self-assigned this
    on Sep 27, 2024
  8. vdusek commented on Sep 27, 2024

    @vdusek
    CollaboratorAuthor

    Thanks for the inputs, based on this and our internal discussions I opened #547 and #548 and closing this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

solutioningThe issue is not being implemented but only analyzed and planned.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions