You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add always_enqueue option to Request for bypassing deduplication #547
I'm interested in working on this enhancement to add an always_enqueue option for bypassing request deduplication. To ensure I understand the requirements correctly, I have a few questions:
Regarding the option name:
Is always_enqueue the preferred name, or would you like suggestions for alternatives?
Should it be implemented as a boolean parameter?
For the implementation of the random unique_key wrapper:
Do you have any preferences for the random key generation method?
Should there be a specific format or prefix for these generated keys?
About the edge cases:
When both unique_key and always_enqueue=True are provided, which should take precedence?
Should we add a warning or raise an exception in this case?
Regarding similarity to Scrapy's dont_filter:
Are there any specific behaviors from Scrapy's implementation that we should replicate or avoid?
Should we maintain exact parity with Scrapy's behavior, or are there Crawlee-specific considerations?
I'm familiar with Python and HTTP clients, and I'd be happy to work on this enhancement. Let me know if you need any clarification or have additional requirements before I start implementing the solution.
Hi @belloibrahv, thanks for your interest in Crawlee.
Is always_enqueue the preferred name, or would you like suggestions for alternatives?
always_enqueue is good, but if you come up with something better, we can consider it.
Should it be implemented as a boolean parameter?
yes
Do you have any preferences for the random key generation method?
Should there be a specific format or prefix for these generated keys?
You can probably generate a standard unique key and then append some random string using crypto_random_object_id.
When both unique_key and always_enqueue=True are provided, which should take precedence?
Should we add a warning or raise an exception in this case?
In that case, raise an exception.
Are there any specific behaviors from Scrapy's implementation that we should replicate or avoid?
Should we maintain exact parity with Scrapy's behavior, or are there Crawlee-specific considerations?
Activity
belloibrahv commentedon Oct 2, 2024
Hi @vdusek,
I'm interested in working on this enhancement to add an
always_enqueueoption for bypassing request deduplication. To ensure I understand the requirements correctly, I have a few questions:Regarding the option name:
always_enqueuethe preferred name, or would you like suggestions for alternatives?For the implementation of the random
unique_keywrapper:About the edge cases:
unique_keyandalways_enqueue=Trueare provided, which should take precedence?Regarding similarity to Scrapy's
dont_filter:I'm familiar with Python and HTTP clients, and I'd be happy to work on this enhancement. Let me know if you need any clarification or have additional requirements before I start implementing the solution.
Thank you for considering my contribution!
vdusek commentedon Oct 2, 2024
Hi @belloibrahv, thanks for your interest in Crawlee.
always_enqueueis good, but if you come up with something better, we can consider it.yes
You can probably generate a standard unique key and then append some random string using crypto_random_object_id.
In that case, raise an exception.
I am not aware of any.
paradoxxx09 commentedon Oct 12, 2024
/assign
vdusek commentedon Oct 14, 2024
@paradoxxx09 We don't assign issues for hacktoberfest. If you want to work on this, open a PR. First mergeable one gets merged.
feat: Add `always_enqueue` option to bypass URL deduplication (apify#621