You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add Support for Specifying Headers in context.enqueue_links Method #894
When using the context.enqueue_links method, there is currently no way to specify custom headers (e.g., cookies or other required headers) for the links being enqueued. Some websites require specific headers or cookies to properly retrieve data from detail pages. It would be helpful to add support for passing headers or request options to context.enqueue_links to handle such cases.
Activity
janbuchar commentedon Jan 10, 2025
I believe that adding a
transform_request_functionparameter toenqueue_linkswould help, am I correct? The JS version of Crawlee has it - https://crawlee.dev/api/core/interface/EnqueueLinksOptions#transformRequestFunctionzhouya2010 commentedon Jan 10, 2025
Yes, that's right
feat: add `transform_request_function` for `enqueue_links` (#923)