Add to the context, the cookie of the session from which the request was made, both for HTTP crawlers and Playwright
Activity
[-]Add response headers in context[/-][+]Add response headers to crawling context[/+]janbuchar commentedon Nov 19, 2024
Could you elaborate please? For plain HTTP crawlers, you can use
context.http_response.headers. Is this about accessing the request in Playwright?Mantisus commentedon Nov 19, 2024
I missed that, for HTTP crawlers we have
context.http_response.headersfor Playwright this is not as relevant.janbuchar commentedon Nov 19, 2024
It is true that it would be hard to reach the headers via
PlaywrightCrawlingContextthough.Mantisus commentedon Nov 19, 2024
It can be useful for Playwright to have access to the cookie of the session from which the request was made.
But not directly to the headers
janbuchar commentedon Nov 19, 2024
Feel free to rephrase the issue title and description then 🙂
[-]Add response headers to crawling context[/-][+]Add session cookies to crawling context[/+]oldsiks commentedon Dec 25, 2024
When using PlaywrightCrawler in Crawlee for web scraping, how can I add cookies? Could you provide an example?
Mantisus commentedon Dec 25, 2024
Hey @oldsiks
Thank you for your interest in crawlee.
Here is an example of using a cookie at the request header level
Also after release 0.5 it will be possible to set cookie in
PlaywrightBrowserPluginusingbrowser_new_context_optionsYou can try this using the pre-relise version - 0.5.0b30
Example for 0.5:
barabacker commentedon Jan 22, 2025
I would like to know how to use cookies when using CurlImpersonateHttpClient
Mantisus commentedon Jan 22, 2025
Hey @barabaker
You can pass cookies in the Request headers
refactor!: change `Session` cookies from `dict` to `SessionCookies` w…