Skip to content

Add session cookies to crawling context #710

Closed

Description

@Mantisus

Add to the context, the cookie of the session from which the request was made, both for HTTP crawlers and Playwright

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Nov 18, 2024
  2. changed the title [-]Add response headers in context[/-] [+]Add response headers to crawling context[/+] on Nov 19, 2024
  3. janbuchar commented on Nov 19, 2024

    @janbuchar
    Collaborator

    Could you elaborate please? For plain HTTP crawlers, you can use context.http_response.headers. Is this about accessing the request in Playwright?

  4. Mantisus commented on Nov 19, 2024

    @Mantisus
    CollaboratorAuthor

    I missed that, for HTTP crawlers we have context.http_response.headers for Playwright this is not as relevant.

  5. janbuchar commented on Nov 19, 2024

    @janbuchar
    Collaborator

    It is true that it would be hard to reach the headers via PlaywrightCrawlingContext though.

  6. Mantisus commented on Nov 19, 2024

    @Mantisus
    CollaboratorAuthor

    It can be useful for Playwright to have access to the cookie of the session from which the request was made.

    But not directly to the headers

  7. janbuchar commented on Nov 19, 2024

    @janbuchar
    Collaborator

    Feel free to rephrase the issue title and description then 🙂

  8. changed the title [-]Add response headers to crawling context[/-] [+]Add session cookies to crawling context[/+] on Nov 19, 2024
  9. oldsiks commented on Dec 25, 2024

    @oldsiks

    When using PlaywrightCrawler in Crawlee for web scraping, how can I add cookies? Could you provide an example?

  10. Mantisus commented on Dec 25, 2024

    @Mantisus
    CollaboratorAuthor

    Hey @oldsiks

    Thank you for your interest in crawlee.

    Here is an example of using a cookie at the request header level

    import asyncio
    
    from crawlee import Request
    from crawlee.playwright_crawler import PlaywrightCrawler, PlaywrightCrawlingContext
    
    
    async def main() -> None:
        crawler = PlaywrightCrawler()
    
        @crawler.router.default_handler
        async def request_handler(context: PlaywrightCrawlingContext) -> None:
            print(await context.page.content())
    
        await crawler.run([
            Request.from_url(url='https://httpbin.org/get', headers={'cookie': 'my_cookies'})
            ])
    
    asyncio.run(main())

    Also after release 0.5 it will be possible to set cookie in PlaywrightBrowserPlugin using browser_new_context_options

    You can try this using the pre-relise version - 0.5.0b30

    Example for 0.5:

    import asyncio
    
    from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
    from crawlee.browsers import BrowserPool, PlaywrightBrowserPlugin
    
    
    async def main() -> None:
        user_plugin = PlaywrightBrowserPlugin(
            browser_new_context_options={"extra_http_headers": {'cookie': 'my_cookies'}}
            )
    
        browser_pool = BrowserPool(plugins=[user_plugin])
    
        crawler = PlaywrightCrawler(browser_pool=browser_pool)
    
        @crawler.router.default_handler
        async def request_handler(context: PlaywrightCrawlingContext) -> None:
            print(await context.page.content())
    
        await crawler.run(['https://httpbin.org/get'])
    
    asyncio.run(main())
  11. barabacker commented on Jan 22, 2025

    @barabacker

    I would like to know how to use cookies when using CurlImpersonateHttpClient

  12. Mantisus commented on Jan 22, 2025

    @Mantisus
    CollaboratorAuthor

    Hey @barabaker

    You can pass cookies in the Request headers

    import asyncio
    
    from crawlee import Request
    from crawlee.crawlers import HttpCrawler, HttpCrawlingContext
    from crawlee.http_clients import CurlImpersonateHttpClient
    
    
    async def main() -> None:
        crawler = HttpCrawler(http_client=CurlImpersonateHttpClient())
    
        @crawler.router.default_handler
        async def request_handler(context: HttpCrawlingContext) -> None:
            print(context.http_response.read())
    
        await crawler.run([Request.from_url(url='https://httpbin.org/get', headers={'cookie': 'my_cookies'})])
    
    
    asyncio.run(main())
  13. added a commit that references this issue on Feb 25, 2025
    6523b3a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions