Skip to content

Integrate Camoufox into PlaywrightCrawler #684

Closed

Description

@Ehsan-U

crawlee-python is great but often lacks stealth against modern anti-bot measures when using vanilla playwright.
Camoufox is fully compatible with playwright, only have to change browser initialization. It bypasses all currently available anti-bot measures. It would be a great opportunity to make crawlee-python playwright stealthy.

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Nov 12, 2024
  2. janbuchar commented on Nov 12, 2024

    @janbuchar
    Collaborator

    Hello @Ehsan-U, thank you for the suggestion, we will consider it!

  3. changed the title [-]Camoufox Integration[/-] [+]Integrate Camoufox into PlaywrightCrawler[/+] on Nov 12, 2024
  4. Pijukatel commented on Dec 4, 2024

    @Pijukatel
    Collaborator

    Hi, I looked into how much effort it will be and seems like it will work almost out of the box.

    If you want to try it asap, then you can check this branch:
    https://github.com/apify/crawlee-python/tree/camoufox
    If you run this playwright_crawler_example from that branch, then it will use camoufox.

    It will take me more time to polish it merge to master. Word of warning: When you run camoufox it will download latest release which is almost 700MB large!

  5. Pijukatel commented on Dec 6, 2024

    @Pijukatel
    Collaborator

    I tried two approaches:

    Tight integration into crawlee:
    #782
    Pros:

    • Super easy to use.

    Cons:

    • Does not give much flexibility in customizing the Camoufox behavior. (Like turning off crawlee header generation and use Camoufox header generation)
    • Additional dependency for crawlee (currently does not install on Python3.13)

    No integration, but creating example code that shows how to do it with current codebase:
    #789
    Pros:

    • No extra dependency for crawlee
    • No need to modify existing code
    • Big flexibility

    Cons

    • Not as easy to use as the first approach. But still straightforward with the example.
  6. MarhyCZ commented on Jan 8, 2025

    @MarhyCZ
  7. murilomulato commented on Aug 22, 2025

    @murilomulato

    Hey @Ehsan-U I wanted to look into scrapy-camoufox. Have you taken the repo offline?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions