Skip to content

feat: add an internal HttpClient to be used in send_request for PlaywrightCrawler using APIRequestContext bound to the browser context - #1134

Merged
vdusek merged 9 commits into
apify:masterfrom
Mantisus:playwright-send-request
Apr 25, 2025
Merged

vdusek merged 9 commits into
apify:masterfrom
Mantisus:playwright-send-request

Conversation

@Mantisus

@Mantisus Mantisus commented Apr 3, 2025 •

Copy link
Copy Markdown
Collaborator

Description

  • This PR, adds an internal HttpClient implementation for PlaywrightCrawler using the APIRequestContext from Playwright. This ensures that HTTP requests use the same proxies as the browser context.

Issues

@Mantisus

Mantisus commented Apr 3, 2025 •

Copy link
Copy Markdown
Collaborator Author

After discussions with @janbuchar in Slack, we came to the decision that this approach is best for PlaywrightCrawler. The requests will be closer to the browser context, and will use the same proxies that were set when the context was opened.

However, there is a problem. Playwright does not propagate the headers set with set_extra_http_headers to the request context. This results in the request context not receiving fingerprint headers.

It relevant #1055

Refactoring would be required to pass the headers set when opening the browser context to the crawling context. Or wait for Playwright to do something about it on their end

@Mantisus
Mantisus requested review from Copilot and janbuchar April 3, 2025 12:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

tests/unit/crawlers/_playwright/test_playwright_crawler.py:486

  • Ensure that send_request_response.read() is decoded (e.g. using .decode('utf-8')) before passing it to json.loads to avoid potential type errors when parsing bytes.
check_data['send_request'] = dict(json.loads(send_request_response.read()))

@Mantisus Mantisus self-assigned this Apr 4, 2025
@Mantisus Mantisus changed the title feat: add helper function send_request for PlaywrightCrawler using APIRequestContext bound to the browser context feat: add an internal HttpClient to be used in send_request for PlaywrightCrawler using APIRequestContext bound to the browser context Apr 21, 2025

@janbuchar janbuchar left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, just a couple of nits 🙂

@Mantisus
Mantisus requested a review from vdusek April 24, 2025 11:40

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement a playwright-based HttpClient implementation

4 participants