Skip to content

feat: add new curl impersonate HTTP client - #387

Merged
vdusek merged 9 commits into
masterfrom
add-curl-cffi-client
Aug 5, 2024
Merged

vdusek merged 9 commits into
masterfrom
add-curl-cffi-client

Conversation

@vdusek

@vdusek vdusek commented Aug 1, 2024 •

Copy link
Copy Markdown
Collaborator

Description

Issues

Testing

  • New unit tests were written.
  • Or check the example below utilizing curl impersonate client with BeautifulSoup crawler.
import asyncio

from crawlee.beautifulsoup_crawler import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from crawlee.http_clients import CurlImpersonateHttpClient
from crawlee.proxy_configuration import ProxyConfiguration


async def main() -> None:
    proxy_configuration = ProxyConfiguration(
        proxy_urls=[
            'http://username:password@proxy.apify.com:8000',
        ],
    )

    crawler = BeautifulSoupCrawler(
        max_requests_per_crawl=10,
        proxy_configuration=proxy_configuration,
        http_client=CurlImpersonateHttpClient(),
    )

    @crawler.router.default_handler
    async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url}...')
        await context.enqueue_links()

        data = {
            'url': context.request.url,
            'title': context.soup.title.text if context.soup.title else '',
        }

        context.log.info(f'Extracted data: {data}')
        await context.push_data(data)

    await crawler.run(['https://apify.com', 'https://crawlee.dev/'])
    crawler.log.info('Finished crawling.')


if __name__ == '__main__':
    asyncio.run(main())

TODO

  • Before merging add better documentation of HTTP clients, an option of switching them, and implementing a new one.

Checklist

  • CI passed

@vdusek
vdusek requested a review from janbuchar August 1, 2024 15:23
@vdusek vdusek self-assigned this Aug 1, 2024
@vdusek
vdusek force-pushed the add-curl-cffi-client branch from 2a8a909 to 179c965 Compare August 1, 2024 15:36
@github-actions github-actions Bot added this to the 95th sprint - Tooling team milestone Aug 1, 2024
@github-actions github-actions Bot added t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics. labels Aug 1, 2024

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

@siddiqkaithodu

Copy link
Copy Markdown
Contributor

Hello.Great effort so far.

Is the impersonate browser argument is missing or avoided for a good reason.

@vdusek

vdusek commented Aug 2, 2024 •

Copy link
Copy Markdown
Collaborator Author

@siddiqkaithodu Hey, good point, I'll expose kwargs for the curl_cffi.requests.AsyncSession.

Comment thread pyproject.toml Resolved

class HttpxClient(BaseHttpClient):
"""A httpx-based HTTP client used for making HTTP calls in crawlers (`BasicCrawler` subclasses)."""
class HttpxHttpClient(BaseHttpClient):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean... does this really contain more information now?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was just following the pattern of using "library name" + "HttpClient" as a suffix for consistency, even though it doesn't add much information in this particular case.

@vdusek
vdusek requested a review from janbuchar August 2, 2024 12:12
Comment thread pyproject.toml Resolved
@janbuchar
janbuchar self-requested a review August 2, 2024 16:31
@vdusek
vdusek merged commit 9c06260 into master Aug 5, 2024
@vdusek
vdusek deleted the add-curl-cffi-client branch August 5, 2024 08:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add new HTTP client curl-cffi

3 participants