Skip to content

chore: Refactor HttpClient implementations tests - #1299

Merged
Pijukatel merged 6 commits into
masterfrom
refactor-http-clients-tests
Jul 15, 2025
Merged

Pijukatel merged 6 commits into
masterfrom
refactor-http-clients-tests

Conversation

@Pijukatel

Copy link
Copy Markdown
Collaborator

Description

  • Parametrize HttpClient tests.
  • Add tests missing for one of the implementations.
  • Delete non-unique, non-parametrized HttpClient tests.

Issues

@Pijukatel Pijukatel added t-tooling Issues with this label are in the ownership of the tooling team. debt Code quality improvement or decrease of technical debt. labels Jul 11, 2025
@github-actions github-actions Bot added this to the 118th sprint - Tooling team milestone Jul 11, 2025
@github-actions github-actions Bot added the tested Temporary label used only programatically for some analytics. label Jul 11, 2025

@Mantisus Mantisus left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job!

@Pijukatel
Pijukatel requested a review from Copilot July 15, 2025 09:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Refactor and consolidate HTTP client implementation tests to remove duplication and improve coverage.

  • Remove separate test files for each HttpClient implementation and delete redundant, non-parameterized tests.
  • Introduce a unified, parameterized suite (test_http_clients.py) covering CurlImpersonateHttpClient, HttpxHttpClient, and ImpitHttpClient.
  • Simplify test_httpx.py to focus on header behavior and remove excess imports and tests.

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
tests/unit/http_clients/test_impit.py Removed standalone tests for ImpitHttpClient.
tests/unit/http_clients/test_httpx.py Simplified Httpx-specific tests, keeping only header checks.
tests/unit/http_clients/test_http_clients.py Added consolidated, parameterized tests for all clients.
tests/unit/http_clients/test_curl_impersonate.py Removed standalone tests for CurlImpersonateHttpClient.
Comments suppressed due to low confidence (2)

tests/unit/http_clients/test_httpx.py:9

  • The imported COMMON_ACCEPT_LANGUAGE constant is not used in this test. Either add assertions to verify the accept-language header logic or remove the unused import to keep the test focused.
from crawlee.fingerprint_suite._consts import COMMON_ACCEPT_LANGUAGE

tests/unit/http_clients/test_httpx.py:39

  • The variable response_headers is used here but never defined. You need to parse the response body (e.g., response_headers = json.loads(await response.read())) before asserting on its contents.
    assert 'user-agent' in response_headers

@Pijukatel
Pijukatel marked this pull request as ready for review July 15, 2025 09:56
@Pijukatel
Pijukatel requested a review from vdusek July 15, 2025 09:56


@pytest.fixture
async def http_client(request: SubRequest) -> AsyncGenerator[HttpClient]:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe you already tried and found a reason why it's not viable, but could this be made into a parametrized fixture? Then each test (or at least each test that uses _DEFAULT_CLIENTS) could just depend on http_client and it would get repeated for each possible value automatically.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, the only minor drawback is that I have to use a different variable name in tests that do not use the default Http clients. But I think it is still worth it.

@Pijukatel
Pijukatel requested a review from janbuchar July 15, 2025 11:40

@janbuchar janbuchar left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One nit, LGTM otherwise

Comment thread tests/unit/http_clients/test_httpx.py Resolved
Comment thread tests/unit/http_clients/test_httpx.py Resolved
Co-authored-by: Jan Buchar <jan.buchar@apify.com>
@Pijukatel
Pijukatel merged commit 47380ef into master Jul 15, 2025
@Pijukatel
Pijukatel deleted the refactor-http-clients-tests branch July 15, 2025 13:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

debt Code quality improvement or decrease of technical debt. t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parametrize HttpClient implementations tests to remove code duplication

4 participants