Repository navigation
chore: Refactor HttpClient implementations tests - #1299
Conversation
…hose that were missing
There was a problem hiding this comment.
Pull Request Overview
Refactor and consolidate HTTP client implementation tests to remove duplication and improve coverage.
- Remove separate test files for each
HttpClientimplementation and delete redundant, non-parameterized tests. - Introduce a unified, parameterized suite (
test_http_clients.py) coveringCurlImpersonateHttpClient,HttpxHttpClient, andImpitHttpClient. - Simplify
test_httpx.pyto focus on header behavior and remove excess imports and tests.
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| tests/unit/http_clients/test_impit.py | Removed standalone tests for ImpitHttpClient. |
| tests/unit/http_clients/test_httpx.py | Simplified Httpx-specific tests, keeping only header checks. |
| tests/unit/http_clients/test_http_clients.py | Added consolidated, parameterized tests for all clients. |
| tests/unit/http_clients/test_curl_impersonate.py | Removed standalone tests for CurlImpersonateHttpClient. |
Comments suppressed due to low confidence (2)
tests/unit/http_clients/test_httpx.py:9
- The imported
COMMON_ACCEPT_LANGUAGEconstant is not used in this test. Either add assertions to verify theaccept-languageheader logic or remove the unused import to keep the test focused.
from crawlee.fingerprint_suite._consts import COMMON_ACCEPT_LANGUAGE
tests/unit/http_clients/test_httpx.py:39
- The variable
response_headersis used here but never defined. You need to parse the response body (e.g.,response_headers = json.loads(await response.read())) before asserting on its contents.
assert 'user-agent' in response_headers
|
|
||
|
|
||
| @pytest.fixture | ||
| async def http_client(request: SubRequest) -> AsyncGenerator[HttpClient]: |
There was a problem hiding this comment.
Maybe you already tried and found a reason why it's not viable, but could this be made into a parametrized fixture? Then each test (or at least each test that uses _DEFAULT_CLIENTS) could just depend on http_client and it would get repeated for each possible value automatically.
There was a problem hiding this comment.
Sure, the only minor drawback is that I have to use a different variable name in tests that do not use the default Http clients. But I think it is still worth it.
janbuchar
left a comment
There was a problem hiding this comment.
One nit, LGTM otherwise
Co-authored-by: Jan Buchar <jan.buchar@apify.com>
Description
HttpClienttests.HttpClienttests.Issues
HttpClientimplementations tests to remove code duplication #1292