Skip to content

Set HTTPX logging to warning level by default #302

Closed

Description

@vdusek

By default, HTTPX's logging level is set to INFO, producing logs such as:

$ python run_beautifulsoup_crawler_el.py 
[crawlee.statistics.statistics] INFO  crawlee.beautifulsoup_crawler.beautifulsoup_crawler request statistics {
        "requests_finished": 0,
        "requests_failed": 0,
        "retry_histogram": [
          0
        ],
        "request_avg_failed_duration": null,
        "request_avg_finished_duration": null,
        "requests_finished_per_minute": 0,
        "requests_failed_per_minute": 0,
        "request_total_duration": 0.0,
        "requests_total": 0,
        "crawler_runtime": 0.008349
      }
[crawlee.autoscaling.autoscaled_pool] INFO  current_concurrency = 0; desired_concurrency = 2; cpu = 0; mem = 0; event_loop = 0.0; client_info = 0.0
[httpx] INFO  HTTP Request: GET https://crawlee.dev "HTTP/1.1 200 OK"
[httpx] INFO  HTTP Request: GET https://crawlee.dev/blog/launching-crawlee-python "HTTP/1.1 200 OK"
[httpx] INFO  HTTP Request: GET https://crawlee.dev/docs/examples "HTTP/1.1 200 OK"
...

We should consider setting it to the WARNING in the basic_crawler.py#L193:L203.

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    enhancementNew feature or request.
    on Jul 15, 2024
  2. self-assigned this
    on Jul 16, 2024
  3. added a commit that references this issue on Jul 16, 2024
    1585def
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions