Skip to content

Improve crawler stats logging #1317

Closed

Description

@vdusek

Maybe it is just a habit but I find the log quite bloated (compated to JS).

crawler stats

  • spread over 10 lines which is quite lot since they are periodic
  • Retry histogram (Which is really useful) is cut after few values unlike in JS
  • Some of the keys of stats are also cut out
  • There are no values for time units, I can guess some are in seconds, some per minute

Originally posted by @metalwarrior665 on Slack - https://apify.slack.com/archives/CD0SF6KD4/p1753007915099809.

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Jul 22, 2025
  2. added this to the 1.0 milestone on Jul 28, 2025
  3. self-assigned this
    on Aug 19, 2025
  4. vdusek commented on Aug 19, 2025

    @vdusek
    CollaboratorAuthor

    Spread over 10 lines which is quite lot since they are periodic

    • This is an optional feature, you can specify the log format using statistics_log_format.
    crawler = ParselCrawler(statistics_log_format='inline')
    ({"requests_finished": 0, "requests_failed": 1, "retry_histogram": [0, 0, 0, 0, 0, 0, 0, 0, 0, 1], "request_avg_failed_duration": 0.762921, "request_avg_finished_duration": null, "requests_finished_per_minute": 0, "requests_failed_per_minute": 8, "request_total_duration": 0.762921, "requests_total": 1, "crawler_runtime": 6.973047})
    
    crawler = ParselCrawler(statistics_log_format='table')
    ┌────────────────────────────┬────────────────────────────┐
    │ requests_finished          │ 0                          │
    │ requests_failed            │ 1                          │
    │ retry_histogram            │ [0, 0, 0, 0, 0, 0, 0, 0... │
    │ request_avg_failed_dura... │ 0.514166                   │
    │ request_avg_finished_du... │ None                       │
    │ requests_finished_per_m... │ 0                          │
    │ requests_failed_per_minute │ 9                          │
    │ request_total_duration     │ 0.514166                   │
    │ requests_total             │ 1                          │
    │ crawler_runtime            │ 6.190427                   │
    └────────────────────────────┴────────────────────────────┘
    
  5. added a commit that references this issue on Aug 19, 2025
    1eb6da5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions