Skip to content

fix: stop truncating fractional robots.txt crawl-delay to an integer - #2392

Merged
vdusek merged 2 commits into
apify:masterfrom
RaphaelFakhri:fix/robots-txt-fractional-crawl-delay
Nov 5, 2026
Merged

vdusek merged 2 commits into
apify:masterfrom
RaphaelFakhri:fix/robots-txt-fractional-crawl-delay

Conversation

@RaphaelFakhri

Copy link
Copy Markdown
Contributor

Description

  • RobotsTxtFile.get_crawl_delay() returns the parsed Crawl-delay value as a float instead of truncating it to an int. Before this change, Crawl-delay: 0.5 returned 0, so ThrottlingRequestManager applied no delay to the domain, and Crawl-delay: 2.5 returned 2.
  • ThrottlingRequestManager.set_crawl_delay() accepts a float for delay_seconds. Existing int callers keep working.

Issues

Testing

  • Added test_get_fractional_crawl_delay in tests/unit/_utils/test_robots.py. It fails without the fix (assert 0 == 0.5) and passes with it.
  • tests/unit/_utils/test_robots.py and tests/unit/test_throttling_request_manager.py pass locally (127 passed). ruff check, ruff format --check and ty check pass.

Checklist

  • CI passed

`RobotsTxtFile.get_crawl_delay()` cast the parsed `Crawl-delay` value to
`int`, so `Crawl-delay: 0.5` became 0 and `ThrottlingRequestManager`
applied no delay to the domain, and `Crawl-delay: 2.5` became 2.
Return the value as a `float` and accept a `float` in
`ThrottlingRequestManager.set_crawl_delay()`.
@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 3, 2026
@janbuchar
janbuchar requested review from Mantisus and vdusek and removed request for vdusek November 3, 2026 17:08

@Mantisus Mantisus left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for PR! Only one nit. Otherwise LGTM

Comment thread src/crawlee/_utils/robots.py Resolved
Comment thread src/crawlee/_utils/robots.py Resolved

@apify-approval-mirror apify-approval-mirror Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mirroring approval from @Mantisus.

@apify-approval-mirror apify-approval-mirror Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mirroring approval from @Mantisus.

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vdusek vdusek changed the title fix: keep fractional robots.txt crawl-delay values fix: stop truncating fractional robots.txt crawl-delay to an integer Nov 5, 2026
@vdusek
vdusek merged commit fa386bb into apify:master Nov 5, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fractional robots.txt Crawl-delay values are truncated to whole seconds

3 participants