You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
max_session_rotations not being respected in AdaptivePlaywrightCrawler.with_beautifulsoup_static_parser #1484
The code above performs exactly 10 session rotations. However, note that run will never terminate because you are using keep_alive=True.
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] ERROR Request to http://httpbingo.org/status/429 failed and reached maximum retries
It is not related to keep_alive=True, it happens with keep_alive=False in the same way. These are my logs (I killed it with ctrl+c). I am using Crawlee 1.0.2 in case it helps
[dip.crawler.main] INFO Starting crawling on 'http://httpbingo.org/status/429'
[crawlee.crawlers._basic._basic_crawler] INFO Crawled 0/1 pages, 0 failed requests, desired concurrency 1.
[crawlee._autoscaling.autoscaled_pool] INFO current_concurrency = 0; desired_concurrency = 1; cpu = 0.0; mem = 0.0; event_loop = 0.0; client_info = 0.0
[crawlee.crawlers._basic._basic_crawler] INFO Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO Before processing http://httpbingo.org/status/429
^C[crawlee.crawlers._basic._basic_crawler] INFO Pausing... Press CTRL+C again to force exit.
[crawlee._autoscaling.autoscaled_pool] INFO Waiting for remaining tasks to finish
??????????????????? 0 5
[crawlee.crawlers._basic._basic_crawler] WARN Retrying request to http://httpbingo.org/status/429 due to: Page.goto: net::ERR_ABORTED at http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 1 retries
[dip.crawler.main] ERROR Exception: Page.goto: net::ERR_ABORTED at http://httpbingo.org/status/429
Call log:
- navigating to "http://httpbingo.org/status/429", waiting until "load"
None
[crawlee.crawlers._basic._basic_crawler] INFO Error analysis: total_errors=1 unique_errors=1
[crawlee.crawlers._basic._basic_crawler] INFO The crawl was interrupted. To resume, do: CRAWLEE_PURGE_ON_START=0 python scripts/crawling/crawl_url.py
[crawlee.crawlers._basic._basic_crawler] INFO Final request statistics:
┌───────────────────────────────┬───────┐
│ requests_finished │ 0 │
│ requests_failed │ 0 │
│ retry_histogram │ [0] │
│ request_avg_failed_duration │ None │
│ request_avg_finished_duration │ None │
│ requests_finished_per_minute │ 0 │
│ requests_failed_per_minute │ 0 │
│ request_total_duration │ 0s │
│ requests_total │ 0 │
│ crawler_runtime │ 7.50s │
└───────────────────────────────┴───────┘
[dip.crawler.main] INFO Finished crawling
[asyncio] ERROR Future exception was never retrieved
future: <Future finished exception=Exception('Connection closed while reading from the driver')>
Exception: Connection closed while reading from the driver
Activity
ericvg97 commentedon Oct 14, 2025
This is my full crawler async def crawl(seed_url: str) -> None:
`async def crawl(seed_url: str) -> None:
concurrency_settings = ConcurrencySettings(
min_concurrency=1,
max_concurrency=1,
desired_concurrency=1,
max_tasks_per_minute=200,
)
statistics = Statistics.with_default_state(log_interval=timedelta(minutes=1), statistics_log_format="table")
config = service_locator.get_configuration()
config.purge_on_start = True
Mantisus commentedon Oct 14, 2025
The code above performs exactly 10 session rotations. However, note that
runwill never terminate because you are usingkeep_alive=True.ericvg97 commentedon Oct 14, 2025
It is not related to keep_alive=True, it happens with keep_alive=False in the same way. These are my logs (I killed it with ctrl+c). I am using Crawlee 1.0.2 in case it helps
Mantisus commentedon Oct 15, 2025
Yes, I was able to reproduce it. Thank you.
This is a problem with
SqlRequestQueueClient.fix: fix `reclaim_request` in `SqlRequestQueueClient` to correctly up…