Skip to content

max_session_rotations not being respected in AdaptivePlaywrightCrawler.with_beautifulsoup_static_parser #1484

Closed

Description

@ericvg97

Hi, its me again, sorry for the amount of issues I am opening:) I think in this case it is a bug. I am using this

concurrency_settings  =  ConcurrencySettings(
    min_concurrency=1,
    max_concurrency=1,
    desired_concurrency=1,
    max_tasks_per_minute=200)
    
crawler = AdaptivePlaywrightCrawler.with_beautifulsoup_static_parser(
    max_requests_per_crawl=10000,
    playwright_crawler_specific_kwargs={
        "browser_type": "chromium",
        "headless": True,
    },
    max_session_rotations=10,
    retry_on_blocked=True,
    concurrency_settings=concurrency_settings,
    use_session_pool=True,
    max_request_retries=5,
    keep_alive=True,
)

and if I crawl this url "http://httpbingo.org/status/429" it keeps rotating the session ("Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...") until infinity

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Oct 14, 2025
  2. ericvg97 commented on Oct 14, 2025

    @ericvg97
    Author

    This is my full crawler async def crawl(seed_url: str) -> None:

    `async def crawl(seed_url: str) -> None:
    concurrency_settings = ConcurrencySettings(
    min_concurrency=1,
    max_concurrency=1,
    desired_concurrency=1,
    max_tasks_per_minute=200,
    )
    statistics = Statistics.with_default_state(log_interval=timedelta(minutes=1), statistics_log_format="table")
    config = service_locator.get_configuration()
    config.purge_on_start = True

    engine = create_async_engine("postgresql+psycopg://postgres:password@localhost:5432/lighthouse")
    sql_client = SqlStorageClient(engine=engine)
    
    service_locator.set_storage_client(sql_client)
    
    crawler = AdaptivePlaywrightCrawler.with_beautifulsoup_static_parser(
        max_requests_per_crawl=10000,
        playwright_crawler_specific_kwargs={
            "browser_type": "chromium",
            "headless": True,
        },
        max_session_rotations=10,
        retry_on_blocked=True,
        concurrency_settings=concurrency_settings,
        statistics=statistics,
        configure_logging=True,
        use_session_pool=True,
        max_request_retries=5,
        request_handler_timeout=timedelta(seconds=30),
        keep_alive=True,
    )
    
    async def default_handler(context: AdaptivePlaywrightCrawlingContext) -> None:
        context.log.info(f"Processing {context.request.url} ...")
    
        context.request.user_data["something"] = "something"
    
        await context.enqueue_links(strategy="same-domain")
    
    crawler.router.default_handler(default_handler)
    
    async def pre_navigation_hook(context: BasicCrawlingContext) -> None:
        context.log.info(f"Before processing {context.request.url}")
    
    crawler.pre_navigation_hook(pre_navigation_hook)
    
    async def error_handler(context: BasicCrawlingContext, error: Exception) -> None:
        logger.error(f"Failed request {context.request.url}, after {context.request.retry_count} retries")
        logger.error(f"Exception: {error}")
    
    async def failed_request_handler(context: BasicCrawlingContext, error: Exception) -> None:
        logger.error(
            f"Not retrying any more for request {context.request.url}, after {context.request.retry_count} retries"
        )
        logger.error(f"Exception was: {error}")
    
        context.request.user_data["error"] = True
    
    crawler.error_handler(error_handler)
    crawler.failed_request_handler(failed_request_handler)
    
    logger.info(f"Starting crawling on '{seed_url}'")
    await crawler.run([seed_url])
    
    logger.info("Finished crawling")`
    
  3. Mantisus commented on Oct 14, 2025

    @Mantisus
    Collaborator

    and if I crawl this url "http://httpbingo.org/status/429" it keeps rotating the session ("Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...") until infinity

    The code above performs exactly 10 session rotations. However, note that run will never terminate because you are using keep_alive=True.

    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] ERROR Request to http://httpbingo.org/status/429 failed and reached maximum retries
  4. ericvg97 commented on Oct 14, 2025

    @ericvg97
    Author

    It is not related to keep_alive=True, it happens with keep_alive=False in the same way. These are my logs (I killed it with ctrl+c). I am using Crawlee 1.0.2 in case it helps

    
    [dip.crawler.main] INFO  Starting crawling on 'http://httpbingo.org/status/429'
    [crawlee.crawlers._basic._basic_crawler] INFO  Crawled 0/1 pages, 0 failed requests, desired concurrency 1.
    [crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 0; desired_concurrency = 1; cpu = 0.0; mem = 0.0; event_loop = 0.0; client_info = 0.0
    [crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
    [dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
    [dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
    [dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
    [dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
    [dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
    [dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
    [dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
    [dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
    [dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
    [dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
    [dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
    [dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
    [dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
    [dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
    [dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
    [dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
    [dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
    [dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
    [dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
    [dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
    [dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
    [dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
    [dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
    [dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
    [dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
    [dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
    [dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
    [dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
    [dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
    [dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
    [dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
    [dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
    [dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
    [dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
    [dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
    [dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
    [dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
    [dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
    [dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
    [dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
    [dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
    [dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
    [crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
    [crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
    ^C[crawlee.crawlers._basic._basic_crawler] INFO  Pausing... Press CTRL+C again to force exit.
    [crawlee._autoscaling.autoscaled_pool] INFO  Waiting for remaining tasks to finish
    ??????????????????? 0 5
    [crawlee.crawlers._basic._basic_crawler] WARN  Retrying request to http://httpbingo.org/status/429 due to: Page.goto: net::ERR_ABORTED at http://httpbingo.org/status/429
    [dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 1 retries
    [dip.crawler.main] ERROR Exception: Page.goto: net::ERR_ABORTED at http://httpbingo.org/status/429
    Call log:
      - navigating to "http://httpbingo.org/status/429", waiting until "load"
    
    None
    [crawlee.crawlers._basic._basic_crawler] INFO  Error analysis: total_errors=1 unique_errors=1
    [crawlee.crawlers._basic._basic_crawler] INFO  The crawl was interrupted. To resume, do: CRAWLEE_PURGE_ON_START=0 python scripts/crawling/crawl_url.py
    [crawlee.crawlers._basic._basic_crawler] INFO  Final request statistics:
    ┌───────────────────────────────┬───────┐
    │ requests_finished             │ 0     │
    │ requests_failed               │ 0     │
    │ retry_histogram               │ [0]   │
    │ request_avg_failed_duration   │ None  │
    │ request_avg_finished_duration │ None  │
    │ requests_finished_per_minute  │ 0     │
    │ requests_failed_per_minute    │ 0     │
    │ request_total_duration        │ 0s    │
    │ requests_total                │ 0     │
    │ crawler_runtime               │ 7.50s │
    └───────────────────────────────┴───────┘
    [dip.crawler.main] INFO  Finished crawling
    [asyncio] ERROR Future exception was never retrieved
    future: <Future finished exception=Exception('Connection closed while reading from the driver')>
          Exception: Connection closed while reading from the driver
    
  5. self-assigned this
    on Oct 15, 2025
  6. Mantisus commented on Oct 15, 2025

    @Mantisus
    Collaborator

    Yes, I was able to reproduce it. Thank you.

    This is a problem with SqlRequestQueueClient.

  7. added a commit that references this issue on Oct 15, 2025
    1502469
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions