Skip to content

failed_request_handler runs and logs but context.push_data(...) does not write to dataset (PlaywrightCrawler) #1532

Closed

Description

@loic-bellinger

When a request fails in PlaywrightCrawler, my @crawler.failed_request_handler runs (I can see context.log output) but await context.push_data(...) does not create any dataset rows. Logging works (I can see the context.log.info(...) lines), only the dataset push appears to be ignored.

Minimal repro attached below.

import asyncio

from crawlee.crawlers import (
    PlaywrightCrawler,
    PlaywrightCrawlingContext,
    BasicCrawlingContext,
)


async def main() -> None:
    crawler = PlaywrightCrawler(
        max_requests_per_crawl=10,
        max_request_retries=2,
        headless=True,
        browser_type='chromium',
    )

    @crawler.router.default_handler
    async def request_handler(context: PlaywrightCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url} ...')

    @crawler.failed_request_handler
    async def failed_request_handler(context: BasicCrawlingContext, error: Exception) -> None:
        context.log.info(f'failed_request_handler: processing {context.request.url} ...')
        await context.push_data(
            dataset_name="failed_request_handler_errors",
            data={
                "failed_url": context.request.url,
                "label": context.request.label,
                "error_type": type(error).__name__,
                "error_message": str(error),
                "retry_count": context.request.retry_count,
                "status": "failed",
            },
        )

    await crawler.run(['https://www.info.gouv.fr/totalnonsense'])


if __name__ == '__main__':
    asyncio.run(main())

I'm using crawlee[playwright]>=1.0.4

Relevant part of the documentation is https://crawlee.dev/python/docs/guides/request-router#failed-request-handler

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Nov 5, 2025
  2. janbuchar commented on Nov 5, 2025

    @janbuchar
    Collaborator

    Hi @loic-bellinger, thanks for reporting this! The issue is that we "buffer" any storage-related calls on the context object so that partial results of failing request handler runs are not stored. This is an unintended side effect of this feature. We'll do our best to fix this soon 🙂

    As a temporary workaround, you may try this:

    dataset = await Dataset.open(name="failed_request_handler_errors")
    await dataset.push_data({...})

    It should bypass the "buffering" mechanism.

  3. loic-bellinger commented on Nov 5, 2025

    @loic-bellinger
    Author

    Wouldn't calling dataset = await Dataset.open(name="failed_request_handler_errors") on each error create some overhead?

  4. janbuchar commented on Nov 5, 2025

    @janbuchar
    Collaborator

    Wouldn't calling dataset = await Dataset.open(name="failed_request_handler_errors") on each error create some overhead?

    There is a lot of caching, you should be fine. But let us know if anything happens!

  5. added a commit that references this issue on Nov 19, 2025
    b830019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions