Skip to content

context.request.user_data mutations are silently dropped on retry #2061

Closed

Description

@vdusek

Description

Mutating context.request.user_data inside a handler doesn't persist across a retry. The change is visible within the same handler run, but silently gone on the next attempt.

Cause: context.request is an isolated deepcopy created for handler execution (RequestHandlerRunResult, src/crawlee/_types.py:277), while the retry reclaims the original request. So in-handler mutations to context.request (including user_data) are dropped on retry. This differs from Crawlee for JS, where request.userData mutations persist across retries.

Surfaced in discussion #2028, where the natural workaround (stash state on request.user_data, read it on the retry) silently fails.

Reproduction

import asyncio

from crawlee.crawlers import BasicCrawler, BasicCrawlingContext


async def main() -> None:
    crawler = BasicCrawler(max_request_retries=3)

    @crawler.router.default_handler
    async def handler(context: BasicCrawlingContext) -> None:
        print(f'retry_count={context.request.retry_count}, flag={context.request.user_data.get("flag")!r}')
        if context.request.retry_count == 0:
            context.request.user_data['flag'] = 'set-on-first-attempt'
            raise RuntimeError('force a retry')

    await crawler.run(['https://example.com/'])


asyncio.run(main())

Actual

retry_count=0, flag=None
retry_count=1, flag=None

Expected

retry_count=0, flag=None
retry_count=1, flag='set-on-first-attempt'

Reproduced on crawlee 1.8.x.

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Jul 10, 2026
  2. Mantisus commented on Jul 10, 2026

    @Mantisus
    Collaborator

    This conflicts with the PR - #1585

    We need to decide what the final behavior should be.

  3. Huaian666 commented on Jul 13, 2026

    @Huaian666

    js/python parity is the bigger deal here tbh — silent data loss on retry is brutal to debug when porting handlers from Crawlee JS. deepcopy isolation is fine for handler safety, but maybe an explicit request.persist_user_data() would cover retry-state without changing mutation semantics

  4. added a commit that references this issue on Jul 15, 2026
    e971279
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions