Skip to content

All crawlee persistence names should be valid names for Apify platform #1354

Closed

Description

@Pijukatel

In FileSystemRequestQueueClient there is persistance to kvs with name persist_state_kvs_name=f'__RQ_STATE_{self._metadata.id}',

When used on Apify platform, it will throw an error:

apify_client._errors.ApifyApiError: Invalid value provided in store: name can only contain letters 'a' through 'z', the digits '0' through '9', and the hyphen ('-') but only in the middle of the string (e.g. 'my-value-1')

which is caused by the underscores in the name.

Fix the name to be compatible with the Apify constraints and check all similar names of RecoverableStates in code base

To reproduce, run this on Apify platform, which create file system base request queue, that persists to Apify kvs:

async def main() -> None:
    async with Actor:
        rq = await RequestQueue.open(
            storage_client=FileSystemStorageClient(),
        )

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Aug 13, 2025
  2. self-assigned this
    on Aug 13, 2025
  3. added this to the 1.0 milestone on Aug 18, 2025
  4. Pijukatel commented on Aug 21, 2025

    @Pijukatel
    CollaboratorAuthor

    This is more of a problem that RecoverableState relies on a service locator to give it suitable storage, and this can lead to scenarios where RecoverableState for XYZ storage is being saved to ABC storage, which makes no sense. A similar problem is waiting to be triggered in KeyValueStore.get_auto_saved_value, which also uses RecoverableState that relies on the service locator.

    Reasonable default should be that RecoverableState is using the same storage for which it is recovering...

  5. changed the title [-]All crawlee persistance names should be valid names for Apify platform[/-] [+]All crawlee persistence names should be valid names for Apify platform[/+] on Oct 7, 2025
  6. added a commit that references this issue on Oct 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions