Skip to content

Improve storage manager and merge it with creation_management module #147

Closed

Description

@vdusek

The current Crawlee / StorageClientManager is more or less just copied from the Python SDK / StorageClientManager and is extremely simple. Its primary role is to maintain and provide access to storage client instances based on specific input parameters.

The Crawlee TS / StorageManager is more complex and it takes care of more things - creating instances of storages & their caching.

Currently, we have a helper module "creation_management" in storages/ which helps with it.

Let's move logic from storages/creation_management to StorageClientManager and improve the creation & caching process.

Functions get_or_create, find_or_create_client_by_id_or_name a create_*_from_directory should be refactored.

Activity

  1. added
    enhancementNew feature or request.
    t-toolingIssues with this label are in the ownership of the tooling team.
    on May 10, 2024
  2. janbuchar commented on May 10, 2024

    @janbuchar
    Collaborator

    I could also imagine putting the functionality into a module instead of a singleton class, so basically StorageManager -> creation_management, not vice versa.

  3. changed the title [-]Improve storage manager[/-] [+]Improve storage manager and merge it with `creation_management` module[/+] on May 13, 2024
  4. janbuchar commented on Jun 3, 2024

    @janbuchar
    Collaborator

    This code should not check the implementation in use - it's a generic storage manager that should not be concerned with the concrete implementation.

  5. added this to the milestone on Jun 10, 2024
  6. removed this from the milestone on Jul 15, 2024
  7. self-assigned this
    on Apr 7, 2025
  8. added a commit that references this issue on Jul 1, 2025
    de1c03f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions