You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Improve storage manager and merge it with creation_management module #147
The current Crawlee / StorageClientManager is more or less just copied from the Python SDK / StorageClientManager and is extremely simple. Its primary role is to maintain and provide access to storage client instances based on specific input parameters.
The Crawlee TS / StorageManager is more complex and it takes care of more things - creating instances of storages & their caching.
Currently, we have a helper module "creation_management" in storages/ which helps with it.
Let's move logic from storages/creation_management to StorageClientManager and improve the creation & caching process.
Functions get_or_create, find_or_create_client_by_id_or_name a create_*_from_directory should be refactored.
I could also imagine putting the functionality into a module instead of a singleton class, so basically StorageManager -> creation_management, not vice versa.
Activity
janbuchar commentedon May 10, 2024
I could also imagine putting the functionality into a module instead of a singleton class, so basically StorageManager -> creation_management, not vice versa.
[-]Improve storage manager[/-][+]Improve storage manager and merge it with `creation_management` module[/+]janbuchar commentedon Jun 3, 2024
This code should not check the implementation in use - it's a generic storage manager that should not be concerned with the concrete implementation.
refactor!: Introduce new storage client system (apify#1194)