Repository navigation
Closed
Labels
t-toolingIssues with this label are in the ownership of the tooling team.Issues with this label are in the ownership of the tooling team.
Description
Problem
I cannot overwrite the default settings of the BasicCrawler through the PlaywrightCrawler. Reason for this is that the PlaywrightCrawler sets up its browser_pool = BrowserPool.with_default_plugin(..) before calling super().__init__() which will set the configuration through the BasicCrawler's init, however the browserpool code calls service_locator.get_configuration() which in turn locks setting the configuration preventing me from changing it.
Code
from crawlee.configuration import Configuration
from crawlee.crawlers import PlaywrightCrawler
configuration = Configuration(default_dataset_id='my_dataset_id')
PlaywrightCrawler(configuration=configuration)Error message
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "~/crawlee-python/src/crawlee/crawlers/_playwright/_playwright_crawler.py", line 177, in __init__
super().__init__(**kwargs)
File "~/crawlee-python/src/crawlee/crawlers/_basic/_basic_crawler.py", line 310, in __init__
service_locator.set_configuration(configuration)
File "~/crawlee-python/src/crawlee/_service_locator.py", line 45, in set_configuration
raise ServiceConflictError(Configuration, configuration, self._configuration)
crawlee.errors.ServiceConflictError: Service Configuration is already in use. Existing value: internal_timeout=None default_browser_path=None disable_browser_sandbox=False log_level='INFO' default_dataset_id='default' default_key_value_store_id='default' default_request_queue_id='default' purge_on_start=True write_metadata=True persist_storage=True persist_state_interval=datetime.timedelta(seconds=60) system_info_interval=datetime.timedelta(seconds=1) max_used_cpu_ratio=0.95 max_used_memory_ratio=0.9 max_event_loop_delay=datetime.timedelta(microseconds=50000) max_client_errors=1 memory_mbytes=None available_memory_ratio=0.25 storage_dir='./storage' headless=True, attempted new value: internal_timeout=None default_browser_path=None disable_browser_sandbox=False log_level='INFO' default_dataset_id='my_dataset_id' default_key_value_store_id='default' default_request_queue_id='default' purge_on_start=True write_metadata=True persist_storage=True persist_state_interval=datetime.timedelta(seconds=60) system_info_interval=datetime.timedelta(seconds=1) max_used_cpu_ratio=0.95 max_used_memory_ratio=0.9 max_event_loop_delay=datetime.timedelta(microseconds=50000) max_client_errors=1 memory_mbytes=None available_memory_ratio=0.25 storage_dir='./storage' headless=True.
Metadata
Metadata
Assignees
Labels
t-toolingIssues with this label are in the ownership of the tooling team.Issues with this label are in the ownership of the tooling team.
Activity
fix: Allow config change on `PlaywrightCrawler` (apify#1186)