Repository navigation
Closed
Labels
bugSomething isn't working.Something isn't working.t-toolingIssues with this label are in the ownership of the tooling team.Issues with this label are in the ownership of the tooling team.
Milestone
Description
I'm getting the following traceback when trying to start a crawl job again after exiting with "CTRL-C":
❯ CRAWLEE_PURGE_ON_START=0 python main.py
Traceback (most recent call last):
File "proj/main.py", line 80, in <module>
asyncio.run(main())
~~~~~~~~~~~^^^^^^^^
File "/opt/homebrew/Cellar/python@3.13/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/runners.py", line 195, in run
return runner.run(main)
~~~~~~~~~~^^^^^^
File "/opt/homebrew/Cellar/python@3.13/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
File "/opt/homebrew/Cellar/python@3.13/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/base_events.py", line 725, in run_until_complete
return future.result()
~~~~~~~~~~~~~^^
File "proj/main.py", line 76, in main
await crawler.run(urls)
File "proj/.venv/lib/python3.13/site-packages/crawlee/crawlers/_basic/_basic_crawler.py", line 474, in run
await self.add_requests(requests)
File "proj/.venv/lib/python3.13/site-packages/crawlee/crawlers/_basic/_basic_crawler.py", line 563, in add_requests
request_manager = await self.get_request_manager()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "proj/.venv/lib/python3.13/site-packages/crawlee/crawlers/_basic/_basic_crawler.py", line 406, in get_request_manager
self._request_manager = await RequestQueue.open()
^^^^^^^^^^^^^^^^^^^^^^^^^
File "proj/.venv/lib/python3.13/site-packages/crawlee/storages/_request_queue.py", line 165, in open
return await open_storage(
^^^^^^^^^^^^^^^^^^^
...<5 lines>...
)
^
File "proj/.venv/lib/python3.13/site-packages/crawlee/storages/_creation_management.py", line 166, in open_storage
storage_info = await resource_collection_client.get_or_create(name=name, id=id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "proj/.venv/lib/python3.13/site-packages/crawlee/storage_clients/_memory/_request_queue_collection_client.py", line 35, in get_or_create
resource_client = await get_or_create_inner(
^^^^^^^^^^^^^^^^^^^^^^^^^^
...<5 lines>...
)
^
File "proj/.venv/lib/python3.13/site-packages/crawlee/storage_clients/_memory/_creation_management.py", line 143, in get_or_create_inner
found = find_or_create_client_by_id_or_name_inner(
resource_client_class=resource_client_class,
...<2 lines>...
id=id,
)
File "proj/.venv/lib/python3.13/site-packages/crawlee/storage_clients/_memory/_creation_management.py", line 102, in find_or_create_client_by_id_or_name_inner
storage_path = _determine_storage_path(resource_client_class, memory_storage_client, id, name)
File "proj/.venv/lib/python3.13/site-packages/crawlee/storage_clients/_memory/_creation_management.py", line 412, in _determine_storage_path
metadata = json.load(metadata_file)
File "/opt/homebrew/Cellar/python@3.13/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python3.13/json/__init__.py", line 293, in load
return loads(fp.read(),
cls=cls, object_hook=object_hook,
parse_float=parse_float, parse_int=parse_int,
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/opt/homebrew/Cellar/python@3.13/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python3.13/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
~~~~~~~~~~~~~~~~~~~~~~~^^^
File "/opt/homebrew/Cellar/python@3.13/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python3.13/json/decoder.py", line 345, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.13/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python3.13/json/decoder.py", line 363, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
It looks like it might be having trouble decoding the metadata file? FWIW, I don't see any metadata files in storage/request_queues/default.
Metadata
Metadata
Assignees
Labels
bugSomething isn't working.Something isn't working.t-toolingIssues with this label are in the ownership of the tooling team.Issues with this label are in the ownership of the tooling team.
Activity
janbuchar commentedon Feb 27, 2025
Hello @yqiang and thanks for the bug report! This seems like the same thing as #968, but that was closed due to inactivity. I hope we can crack it this time 🙂
yqiang commentedon Feb 27, 2025
I hope so too. FWIW crawlee was running with around 25 concurrent headless WebKit clients via playwright. In the meantime, is there anything that I can try to salvage this crawl job?
Mantisus commentedon Feb 27, 2025
Could you double-check, the file should be called
__metadata__.json? It might be lost among the queue files.... :)Because if it's not in folder, it should have been re-created automatically. However, if it is in the folder and it is empty, it will probably trigger your error.
yqiang commentedon Feb 27, 2025
The file exists, but it's empty:
Otherwise the directory is populated with other request files:
Should I remove the file so it can recreate it?
Mantisus commentedon Feb 27, 2025
Apparently сrawl interrupted when the file was supposed to be updated.
Yeah, I think that'll work for you. The queue should be restored from the requests data
fix: fix handling of loading empty `metadata` file for queue (apify#1042