Skip to content

error starting up crawlee - json.decoder.JSONDecodeError: Expecting value #1029

Closed

Description

@yqiang

I'm getting the following traceback when trying to start a crawl job again after exiting with "CTRL-C":

❯ CRAWLEE_PURGE_ON_START=0 python main.py
Traceback (most recent call last):
  File "proj/main.py", line 80, in <module>
    asyncio.run(main())
    ~~~~~~~~~~~^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.13/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ~~~~~~~~~~^^^^^^
  File "/opt/homebrew/Cellar/python@3.13/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/opt/homebrew/Cellar/python@3.13/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/base_events.py", line 725, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File "proj/main.py", line 76, in main
    await crawler.run(urls)
  File "proj/.venv/lib/python3.13/site-packages/crawlee/crawlers/_basic/_basic_crawler.py", line 474, in run
    await self.add_requests(requests)
  File "proj/.venv/lib/python3.13/site-packages/crawlee/crawlers/_basic/_basic_crawler.py", line 563, in add_requests
    request_manager = await self.get_request_manager()
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "proj/.venv/lib/python3.13/site-packages/crawlee/crawlers/_basic/_basic_crawler.py", line 406, in get_request_manager
    self._request_manager = await RequestQueue.open()
                            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "proj/.venv/lib/python3.13/site-packages/crawlee/storages/_request_queue.py", line 165, in open
    return await open_storage(
           ^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
    )
    ^
  File "proj/.venv/lib/python3.13/site-packages/crawlee/storages/_creation_management.py", line 166, in open_storage
    storage_info = await resource_collection_client.get_or_create(name=name, id=id)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "proj/.venv/lib/python3.13/site-packages/crawlee/storage_clients/_memory/_request_queue_collection_client.py", line 35, in get_or_create
    resource_client = await get_or_create_inner(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
    )
    ^
  File "proj/.venv/lib/python3.13/site-packages/crawlee/storage_clients/_memory/_creation_management.py", line 143, in get_or_create_inner
    found = find_or_create_client_by_id_or_name_inner(
        resource_client_class=resource_client_class,
    ...<2 lines>...
        id=id,
    )
  File "proj/.venv/lib/python3.13/site-packages/crawlee/storage_clients/_memory/_creation_management.py", line 102, in find_or_create_client_by_id_or_name_inner
    storage_path = _determine_storage_path(resource_client_class, memory_storage_client, id, name)
  File "proj/.venv/lib/python3.13/site-packages/crawlee/storage_clients/_memory/_creation_management.py", line 412, in _determine_storage_path
    metadata = json.load(metadata_file)
  File "/opt/homebrew/Cellar/python@3.13/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python3.13/json/__init__.py", line 293, in load
    return loads(fp.read(),
        cls=cls, object_hook=object_hook,
        parse_float=parse_float, parse_int=parse_int,
        parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/opt/homebrew/Cellar/python@3.13/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python3.13/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^
  File "/opt/homebrew/Cellar/python@3.13/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python3.13/json/decoder.py", line 345, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.13/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python3.13/json/decoder.py", line 363, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

It looks like it might be having trouble decoding the metadata file? FWIW, I don't see any metadata files in storage/request_queues/default.

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Feb 27, 2025
  2. janbuchar commented on Feb 27, 2025

    @janbuchar
    Collaborator

    Hello @yqiang and thanks for the bug report! This seems like the same thing as #968, but that was closed due to inactivity. I hope we can crack it this time 🙂

  3. yqiang commented on Feb 27, 2025

    @yqiang
    Author

    I hope so too. FWIW crawlee was running with around 25 concurrent headless WebKit clients via playwright. In the meantime, is there anything that I can try to salvage this crawl job?

  4. Mantisus commented on Feb 27, 2025

    @Mantisus
    Collaborator

    I don't see any metadata files in

    Could you double-check, the file should be called __metadata__.json? It might be lost among the queue files.... :)

    Because if it's not in folder, it should have been re-created automatically. However, if it is in the folder and it is empty, it will probably trigger your error.

  5. yqiang commented on Feb 27, 2025

    @yqiang
    Author

    The file exists, but it's empty:

    storage/request_queues/default on main via 🐍 v3.13.2
    ❯ file __metadata__.json
    __metadata__.json: empty
    

    Otherwise the directory is populated with other request files:

    ❯ ls -l | wc -l
        1558
    

    Should I remove the file so it can recreate it?

  6. Mantisus commented on Feb 27, 2025

    @Mantisus
    Collaborator

    The file exists, but it's empty:

    Apparently сrawl interrupted when the file was supposed to be updated.

    Should I remove the file so it can recreate it?

    Yeah, I think that'll work for you. The queue should be restored from the requests data

  7. removed this from the 109th sprint - Tooling team milestone on Mar 18, 2025
  8. added a commit that references this issue on Mar 18, 2025
    b00876e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions