You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When running the httpx crawler on v0.3.7
My request_dict '{"url": "XXXXXX", "unique_key": "YYYYY", "method": "GET", "headers": {}, "query_params": {}, "data": {}, "user_data": {"__crawlee": {"state": "RequestState.ERROR_HANDLER"}}, "retry_count": 1, "no_retry": false, "id": "doOfa0tGsjBwT90"}'
create the exception below when running Request.model_validate(request_dict)
user_data.__crawlee.state
Input should be 0, 1, 2, 3, 4, 5, 6 or 7 [type=enum, input_value='RequestState.ERROR_HANDLER', input_type=str]
For further information visit https://errors.pydantic.dev/2.9/v/enum
Hello - the request_dict comes from a deserialised request from my crawlee storage. There is a bug in Crawlee enum serialisation/deserialisation through Pydantic (I am using v2.9.2, compatible with crawlee pyproject constraints)
Activity
janbuchar commentedon Sep 29, 2024
Hello and thank you for your interest in Crawlee! Where does that request_dict come from? The state property is suspicious.
cadlagtrader commentedon Sep 30, 2024
Hello - the request_dict comes from a deserialised request from my crawlee storage. There is a bug in Crawlee enum serialisation/deserialisation through Pydantic (I am using v2.9.2, compatible with crawlee pyproject constraints)
I have modifed
test_complex_user_data_serializationto highlight the issue: https://github.com/cadlagtrader/crawlee-python/blob/bug_with_pydantic/tests/unit/storages/test_request_queue.pyfix: use StrEnum for RequestState to fix serialization