You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I want to use fastapi+crawlee to scrape the posts content of a forum. I hope that my crawler will re-run every time a post request is made, but actually, when I request a second time, the links requested before will not be requested again. I have learned two solutions from this post:
Set environment variables
I did this, but it didn't work
Hi, and thanks for your interest in Crawlee! Setting persist_storage = False and purge_on_start = True should be enough for the RequestQueue not to be persisted to the filesystem. Could you check what's in your storage/request-queues/default directory?
Even if I delete these files, I find it still doesn't work as expected. Since I'm writing this project just for fun, you can see my complete code here (currently, I'm trying to restart the server after each post request).
I am having the same issue here, I tried as well using data_set.drop() but crawlee keep looking in cache and return an empty content instead of crawling the website again
crawler.run cannot be called in parallel (from multiple HTTP calls) - we use FastAPI dependencies to limit this
Currently, if you try to work around this limitation by making a new crawler instance for each HTTP call, only the first call will return a non-empty result
this is because of how the StorageManager, an internal mechanism in Crawlee, works - it is supposed to prevent problems caused by opening the same storage with different configurations
to make this work, you need to open a separate RequestQueue for each request, like this:
Activity
janbuchar commentedon Jul 24, 2024
Hi, and thanks for your interest in Crawlee! Setting
persist_storage = Falseandpurge_on_start = Trueshould be enough for the RequestQueue not to be persisted to the filesystem. Could you check what's in yourstorage/request-queues/defaultdirectory?GHOST-LOVE-YOU commentedon Jul 24, 2024
Thank you for your quick response!
Below is the content of my storage/request-queues/default:
Here are two representative files:
{ "id": "default", "name": null, "accessed_at": "2024-07-24 08:33:18.665059+00:00", "created_at": "2024-07-24 08:33:10.555299+00:00", "modified_at": "2024-07-24 08:33:18.664195+00:00", "had_multiple_clients": false, "handled_request_count": 23, "pending_request_count": 0, "stats": {}, "total_request_count": 23, "user_id": "1", "resource_directory": "./storage/request_queues/default" }{ "url": "https://xxxxxxxx/article/IWhisper/7420120?p=3", "unique_key": "/article/iwhisper/7420120?p=3", "method": "get", "payload": null, "headers": {}, "user_data": {}, "retry_count": 0, "no_retry": false, "loaded_url": null, "handled_at": null, "id": "NejkxElN87hwno9", "json_": "{\n \"url\": \"https://xxxxxxxx/article/IWhisper/7420120?p=3\",\n \"unique_key\": \"/article/iwhisper/7420120?p=3\",\n \"method\": \"get\",\n \"payload\": null,\n \"headers\": {},\n \"user_data\": {\n \"label\": \"DETAIL\",\n \"__crawlee\": {\n \"state\": \"RequestState.REQUEST_HANDLER\"\n }\n },\n \"retry_count\": 0,\n \"no_retry\": false,\n \"loaded_url\": \"https://xxxxxxxxxxx/#!article/IWhisper/7420120?p=3\",\n \"handled_at\": \"2024-07-24 08:33:17.498771+00:00\",\n \"id\": \"NejkxElN87hwno9\",\n \"json_\": null,\n \"order_no\": null\n}", "order_no": null }Even if I delete these files, I find it still doesn't work as expected. Since I'm writing this project just for fun, you can see my complete code here (currently, I'm trying to restart the server after each post request).
janbuchar commentedon Jul 24, 2024
Thanks for the code, we'll look into this! There is a good chance this is a bug in Crawlee.
1hachem commentedon Jul 27, 2024
I am having the same issue here, I tried as well using
data_set.drop()but crawlee keep looking in cache and return an empty content instead of crawling the website againfix: pass crawler configuration to storages (#375)
janbuchar commentedon Jul 30, 2024
With crawlee 0.1.2, you should be able to do this:
The caveats are:
crawler.runcannot be called in parallel (from multiple HTTP calls) - we use FastAPI dependencies to limit thisStorageManager, an internal mechanism in Crawlee, works - it is supposed to prevent problems caused by opening the same storage with different configurationsRequestQueuefor each request, like this:Let me know if this helps!
GHOST-LOVE-YOU commentedon Jul 31, 2024
It does work! Thanks a lot.
test: Test persist_storage more thoroughly (#383)