You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am trying to write a simple function to crawl a website and I don't want crawlee to cache anything (each time I call this function it will do everything from scratch).
here is my attempt so far, I tried with persist_storage=False and purge_on_start=True in the configuration, and with removing the storage directory entirely, but I keep getting either a concatenated result of all the requests or and empty result in case I delete the storage directory.
asyncdefmain(
website: str,
include_links: list[str],
exclude_links: list[str],
depth: int=5,
) ->str:
crawler=BeautifulSoupCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.max_requests_per_crawl=depth,
)
dataset=awaitDataset.open(
configuration=Configuration(
persist_storage=False,
purge_on_start=True,
),
)
# Define the default request handler, which will be called for every request.@crawler.router.default_handlerasyncdefrequest_handler(context: BeautifulSoupCrawlingContext) ->None: # type: ignore# Extract data from the page.text=context.soup.get_text()
awaitdataset.push_data({"content": text})
# Enqueue all links found on the page.awaitcontext.enqueue_links(
include=[Glob(url) forurlininclude_links],
exclude=[Glob(url) forurlinexclude_links],
)
# Run the crawler with the initial list of URLs.awaitcrawler.run([website])
data=awaitdataset.get_data()
content="\n".join([item["content"] foritemindata.items]) # type: ignorereturncontent
also is there a way to simple get the result of the crawl as a string, and not use Dataset ?
Hello and thank you for your interest in Crawlee! This seems closely related to #351. Could you please re-check that you get an empty string if you run this after removing the storage directory? I can imagine getting an empty string on a second run without deleting the storage (because of both persist_storage=False and purge_on_start functioning incorrectly), but what you're describing sounds strange.
After some debugging I found a workaround to avoid re-using the cache. Basically we have to ensure that each time the crawler runs it uses a different request queue, e.g. like this:
Thanks. If it helps, I found out during debugging that the problem seems to be in that there’s the same instance of MemoryStorageClient used across runs. There must be some reference left out after the first run.
Thanks. If it helps, I found out during debugging that the problem seems to be in that there’s the same instance of MemoryStorageClient used across runs. There must be some reference left out after the first run.
Yes, that is the case. We're carrying a lot o historical baggage here and maybe this mechanism won't even be necessary in the end. Until then, I'm happy that you found a workaround.
Has anyone found any solutions instead of just changing the request_queue_id?
I don't know of anything, but we're refactoring the storage code so that this can work as expected out of the box. #1107 is the first step towards this.
Activity
janbuchar commentedon Jul 29, 2024
Hello and thank you for your interest in Crawlee! This seems closely related to #351. Could you please re-check that you get an empty string if you run this after removing the storage directory? I can imagine getting an empty string on a second run without deleting the storage (because of both
persist_storage=Falseandpurge_on_startfunctioning incorrectly), but what you're describing sounds strange.tlinhart commentedon Sep 24, 2024
After some debugging I found a workaround to avoid re-using the cache. Basically we have to ensure that each time the crawler runs it uses a different request queue, e.g. like this:
It would be great if we could actually disable caching at all but this works for now.
vdusek commentedon Sep 25, 2024
@tlinhart Thanks. I will also link #541 here as it provides additional context.
tlinhart commentedon Sep 25, 2024
Thanks. If it helps, I found out during debugging that the problem seems to be in that there’s the same instance of
MemoryStorageClientused across runs. There must be some reference left out after the first run.janbuchar commentedon Sep 25, 2024
Yes, that is the case. We're carrying a lot o historical baggage here and maybe this mechanism won't even be necessary in the end. Until then, I'm happy that you found a workaround.
[-]how can I disable cache completely ? [/-][+]How can I disable cache completely?[/+]fix!: Refactor service usage to rely on `service_locator` (apify#691)
amindadgar commentedon Apr 2, 2025
Has anyone found any solutions instead of just changing the
request_queue_id?janbuchar commentedon Apr 2, 2025
I don't know of anything, but we're refactoring the storage code so that this can work as expected out of the box. #1107 is the first step towards this.