You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add local caching for crawled pages to enhance development efficiency #861
It would be a huge time saver during development if the website only needed to be crawled once. While iterating on my data extraction code, I wish I could load the pages from my local disk instead of the Internet.
During development, repeatedly crawling the website to test data extraction code is time-consuming and inefficient. A caching mechanism that stores previously downloaded pages on the local disk would be a huge time saver. Instead of fetching the pages from the Internet, the development environment could simply load them from local storage.
This approach would not only speed up the iterative development process but also reduce load on the website's server. Such a feature would be particularly useful for debugging and refining data extraction scripts.
changed the title [-]Is there an option to cache downloaded pages locally?[/-][+]Add local caching for crawled pages to enhance development efficiency[/+]on Jan 6, 2025
I think this can already be achieved by the approach described in the documentation related to web archiving.
During the first crawl, follow these steps: Crawling through proxy recording server
To just re-use already archived pages, follow these steps: Using the archived data
Activity
vdusek commentedon Jan 6, 2025
Hi, this sounds reasonable, while it is not currently on our roadmap. We will revise it in the future.
[-]Is there an option to cache downloaded pages locally?[/-][+]Add local caching for crawled pages to enhance development efficiency[/+]Pijukatel commentedon Jan 12, 2026
I think this can already be achieved by the approach described in the documentation related to web archiving.
During the first crawl, follow these steps:
Crawling through proxy recording server
To just re-use already archived pages, follow these steps:
Using the archived data