Skip to content

Add local caching for crawled pages to enhance development efficiency #861

Closed

Description

@matecsaj

It would be a huge time saver during development if the website only needed to be crawled once. While iterating on my data extraction code, I wish I could load the pages from my local disk instead of the Internet.

During development, repeatedly crawling the website to test data extraction code is time-consuming and inefficient. A caching mechanism that stores previously downloaded pages on the local disk would be a huge time saver. Instead of fetching the pages from the Internet, the development environment could simply load them from local storage.

This approach would not only speed up the iterative development process but also reduce load on the website's server. Such a feature would be particularly useful for debugging and refining data extraction scripts.

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Jan 5, 2025
  2. vdusek commented on Jan 6, 2025

    @vdusek
    Collaborator

    Hi, this sounds reasonable, while it is not currently on our roadmap. We will revise it in the future.

  3. changed the title [-]Is there an option to cache downloaded pages locally?[/-] [+]Add local caching for crawled pages to enhance development efficiency[/+] on Jan 6, 2025
  4. Pijukatel commented on Jan 12, 2026

    @Pijukatel
    Collaborator

    I think this can already be achieved by the approach described in the documentation related to web archiving.
    During the first crawl, follow these steps:
    Crawling through proxy recording server
    To just re-use already archived pages, follow these steps:
    Using the archived data

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request.t-toolingIssues with this label are in the ownership of the tooling team.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions