Skip to content

How to reset crawlee URL cache? #351

Closed

Description

@GHOST-LOVE-YOU

I want to use fastapi+crawlee to scrape the posts content of a forum. I hope that my crawler will re-run every time a post request is made, but actually, when I request a second time, the links requested before will not be requested again. I have learned two solutions from this post:

  1. Set environment variables
    I did this, but it didn't work
@app.post("/crawl")
async def crawl(auth_data: AuthData):
    if not authenticate(auth_data.username, auth_data.password):
        raise HTTPException(status_code=401, detail="Unauthorized")
    
    if not crawler_semaphore.locked():
        async with crawler_semaphore:
            try:
                crawl_results.clear()
                await run_crawler()
                return {"status": "success", "data": crawl_results}
            except Exception as e:
                raise HTTPException(status_code=500, detail=str(e))
    else:
        raise HTTPException(status_code=429, detail="Crawler is already running. Please try again later.")
async def run_crawler():
    config = Configuration(
        persist_storage=False,
        purge_on_start=True,
    )
    crawler = PlaywrightCrawler(
        max_requests_per_crawl=500,
        request_handler=router,
        configuration=config,
    )
    unique_key = str(uuid.uuid4())
    request = BaseRequestData(url='https://localhost:8888', unique_key=unique_key)
    
    await crawler.run([request])
  1. Modify the unique_key for each request
    I did this in baseUrl, it's cool and it really worked, but I don't know how to add unique_key for enqueue_links
await context.enqueue_links(
    selector=f"a[href='{post['url']}']",
    label='DETAIL',
)

Thank you very much for your help!

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Jul 24, 2024
  2. janbuchar commented on Jul 24, 2024

    @janbuchar
    Collaborator

    Hi, and thanks for your interest in Crawlee! Setting persist_storage = False and purge_on_start = True should be enough for the RequestQueue not to be persisted to the filesystem. Could you check what's in your storage/request-queues/default directory?

  3. GHOST-LOVE-YOU commented on Jul 24, 2024

    @GHOST-LOVE-YOU
    Author

    Thank you for your quick response!
    Below is the content of my storage/request-queues/default:

    > tree storage/
    storage/
    ├── key_value_stores
    │   └── default
    │       └── __metadata__.json
    └── request_queues
        └── default
            ├── 11ZDDqZVzTDFo3c.json
            ├── 2I0wNHjqjrkrZ3G.json
            ├── __metadata__.json
            ├── NejkxElN87hwno9.json
            ├── ......
    

    Here are two representative files:

    1. metadata.json
    {
      "id": "default",
      "name": null,
      "accessed_at": "2024-07-24 08:33:18.665059+00:00",
      "created_at": "2024-07-24 08:33:10.555299+00:00",
      "modified_at": "2024-07-24 08:33:18.664195+00:00",
      "had_multiple_clients": false,
      "handled_request_count": 23,
      "pending_request_count": 0,
      "stats": {},
      "total_request_count": 23,
      "user_id": "1",
      "resource_directory": "./storage/request_queues/default"
    }
    1. NejkxElN87hwno9.json
    {
      "url": "https://xxxxxxxx/article/IWhisper/7420120?p=3",
      "unique_key": "/article/iwhisper/7420120?p=3",
      "method": "get",
      "payload": null,
      "headers": {},
      "user_data": {},
      "retry_count": 0,
      "no_retry": false,
      "loaded_url": null,
      "handled_at": null,
      "id": "NejkxElN87hwno9",
      "json_": "{\n  \"url\": \"https://xxxxxxxx/article/IWhisper/7420120?p=3\",\n  \"unique_key\": \"/article/iwhisper/7420120?p=3\",\n  \"method\": \"get\",\n  \"payload\": null,\n  \"headers\": {},\n  \"user_data\": {\n    \"label\": \"DETAIL\",\n    \"__crawlee\": {\n      \"state\": \"RequestState.REQUEST_HANDLER\"\n    }\n  },\n  \"retry_count\": 0,\n  \"no_retry\": false,\n  \"loaded_url\": \"https://xxxxxxxxxxx/#!article/IWhisper/7420120?p=3\",\n  \"handled_at\": \"2024-07-24 08:33:17.498771+00:00\",\n  \"id\": \"NejkxElN87hwno9\",\n  \"json_\": null,\n  \"order_no\": null\n}",
      "order_no": null
    }

    Even if I delete these files, I find it still doesn't work as expected. Since I'm writing this project just for fun, you can see my complete code here (currently, I'm trying to restart the server after each post request).

  4. janbuchar commented on Jul 24, 2024

    @janbuchar
    Collaborator

    Thanks for the code, we'll look into this! There is a good chance this is a bug in Crawlee.

  5. 1hachem commented on Jul 27, 2024

    @1hachem

    I am having the same issue here, I tried as well using data_set.drop() but crawlee keep looking in cache and return an empty content instead of crawling the website again

  6. janbuchar commented on Jul 30, 2024

    @janbuchar
    Collaborator

    With crawlee 0.1.2, you should be able to do this:

    from __future__ import annotations
    
    import asyncio
    import uuid
    from contextlib import asynccontextmanager
    from typing import Annotated
    
    import uvicorn
    from fastapi import Depends, FastAPI
    
    from crawlee import Glob
    from crawlee.beautifulsoup_crawler import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
    from crawlee.configuration import Configuration
    from crawlee.models import Request
    
    crawler = BeautifulSoupCrawler(
        max_requests_per_crawl=500,
        configuration=Configuration(
            persist_storage=False,
        ),
    )
    
    
    @crawler.router.default_handler
    async def handler(context: BeautifulSoupCrawlingContext) -> None:
        await context.enqueue_links(
            include=[Glob('https://apify.com/store/**')], user_data={'request_id': context.request.user_data['request_id']}
        )
        await context.push_data({'url': context.request.url}, dataset_name=context.request.user_data['request_id'])
    
    
    @asynccontextmanager
    async def lifespan(app: FastAPI):
        crawler_lock = asyncio.Lock()
        app.dependency_overrides[asyncio.Lock] = lambda: crawler_lock
        yield
    
    
    api = FastAPI(lifespan=lifespan)
    
    
    @api.post('/')
    async def run_crawler(crawler_lock: Annotated[asyncio.Lock, Depends()]):
        request_id = str(uuid.uuid4())
    
        async with crawler_lock:
            await crawler.run([Request.from_url('https://apify.com/store', user_data={'request_id': request_id})])
            return await crawler.get_data(dataset_name=request_id)
    
    
    uvicorn.run(api)

    The caveats are:

    1. crawler.run cannot be called in parallel (from multiple HTTP calls) - we use FastAPI dependencies to limit this
    2. Currently, if you try to work around this limitation by making a new crawler instance for each HTTP call, only the first call will return a non-empty result
      • this is because of how the StorageManager, an internal mechanism in Crawlee, works - it is supposed to prevent problems caused by opening the same storage with different configurations
      • to make this work, you need to open a separate RequestQueue for each request, like this:
    from __future__ import annotations
    
    import uuid
    
    import uvicorn
    from fastapi import FastAPI
    
    from crawlee import Glob
    from crawlee.beautifulsoup_crawler import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
    from crawlee.configuration import Configuration
    from crawlee.models import Request
    from crawlee.storages.request_queue import RequestQueue
    
    api = FastAPI()
    
    
    @api.post('/')
    async def run_crawler():
        request_id = str(uuid.uuid4())
    
        crawler = BeautifulSoupCrawler(
            max_requests_per_crawl=500,
            configuration=Configuration(
                persist_storage=False,
            ),
            request_provider=await RequestQueue.open(name=request_id),
        )
    
        @crawler.router.default_handler
        async def handler(context: BeautifulSoupCrawlingContext) -> None:
            await context.enqueue_links(
                include=[Glob('https://apify.com/store/**')],
            )
            await context.push_data({'url': context.request.url}, dataset_name=request_id)
    
        await crawler.run([Request.from_url('https://apify.com/store')])
        return await crawler.get_data(dataset_name=request_id)
    
    
    uvicorn.run(api)    

    Let me know if this helps!

  7. GHOST-LOVE-YOU commented on Jul 31, 2024

    @GHOST-LOVE-YOU
    Author

    It does work! Thanks a lot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working.t-toolingIssues with this label are in the ownership of the tooling team.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions