Skip to content

Request fetching from RequestQueue is sometimes very slow #203

Closed

Description

@vdusek
  • Fetching requests from RequestQueue is sometimes very slow and can get stuck for a while.
  • I turned on logging and reproduced the issue with the following code:
import asyncio
import logging

from crawlee.beautifulsoup_crawler import BeautifulSoupCrawler, BeautifulSoupCrawlingContext

logging.basicConfig(level=logging.INFO)


async def main() -> None:
    crawler = BeautifulSoupCrawler()

    @crawler.router.default_handler
    async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
        await context.enqueue_links(strategy='same-hostname')
        data = {
            'request_url': context.request.url,
            'soup_url': context.soup.url,
            'soup_title': context.soup.title.string if context.soup.title else None,
        }
        await context.push_data(data)

    await crawler.run(['https://crawlee.dev'])


if __name__ == '__main__':
    asyncio.run(main())
  • In the logs, there are many lines like this:
INFO:crawlee.storages.request_queue:Waiting for 9.988466 for queue finalization, to ensure data consistency.

Questions

  • Is this behavior correct?
  • Is the waiting period necessary?
  • Is it necessary for memory storage as well?

Activity

  1. added
    bugSomething isn't working.
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Jun 20, 2024
  2. self-assigned this
    on Jun 24, 2024
  3. vdusek commented on Jun 26, 2024

    @vdusek
    CollaboratorAuthor

    In #235 we set ensure_consistency to False as a hotfix in the request queue.

    The root cause of this behavior is in #186, before that, the RQ works fine, without waiting. Let's deep dive into that after the public launch and documentation is ready.

    Code to reproduce it:

    import asyncio
    import logging
    
    from crawlee.beautifulsoup_crawler import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
    from crawlee.storages import Dataset
    from crawlee.log_config import CrawleeLogFormatter
    
    # Configuration the logging
    handler = logging.StreamHandler()
    handler.setFormatter(CrawleeLogFormatter(include_logger_name=True))
    root_logger = logging.getLogger()
    root_logger.setLevel(logging.INFO)
    root_logger.addHandler(handler)
    
    
    async def main() -> None:
        crawler = BeautifulSoupCrawler()
        dataset = await Dataset.open()
    
        @crawler.router.default_handler
        async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
            await context.enqueue_links()
            data = {
                'url': context.request.url,
                'title': context.soup.title.string if context.soup.title else '',
            }
            await dataset.push_data(data)
    
        await crawler.run(['https://crawlee.dev/'])
    
    
    if __name__ == '__main__':
        asyncio.run(main())
  4. modified the milestones: 92nd sprint - Tooling team, on Jun 27, 2024
  5. removed this from the milestone on Jul 15, 2024
  6. assigned and unassigned on Aug 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions