Skip to content

Automate default logging configuration for crawlers #214

Closed

Description

@vdusek

Currently, when a user runs their crawler, no log is printed, leaving users unaware of the crawler's progress and actions.

We have a log formatter already ready, but users need to configure it manually. Like this:

import asyncio
import logging

from crawlee.beautifulsoup_crawler import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from crawlee.log_config import CrawleeLogFormatter

# Configuration the logging
handler = logging.StreamHandler()
handler.setFormatter(CrawleeLogFormatter(include_logger_name=True))
root_logger = logging.getLogger()
root_logger.setLevel(logging.INFO)
root_logger.addHandler(handler)


async def main() -> None:
    crawler = BeautifulSoupCrawler()

    @crawler.router.default_handler
    async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
        await context.enqueue_links()
        data = {
            'url': context.request.url,
            'title': context.soup.title.string,
        }
        await context.push_data(data)

    await crawler.run(['https://crawlee.dev'])


if __name__ == '__main__':
    asyncio.run(main())

Can we make this setup a default? Possible solution: importing any module from Crawlee should configure the root logger automatically.

Activity

  1. added
    enhancementNew feature or request.
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Jun 24, 2024
  2. changed the title [-]Set up default logging[/-] [+]Automate default logging configuration for crawlers[/+] on Jun 24, 2024
  3. self-assigned this
    on Jun 24, 2024
  4. added a commit that references this issue on Jun 25, 2024
    a67b72f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions