You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add support for AI/LLM-based HTML parsing (selectors) #1593
At recent events I attended, I was asked about AI/LLM-based HTML parsing. I also found a few dedicated AI-based scraping frameworks, such as ScrapeGraphAI and Parsera, that appear to be gaining traction.
Right now, we provide an AI-selector workflow only through the PlaywrightCrawler via Stagehand guide.
This means:
AI-based selectors are supported only for Playwright, not for HTTP-based crawlers.
Even for PlaywrightCrawler, the integration is not very smooth compared to the tools mentioned above.
Example from the ScrapeGraphAI:
# Create the SmartScraperGraph instancesmart_scraper_graph=SmartScraperGraph(
prompt="Extract useful information from the webpage, including a description of what the company does, founders and social media links",
source="https://scrapegraphai.com/",
config=graph_config
)
# Run the pipelineresult=smart_scraper_graph.run()
It might be worth exploring a more native solution:
Better Stagehand integration so that AI-based selectors in Playwright crawlers are as straightforward as in the dedicated AI-scraping libraries.
Introduce an AI/LLM-powered crawler built on top of AbstractHttpCrawler, enabling AI/LLM selectors for HTTP-based scraping as well.
This could make Crawlee more usable for AI/LLM-based extractions, and/or for faster prototype scrapers without manual CSS/XPath selectors.
This could make Crawlee more usable for AI/LLM-based extractions, and/or for faster prototype scrapers without manual CSS/XPath selectors.
If you want to avoid writing selectors, I'd argue that using a coding assistant with Playwright MCP will get you a faster, cheaper crawler. However, if you're aiming for resilience in the face of changing markup, or if the target website is trash and there's no usable structure, just piping the whole thing into an LLM might make sense.
Activity
janbuchar commentedon Dec 2, 2025
If you want to avoid writing selectors, I'd argue that using a coding assistant with Playwright MCP will get you a faster, cheaper crawler. However, if you're aiming for resilience in the face of changing markup, or if the target website is trash and there's no usable structure, just piping the whole thing into an LLM might make sense.
feat: Add PydanticAiCrawler with AI-powered HTML extraction (apify#1964)