You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
BeautifulSoup lacks proper type hints, mostly Any type, hence not effective IDE autocompletion. A solid alternative is Parsel. It supports CSS selectors, XPath expressions for HTML and XML, JMESPath for JSON documents, and Regex expressions. Additionally, Parsel is the parser used by Scrapy.
We started out with BeautifulSoup because of its popularity, but you're right that it has its shortcomings. Adding support for either selectolax or parsel as a new crawler type should be fairly easy - we'll consider it.
Activity
siddiqkaithodu commentedon Jul 21, 2024
I was thinking about selectolax
Ehsan-U commentedon Jul 21, 2024
selectolax doesn't support XPATH selector nor JMESPath for JSON.
janbuchar commentedon Jul 22, 2024
We started out with
BeautifulSoupbecause of its popularity, but you're right that it has its shortcomings. Adding support for either selectolax or parsel as a new crawler type should be fairly easy - we'll consider it.asymness commentedon Jul 22, 2024
+1 for Parsel
asymness commentedon Jul 22, 2024
@janbuchar, I'd like to help out by adding Parsel support as a new crawler type. Would you be open to a PR from me for this?
janbuchar commentedon Jul 22, 2024
Absolutely 🙂
feat: Implement ParselCrawler that adds support for Parsel (#348)