Skip to content

fix(crawlers): Parse HTML responses starting with an XML declaration as HTML in ParselCrawler - #2262

Merged
vdusek merged 1 commit into
apify:masterfrom
Mantisus:fix-parsel-112
Oct 1, 2026
Merged

vdusek merged 1 commit into
apify:masterfrom
Mantisus:fix-parsel-112

Conversation

@Mantisus

Copy link
Copy Markdown
Collaborator

Description

  • ParselCrawler now parses text/html and application/xhtml+xml responses as HTML, even when the page starts with an XML declaration. Since parsel 1.12, such pages were parsed as XML, so CSS selectors and enqueue_links found nothing on XHTML pages.

@Mantisus Mantisus self-assigned this Sep 30, 2026
@Mantisus
Mantisus requested a review from Pijukatel September 30, 2026 15:34
@vdusek
vdusek merged commit df1178c into apify:master Oct 1, 2026
34 checks passed
Pijukatel pushed a commit that referenced this pull request Oct 2, 2026
…ponses in `ParselCrawler` (#2266)

### Description

`ParselCrawler` detects a JSON body served as `text/html` as JSON again,
so `context.selector.jmespath` works on it. HTML responses starting with
an XML declaration are still parsed as HTML.

An empty response body no longer fails the request. It's parsed as an
empty HTML document.

Follow-up to #2262

### Testing

- Added new regression tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants