Skip to content

Sitemap.from_xml_string cannot opt into enqueue host filtering #2118

Closed

Description

@janbuchar

Found while triaging GHSA-c533-9w4q-667c (which I'm closing as informative - a raw XML string has no origin, so there's no host policy to bypass). It does point at a real API gap though.

  • Sitemap.from_xml_string takes only content and hardcodes SitemapSource(type='raw', content=content) - no url, no ParseSitemapOptions
  • host filtering is conditional on source.get('url')
  • so if you have raw sitemap XML and you know where it came from, there's no way to say so - filtering is silently skipped and the same-hostname default never applies
  • suggestion: from_xml_string(content, *, sitemap_url: str | None = None, parse_sitemap_options: ParseSitemapOptions | None = None), forwarded as SitemapSource(type='raw', content=content, url=sitemap_url)
  • explicitly not worth doing: rejecting <loc> values when the origin is unknown - that makes the default behaviour return an empty list
  • while in there: _get_origin_url stamps raw sources with raw://<sha256>, so SitemapUrl.origin_sitemap_url is a pseudo-URL that would fail scheme validation if anyone fed it back into filter_url - harmless today, footgun later

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Aug 4, 2026
  2. arpitjain099 commented on Aug 6, 2026

    @arpitjain099

    While tracing the advisory I noticed most of the plumbing for this already exists, in case it helps whoever picks it up.

    SitemapSource already declares url as NotRequired, and the filter in _process_sitemap_item keys on source.get('url') rather than on the source type, so a raw source that carries a url would be filtered today with no change to the filtering code. What blocks it is only that from_xml_string hardcodes the source:

    async def from_xml_string(cls, content: str) -> Sitemap:
        return await cls.parse([SitemapSource(type='raw', content=content)])

    so the shape could be as small as an optional origin plus the options passthrough that is currently missing:

    @classmethod
    async def from_xml_string(
        cls,
        content: str,
        url: str | None = None,
        parse_sitemap_options: ParseSitemapOptions | None = None,
    ) -> Sitemap:
        source = SitemapSource(type='raw', content=content)
        if url is not None:
            source['url'] = url
        return await cls.parse([source], parse_sitemap_options=parse_sitemap_options)

    Two things worth deciding before anyone writes it, since they change behaviour beyond filtering:

    • _get_origin_url() returns raw://<sha256> for any raw source. If a caller declares an origin, should SitemapUrl.origin_sitemap_url become that URL instead, or keep the content hash so raw sources stay distinguishable in output?
    • That same value is the dedup key for nested sitemaps. Switching it for url-bearing raw sources means a raw sitemap and the fetched copy of the same sitemap would dedup against each other, which may be what you want or may not.

    Happy to send the PR if you tell me which way you want those two, otherwise feel free to use any of this.

  3. added a commit that references this issue on Aug 18, 2026
    a3c20c1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    t-toolingIssues with this label are in the ownership of the tooling team.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions