# SitemapRequestLoader<!-- -->

A request loader that reads URLs from sitemap(s).

The loader is designed to handle sitemaps that follow the format described in the Sitemaps protocol (<https://www.sitemaps.org/protocol.html>). It supports both XML and plain text sitemap formats. Note that HTML pages containing links are not supported - those should be handled by regular crawlers and the `enqueue_links` functionality.

The loader fetches and parses sitemaps in the background, allowing crawling to start before all URLs are loaded. It supports filtering URLs using glob and regex patterns.

The loader supports state persistence, allowing it to resume from where it left off after interruption when a `persist_state_key` is provided during initialization.

### Hierarchy

* [RequestLoader](https://crawlee.dev/python/api/class/RequestLoader)
  * *SitemapRequestLoader*

## Index[](#Index)

### Methods

* [\_\_aenter\_\_](https://docs.apify.com/sdk/python/reference/class/SitemapRequestLoader.md#__aenter__)
* [\_\_aexit\_\_](https://docs.apify.com/sdk/python/reference/class/SitemapRequestLoader.md#__aexit__)
* [\_\_init\_\_](https://docs.apify.com/sdk/python/reference/class/SitemapRequestLoader.md#__init__)
* [abort_loading](https://docs.apify.com/sdk/python/reference/class/SitemapRequestLoader.md#abort_loading)
* [close](https://docs.apify.com/sdk/python/reference/class/SitemapRequestLoader.md#close)
* [fetch_next_request](https://docs.apify.com/sdk/python/reference/class/SitemapRequestLoader.md#fetch_next_request)
* [get_handled_count](https://docs.apify.com/sdk/python/reference/class/SitemapRequestLoader.md#get_handled_count)
* [get_total_count](https://docs.apify.com/sdk/python/reference/class/SitemapRequestLoader.md#get_total_count)
* [is_empty](https://docs.apify.com/sdk/python/reference/class/SitemapRequestLoader.md#is_empty)
* [is_finished](https://docs.apify.com/sdk/python/reference/class/SitemapRequestLoader.md#is_finished)
* [mark_request_as_handled](https://docs.apify.com/sdk/python/reference/class/SitemapRequestLoader.md#mark_request_as_handled)
* [start](https://docs.apify.com/sdk/python/reference/class/SitemapRequestLoader.md#start)
* [to_tandem](https://docs.apify.com/sdk/python/reference/class/SitemapRequestLoader.md#to_tandem)

## Methods<!-- -->[](#Methods)

### [](#__aenter__)[](https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/request_loaders/_sitemap_request_loader.py#L181)\_\_aenter\_\_

* async **\_\_aenter\_\_**(): [SitemapRequestLoader](https://crawlee.dev/python/api/class/SitemapRequestLoader)

- Enter the context manager.

  ***

  #### Returns [SitemapRequestLoader](https://crawlee.dev/python/api/class/SitemapRequestLoader)

### [](#__aexit__)[](https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/request_loaders/_sitemap_request_loader.py#L186)\_\_aexit\_\_

* async **\_\_aexit\_\_**(exc_type, exc_value, exc_traceback): None

- Exit the context manager.

  ***

  #### Parameters

  * ##### exc_type: [type](https://crawlee.dev/python/api/class/SitemapSource#type)\[BaseException] | None
  * ##### exc_value: BaseException | None
  * ##### exc_traceback: TracebackType | None

  #### Returns None

### [](#__init__)[](https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/request_loaders/_sitemap_request_loader.py#L119)\_\_init\_\_

* **\_\_init\_\_**(sitemap_urls, http_client, \*, proxy_info, include, exclude, enqueue_strategy, max_buffer_size, persist_state_key, transform_request_function): None

- Initialize the sitemap request loader.

  ***

  #### Parameters

  * ##### sitemap_urls: list\[str]

    List of sitemap URLs to fetch and parse for request URLs.

  * ##### http_client: [HttpClient](https://crawlee.dev/python/api/class/HttpClient)

    the instance of `HttpClient` to use for fetching sitemaps.

  * ##### optionalkeyword-onlyproxy_info: [ProxyInfo](https://crawlee.dev/python/api/class/ProxyInfo) | None = <!-- -->None

    Optional proxy to use for fetching sitemaps.

  * ##### optionalkeyword-onlyinclude: list\[re.Pattern\[Any] | [Glob](https://crawlee.dev/python/api/class/Glob)] | None = <!-- -->None

    List of glob or regex patterns to include URLs.

  * ##### optionalkeyword-onlyexclude: list\[re.Pattern\[Any] | [Glob](https://crawlee.dev/python/api/class/Glob)] | None = <!-- -->None

    List of glob or regex patterns to exclude URLs.

  * ##### optionalkeyword-onlyenqueue_strategy: [EnqueueStrategy](https://crawlee.dev/python/api#EnqueueStrategy) = <!-- -->'same-hostname'

    Strategy used to decide which sitemap-derived URLs (both nested-sitemap entries and URL entries) are kept relative to the parent sitemap URL. Defaults to `'same-hostname'`, matching the sitemap protocol's same-host expectation and the `enqueue_links` default; pass `'all'` to disable filtering. Note: regardless of `enqueue_strategy`, entries with non-`http(s)` schemes are always filtered out.

  * ##### optionalkeyword-onlymax_buffer_size: int = <!-- -->200

    Maximum number of URLs to buffer in memory.

  * ##### optionalkeyword-onlypersist_state_key: str | None = <!-- -->None

    A key for persisting the loader's state in the KeyValueStore. When provided, allows resuming from where it left off after interruption. If None, no state persistence occurs.

  * ##### optionalkeyword-onlytransform_request_function: Callable\[\[RequestOptions], [RequestOptions](https://crawlee.dev/python/api/class/RequestOptions) | [RequestTransformAction](https://crawlee.dev/python/api#RequestTransformAction)] | None = <!-- -->None

    An optional function to transform requests generated by the loader. It receives `RequestOptions` with `url` and should return either modified `RequestOptions` or a `RequestTransformAction`.

  #### Returns None

### [](#abort_loading)[](https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/request_loaders/_sitemap_request_loader.py#L269)abort_loading

* async **abort_loading**(): None

- Abort the sitemap loading process.

  ***

  #### Returns None

### [](#close)[](https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/request_loaders/_sitemap_request_loader.py#L276)close

* async **close**(): None

- Close the request loader.

  ***

  #### Returns None

### [](#fetch_next_request)[](https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/request_loaders/_sitemap_request_loader.py#L220)fetch_next_request

* async **fetch_next_request**(): [Request](https://crawlee.dev/python/api/class/Request) | None

- Overrides [RequestLoader.fetch_next_request](https://crawlee.dev/python/api/class/RequestLoader#fetch_next_request)

  Fetch the next request to process.

  ***

  #### Returns [Request](https://crawlee.dev/python/api/class/Request) | None

### [](#get_handled_count)[](https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/request_loaders/_sitemap_request_loader.py#L202)get_handled_count

* async **get_handled_count**(): int

- Overrides [RequestLoader.get_handled_count](https://crawlee.dev/python/api/class/RequestLoader#get_handled_count)

  Return the number of URLs that have been handled.

  ***

  #### Returns int

### [](#get_total_count)[](https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/request_loaders/_sitemap_request_loader.py#L196)get_total_count

* async **get_total_count**(): int

- Overrides [RequestLoader.get_total_count](https://crawlee.dev/python/api/class/RequestLoader#get_total_count)

  Return the total number of URLs found so far.

  ***

  #### Returns int

### [](#is_empty)[](https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/request_loaders/_sitemap_request_loader.py#L208)is_empty

* async **is_empty**(): bool

- Overrides [RequestLoader.is_empty](https://crawlee.dev/python/api/class/RequestLoader#is_empty)

  Check if there are no more URLs to process.

  ***

  #### Returns bool

### [](#is_finished)[](https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/request_loaders/_sitemap_request_loader.py#L214)is_finished

* async **is_finished**(): bool

- Overrides [RequestLoader.is_finished](https://crawlee.dev/python/api/class/RequestLoader#is_finished)

  Check if all URLs have been processed.

  ***

  #### Returns bool

### [](#mark_request_as_handled)[](https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/request_loaders/_sitemap_request_loader.py#L255)mark_request_as_handled

* async **mark_request_as_handled**(request): [ProcessedRequest](https://crawlee.dev/python/api/class/ProcessedRequest) | None

- Overrides [RequestLoader.mark_request_as_handled](https://crawlee.dev/python/api/class/RequestLoader#mark_request_as_handled)

  Mark a request as successfully handled.

  ***

  #### Parameters

  * ##### request: [Request](https://crawlee.dev/python/api/class/Request)

  #### Returns [ProcessedRequest](https://crawlee.dev/python/api/class/ProcessedRequest) | None

### [](#start)[](https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/request_loaders/_sitemap_request_loader.py#L263)start

* async **start**(): None

- Start the sitemap loading process.

  ***

  #### Returns None

### [](#to_tandem)[](https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/request_loaders/_request_loader.py#L56)to_tandem

* async **to_tandem**(request_manager): [RequestManagerTandem](https://crawlee.dev/python/api/class/RequestManagerTandem)

- Inherited from [RequestLoader.to_tandem](https://crawlee.dev/python/api/class/RequestLoader#to_tandem)

  Combine the loader with a request manager to support adding and reclaiming requests.

  ***

  #### Parameters

  * ##### optionalrequest_manager: RequestManager | None = <!-- -->None

    Request manager to combine the loader with. If None is given, the default request queue is used.

  #### Returns [RequestManagerTandem](https://crawlee.dev/python/api/class/RequestManagerTandem)
