> ## Documentation Index
> Fetch the complete documentation index at: https://help.decodo.com/llms.txt
> Use this file to discover all available pages before exploring further.

# LangChain (Python)

> The langchain-decodo package: Decodo scrape, search, map and crawl tools for LangChain agents, plus a document loader for RAG pipelines.

`langchain-decodo` is the Python LangChain package for the Decodo Web Data API. It gives LangChain agents six tools — scraping, two kinds of search, site mapping and crawling — and adds a document loader that turns web pages into LangChain `Document` objects for RAG pipelines.

> [!NOTE]
>
> View the [**langchain repository**](https://github.com/Decodo/langchain/tree/master/libs/partners/decodo) for the source code and latest releases. Building in Node.js? See [**LangChain (TypeScript)**](https://help.decodo.com/docs/langchain.md).

## Features

* **Web scraping** – scrape any URL and get the page back as Markdown
* **Parsed search** – Google Search and Amazon Search with structured results
* **Web search** – search the web and get ranked results with title, link and description
* **Site mapping** – list the URLs on a website without fetching their content
* **Crawling** – crawl a website and retrieve the content of every page found
* **Document loader** – load one or more URLs as LangChain `Document` objects for RAG
* **LangChain integration** – standard `BaseTool` and `BaseLoader` classes that plug into any agent or chain

## Requirements

* Python 3.10+
* A Decodo [**Web Data API plan**](https://dashboard.decodo.com/web-data/pricing). Free trials are available on the [**dashboard**](https://dashboard.decodo.com/).

## Installation

```bash theme={null}
pip install langchain-decodo
```

## Authentication

Copy your Web Data API key from the [**Web Data API playground**](https://dashboard.decodo.com/web-data/playground) and store it in the `DECODO_API_TOKEN` environment variable. Every class in the package reads it from there:

```bash theme={null}
export DECODO_API_TOKEN=your_api_key
```

How you then pass it depends on which group the class belongs to.

### Scrape tools and the loader

`DecodoWebScrapeTool`, `DecodoSearchTool` and `DecodoLoader` support two credential types, chosen with `auth_mode`. Pass `auth_mode="token"` to use an API key:

```python theme={null}
from langchain_decodo import DecodoWebScrapeTool

tool = DecodoWebScrapeTool(auth_mode="token")
```

> [!WARNING]
>
> `auth_mode` defaults to `"basic"`, not `"token"`. If you leave it out, the API key is sent as a basic authentication token and every request fails with `Decodo API error: Username invalid.`

Older plans only have a basic authentication token (a base64-encoded `username:password`). Put that in `DECODO_API_TOKEN` and leave `auth_mode` out, since `"basic"` is the default.

You can also pass the credential directly instead of using the environment variable: `decodo_api_token=` on the two tools, `api_token=` on the loader.

### Web Data API tools

`DecodoWebSearchTool`, `DecodoMapTool`, `DecodoCrawlTool` and `DecodoCrawlCancelTool` call endpoints that exist only on `data.decodo.com`, which accepts API keys and nothing else. These four have **no `auth_mode` argument** and do not work with a basic authentication token. They read `DECODO_API_TOKEN` from the environment, or take `decodo_api_token=` directly:

```python theme={null}
from langchain_decodo import DecodoWebSearchTool

tool = DecodoWebSearchTool()  # reads the API key from DECODO_API_TOKEN
```

Construct one without a key and it raises `ValueError: A Decodo Web Data API key is required.` Give it a basic authentication token and the API rejects the request, with an error saying that basic authentication tokens from older plans do not work here.

## Quick start

A LangChain agent that searches the web, picks a page and scrapes it. This example uses OpenAI through `langchain-openai`, but any chat model LangChain supports works:

```bash theme={null}
pip install langchain-decodo langchain langchain-openai
export DECODO_API_TOKEN=your_api_key
export OPENAI_API_KEY=your_openai_key
```

```python theme={null}
# main.py
from langchain.agents import create_agent
from langchain_decodo import DecodoWebScrapeTool, DecodoWebSearchTool

agent = create_agent(
    model="openai:gpt-5-mini",
    tools=[
        DecodoWebSearchTool(),
        DecodoWebScrapeTool(auth_mode="token"),
    ],
)

result = agent.invoke(
    {
        "messages": [
            {
                "role": "user",
                "content": "Find out who won the 2025 NBA Finals, then scrape the page you find and summarise it.",
            }
        ]
    }
)

print(result["messages"][-1].content)
```

Note the mixed authentication in that example: `DecodoWebSearchTool` takes no `auth_mode`, while `DecodoWebScrapeTool` needs `auth_mode="token"`. Both read the same `DECODO_API_TOKEN`.

## Available tools

| Class | Tool name exposed to the agent | Purpose |
| - | - | - |
| `DecodoWebScrapeTool` | `decodo_scrape_url` | Scrape any URL and return Markdown |
| `DecodoSearchTool` | `decodo_search` | Google Search or Amazon Search with parsed results |
| `DecodoWebSearchTool` | `decodo_web_search` | Search the web and return ranked results |
| `DecodoMapTool` | `decodo_map` | List the URLs on a website, 100 by default and up to 500 |
| `DecodoCrawlTool` | `decodo_crawl` | Crawl a website and return every page, 10 by default and up to 100 |
| `DecodoCrawlCancelTool` | `decodo_crawl_cancel` | Stop a running crawl |

The tools fall into two groups, which differ in how they authenticate and what they return:

* **Scrape tools** – `DecodoWebScrapeTool` and `DecodoSearchTool`. They scrape a page or a search engine's results page through the scrape endpoint, and accept either credential type.
* **Web Data API tools** – `DecodoWebSearchTool`, `DecodoMapTool`, `DecodoCrawlTool` and `DecodoCrawlCancelTool`. Each has its own input schema, and all four require an API key.

`DecodoLoader` is not a tool. It is a LangChain document loader, covered in [Document loader](#document-loader) below.

You can call any tool directly, without an agent:

```python theme={null}
from langchain_decodo import DecodoMapTool, DecodoWebScrapeTool

scrape = DecodoWebScrapeTool(auth_mode="token")
markdown = scrape.invoke("https://quotes.toscrape.com")

site_map = DecodoMapTool()
urls = site_map.invoke({"url": "https://quotes.toscrape.com", "limit": 20})
```

## Scrape tools

### `DecodoWebScrapeTool`

Takes a single `url`, which must include `http://` or `https://`. It calls the `universal` target with `markdown: true` and returns the page as a Markdown string. When the API returns no content, the tool returns the literal string `(No content returned by Decodo API)` rather than raising.

| Constructor argument | Default | Description |
| - | - | - |
| `decodo_api_token` | `DECODO_API_TOKEN` | API key, or basic authentication token on older plans |
| `auth_mode` | `"basic"` | Set to `"token"` for an API key |
| `base_url` | – | Overrides the host that `auth_mode` selects. Rarely needed |

### `DecodoSearchTool`

Takes these inputs:

| Input | Default | Description |
| - | - | - |
| `query` | – | Search query. Required |
| `engine` | `google` | `google` calls the `google_search` target, `amazon` calls `amazon_search` |
| `num_results` | `10` | Sent to the API as `limit`. Accepts 1 to 100 |

It always sends `parse: true` and returns a JSON string: a list with one entry per result, each holding `content`, `status_code` and `url`. The parsed results are in `content`, for example the Google organic results at `content.results.results.organic`.

If the search fails, the tool raises a `RuntimeError` with the status code. A failed search can come back from the API as HTTP 200, so raising lets the agent retry instead of treating it as "no results".

> [!NOTE]
>
> Pass `engine` with each call, as in the example above. Setting it when you create the tool (`DecodoSearchTool(engine="amazon")`) has no effect, and the search still goes to Google.

> [!NOTE]
>
> An `engine` value the tool does not recognise is not rejected. It falls back to Google.

> [!NOTE]
>
> `num_results` doesn't cap the results returned. A Google search with `num_results` set to 3 still returns the full first page of organic results.

## Web Data API tools

### `DecodoWebSearchTool`

Searches the web and returns one normalized result document, rather than scraping a particular engine's results page the way `DecodoSearchTool` does.

| Input | Default | Description |
| - | - | - |
| `query` | – | Search query. Required |
| `geo` | – | Country to send the request from, as a full name, for example `United States` |
| `locale` | – | Interface language, for example `en-US` |
| `page` | `1` | Results page to return. Must be 1 or higher |

Returns a JSON string. The ranked results are under `organic`, each with a title, link and description, alongside `top_stories`, `people_also_ask` and `related_searches` when the search has them. To read a result, scrape its link with `decodo_scrape_url`.

A search the API reports as failed raises `RuntimeError: Decodo search failed: ...`.

### `DecodoMapTool`

Lists the URLs on a website without fetching their content — useful for narrowing a large site before scraping only the pages you need.

`DecodoMapTool` and `DecodoCrawlTool` both start from a URL and discover pages from there, so they share these inputs:

| Input | Default | Description |
| - | - | - |
| `url` | – | Absolute http or https URL of the site to start from. Required |
| `sitemap` | – | `include`, `only` or `skip` – whether to use the site's sitemap alongside its links, only the sitemap, or skip it |
| `domain_filter` | – | `domain`, `subdomain` or `all` – which hosts count as part of the site |
| `select_paths` | – | Regular expressions matched against URL paths. Only URLs matching one of them are kept |
| `exclude_paths` | – | Regular expressions matched against URL paths. URLs matching one of them are dropped |
| `geo` | – | Country to send the request from, as a full name |
| `locale` | – | Interface language, for example `en-US` |

`DecodoMapTool` adds:

| Input | Default | Description |
| - | - | - |
| `limit` | `100` | Maximum number of URLs to return. Accepts 1 to 500 |
| `js_render` | `False` | When `True`, renders pages with JavaScript to find links. Slower, so turn it on only when a site needs it |

```python theme={null}
from langchain_decodo import DecodoMapTool

tool = DecodoMapTool()
result = tool.invoke(
    {
        "url": "https://www.python.org",
        "select_paths": ["^/downloads/"],
        "limit": 50,
    }
)
```

Returns the API's map response as a JSON string. If the site could not be mapped — for example when it blocks the request — the tool raises `RuntimeError: Decodo map failed (status ...)`.

### `DecodoCrawlTool`

Crawls a website from a start URL and returns the content of every page found. Takes the shared site-discovery inputs above, plus:

| Input | Default | Description |
| - | - | - |
| `limit` | `10` | Maximum number of pages to crawl. Accepts 1 to 100 |
| `max_depth` | – | How many links deep to follow from the start URL. Accepts 1 to 10000 |
| `output` | `markdown` | `markdown` or `html` – format of the page content |

Unlike the other tools, it also takes two constructor arguments that control how long it waits:

| Constructor argument | Default | Description |
| - | - | - |
| `poll_interval` | `5.0` | Seconds between status checks |
| `timeout` | `600.0` | Seconds to wait for the crawl before giving up |

```python theme={null}
from langchain_decodo import DecodoCrawlTool

tool = DecodoCrawlTool(timeout=300, poll_interval=5)
result = tool.invoke({"url": "https://example.com", "limit": 10, "max_depth": 2})
```

> [!TIP]
>
> On a large site, narrow the crawl with `select_paths` or `exclude_paths` first, or use `DecodoMapTool` to list the URLs and then scrape only the pages you need.

#### Crawl behaviour

The tool waits until the crawl finishes, then returns a JSON string shaped like `{"status": {...}, "pages": [...]}`. `status` is the crawl's status response, including its `crawl_id`, and each entry in `pages` is a page as the API returns it, with its own status.

> [!WARNING]
>
> A crawl that ends `faulted` or `cancelled`, or is still `pending` when `timeout` runs out, returns the pages collected so far instead of raising. Check `status.status` in the result: anything other than `done` means the crawl stopped early and `pages` holds only part of the site.

`status` is `null` when it could not be read, for example when the crawl collected no pages at all.

`timeout` is also sent to the API as the crawl's `max_duration`, capped at the API's four-hour limit, so Decodo stops a crawl the tool stopped waiting for. A crawl still running when `timeout` runs out comes back as `pending` with the pages collected so far, and Decodo stops it at the same limit shortly after. To stop it sooner, pass `status.crawl_id` to `DecodoCrawlCancelTool`.

### `DecodoCrawlCancelTool`

Stops a running crawl, so it crawls no more pages.

| Input | Default | Description |
| - | - | - |
| `crawl_id` | – | Id of the crawl to stop: `status.crawl_id` from a `decodo_crawl` result. Required |

```python theme={null}
import json

from langchain_decodo import DecodoCrawlCancelTool, DecodoCrawlTool

result = json.loads(DecodoCrawlTool(timeout=60).invoke({"url": "https://example.com"}))
status = result["status"]
if status and status["status"] == "pending":
    DecodoCrawlCancelTool().invoke({"crawl_id": status["crawl_id"]})
```

Register it alongside `DecodoCrawlTool` so an agent can stop a crawl it stopped waiting for. Anything that is not a crawl id raises `ValueError: Not a crawl id: ...` before a request is sent.

## Document loader

`DecodoLoader` scrapes one or more URLs and returns each page as a LangChain `Document`, ready to split, embed and store in a RAG pipeline:

```python theme={null}
from langchain_decodo import DecodoLoader

loader = DecodoLoader(
    urls=["https://quotes.toscrape.com", "https://books.toscrape.com"],
    auth_mode="token",
)

for doc in loader.lazy_load():
    print(doc.metadata["url"], doc.metadata["status_code"], len(doc.page_content))
```

Each `Document` holds the page as Markdown in `page_content`. Its `metadata` has `source`, `url` and `status_code`, plus `task_id`, `created_at` and `updated_at` when the API returns them. URLs are scraped one at a time, in order. Use `load()` for a list of every document, or `lazy_load()` to process them as they arrive.

| Argument | Default | Description |
| - | - | - |
| `urls` | – | A single URL or a list of URLs. Required |
| `api_token` | `DECODO_API_TOKEN` | API key, or basic authentication token on older plans |
| `auth_mode` | `"basic"` | Set to `"token"` for an API key |
| `timeout` | `180` | Request timeout in seconds, per URL |
| `continue_on_error` | `True` | Keep loading the other URLs when one fails |

With `continue_on_error` on, a failed URL still produces a `Document`, with empty `page_content` and the error message in `metadata["error"]`. Filter those out before embedding. Set `continue_on_error=False` to raise on the first failure instead.

## Error handling

Every tool raises `RuntimeError` for API errors, timeouts and network failures, with the message prefixed by what went wrong, for example `Decodo API error: ...` or `Decodo API network error: ...`. The four Web Data API tools also raise it for a response that is not JSON.

Two cases raise `ValueError` instead, before any request is sent: a missing API key, and a `crawl_id` that is not a crawl id.

> [!NOTE]
>
> An authentication failure on one of the four Web Data API tools returns an error that names the likely cause — an invalid key, a plan without that endpoint, or a basic authentication token used where only an API key works.

For the full list of API parameters, see [**Parameters**](https://help.decodo.com/docs/web-scraping-api-parameters-overview.md).

***

## Related

* [**LangChain (TypeScript)**](https://help.decodo.com/docs/langchain.md) – The Node.js LangChain tools.
* [**Python SDK**](https://help.decodo.com/docs/python-sdk.md) – Typed access to every Decodo target from Python.
* [**LlamaIndex**](https://help.decodo.com/docs/llamaindex.md) – Decodo readers and agent tools for LlamaIndex.
* [**CrewAI**](https://help.decodo.com/docs/crewai.md) – Decodo tools for CrewAI agents and crews.
* [**MCP Server**](https://help.decodo.com/docs/mcp.md) – Connect AI agents and MCP-compatible tools to Decodo.
* [GitHub repository](https://github.com/Decodo/langchain/tree/master/libs/partners/decodo) – View the source code and latest releases.

***

> [card] [Support](https://direct.lc.chat/12092754)
>
> Need help or just want to say hello? Our support is available 24/7. \
> You can also reach us anytime via email at [support@decodo.com](mailto:support@decodo.com).

> [card] [Feedback](mailto:feedback@decodo.com)
>
> Can't find what you're looking for? Request an article! \
> Have feedback? Share your thoughts on how we can improve.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.