Skip to main content
langchain-decodo is the Python LangChain package for the Decodo Web Data API. It gives LangChain agents six tools — scraping, two kinds of search, site mapping and crawling — and adds a document loader that turns web pages into LangChain Document objects for RAG pipelines.
View the langchain repository for the source code and latest releases. Building in Node.js? See LangChain (TypeScript).

Features

  • Web scraping – scrape any URL and get the page back as Markdown
  • Parsed search – Google Search and Amazon Search with structured results
  • Web search – search the web and get ranked results with title, link and description
  • Site mapping – list the URLs on a website without fetching their content
  • Crawling – crawl a website and retrieve the content of every page found
  • Document loader – load one or more URLs as LangChain Document objects for RAG
  • LangChain integration – standard BaseTool and BaseLoader classes that plug into any agent or chain

Requirements

Installation

Authentication

Copy your Web Data API key from the Web Data API playground and store it in the DECODO_API_TOKEN environment variable. Every class in the package reads it from there:
How you then pass it depends on which group the class belongs to.

Scrape tools and the loader

DecodoWebScrapeTool, DecodoSearchTool and DecodoLoader support two credential types, chosen with auth_mode. Pass auth_mode="token" to use an API key:
auth_mode defaults to "basic", not "token". If you leave it out, the API key is sent as a basic authentication token and every request fails with Decodo API error: Username invalid.
Older plans only have a basic authentication token (a base64-encoded username:password). Put that in DECODO_API_TOKEN and leave auth_mode out, since "basic" is the default. You can also pass the credential directly instead of using the environment variable: decodo_api_token= on the two tools, api_token= on the loader.

Web Data API tools

DecodoWebSearchTool, DecodoMapTool, DecodoCrawlTool and DecodoCrawlCancelTool call endpoints that exist only on data.decodo.com, which accepts API keys and nothing else. These four have no auth_mode argument and do not work with a basic authentication token. They read DECODO_API_TOKEN from the environment, or take decodo_api_token= directly:
Construct one without a key and it raises ValueError: A Decodo Web Data API key is required. Give it a basic authentication token and the API rejects the request, with an error saying that basic authentication tokens from older plans do not work here.

Quick start

A LangChain agent that searches the web, picks a page and scrapes it. This example uses OpenAI through langchain-openai, but any chat model LangChain supports works:
Note the mixed authentication in that example: DecodoWebSearchTool takes no auth_mode, while DecodoWebScrapeTool needs auth_mode="token". Both read the same DECODO_API_TOKEN.

Available tools

The tools fall into two groups, which differ in how they authenticate and what they return:
  • Scrape tools – DecodoWebScrapeTool and DecodoSearchTool. They scrape a page or a search engine’s results page through the scrape endpoint, and accept either credential type.
  • Web Data API tools – DecodoWebSearchTool, DecodoMapTool, DecodoCrawlTool and DecodoCrawlCancelTool. Each has its own input schema, and all four require an API key.
DecodoLoader is not a tool. It is a LangChain document loader, covered in Document loader below. You can call any tool directly, without an agent:

Scrape tools

DecodoWebScrapeTool

Takes a single url, which must include http:// or https://. It calls the universal target with markdown: true and returns the page as a Markdown string. When the API returns no content, the tool returns the literal string (No content returned by Decodo API) rather than raising.

DecodoSearchTool

Takes these inputs: It always sends parse: true and returns a JSON string: a list with one entry per result, each holding content, status_code and url. The parsed results are in content, for example the Google organic results at content.results.results.organic. If the search fails, the tool raises a RuntimeError with the status code. A failed search can come back from the API as HTTP 200, so raising lets the agent retry instead of treating it as “no results”.
Pass engine with each call, as in the example above. Setting it when you create the tool (DecodoSearchTool(engine="amazon")) has no effect, and the search still goes to Google.
An engine value the tool does not recognise is not rejected. It falls back to Google.
num_results doesn’t cap the results returned. A Google search with num_results set to 3 still returns the full first page of organic results.

Web Data API tools

DecodoWebSearchTool

Searches the web and returns one normalized result document, rather than scraping a particular engine’s results page the way DecodoSearchTool does. Returns a JSON string. The ranked results are under organic, each with a title, link and description, alongside top_stories, people_also_ask and related_searches when the search has them. To read a result, scrape its link with decodo_scrape_url. A search the API reports as failed raises RuntimeError: Decodo search failed: ....

DecodoMapTool

Lists the URLs on a website without fetching their content — useful for narrowing a large site before scraping only the pages you need. DecodoMapTool and DecodoCrawlTool both start from a URL and discover pages from there, so they share these inputs: DecodoMapTool adds:
Returns the API’s map response as a JSON string. If the site could not be mapped — for example when it blocks the request — the tool raises RuntimeError: Decodo map failed (status ...).

DecodoCrawlTool

Crawls a website from a start URL and returns the content of every page found. Takes the shared site-discovery inputs above, plus: Unlike the other tools, it also takes two constructor arguments that control how long it waits:
On a large site, narrow the crawl with select_paths or exclude_paths first, or use DecodoMapTool to list the URLs and then scrape only the pages you need.

Crawl behaviour

The tool waits until the crawl finishes, then returns a JSON string shaped like {"status": {...}, "pages": [...]}. status is the crawl’s status response, including its crawl_id, and each entry in pages is a page as the API returns it, with its own status.
A crawl that ends faulted or cancelled, or is still pending when timeout runs out, returns the pages collected so far instead of raising. Check status.status in the result: anything other than done means the crawl stopped early and pages holds only part of the site.
status is null when it could not be read, for example when the crawl collected no pages at all. timeout is also sent to the API as the crawl’s max_duration, capped at the API’s four-hour limit, so Decodo stops a crawl the tool stopped waiting for. A crawl still running when timeout runs out comes back as pending with the pages collected so far, and Decodo stops it at the same limit shortly after. To stop it sooner, pass status.crawl_id to DecodoCrawlCancelTool.

DecodoCrawlCancelTool

Stops a running crawl, so it crawls no more pages.
Register it alongside DecodoCrawlTool so an agent can stop a crawl it stopped waiting for. Anything that is not a crawl id raises ValueError: Not a crawl id: ... before a request is sent.

Document loader

DecodoLoader scrapes one or more URLs and returns each page as a LangChain Document, ready to split, embed and store in a RAG pipeline:
Each Document holds the page as Markdown in page_content. Its metadata has source, url and status_code, plus task_id, created_at and updated_at when the API returns them. URLs are scraped one at a time, in order. Use load() for a list of every document, or lazy_load() to process them as they arrive. With continue_on_error on, a failed URL still produces a Document, with empty page_content and the error message in metadata["error"]. Filter those out before embedding. Set continue_on_error=False to raise on the first failure instead.

Error handling

Every tool raises RuntimeError for API errors, timeouts and network failures, with the message prefixed by what went wrong, for example Decodo API error: ... or Decodo API network error: .... The four Web Data API tools also raise it for a response that is not JSON. Two cases raise ValueError instead, before any request is sent: a missing API key, and a crawl_id that is not a crawl id.
An authentication failure on one of the four Web Data API tools returns an error that names the likely cause — an invalid key, a plan without that endpoint, or a basic authentication token used where only an API key works.
For the full list of API parameters, see Parameters.

Support

Need help or just want to say hello? Our support is available 24/7.
You can also reach us anytime via email at support@decodo.com.

Feedback

Can’t find what you’re looking for? Request an article!
Have feedback? Share your thoughts on how we can improve.