Batch

POST /v1/web/batch reads a list of URLs you give it, as one background job. There is no discovery: you choose exactly which pages. You get a batchId straight away, then poll it or let a webhook tell you as pages finish. 1 credit per page read.

Batch or crawl?

You give the URLs. Best when you already know which pages you want.

Crawl

We find the URLs from a starting page. Best for a whole site or section.

When to use it

You have a set of URLs, from your own data, a sitemap or a Map call, and want them all read in one job.
You want the same options applied to every URL.

Example

curl -X POST https://api.hydrafetch.com/v1/web/batch \
  -H "X-API-Key: hf_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://example.com/a",
      "https://example.com/b",
      "https://example.com/c"
    ],
    "scrapeOptions": { "formats": ["markdown", "structured"] }
  }'

import { Hydrafetch } from "@hydrafetch/node-sdk";

const hf = new Hydrafetch(process.env.HYDRAFETCH_API_KEY);
const job = await hf.batchAndWait(
  ["https://example.com/a", "https://example.com/b", "https://example.com/c"],
  { scrapeOptions: { formats: ["markdown", "structured"] } },
);
job.pages.forEach((p) => console.log(p.url, p.status));

import requests

res = requests.post(
    "https://api.hydrafetch.com/v1/web/batch",
    headers={"X-API-Key": "hf_your_key_here"},
    json={
        "urls": [
            "https://example.com/a",
            "https://example.com/b",
            "https://example.com/c",
        ],
        "scrapeOptions": {"formats": ["markdown", "structured"]},
    },
)
batch_id = res.json()["batchId"]

The batch starts at once:

{ "batchId": "019f3c09-6fae-740f-9257-10c2b6af7f43", "status": "queued" }

Options

string[]

required

The URLs to read, from 1 to 25,000. Each must be http or https.

object

How to read each page. The same options as a Scrape, with formats limited to markdown, html, rawHtml, links and structured.

object

Have us call your URL as pages finish and when the job ends, instead of polling. See Jobs & webhooks.

Getting the results

Poll GET /v1/web/batch/{id}. It answers in the same shape as a crawl’s status, with kind set to batch and every page’s discoveredVia set to requested. Results stay available for 90 days after the batch last changes. After that the call answers 404.

curl https://api.hydrafetch.com/v1/web/batch/019f3c09-6fae-740f-9257-10c2b6af7f43 \
  -H "X-API-Key: hf_your_key_here"

{
  "data": {
    "id": "019f3c09-6fae-740f-9257-10c2b6af7f43",
    "kind": "batch",
    "status": "running",
    "seedUrl": null,
    "total": 3,
    "completed": 2,
    "failed": 0,
    "creditsUsed": 2,
    "pages": [
      {
        "url": "https://example.com/a",
        "status": "completed",
        "depth": 0,
        "discoveredVia": "requested",
        "error": null,
        "data": { "url": "https://example.com/a", "status": 200, "markdown": "# Page A\n\n..." }
      }
    ]
  }
}

Cost

1 credit per page read. Pages that fail are free. The job’s creditsUsed shows the total so far.

Next steps

Map a site

Get the URLs to send to a batch.

Crawl a whole site

When you want us to find the pages.

Jobs & webhooks

Polling, webhooks and checking deliveries.

API reference

The full schema, with a live playground.

Get started

Core concepts

Web Scraping

Web Extraction

Data indexes

Batch

Batch or crawl?

Batch

Crawl

When to use it

Example

Options

Getting the results

Cost

Next steps

Map a site

Crawl a whole site

Jobs & webhooks

API reference