Skip to main content
POST
Best for reading a whole section of a site without holding a call open.

Authorizations

Authorization
string
header
required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Headers

Idempotency-Key
string

A value you choose for one call. Send the same key with the same method, path and body within 24 hours and the first answer comes back with idempotent-replayed: true, with the same cost headers and no second charge. A call that sends zeroDataRetention and answers with page data is not kept, so a retry runs again. Needs an API key.

Query Parameters

maxTextLength
integer

Cut every text value in the response to this many characters. A value that was cut ends with …, and a page whose text was cut has isTruncated set to true. A screenshot is never cut. Leave it out to get the full text.

Required range: x >= 1

Body

application/json
url
string
required
prompt
string

What you are looking for, such as sqlite performance. Pages about it are read first, and the crawl stops once new pages stop mentioning it.

Required string length: 1 - 1000
includePaths
string[]

Read only pages whose path matches one of these regular expressions, such as blog/.*.

Maximum array length: 100
Required string length: 1 - 200
excludePaths
string[]

Skip pages whose path matches one of these regular expressions, such as tag/.*.

Maximum array length: 100
Required string length: 1 - 200
maxDiscoveryDepth
integer

How many links away from the start page to go. Without it there is no depth limit, and limit bounds the crawl.

Required range: 0 <= x <= 9007199254740991
sitemap
enum<string>
default:include

include also uses the site's sitemaps. only reads just the pages the sitemaps name. skip ignores them.

Available options:
skip,
include,
only
ignoreQueryParameters
boolean
default:false

Treat addresses that differ only in their ?query part as one page.

regexOnFullURL
boolean
default:false

Match includePaths and excludePaths against the whole address, not only the path.

limit
integer
default:10000

Stop after reading this many pages in total.

Required range: 1 <= x <= 10000
crawlEntireDomain
boolean
default:false

Read the whole site, not only pages under the start page's path.

deduplicateSimilarURLs
boolean
default:true

Read one page when addresses differ only by www, a trailing slash, a #fragment, tracking parameters such as utm_source, or the order of the ?query. Send false to read each address.

Also follow links to other sites. They count toward limit and maxDiscoveryDepth. Each other site's robots.txt is read too, for up to 25 new sites per page. A page on a site whose robots.txt cannot be read, or that comes after those 25, is skipped.

allowSubdomains
boolean
default:false

Also read pages on subdomains, such as docs.bun.com for bun.com.

delay
number

Seconds to wait between pages. With delay the crawl reads one page at a time. At most 60.

Required range: 0 <= x <= 60
maxConcurrency
integer

Most pages read at once. We read at most 4 at once, so a larger number gives the same speed.

Required range: 1 <= x <= 100
scrapeOptions
object

How to read each page, the same options and formats as scrape. Without formats, each page comes back as markdown.

zeroDataRetention
boolean
default:false

Send true to keep nothing from this crawl: no stored copy is read or made, page results are held for one hour at most, and nothing is logged. Adds $2 per 1,000 pages.

webhook

Where to be told about the job: a plain address, or { url } which gets a JSON POST when the job starts, for each finished page, and when the job ends. headers are sent with every call, metadata comes back in every call, and events picks which of started, page, action, completed, failed and cancelled to send (all by default). An agent run sends action for each step it takes, and page too. Every call is signed: read Stophy-Signature (t=,v1=), where v1 is the HMAC-SHA256 of "." with your webhook secret from webhook.secret. A call that gets no 2xx is sent again, up to 3 times. Every call has an eventId that stays the same when the call is sent again, so you can skip one you already handled. If your address answers slowly, a page event can carry only the page url. Read that page with the status call.

Response

The data, and what this call cost in dollars.

success
boolean
required
id
string
url
string
status
string
Allowed value: "scraping"
queuePosition
integer
Required range: -9007199254740991 <= x <= 9007199254740991