Crawl
Crawl a website in the background.
Authorizations
Headers
A value you choose for one call. Send the same key with the same method, path and body within 24 hours and the first answer comes back with idempotent-replayed: true, with the same cost headers and no second charge. A call that sends zeroDataRetention and answers with page data is not kept, so a retry runs again. Needs an API key.
Query Parameters
Body
The page to start from, such as https://bun.com/docs.
Where to be told about the job: a plain address, or { url } which gets a JSON POST when the job starts, for each finished page, and when the job ends. headers are sent with every call, metadata comes back in every call, and events picks which of started, page, action, completed, failed and cancelled to send (all by default). An agent run sends action for each step it takes, and page too. Every call is signed: read Stophy-Signature (t=,v1=), where v1 is the HMAC-SHA256 of "." with your webhook secret from webhook.secret. A call that gets no 2xx is sent again, up to 3 times. Every call has an eventId that stays the same when the call is sent again, so you can skip one you already handled. If your address answers slowly, a page event can carry only the page url. Read that page with the status call.