id returns 404 notFound.
A batch works the same way at /v1/batch/scrape/{id}, with /results and /errors under it. This page is the home for the shared behavior.
Read every page
PollGET /v1/crawl/{id} until status is completed, failed or cancelled, then follow next until it is missing.
limit. next appears only when more finished pages wait, so it never leads to an empty data. It is a full URL: call it with the same key. A cursor works only for its own job and limit.
The SDK iterators read through a checkpoint: checkpoint=ck_start starts, and each answer returns the next checkpoint and only pages you have not seen. They wait while the job runs and end when it is over and every page was delivered. After 15 minutes without finishing they stop with resultsTimeout. Pass timeoutMs (timeout in Python) to change that, and onCheckpoint (on_checkpoint) to resume after a restart. For a batch use batchScrapeResults (batch_scrape_results).
Status fields
Results with depth
GET /v1/crawl/{id}/results returns finished pages with more detail: the url the crawl ended on, the requestedUrl it followed, depth and a costUsd per page. It returns 20 at a time. Send limit up to 100 and cursor from the answer, with the same limit, until cursor is missing. A part can hold fewer than limit.
Errors
GET /v1/crawl/{id}/errors lists the pages that could not be read, up to 1,000, with isTruncated when there were more. retryable says whether trying again can help. Error codes and fixes are in Errors. robotsBlocked lists up to 1,000 links that robots rules disallow. They are skipped, not failures.
A 404 or 410 page is not an error. It is a finished page with its metadata.statusCode and a metadata.error.
Cancel and list
DELETE /v1/crawl/{id}cancels a running crawl and keeps the pages already read, which are charged. A finished job returns409 jobFinished.GET /v1/crawllists your crawls, newest first, 20 at a time. Sendstatus,limitup to 100 andcursor. A part can hold fewer thanlimit.GET /v1/crawl/activelists crawls that are waiting or running:id,url,status(queuedorscraping),createdAtandoptions.
Webhooks
Sendwebhook when you start a job instead of polling. It is a URL, or an object.
A call that gets no
2xx is sent again, up to 3 times, with the same eventId. If your endpoint answers slowly, a page event can carry only the page’s url: read that page with the status call. Signatures and payloads are in Webhooks.