Skip to main content
A crawl runs in the background. These calls read, cancel and list it, and none of them is charged. They need your API key. A crawl and its pages are kept for 24 hours, and an unknown, expired or other account’s id returns 404 notFound. A batch works the same way at /v1/batch/scrape/{id}, with /results and /errors under it. This page is the home for the shared behavior.

Read every page

Poll GET /v1/crawl/{id} until status is completed, failed or cancelled, then follow next until it is missing.
A status answer holds up to 100 finished pages or 2 MB, whichever comes first, which is separate from the crawl’s limit. next appears only when more finished pages wait, so it never leads to an empty data. It is a full URL: call it with the same key. A cursor works only for its own job and limit. The SDK iterators read through a checkpoint: checkpoint=ck_start starts, and each answer returns the next checkpoint and only pages you have not seen. They wait while the job runs and end when it is over and every page was delivered. After 15 minutes without finishing they stop with resultsTimeout. Pass timeoutMs (timeout in Python) to change that, and onCheckpoint (on_checkpoint) to resume after a restart. For a batch use batchScrapeResults (batch_scrape_results).

Status fields

Results with depth

GET /v1/crawl/{id}/results returns finished pages with more detail: the url the crawl ended on, the requestedUrl it followed, depth and a costUsd per page. It returns 20 at a time. Send limit up to 100 and cursor from the answer, with the same limit, until cursor is missing. A part can hold fewer than limit.

Errors

GET /v1/crawl/{id}/errors lists the pages that could not be read, up to 1,000, with isTruncated when there were more. retryable says whether trying again can help. Error codes and fixes are in Errors. robotsBlocked lists up to 1,000 links that robots rules disallow. They are skipped, not failures. A 404 or 410 page is not an error. It is a finished page with its metadata.statusCode and a metadata.error.

Cancel and list

  • DELETE /v1/crawl/{id} cancels a running crawl and keeps the pages already read, which are charged. A finished job returns 409 jobFinished.
  • GET /v1/crawl lists your crawls, newest first, 20 at a time. Send status, limit up to 100 and cursor. A part can hold fewer than limit.
  • GET /v1/crawl/active lists crawls that are waiting or running: id, url, status (queued or scraping), createdAt and options.

Webhooks

Send webhook when you start a job instead of polling. It is a URL, or an object.
A call that gets no 2xx is sent again, up to 3 times, with the same eventId. If your endpoint answers slowly, a page event can carry only the page’s url: read that page with the status call. Signatures and payloads are in Webhooks.