Skip to main content
POST
Batch scrape web pages

POST /api/v1/platform/scrapers/web/scrape/batch

Queue multiple web page scrapes and poll for per-URL results.
Use batch scrape when you have many URLs and do not need the results in the initial HTTP response. The endpoint returns an operation ID immediately; poll GET /api/operations/:id until the operation reaches a terminal status.

Example

Authorizations

X-Access-Key
string
header
required

The public API key from the Credentials page.

X-Secret-Key
string
header
required

The API secret shown when the credential is created.

Headers

Idempotency-Key
string

Optional. Reusing the same key for the same organization returns the existing queued batch operation.

Body

application/json
urls
string[]

URLs to scrape using the top-level renderJs and cache freshness options.

items
object[]

Per-item scrape requests. Use either items or urls, not both.

renderJs
boolean
default:false

Render JavaScript before scraping for urls entries.

maxAgeMs
number
default:0
Required range: 0 <= x <= 2592000000
maxAgeDays
number
default:0
Required range: 0 <= x <= 30
maxConcurrency
number
default:3

Maximum number of batch items processed concurrently.

Required range: 1 <= x <= 10

Response

Batch scrape accepted. Poll the returned operation URL for results.

data
object
required
meta
object
required