Skip to main content
POST
/
api
/
v1
/
platform
/
scrapers
/
web
/
scrape
/
batch
Batch scrape web pages
curl --request POST \
  --url https://developer.thehog.ai/api/v1/platform/scrapers/web/scrape/batch \
  --header 'Content-Type: application/json' \
  --header 'X-Access-Key: <api-key>' \
  --header 'X-Secret-Key: <api-key>' \
  --data '
{
  "urls": [
    "<string>"
  ],
  "items": [
    {
      "url": "https://example.com/page",
      "renderJs": false,
      "maxAgeMs": 0,
      "maxAgeDays": 0
    }
  ],
  "renderJs": false,
  "maxAgeMs": 0,
  "maxAgeDays": 0,
  "maxConcurrency": 3
}
'
{
  "data": {
    "id": "<string>",
    "operationId": "<string>",
    "pollUrl": "<string>"
  },
  "meta": {
    "requestId": "<string>"
  }
}

​
POST /api/v1/platform/scrapers/web/scrape/batch

Queue multiple web page scrapes and poll for per-URL results.
Use batch scrape when you have many URLs and do not need the results in the initial HTTP response. The endpoint returns an operation ID immediately; poll GET /api/operations/:id until the operation reaches a terminal status.

​
Example

curl -X POST https://developer.thehog.ai/api/v1/platform/scrapers/web/scrape/batch \
  -H "Authorization: Bearer <key>" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: batch-2026-05-22-001" \
  -d '{"urls": ["https://example.com/pricing", "https://example.com/about"], "maxConcurrency": 2}'

Authorizations

​
X-Access-Key
string
header
required

The public API key from the Credentials page.

​
X-Secret-Key
string
header
required

The API secret shown when the credential is created.

Headers

​
Idempotency-Key
string

Optional. Reusing the same key for the same organization returns the existing queued batch operation.

Body

application/json
​
urls
string[]

URLs to scrape using the top-level renderJs and cache freshness options.

​
items
object[]

Per-item scrape requests. Use either items or urls, not both.

​
renderJs
boolean
default:false

Render JavaScript before scraping for urls entries.

​
maxAgeMs
number
default:0
Required range: 0 <= x <= 2592000000
​
maxAgeDays
number
default:0
Required range: 0 <= x <= 30
​
maxConcurrency
number
default:3

Maximum number of batch items processed concurrently.

Required range: 1 <= x <= 10

Response

Batch scrape accepted. Poll the returned operation URL for results.

​
data
object
required
​
meta
object
required