POST

api

platform

scrapers

web

scrape

jobs

Queue a deep web scrape job

curl --request POST \
  --url https://developer.thehog.ai/api/v1/platform/scrapers/web/scrape/jobs \
  --header 'Content-Type: application/json' \
  --header 'X-Access-Key: <api-key>' \
  --header 'X-Secret-Key: <api-key>' \
  --data '
{
  "url": "https://example.com/page",
  "formats": [
    "text",
    "metadata"
  ],
  "jsonSchema": {},
  "instructions": "<string>",
  "maxAgeMs": 0,
  "maxDurationMs": 120000,
  "maxScrolls": 40,
  "scrollWaitMs": 500,
  "contentStableRounds": 3,
  "expandClickableContent": true,
  "maxExpansionClicks": 25
}
'

const options = {
  method: 'POST',
  headers: {
    'X-Access-Key': '<api-key>',
    'X-Secret-Key': '<api-key>',
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    url: 'https://example.com/page',
    formats: ['text', 'metadata'],
    jsonSchema: {},
    instructions: '<string>',
    maxAgeMs: 0,
    maxDurationMs: 120000,
    maxScrolls: 40,
    scrollWaitMs: 500,
    contentStableRounds: 3,
    expandClickableContent: true,
    maxExpansionClicks: 25
  })
};

fetch('https://developer.thehog.ai/api/v1/platform/scrapers/web/scrape/jobs', options)
  .then(res => res.json())
  .then(res => console.log(res))
  .catch(err => console.error(err));

import requests

url = "https://developer.thehog.ai/api/v1/platform/scrapers/web/scrape/jobs"

payload = {
    "url": "https://example.com/page",
    "formats": ["text", "metadata"],
    "jsonSchema": {},
    "instructions": "<string>",
    "maxAgeMs": 0,
    "maxDurationMs": 120000,
    "maxScrolls": 40,
    "scrollWaitMs": 500,
    "contentStableRounds": 3,
    "expandClickableContent": True,
    "maxExpansionClicks": 25
}
headers = {
    "X-Access-Key": "<api-key>",
    "X-Secret-Key": "<api-key>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.text)

{
  "data": {
    "id": "<string>",
    "operationId": "<string>",
    "status": "queued",
    "pollUrl": "<string>"
  },
  "meta": {
    "requestId": "<string>"
  }
}

{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "Validation failed",
  "path": "/api/v1/search",
  "timestamp": "2026-05-21T09:08:10.000Z",
  "requestId": "506af9b3-01a9-43be-9eb4-8458fe3e4f5b",
  "errors": [
    {
      "property": "body.query",
      "message": "query must be a string",
      "constraints": {
        "isString": "query must be a string"
      }
    }
  ]
}

{
  "statusCode": 401,
  "error": "Unauthorized",
  "message": "Authentication is required.",
  "path": "/api/v1/search",
  "timestamp": "2026-05-21T09:08:10.000Z",
  "requestId": "506af9b3-01a9-43be-9eb4-8458fe3e4f5b"
}

{
  "statusCode": 402,
  "error": "Payment Required",
  "message": "Insufficient credits.",
  "path": "/api/v1/search",
  "timestamp": "2026-05-21T09:08:10.000Z",
  "requestId": "506af9b3-01a9-43be-9eb4-8458fe3e4f5b"
}

{
  "statusCode": 500,
  "error": "Internal Server Error",
  "message": "An unexpected error occurred.",
  "path": "/api/v1/search",
  "timestamp": "2026-05-21T09:08:10.000Z",
  "requestId": "506af9b3-01a9-43be-9eb4-8458fe3e4f5b"
}

{
  "statusCode": 503,
  "error": "Service Unavailable",
  "message": "Service temporarily unavailable.",
  "path": "/api/v1/search",
  "timestamp": "2026-05-21T09:08:10.000Z",
  "requestId": "506af9b3-01a9-43be-9eb4-8458fe3e4f5b"
}

POST /api/v1/platform/scrapers/web/scrape/jobs

Queue a deep web scrape for long or dynamic pages and poll for the result.

Use deep scrape when a page needs more rendering time than the synchronous scrape endpoint should spend, such as long comment threads, lazy-loaded content, or pages with repeated “load more” controls. The endpoint returns an operation ID immediately; poll GET /api/operations/:id until the operation reaches a terminal status. The result uses the same requested formats as single-page scrape. Metadata may include capture details such as how many scrolls were completed and why capture stopped. A deep scrape is still bounded by the limits you send, so use the capture metadata to decide whether to run again with higher limits.

Example

curl -X POST https://developer.thehog.ai/api/v1/platform/scrapers/web/scrape/jobs \
  -H "X-Access-Key: ak_xxxxxxxxxxxxxxxx" \
  -H "X-Secret-Key: sk_xxxxxxxxxxxxxxxx" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: deep-scrape-2026-06-24-001" \
  -d '{
    "url": "https://example.com/thread",
    "formats": ["markdown", "metadata"],
    "maxDurationMs": 120000,
    "maxScrolls": 40,
    "contentStableRounds": 3,
    "expandClickableContent": true
  }'

Use formats: ["json"] with jsonSchema when you want schema-guided extraction from the captured page content. The schema defines the returned keys and shape.

Authorizations

X-Access-Key

string

header

required

The public API key from the Credentials page.

X-Secret-Key

string

header

required

The API secret shown when the credential is created.

Headers

Idempotency-Key

string

Optional. Reusing the same key for the same organization returns the existing queued deep scrape operation.

Body

application/json

url

string

required

URL to scrape with an async deep browser acquisition job.

Example:

"https://example.com/page"

formats

enum<string>[]

Output formats to store on the completed operation. Defaults to ["text", "metadata"]. Request "json" only with jsonSchema.

Available options:

text,

markdown,

html,

links,

metadata,

json

jsonSchema

object

JSON Schema used for schema-guided extraction when formats includes "json".

instructions

string

Additional extraction instructions used only when formats includes "json".

Maximum string length: 8000

maxAgeMs

number

default:0

Maximum accepted cache age in milliseconds. Defaults to 0 for deep jobs so dynamic pages are fetched fresh.

Required range: 0 <= x <= 2592000000

maxDurationMs

number

default:120000

Maximum deep render wall-clock duration in milliseconds.

Required range: 5000 <= x <= 600000

maxScrolls

number

default:40

Maximum viewport scroll steps during deep render.

Required range: 0 <= x <= 200

scrollWaitMs

number

default:500

Delay after each deep-render scroll step.

Required range: 100 <= x <= 5000

contentStableRounds

number

default:3

Stop after this many consecutive scroll rounds without meaningful content growth.

Required range: 1 <= x <= 20

expandClickableContent

boolean

default:true

Click generic visible "load more" or "show more" controls during deep render.

maxExpansionClicks

number

default:25

Maximum generic expansion clicks during deep render.

Required range: 0 <= x <= 200

Response

Deep scrape accepted. Poll the returned operation URL for results.

data

object

required

Show child attributes

meta

object

required

Show child attributes

⌘I

Queue a deep web scrape job

curl --request POST \
  --url https://developer.thehog.ai/api/v1/platform/scrapers/web/scrape/jobs \
  --header 'Content-Type: application/json' \
  --header 'X-Access-Key: <api-key>' \
  --header 'X-Secret-Key: <api-key>' \
  --data '
{
  "url": "https://example.com/page",
  "formats": [
    "text",
    "metadata"
  ],
  "jsonSchema": {},
  "instructions": "<string>",
  "maxAgeMs": 0,
  "maxDurationMs": 120000,
  "maxScrolls": 40,
  "scrollWaitMs": 500,
  "contentStableRounds": 3,
  "expandClickableContent": true,
  "maxExpansionClicks": 25
}
'

const options = {
  method: 'POST',
  headers: {
    'X-Access-Key': '<api-key>',
    'X-Secret-Key': '<api-key>',
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    url: 'https://example.com/page',
    formats: ['text', 'metadata'],
    jsonSchema: {},
    instructions: '<string>',
    maxAgeMs: 0,
    maxDurationMs: 120000,
    maxScrolls: 40,
    scrollWaitMs: 500,
    contentStableRounds: 3,
    expandClickableContent: true,
    maxExpansionClicks: 25
  })
};

fetch('https://developer.thehog.ai/api/v1/platform/scrapers/web/scrape/jobs', options)
  .then(res => res.json())
  .then(res => console.log(res))
  .catch(err => console.error(err));

import requests

url = "https://developer.thehog.ai/api/v1/platform/scrapers/web/scrape/jobs"

payload = {
    "url": "https://example.com/page",
    "formats": ["text", "metadata"],
    "jsonSchema": {},
    "instructions": "<string>",
    "maxAgeMs": 0,
    "maxDurationMs": 120000,
    "maxScrolls": 40,
    "scrollWaitMs": 500,
    "contentStableRounds": 3,
    "expandClickableContent": True,
    "maxExpansionClicks": 25
}
headers = {
    "X-Access-Key": "<api-key>",
    "X-Secret-Key": "<api-key>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.text)

{
  "data": {
    "id": "<string>",
    "operationId": "<string>",
    "status": "queued",
    "pollUrl": "<string>"
  },
  "meta": {
    "requestId": "<string>"
  }
}

{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "Validation failed",
  "path": "/api/v1/search",
  "timestamp": "2026-05-21T09:08:10.000Z",
  "requestId": "506af9b3-01a9-43be-9eb4-8458fe3e4f5b",
  "errors": [
    {
      "property": "body.query",
      "message": "query must be a string",
      "constraints": {
        "isString": "query must be a string"
      }
    }
  ]
}

{
  "statusCode": 401,
  "error": "Unauthorized",
  "message": "Authentication is required.",
  "path": "/api/v1/search",
  "timestamp": "2026-05-21T09:08:10.000Z",
  "requestId": "506af9b3-01a9-43be-9eb4-8458fe3e4f5b"
}

{
  "statusCode": 402,
  "error": "Payment Required",
  "message": "Insufficient credits.",
  "path": "/api/v1/search",
  "timestamp": "2026-05-21T09:08:10.000Z",
  "requestId": "506af9b3-01a9-43be-9eb4-8458fe3e4f5b"
}

{
  "statusCode": 500,
  "error": "Internal Server Error",
  "message": "An unexpected error occurred.",
  "path": "/api/v1/search",
  "timestamp": "2026-05-21T09:08:10.000Z",
  "requestId": "506af9b3-01a9-43be-9eb4-8458fe3e4f5b"
}

{
  "statusCode": 503,
  "error": "Service Unavailable",
  "message": "Service temporarily unavailable.",
  "path": "/api/v1/search",
  "timestamp": "2026-05-21T09:08:10.000Z",
  "requestId": "506af9b3-01a9-43be-9eb4-8458fe3e4f5b"
}

Overview

Companies

People

Research

Enrichments

Monitors

Search

LinkedIn

X

Instagram

TikTok

Facebook

YouTube

Web

SEO

Image

Operations

Queue a deep web scrape job

POST /api/v1/platform/scrapers/web/scrape/jobs

Example

Authorizations

Headers

Body

Response

​POST /api/v1/platform/scrapers/web/scrape/jobs

​Example

Authorizations

Headers

Body

Response

POST /api/v1/platform/scrapers/web/scrape/jobs

Example