> ## Documentation Index
> Fetch the complete documentation index at: https://docs.thehog.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Queue a deep web scrape job

> Queue an async browser scrape for dynamic or long pages and poll the returned operation URL for results.

# POST /api/v1/platform/scrapers/web/scrape/jobs

> Queue a deep web scrape for long or dynamic pages and poll for the result.

Use deep scrape when a page needs more rendering time than the synchronous
scrape endpoint should spend, such as long comment threads, lazy-loaded content,
or pages with repeated "load more" controls. The endpoint returns an operation
ID immediately; poll `GET /api/operations/:id` until the operation reaches a
terminal status.

The result uses the same requested formats as single-page scrape. Metadata may
include capture details such as how many scrolls were completed and why capture
stopped. A deep scrape is still bounded by the limits you send, so use the
capture metadata to decide whether to run again with higher limits.

## Example

```bash theme={null} theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://developer.thehog.ai/api/v1/platform/scrapers/web/scrape/jobs \
  -H "X-Access-Key: ak_xxxxxxxxxxxxxxxx" \
  -H "X-Secret-Key: sk_xxxxxxxxxxxxxxxx" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: deep-scrape-2026-06-24-001" \
  -d '{
    "url": "https://example.com/thread",
    "formats": ["markdown", "metadata"],
    "maxDurationMs": 120000,
    "maxScrolls": 40,
    "contentStableRounds": 3,
    "expandClickableContent": true
  }'
```

Use `formats: ["json"]` with `jsonSchema` when you want schema-guided
extraction from the captured page content. The schema defines the returned keys
and shape.


## OpenAPI

````yaml api-reference/openapi.json POST /api/v1/platform/scrapers/web/scrape/jobs
openapi: 3.0.0
info:
  title: The Hog API
  description: Public API reference for The Hog.
  version: '1.0'
  contact: {}
servers:
  - url: https://developer.thehog.ai
security: []
tags: []
paths:
  /api/v1/platform/scrapers/web/scrape/jobs:
    post:
      tags:
        - Scrapers
      summary: Queue a deep web scrape job
      description: >-
        Queue an async browser scrape for dynamic or long pages and poll the
        returned operation URL for results.
      operationId: createWebScrapeJob
      parameters:
        - name: Idempotency-Key
          in: header
          description: >-
            Optional. Reusing the same key for the same organization returns the
            existing queued deep scrape operation.
          required: false
          schema:
            type: string
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/PlatformWebScrapeJobDto'
      responses:
        '202':
          description: Deep scrape accepted. Poll the returned operation URL for results.
          content:
            application/json:
              schema:
                type: object
                required:
                  - data
                  - meta
                additionalProperties: false
                properties:
                  data:
                    $ref: '#/components/schemas/WebScrapeJobAcceptedResponseDto'
                  meta:
                    $ref: '#/components/schemas/PublicResponseMetaDto'
        '400':
          description: The request body or parameters are invalid.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/PublicErrorResponseDto'
              example:
                statusCode: 400
                error: Bad Request
                message: Validation failed
                path: /api/v1/search
                timestamp: '2026-05-21T09:08:10.000Z'
                requestId: 506af9b3-01a9-43be-9eb4-8458fe3e4f5b
                errors:
                  - property: body.query
                    message: query must be a string
                    constraints:
                      isString: query must be a string
        '401':
          description: Authentication is required.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/PublicErrorResponseDto'
              example:
                statusCode: 401
                error: Unauthorized
                message: Authentication is required.
                path: /api/v1/search
                timestamp: '2026-05-21T09:08:10.000Z'
                requestId: 506af9b3-01a9-43be-9eb4-8458fe3e4f5b
        '402':
          description: The organization does not have enough credits for this request.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/PublicErrorResponseDto'
              example:
                statusCode: 402
                error: Payment Required
                message: Insufficient credits.
                path: /api/v1/search
                timestamp: '2026-05-21T09:08:10.000Z'
                requestId: 506af9b3-01a9-43be-9eb4-8458fe3e4f5b
        '500':
          description: An unexpected error occurred.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/PublicErrorResponseDto'
              example:
                statusCode: 500
                error: Internal Server Error
                message: An unexpected error occurred.
                path: /api/v1/search
                timestamp: '2026-05-21T09:08:10.000Z'
                requestId: 506af9b3-01a9-43be-9eb4-8458fe3e4f5b
        '503':
          description: The service is temporarily unavailable.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/PublicErrorResponseDto'
              example:
                statusCode: 503
                error: Service Unavailable
                message: Service temporarily unavailable.
                path: /api/v1/search
                timestamp: '2026-05-21T09:08:10.000Z'
                requestId: 506af9b3-01a9-43be-9eb4-8458fe3e4f5b
      security:
        - AccessKey: []
          SecretKey: []
components:
  schemas:
    PlatformWebScrapeJobDto:
      type: object
      properties:
        url:
          type: string
          description: URL to scrape with an async deep browser acquisition job.
          example: https://example.com/page
        formats:
          type: array
          description: >-
            Output formats to store on the completed operation. Defaults to
            ["text", "metadata"]. Request "json" only with jsonSchema.
          default:
            - text
            - metadata
          items:
            type: string
            enum:
              - text
              - markdown
              - html
              - links
              - metadata
              - json
        jsonSchema:
          type: object
          description: >-
            JSON Schema used for schema-guided extraction when formats includes
            "json".
          additionalProperties: true
        instructions:
          type: string
          description: >-
            Additional extraction instructions used only when formats includes
            "json".
          maxLength: 8000
        maxAgeMs:
          type: number
          description: >-
            Maximum accepted cache age in milliseconds. Defaults to 0 for deep
            jobs so dynamic pages are fetched fresh.
          minimum: 0
          maximum: 2592000000
          default: 0
        maxDurationMs:
          type: number
          description: Maximum deep render wall-clock duration in milliseconds.
          minimum: 5000
          maximum: 600000
          default: 120000
        maxScrolls:
          type: number
          description: Maximum viewport scroll steps during deep render.
          minimum: 0
          maximum: 200
          default: 40
        scrollWaitMs:
          type: number
          description: Delay after each deep-render scroll step.
          minimum: 100
          maximum: 5000
          default: 500
        contentStableRounds:
          type: number
          description: >-
            Stop after this many consecutive scroll rounds without meaningful
            content growth.
          minimum: 1
          maximum: 20
          default: 3
        expandClickableContent:
          type: boolean
          description: >-
            Click generic visible "load more" or "show more" controls during
            deep render.
          default: true
        maxExpansionClicks:
          type: number
          description: Maximum generic expansion clicks during deep render.
          minimum: 0
          maximum: 200
          default: 25
      required:
        - url
    WebScrapeJobAcceptedResponseDto:
      type: object
      properties:
        id:
          type: string
        operationId:
          type: string
        status:
          type: string
          enum:
            - queued
        pollUrl:
          type: string
      required:
        - id
        - operationId
        - status
        - pollUrl
    PublicResponseMetaDto:
      type: object
      properties:
        requestId:
          type: string
      required:
        - requestId
    PublicErrorResponseDto:
      type: object
      properties:
        statusCode:
          type: number
          example: 400
        error:
          type: string
          example: Bad Request
        message:
          type: string
          example: Validation failed
        path:
          type: string
          example: /api/v1/search
        requestId:
          type: string
          example: 506af9b3-01a9-43be-9eb4-8458fe3e4f5b
        timestamp:
          type: string
          example: '2026-05-21T09:08:10.000Z'
        errors:
          type: array
          items:
            $ref: '#/components/schemas/PublicValidationErrorItemDto'
      required:
        - statusCode
        - error
        - message
        - path
        - timestamp
    PublicValidationErrorItemDto:
      type: object
      properties:
        property:
          type: string
          example: body.query
        message:
          type: string
          example: query must be a string
        constraints:
          type: object
          additionalProperties:
            type: string
          example:
            isString: query must be a string
      required:
        - property
        - message
  securitySchemes:
    AccessKey:
      type: apiKey
      in: header
      name: X-Access-Key
      description: The public API key from the Credentials page.
    SecretKey:
      type: apiKey
      in: header
      name: X-Secret-Key
      description: The API secret shown when the credential is created.

````