> ## Documentation Index
> Fetch the complete documentation index at: https://docs.thehog.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Scrape web page

> Fetch a web page and return readable text plus requested markdown, HTML, links, metadata, or schema-guided JSON.

# POST /api/v1/platform/scrapers/web/scrape

> Scrape a single web page and return its content.

Scrape a single web page and return readable content. By default the endpoint
returns the backward-compatible `url`, `text`, and `statusCode` fields. Use
`formats` when you need markdown, HTML, discovered links, page metadata, or
schema-guided JSON extraction.

## Example

```bash theme={null} theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://developer.thehog.ai/api/v1/platform/scrapers/web/scrape \
  -H "X-Access-Key: ak_xxxxxxxxxxxxxxxx" \
  -H "X-Secret-Key: sk_xxxxxxxxxxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/pricing", "renderJs": true, "formats": ["markdown", "metadata"]}'
```

## Formats

`formats` is optional and defaults to `["text"]`.

| Format     | Returns                                                                                                         |
| ---------- | --------------------------------------------------------------------------------------------------------------- |
| `text`     | Readable text. This is always included for backward compatibility.                                              |
| `markdown` | Markdown-friendly page content for agents and LLM workflows.                                                    |
| `html`     | Fetched or rendered HTML when available.                                                                        |
| `links`    | Normalized links discovered on the page.                                                                        |
| `metadata` | Page title, description, canonical URL, final URL, status code, content type, and size metadata when available. |
| `json`     | Data extracted into your provided JSON Schema.                                                                  |

Requested optional formats appear as additional fields under `data`. If a
requested format is unavailable, the field may be `null` or an empty array.

## Schema-guided JSON

Request `json` only when you provide `jsonSchema`. The schema defines the
returned keys and shape; page content is mapped into that schema and validated
before the response is returned. The root schema must be an object; use array
properties for repeated data such as comments, products, or plans.

```bash theme={null} theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://developer.thehog.ai/api/v1/platform/scrapers/web/scrape \
  -H "X-Access-Key: ak_xxxxxxxxxxxxxxxx" \
  -H "X-Secret-Key: sk_xxxxxxxxxxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "formats": ["markdown", "json"],
    "jsonSchema": {
      "type": "object",
      "additionalProperties": false,
      "required": ["plans"],
      "properties": {
        "plans": {
          "type": "array",
          "items": {
            "type": "object",
            "additionalProperties": false,
            "required": ["name", "price"],
            "properties": {
              "name": { "type": "string" },
              "price": { "type": "string" }
            }
          }
        }
      }
    },
    "instructions": "Extract public pricing plans visible on the page."
  }'
```

`jsonSchema` and `instructions` are only valid when `formats` includes `json`.
Schema-guided extraction performs additional model work and may use more credits
than a plain text or markdown scrape.


## OpenAPI

````yaml api-reference/openapi.json POST /api/v1/platform/scrapers/web/scrape
openapi: 3.0.0
info:
  title: The Hog API
  description: Public API reference for The Hog.
  version: '1.0'
  contact: {}
servers:
  - url: https://developer.thehog.ai
security: []
tags: []
paths:
  /api/v1/platform/scrapers/web/scrape:
    post:
      tags:
        - Scrapers
      summary: Scrape web page
      description: >-
        Fetch a web page and return readable text plus requested markdown, HTML,
        links, metadata, or schema-guided JSON.
      operationId: scrapeWebPage
      parameters: []
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/PlatformWebScrapeDto'
      responses:
        '200':
          description: Scraped web page content.
          content:
            application/json:
              schema:
                type: object
                required:
                  - data
                  - meta
                additionalProperties: false
                properties:
                  data:
                    $ref: '#/components/schemas/WebScrapeResponseDto'
                  meta:
                    $ref: '#/components/schemas/PublicResponseMetaDto'
        '400':
          description: The request body or parameters are invalid.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/PublicErrorResponseDto'
              example:
                statusCode: 400
                error: Bad Request
                message: Validation failed
                path: /api/v1/search
                timestamp: '2026-05-21T09:08:10.000Z'
                requestId: 506af9b3-01a9-43be-9eb4-8458fe3e4f5b
                errors:
                  - property: body.query
                    message: query must be a string
                    constraints:
                      isString: query must be a string
        '401':
          description: Authentication is required.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/PublicErrorResponseDto'
              example:
                statusCode: 401
                error: Unauthorized
                message: Authentication is required.
                path: /api/v1/search
                timestamp: '2026-05-21T09:08:10.000Z'
                requestId: 506af9b3-01a9-43be-9eb4-8458fe3e4f5b
        '402':
          description: The organization does not have enough credits for this request.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/PublicErrorResponseDto'
              example:
                statusCode: 402
                error: Payment Required
                message: Insufficient credits.
                path: /api/v1/search
                timestamp: '2026-05-21T09:08:10.000Z'
                requestId: 506af9b3-01a9-43be-9eb4-8458fe3e4f5b
        '500':
          description: An unexpected error occurred.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/PublicErrorResponseDto'
              example:
                statusCode: 500
                error: Internal Server Error
                message: An unexpected error occurred.
                path: /api/v1/search
                timestamp: '2026-05-21T09:08:10.000Z'
                requestId: 506af9b3-01a9-43be-9eb4-8458fe3e4f5b
        '502':
          description: The upstream service could not complete the request.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/PublicErrorResponseDto'
              example:
                statusCode: 502
                error: Bad Gateway
                message: An unexpected error occurred.
                path: /api/v1/search
                timestamp: '2026-05-21T09:08:10.000Z'
                requestId: 506af9b3-01a9-43be-9eb4-8458fe3e4f5b
        '503':
          description: The service is temporarily unavailable.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/PublicErrorResponseDto'
              example:
                statusCode: 503
                error: Service Unavailable
                message: Service temporarily unavailable.
                path: /api/v1/search
                timestamp: '2026-05-21T09:08:10.000Z'
                requestId: 506af9b3-01a9-43be-9eb4-8458fe3e4f5b
        '504':
          description: The upstream service timed out before completing the request.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/PublicErrorResponseDto'
              example:
                statusCode: 504
                error: Gateway Timeout
                message: An unexpected error occurred.
                path: /api/v1/search
                timestamp: '2026-05-21T09:08:10.000Z'
                requestId: 506af9b3-01a9-43be-9eb4-8458fe3e4f5b
      security:
        - AccessKey: []
          SecretKey: []
components:
  schemas:
    PlatformWebScrapeDto:
      type: object
      properties:
        url:
          type: string
          description: URL to scrape
          example: https://example.com/page
        renderJs:
          type: boolean
          description: Render JavaScript before scraping
          default: false
        formats:
          type: array
          description: >-
            Output formats to return. Defaults to ["text"]. Request "json" only
            with jsonSchema.
          default:
            - text
          items:
            type: string
            enum:
              - text
              - markdown
              - html
              - links
              - metadata
              - json
        jsonSchema:
          type: object
          description: >-
            JSON Schema used for schema-guided extraction when formats includes
            "json".
          additionalProperties: true
        instructions:
          type: string
          description: >-
            Additional extraction instructions used only when formats includes
            "json".
          maxLength: 8000
        maxAgeMs:
          type: number
          description: >-
            Maximum accepted cache age in milliseconds. Use 0 or omit to force a
            fresh scrape.
          minimum: 0
          maximum: 2592000000
          default: 0
        maxAgeDays:
          type: number
          description: >-
            Maximum accepted cache age in days. Use 0 or omit to force a fresh
            scrape. Ignored when maxAgeMs is provided.
          minimum: 0
          maximum: 30
          default: 0
      required:
        - url
    WebScrapeResponseDto:
      type: object
      properties:
        url:
          type: string
        text:
          type: string
        statusCode:
          type: number
          nullable: true
        markdown:
          type: string
          nullable: true
        html:
          type: string
          nullable: true
        links:
          type: array
          items:
            type: string
        metadata:
          $ref: '#/components/schemas/WebScrapeMetadataResponseDto'
        json:
          type: object
          additionalProperties: true
      required:
        - url
        - text
    PublicResponseMetaDto:
      type: object
      properties:
        requestId:
          type: string
      required:
        - requestId
    PublicErrorResponseDto:
      type: object
      properties:
        statusCode:
          type: number
          example: 400
        error:
          type: string
          example: Bad Request
        message:
          type: string
          example: Validation failed
        path:
          type: string
          example: /api/v1/search
        requestId:
          type: string
          example: 506af9b3-01a9-43be-9eb4-8458fe3e4f5b
        timestamp:
          type: string
          example: '2026-05-21T09:08:10.000Z'
        errors:
          type: array
          items:
            $ref: '#/components/schemas/PublicValidationErrorItemDto'
      required:
        - statusCode
        - error
        - message
        - path
        - timestamp
    WebScrapeMetadataResponseDto:
      type: object
      properties:
        title:
          type: string
          nullable: true
        description:
          type: string
          nullable: true
        canonicalUrl:
          type: string
          nullable: true
        finalUrl:
          type: string
          nullable: true
        contentType:
          type: string
          nullable: true
        statusCode:
          type: number
          nullable: true
        textLength:
          type: number
          nullable: true
        responseBytes:
          type: number
          nullable: true
        capture:
          $ref: '#/components/schemas/WebScrapeCaptureMetadataResponseDto'
    PublicValidationErrorItemDto:
      type: object
      properties:
        property:
          type: string
          example: body.query
        message:
          type: string
          example: query must be a string
        constraints:
          type: object
          additionalProperties:
            type: string
          example:
            isString: query must be a string
      required:
        - property
        - message
    WebScrapeCaptureMetadataResponseDto:
      type: object
      properties:
        mode:
          type: string
        completionReason:
          type: string
        scrollsCompleted:
          type: number
        maxScrolls:
          type: number
        expansionClicks:
          type: number
        maxExpansionClicks:
          type: number
        durationMs:
          type: number
      required:
        - mode
        - completionReason
        - scrollsCompleted
        - maxScrolls
        - expansionClicks
        - maxExpansionClicks
        - durationMs
  securitySchemes:
    AccessKey:
      type: apiKey
      in: header
      name: X-Access-Key
      description: The public API key from the Credentials page.
    SecretKey:
      type: apiKey
      in: header
      name: X-Secret-Key
      description: The API secret shown when the credential is created.

````