Fetch & pagination

The request shape, five pagination strategies, and how records are extracted from a response.

A resource's fetch block defines the list request, how to page through it, and how to pull records out of each response.

fetch:
  id: list
  method: GET                     # GET | POST | PUT | PATCH | DELETE | HEAD
  path: /api/Deals                # templated; joined onto defaults.baseUrl
  query:
    PageSize: 100
  headers: { }
  pagination:
    strategy: page
    pageSize: 100
    pageParam: PageNumber
    sizeParam: PageSize
    startPage: 1
    stopWhen: { path: "$.hasNextPage", equals: false }
  extract:
    recordsPath: "$.items"        # JSONPath to the array of records
    idPath: "$.id"                # per-record unique id (relative to a record)
    updatedAtPath: "$.updatedAt"  # optional — feeds the watermark

Request fields

method (default GET), path (templated), query, headers, body, timeoutMs. The same request-field shape is used by enrich calls.

POST-to-query APIs (templated bodies)

Some sources are queried with POST and a JSON body rather than GET with query params — and page from the body (an offset field), not the query string. The request body is deep-templated per page: a pure {{ expr }} leaf keeps its type (a number stays a number), so {{ page.offset }}, {{ page.size }}, {{ watermark }}, and {{ params.* }} resolve inside the body.

fetch:
  method: POST
  path: /workorders/v1/query
  body:
    segments: ["{{ params.segmentId }}"]        # → [223]  (number, not "223")
    properties:
      startIndex: "{{ page.offset }}"           # → 0, 100, 200 …
      pageSize: "{{ page.size }}"
  pagination:
    strategy: offset
    pageSize: 100
    offsetParam: startIndex   # required by the schema; harmless in the query when the body drives paging

The paginator still advances offset and stops on a short page; you just surface page.offset in the body. See the fsi-cmms example connector for a full OAuth2-password + POST-query spec.

Pagination strategies

pagination is a discriminated union on strategy. Every strategy has a maxPages safety cap (default 10000).

`page`

Page-number pagination.

pagination:
  strategy: page
  pageSize: 100
  pageParam: PageNumber
  sizeParam: PageSize      # optional
  startPage: 1
  stopWhen: { path: "$.hasNextPage", equals: false }

`offset`

Offset/limit pagination.

pagination:
  strategy: offset
  pageSize: 100
  offsetParam: offset
  limitParam: limit        # optional
  startOffset: 0
  stopWhen: { path: "$.data", isEmpty: true }

`cursor`

Follow a cursor token pulled from each response.

pagination:
  strategy: cursor
  cursorParam: after
  cursorPath: "$.page.next"   # JSONPath to the next cursor
  pageSize: 100
  sizeParam: limit            # optional

`link_header`

Follow RFC 5988 Link: headers.

pagination:
  strategy: link_header
  rel: next        # default
  pageSize: 100
  sizeParam: per_page

`none`

A single request — no pagination.

pagination: { strategy: none }

Stop conditions

page and offset strategies accept a stopWhen — a JSONPath plus one of equals or isEmpty:

stopWhen: { path: "$.hasNextPage", equals: false }
stopWhen: { path: "$.data", isEmpty: true }

Extraction

extract pulls records out of each page:

Field	Purpose
`recordsPath`	JSONPath to the array of records. Use `$` for a bare-array response.
`idPath`	Per-record unique id (evaluated relative to each record).
`updatedAtPath`	Optional per-record last-modified timestamp — feeds the watermark and client-side change filtering.

Bare arrays

If the endpoint returns a top-level JSON array rather than an envelope, set recordsPath: "$".

On this page