Anatomy of a spec

The top-level shape of a ConnectorSpec — metadata, connection params, auth, defaults, and resources.

A ConnectorSpec is one declarative document (YAML or JSON) that fully describes a source. The Zod schema in @siphon/spec is the single source of truth — every field below is validated, and every platform type is inferred from it.

Try it live

Paste a spec into the Spec Playground — it validates and draws the pipeline in your browser, no sign-in.

Top-level shape

apiVersion: siphon/v1
kind: ConnectorSpec
metadata:
  key: trexo-crm
  name: Trexo CRM
  version: 1.0.0
connectionParams:
  - { key: baseUrl, type: url, required: true }
  - { key: apiToken, type: string, required: true, secret: true }
auth:
  strategy: bearer_static
  token: "{{ secrets.apiToken }}"
defaults:
  baseUrl: "{{ params.baseUrl }}"
  timeoutMs: 30000
resources:
  - key: deals
    canonicalEntity: Deal
    sync: { mode: full }
    fetch: { ... }
    transform: { ... }

Field	Required	Purpose
`apiVersion`	✅	Literal `siphon/v1`.
`kind`	✅	Literal `ConnectorSpec`.
`metadata`	✅	Identity + version (see below).
`connectionParams`	—	Per-connection configuration + secrets (default `[]`).
`auth`	✅	How every request authenticates. See Auth.
`defaults`	—	Base URL, headers, timeout, retry, rate limit.
`resources`	✅	One or more things to pull (min 1).
`tests`	—	Offline fixture test cases.

metadata

metadata:
  key: trexo-crm          # identifier: [A-Za-z_][A-Za-z0-9_-]*
  name: Trexo CRM         # human label
  version: 1.0.0          # semver — each published version is immutable
  vendorDocs: https://... # optional link to the vendor's API docs
  description: ...        # optional

connectionParams

Each param renders a field in the connection's credential form — so adding a connector adds zero frontend code.

connectionParams:
  - key: baseUrl
    type: url            # string | url | number | boolean
    required: true
    secret: false        # secret params are encrypted at rest + redacted
    label: API base URL
    default: https://app.example.com
    validate: { pattern: "^https://" }
    description: The vendor host.
  - key: apiToken
    type: string
    required: true
    secret: true         # → stored envelope-encrypted, never logged

Params are available to templates as {{ params.baseUrl }}; secrets as {{ secrets.apiToken }} (only inside auth).

defaults

Applied to every fetch and enrich request unless the request overrides them.

defaults:
  baseUrl: "{{ params.baseUrl }}"
  headers:
    Accept: application/json
  timeoutMs: 30000        # default
  retry:
    maxAttempts: 5
    strategy: exponential_jitter   # exponential_jitter | exponential | fixed | none
    baseDelayMs: 500
    maxDelayMs: 20000
    retryOn: [429, 500, 502, 503, 504, "NETWORK", "TIMEOUT"]
    respectRetryAfter: true
  rateLimit:
    requestsPerSecond: 5
    burst: 10
    scope: connection     # token bucket keyed per connection

retryOn entries are HTTP status codes (100–599) or the sentinels "NETWORK" / "TIMEOUT". Retries and rate limiting are enforced by the engine's HTTP client stack, so they apply to auth token exchanges too.

resources

Each resource is one thing to pull, mapped to a canonical entity (a free-form string naming your org's schema — Siphon knows no fixed vocabulary).

resources:
  - key: deals
    canonicalEntity: Deal
    sync: { ... }        # full vs incremental + change filtering
    fetch: { ... }       # request + pagination + extraction
    enrich: [ ... ]      # optional dependent-call DAG
    transform: { ... }   # JSONata mapping to canonical fields
    validate: { ... }    # required fields + what to do on failure
    output: { ... }      # optional typed output columns

Each block has its own page:

Auth

Nine strategies, from bearer to OAuth to custom token exchange.

Fetch & pagination

Request shape, five pagination strategies, extraction.

Sync & change filtering

Full vs incremental; server- vs client-side change detection.

Enrich

Dependent-call DAG, batching, caching, error policy.

Transform

Map raw records to canonical fields with JSONata.

Validate & output

Required fields, quarantine, and typed columns.

Validation

validateSpec runs schema validation plus semantic checks — enrichment must form an acyclic DAG, non-full sync modes require a watermarkField, changeDetection is only valid with client-side filtering, and so on. Publish is blocked until a spec is valid, and the merged result of any connection override is re-validated at save time.