Sync & change filtering

The sync block on a resource controls how much each run pulls and how Siphon decides a record is new or changed. Getting this right is the difference between a cheap incremental poll and a full re-scan every hour.

sync:
  mode: incremental
  watermarkField: updatedAt
  watermarkType: iso8601
  initialWatermark: "2000-01-01T00:00:00Z"

Sync modes

watermarkType is one of iso8601, unix_seconds, unix_millis, integer, or opaque. initialWatermark seeds {{ watermark }} on the first run.

The two axes of filtering

There are two independent questions:

1. Where does filtering happen — on the server (the API returns only the window) or on the client (Siphon fetches everything and drops the unchanged)?
2. How is "changed" decided — by a timestamp watermark, or by a content hash for sources that expose no modified timestamp?

sync:
  mode: full
  filter: client          # server (default) | client
  changeDetection: hash    # watermark (default) | hash
  hashFields: [stage, totalAmount, paidAmount, unitsCount]

Server-side filtering (default)

If the API supports a updatedSince/cursor parameter, template the watermark into the request and let the server return only the window. This is the cheapest path — the source sends nothing you've already seen.

sync:
  mode: incremental
  watermarkField: timestamp
fetch:
  path: /usage
  query:
    from: "{{ watermark }}"   # the API windows the results

filter defaults to server, so incremental specs need no extra fields.

Client-side filtering

When the API cannot filter — it always returns the full set — set filter: client. Siphon fetches every page and drops unchanged records before the expensive enrich stage, so you only pay to enrich and deliver what actually changed.

Two detection methods:

• changeDetection: watermark — keep records whose watermarkField is newer than the cursor. Requires a usable per-record timestamp.
• changeDetection: hash (default for filter: client) — fingerprint each record (a SHA-256 over hashFields, or the whole record if omitted) and keep only new or changed fingerprints. This is the option for sources with no modified timestamp at all.

How it fits the pipeline

Filtering runs between fetch and enrich:

fetch pages → change-filter (drop unchanged) → enrich → transform → validate → deliver

For hash detection, Siphon stores one fingerprint per record (connection × resource × external_id). The fingerprints are loaded before the run and persisted only after successful delivery — the same advance-after-delivery rule the watermark follows, so a failed run never marks records as "seen."

The steady state

Once a client-hash resource is seeded, an unchanged source is nearly free to poll: it still fetches the list (filtering is client-side, so it must pull to compare), but skips all enrichment and delivery. A typical hourly run of an unchanging source looks like:

pages: 5   recordsIn: 122   skipped: 122   recordsOut: 0   (0 enrich calls)

recordsOut climbs above zero only when a hashed field changes, or a brand-new record appears.

Re-applying a spec change

Fingerprints key on a record's source fields, not on your spec version — so if you republish a connector with a new transform (or edit a connection override), a record whose source fields are unchanged would otherwise be skipped and never re-delivered with the new mapping. Two mechanisms handle this:

• Auto-invalidation (default). The engine records the effective-spec checksum (base + connection override) each run. When it changes, the next run treats prior fingerprints as stale and reprocesses all records once — then returns to normal skipping. Opt a resource out with sync.autoInvalidate: false.
• Force full re-sync (manual). From the connection detail page, clear a resource's change-detection state (fingerprints + watermark) to reprocess everything on the next run — the escape hatch when you want a re-pull on demand.

sync:
  mode: full
  filter: client
  autoInvalidate: true   # default — reprocess once when the effective spec changes

Backfill / historical load

Go-forward sync only pulls new data; to load history, backfill a resource from a date: it resets change-detection and sets the watermark to your since date, so the next sync re-pulls everything modified on/after it (SyncService.backfill / backfillAction). Trigger a sync afterward to run it.

Detecting deletions

Pull-based syncs upsert; a record deleted at the source lingers in your destination as a ghost. For a hash-fingerprinted full fetch the signal is available — an id present last run but absent this run was removed. Opt in per resource:

sync:
  mode: full
  filter: client
  changeDetection: hash
  detectDeletions: true   # default false

On a run that fetched the complete set, the engine reports the vanished ids as deletions (stats.deleted, plus a records.deleted host webhook), and drops their fingerprints so they aren't re-reported. In a bring-your-own Postgres warehouse the vanished rows are soft-deleted — a _siphon_deleted_at timestamp is set, the row is kept for audit, and a re-created record clears its own tombstone on the next delivery. Deletion detection is a soft signal, never a hard delete.

Choosing a strategy