Destinations
Deliver to a customer's Postgres or SQL Server warehouse, or a webhook — idempotent, deduped, typed.
A destination is where a resource's records land. Delivery is at-least-once
and idempotent on (connectionId, resourceKey, externalId), so replays overwrite
rather than duplicate. A connection routes each resource to a destination.
Postgres warehouse (bring-your-own)
The customer supplies a Postgres DSN (envelope-encrypted at rest — only display metadata like host/database is stored in the clear). The loader is a two-stage, Airbyte-Destinations-V2-style pipeline:
- Raw landing — records are written to a raw table in a dedicated schema
(default
_siphon_raw). - Typing & Deduping — a projection produces a typed, deduped final table
in your schema (default
public), upserting on the external id.
Declared output columns become real typed columns;
undeclared fields stay in the data JSONB. Typing is soft (a bad value → NULL,
full record preserved), and columns are added with ADD COLUMN IF NOT EXISTS so
the schema evolves additively.
When deletion detection is enabled, records that
vanish from the source are soft-deleted — the final table's _siphon_deleted_at
column is stamped rather than the row removed, and a re-created record clears it on
the next sync. Filter it out in your queries (WHERE _siphon_deleted_at IS NULL) to
see only live records.
Sync modes:
syncMode | Behavior |
|---|---|
append_dedup (default) | Upsert on the external id — one current row per record. |
overwrite | Truncate + reload each run. |
append | Keep every version (history). |
SQL Server warehouse
The same idea for SQL Server: records land in a table via a single MERGE on
(connection_id, resource_key, external_id). The full record is stored as JSON in
a data column plus the key columns; batches are chunked to stay under the 2100
bind-parameter limit. Configure the table, schema (default dbo), and the key
column names.
Webhook
Deliver each batch to an HTTP endpoint — for pushing records into your own system rather than a warehouse.
Testing a destination
The control plane can test a connection (connect + probe) and introspect the schema/tables/columns before you route to it, so you catch a bad DSN or a missing permission at setup time — not on the first run.
Where decryption happens
For a bring-your-own warehouse, the DSN is decrypted in the worker's handler body — never inside a journaled step — so the plaintext connection string never lands in the durable journal.