Architecture

Siphon is a pnpm-workspaces monorepo. The design principle that shapes everything is injectable seams: every side effect — HTTP, the clock, DNS, the durable context, the database — enters through an interface, so the entire pipeline runs and is tested offline (in-memory HTTP fixtures, PGlite, a fake durable context). The only surfaces that need a live environment are two runtime bindings (Restate, the Next.js server) and the browser.

The two planes

┌──────────────────────────────────────────────────────────────┐
│  CONTROL PLANE — Next.js app + @siphon/api                    │
│  spec CRUD/validate/publish · connection lifecycle · runs +   │
│  inspector · quarantine · schedules · tokens · OAuth · webhooks│
└───────────────────────┬──────────────────────────────────────┘
                        │ trigger a run
                        ▼
┌──────────────────────────────────────────────────────────────┐
│  DATA PLANE — @siphon/engine, wrapped by @siphon/worker       │
│  resolveAuth → fetch pages → change-filter → enrich DAG →     │
│  transform → validate → deliver → advance watermark.          │
│  Restate journals each page.                                  │
└───────────────────────┬──────────────────────────────────────┘
                        ▼
        DESTINATIONS: postgres · sqlserver · webhook

Package layout

The dependency graph is a DAG rooted at spec:

spec  ──────────────┐         (zero deps; the contract)
  ▲                 │
engine ──────────┐  │         (imports spec; the executor)
  ▲              │  │
api  ── db       │  │         (services + repositories + HTTP support)
  ▲     ▲        │  │
worker ─┘        │  │         (Restate binding + dispatcher)
app (Next) ──────┴──┘         (route handlers + admin UI)

The request → run → inspect path

1. Author — POST /v1/specs validates and stores a draft version; …/publish makes it immutable and checksummed.
2. Onboard — POST /v1/connections validates params against the pinned published spec, encrypts the secrets (envelope), and lands the connection pending. …/activate enforces the plan cap.
3. Sync — a trigger (manual, schedule, webhook) dispatches a run to the durable worker via the Restate ingress.
4. Execute — the engine loads the effective spec (base + any connection override), resolves auth, pages through the source, drops unchanged records (change filtering), enriches, transforms, validates, and delivers. Every HTTP call flows through the guarded, logging, retrying, rate-limited client stack.
5. Persist — the worker writes sync_runs, redacted call_logs, run_errors, quarantined_records, the watermark, and record fingerprints.
6. Notify — outbound host webhooks fire (run.succeeded / records.quarantined …), each attempt persisted to the webhook_deliveries log.
7. Inspect — the Run Inspector reads the persisted run: stats, the call timeline, quarantine — debug a failure without a terminal.

The HTTP client stack

Composed by withResilience, innermost first:

FetchHttpClient          egress guard (SSRF) + timeout + JSON parse + redirect re-validation
  └ LoggingHttpClient    redacted call log (this is what the inspector renders)
      └ RateLimited      per-connection token bucket
          └ Retrying     backoff + Retry-After; never retries an egress block

The auth token exchange is wrapped too — which is why credential redaction in the logger is not optional.

Durability

The engine's syncRunWorkflow is written against a minimal DurableContext seam (run(name, fn) + keyed state), so the durable orchestration is unit-tested with an in-memory context — including crash-and-resume. @siphon/worker binds that seam to Restate. Full-replay model: on resume, completed pages replay from the journal (no re-fetch), the crashed page re-executes, and delivery never re-fires. See Durable engine.

Persistence & multi-tenancy

Drizzle over Postgres (~20 tables). The tenancy model is organization → workspace → connection; users are global identities, org membership via a members table. Org isolation is enforced in the service layer today — every run/quarantine/connection query is org-scoped, and an IDOR regression test proves cross-tenant access is denied. Row-level-security policies are the planned belt-and-suspenders layer. See Security.

Testing strategy

Unit + property-based (fast-check — ~30 properties generating thousands of cases)

• integration on PGlite (a real in-memory Postgres running the actual migrations and adapters). next build is the app-level check. Because the engine is composed from seams, the whole pipeline is exercised offline against fixtures.