Validate & output

Require fields, decide what happens on failure, and declare typed output columns.

validate

After the transform, validate checks each record and decides what to do with the ones that don't pass.

validate:
  required: [externalId, DEAL_ID]
  onInvalid: quarantine    # quarantine (default) | drop | fail_run

`onInvalid`	Effect
`quarantine` (default)	Set the record aside in `quarantined_records` with its errors — visible and requeueable in the inspector.
`drop`	Silently discard the record.
`fail_run`	Abort the whole run.

Quarantine is the default because one malformed record shouldn't fail a 10,000-row sync — you triage it later without losing the good data.

output.fields optionally declares the resource's typed columns. A typed destination (a Postgres or SQL Server warehouse) projects the canonical data into real, typed columns; undeclared fields still live in the data JSON, so this block is purely additive.

output:
  fields:
    - { name: DEAL_ID, type: string }
    - { name: TOTAL_AMOUNT, type: number }
    - { name: PAID_INSTALLMENTS, type: integer }
    - { name: IS_ACTIVE, type: boolean }
    - { name: CREATED_AT_UTC, type: timestamp }
    - { name: UNITS_JSON, type: json }

Column types map to warehouse types:

`type`	Postgres	SQL Server
`string`	`text`	`nvarchar`
`integer`	`bigint`	`bigint`
`number`	`double precision`	`float`
`boolean`	`boolean`	`bit`
`timestamp`	`timestamptz`	`datetime2`
`json`	`jsonb`	`nvarchar(max)`

Typing is soft: a value that can't be cast lands as NULL, and the full record is always preserved in the data column — so a bad value never breaks a load. Column names must be plain identifiers (letters, digits, underscore). See Destinations for how the columns are created and evolved.

validate

output

On this page