Pipeline spec
Zipline has three places you set configuration: the pipeline spec (the YAML you zipline pipeline apply - what streams where), the runtime config (the node's own YAML file - how the runtime runs), and the console flags (how the optional web console starts). This page covers all three, in that order.
Top level
| Field | Type | Default | Req | Description |
|---|---|---|---|---|
| name | string | - | req | Pipeline id; unique, immutable, must not contain #. |
| commit_mode | on_delivery|on_buffer | on_buffer if any external sink, else on_delivery | opt | Source resume policy - when the source may advance. See Handbook → Delivery guarantees. |
| buffer_max_mb | int | 4096 | opt | Local backlog cap for the pipeline (MiB). |
| free_disk_reserve_mb | int | 10240 | opt | Node free-disk floor (MiB). |
| source | object | - | req | The source connector slot. |
| routes | list | - | req ≥1 | One or more delivery routes. |
Each source and each route's sink share the same slot shape:
| Field | Type | Default | Req | Description |
|---|---|---|---|---|
| type | string | - | req | mssql|postgres|kafka|file. A source's type is immutable per pipeline. |
| config | map | - | req* | Connector config (§Connectors). With profile set, this is just the override delta. |
| secret_refs | [{field, ref}] | - | opt | Point a config field-path at a named secret; values are never inlined. |
| placement | runtime|external | resolved | opt | Where the connector runs. Required when the connector supports both. See Handbook → Placement. |
| profile | string | - | opt | Inherit a reusable Profile (§Profiles). |
* not required when profile is set - the profile supplies the base config.
Each entry in routes:
| Field | Type | Default | Req | Description |
|---|---|---|---|---|
| name | string | - | req | Route/cursor id; unique in the pipeline, immutable, no #. |
| sink | object | - | req | Destination slot - same shape as source, above. |
| dlq | object | - | opt | Dead-letter sink. A DLQ may not declare its own dlq. |
| dlq_failure | halt_route|drop_and_continue | halt_route | opt | What happens if the DLQ itself fails. See Handbook → DLQ vs Skip. |
| retry.max | int | - | opt | Max delivery retries before the DLQ. |
| retry.backoff_ms | int | - | opt | Backoff between retries (ms). |
mapping: is not a pipeline-spec field. It's authored through the console or the API, never parsed from this YAML - see §Mapping below for its shape. Connectors
Four connectors ship today. Each has a fixed direction: source, sink, or both. Each connector's full configuration reference, prerequisites, and examples now live on its own page. Browse them at /connectors, or jump straight to Postgres, SQL Server, Kafka, or File.
Profiles & secrets
A Profile is a reusable, named connector configuration. A slot references it via profile: and supplies its override delta as the slot's own config:.
| Field | Type | Default | Req | Description |
|---|---|---|---|---|
| name | string | - | req | Profile id; referenced by a slot's profile: field. |
| type | mssql|postgres|kafka|file | - | req | Connector type this profile is for. |
| config | map | - | req | Base connector config every referencing slot inherits. |
| overridable | []string | - | opt | Field-paths a slot is allowed to override in its own config: delta. Everything else is fixed by the profile. |
| secret_refs | [{field, ref}] | - | opt | Same shape as a slot's secret_refs - attach secrets to the profile's base config. |
Secrets. secret_refs points a field-path - e.g. password, or sql_server.primary.password - at a named secret. The value itself is never written inline, in a profile or in a pipeline spec.
Mapping & transforms
mapping: is not a pipeline-spec field. It's authored per route through the console or the API - never in the YAML you apply. This section documents its shape so you know what you're setting, wherever you set it. | Field | Type | Default | Req | Description |
|---|---|---|---|---|
| unmatchedPolicy | passthrough|drop | passthrough | opt | Streams with no matching rule: pass through unchanged, or drop. |
| rules[].sourceStream | string | - | req | Stream name to match from the source. |
| rules[].targetStream | string | - | opt | Rename the stream, or fan multiple source streams into one target stream. |
| rules[].unlistedPolicy | passthrough|drop|error | - | opt | Columns not explicitly listed in this rule: pass through, drop, or fail with an error. |
| columns[].source | string | - | req | Source column name. |
| columns[].target | string | - | opt | Rename the column on delivery. The primary key column may not be renamed. |
| columns[].targetKind | bool|int|float|decimal|string|bytes|uuid|json|date|time|timestamp|timestamptz | - | opt | Retype the column on delivery. The primary key column may not be retyped. |
| columns[].targetSourceType | string | - | opt | Advisory hint for the sink's native column type. Informational only - doesn't change delivery behavior. |
drop is permanent. A dropped stream or column's data is not buffered, not queued, and not recoverable - it never leaves the source. Runtime config
The runtime's own YAML file - how ziplined itself runs on this node. Separate from any pipeline spec.
| Field | Type | Default | Req | Description |
|---|---|---|---|---|
| config_version | int | - | req ≥2 | Runtime config schema version this file was written for. |
| log_level | debug|info|warn|error | info | opt | Runtime log verbosity. |
| state_dir | path | ./data/state | opt | Where the runtime persists its local state. |
| connector | string | - | - | This node's connector identity. |
| control_listen | host:port | - | - | Address this node's control API listens on. |
| data_listen | host:port | - | - | Address this node's data plane listens on. |
| enroll_listen | host:port | control port +2 | opt | Enrollment listen address, derived from control_listen's port. |
| advertise_host | bare host | - | opt | Externally-reachable hostname this node advertises to peers - no port, no scheme. |
| wal.dir | path | ./data/wal | opt | The durable data location - where the runtime keeps its durable local queue. (setup default - the setup wizard writes this value; not defaulted if you hand-write the config) |
| pipeline.commit_mode | on_delivery|on_buffer | on_delivery | opt | Node-wide default commit_mode for pipelines that don't set their own. |
| pipeline.buffer_max_mb | int | 4096 | opt | Node-wide default for buffer_max_mb. |
| pipeline.free_disk_reserve_mb | int | 10240 | opt | Node-wide default for free_disk_reserve_mb. |
| runtime.max_pipelines | int | 256 | opt | Cap on concurrently bound pipelines this node will run; 0 = unlimited. |
| runtime.secret_dir | path | <wal.dir>/secrets | opt | Where the runtime reads locally-stored secret material referenced by secret_refs. |
| runtime.save_interval | duration | 1s | opt | How often the runtime checkpoints its local state to disk. |
| runtime.unbound_idle_ttl | duration | 0 (disabled) | opt | How long an unbound pipeline's state may sit idle before the runtime reclaims it. |
| runtime.in_flight_budget | auto|512MB|20%|off | auto | opt | In-flight memory cap for node protection. |
| transport.insecure | bool | false | opt | Disable transport security. Loopback-only - development use only. |
Fields with no default and no req/opt badge above (connector, control_listen, data_listen) are operator-set - there is no default to fall back to.
Console config
The optional web console is its own binary, started with its own flags - separate from both the pipeline spec and the runtime config.
| Flag | Env | Default | Description |
|---|---|---|---|
| --listen | - | :8080 | Console HTTP listen address. |
| --runtime | - | localhost:9090 | Address of the Zipline runtime this console connects to. |
| --enroll | - | control port +2 | Enrollment endpoint, derived from --runtime's port. |
| --gateway-cert-dir | - | ./data/gateway/gw-certs | Where the gateway's TLS certificates live. |
| --gateway-token | - | - | Shared token gating access between the gateway and the runtime. |
| --assets | - | - | Path to the console's static asset bundle. |
| --user-store-dir | - | ./data/gateway/users | Where console user accounts are stored. |
| --admin-user | ZIPLINE_CONSOLE_ADMIN_USER | - | Bootstrap admin username. |
| --admin-pass | ZIPLINE_CONSOLE_ADMIN_PASS | - | Bootstrap admin password. |
| --seed-users | ZIPLINE_CONSOLE_SEED_USERS | - | Seed additional accounts at boot. Format: user:pass:role;…. |
| --jwt-secret | ZIPLINE_CONSOLE_JWT_SECRET | auto-generated if empty | Signs console session tokens. If set, must be at least 32 bytes. |
| --metrics-db | ZIPLINE_CONSOLE_METRICS_DB | <user-store-dir>/metrics.db | Path to the console's metrics database. |
| --metrics-retention | ZIPLINE_CONSOLE_METRICS_RETENTION | 720h | How long metrics history is kept. |
| --metrics-histogram-interval | ZIPLINE_CONSOLE_METRICS_HISTOGRAM_INTERVAL | 10s | Bucket interval for histogram metrics. |
Next steps
Hi, I'm Pip - your guide to Zipline. Ask me about connecting a source, the apply → bind lifecycle, delivery guarantees, or day-2 ops, and I'll answer with diagrams, runnable commands, and links back to the Guide.