Define sources, sinks, and routing logic in YAML or Terraform. Commit, review, deploy. Pipelines evolve like code - diff, rollback, test in staging.
The entire pipeline lives in a single declarative config. Sources, sinks, routing rules, and delivery guarantees - all auditable in version control.
Push to main and Zipline converges. Every deploy is atomic and logged.
# zipline.yaml version: "2" connectors: - name: orders-postgres type: source/postgres dsn: "${POSTGRES_DSN}" tables: [orders.*, customers.profiles] exactly_once: true - name: analytics-kafka type: sink/kafka brokers: "${KAFKA_BROKERS}" topic_template: "zipline.{table}" exactly_once: true - name: warehouse-bigquery type: sink/bigquery project: "${GCP_PROJECT}" dataset: "analytics" write_mode: merge # Fan-out: one source, three sinks routes: - from: orders-postgres to: [analytics-kafka, warehouse-bigquery]
No separate pipeline UI to maintain. Your pipelines live where your code does.
Run zipline pipeline plan -f payments.yaml to preview what applying the spec would do, then zipline pipeline apply -f payments.yaml to record it. Roll back by reverting the commit and applying again.
zipline pipeline plan · zipline pipeline apply · git revert
Route one source to Kafka, BigQuery, and Redis simultaneously. Each sink gets its own exactly-once guarantee, independent retry policy, and lag metric. No duplicated connectors.
1 source → N sinks · exactly-once per sink
Run zipline context use staging to point the CLI at your staging runtime, then apply the same spec there. Identical connector config, isolated runtime, zero production risk.
zipline context use staging · zipline pipeline apply
The entire deploy lifecycle lives in the CLI: write, preview, apply, bind, watch. No dashboards, no point-and-click, no drift.
No wizard, no generator. A pipeline is a YAML file: connectors in, a route out. Write it, review it, commit it like any other code.
$ cat payments.yaml # payments.yaml - one source feeding one route name: payments connectors: - name: payments-db type: source/postgres dsn: "${POSTGRES_DSN}" tables: - payments.* - name: kafka-out type: sink/kafka brokers: "${KAFKA_BROKERS}" topic: cdc.payments routes: - from: payments-db to: [kafka-out]
zipline pipeline plan -f payments.yaml checks the spec against what is already applied and lists the routes it would create. Nothing is written. Safe to run in CI on every pull request.
$ zipline pipeline plan -f payments.yaml + pipeline payments + route kafka-out plan only. nothing was applied.
zipline pipeline apply -f payments.yaml registers the pipeline. It does not start it - nothing streams until you bind a runtime instance.
$ zipline pipeline apply -f payments.yaml applied payments the spec is recorded. nothing streams yet. bind a runtime: `zipline pipeline bind payments --instance <id>`
zipline pipeline bind payments --instance src-7f3a attaches the applied spec to a source instance. Binding starts in pending and settles to running a moment later - nothing streams until that settle happens.
$ zipline pipeline bind payments --instance src-7f3a binding src-7f3a to payments... bound payments <- src-7f3a state: pending
zipline pipeline status payments shows the pipeline's state, its bound instance, and every route. Once the bind settles, the route is running.
$ zipline pipeline status payments PIPELINE payments STATE running SOURCE payments-db (src-7f3a) ROUTE SINK STATE kafka-out sink/kafka running
Manage Zipline connectors alongside your cloud infrastructure. The official provider lets you declare pipelines in HCL, import existing ones, and apply changes through your existing Terraform workflow.
# terraform/pipelines.tf terraform { required_providers { zipline = { source = "hashicorp/zipline" version = "~> 1.0" } } } resource "zipline_connector" "orders" { name = "orders-postgres" type = "source/postgres" dsn = var.postgres_dsn tables = ["orders.*", "customers.profiles"] exactly_once = true environment = "prod" } resource "zipline_connector" "kafka_sink" { name = "analytics-kafka" type = "sink/kafka" brokers = var.kafka_brokers topic_template = "zipline.{table}" exactly_once = true environment = "prod" } resource "zipline_route" "orders_fanout" { from = zipline_connector.orders.id to = [zipline_connector.kafka_sink.id] }