Database replication

Every row, in the order it was committed.

Stream every insert, update, and delete from one database into another, in commit order, without a nightly dump and without writing a sync job.

Source · WAL
0/1A2F3C8ins 4471
0/1A2F3D1upd 4471
0/1A2F3E9del 2210
0/1A2F401ins 4472
0/1A2F418upd 4472
0/1A2F42Fupd 4471
0/1A2F447del 4472
Replica · applied
0/1A2F3C8ins 4471
0/1A2F3D1upd 4471
0/1A2F3E9del 2210
0/1A2F401ins 4472
0/1A2F418upd 4472
0/1A2F42Fupd 4471
0/1A2F447del 4472
Three ways ordering fails
Applied out of order
The update lands before the insert. The row does not exist yet, so the write is dropped and nobody logs it.
Changes are routed per key, so two writes to one row can never overtake each other.
Duplicated on retry
A network partition retries a batch. The insert runs twice and the replica now has two of a row the source has one of.
The sink upserts, so a retry lands the same row rather than a second one.
Missed delete
Incremental sync by updated_at never sees a deleted row, because the row is gone. The replica keeps it forever.
A delete is a change like any other and travels the same path as an insert.
The guarantee, stated
01
Commit order, per key
Every insert, update and delete becomes an ordered change row. Ordering is guaranteed within a key, which is the ordering that decides whether a row is correct.
02
Idempotent apply
The sink upserts. Applying the same change twice leaves the same row, so a retry after a partition is safe rather than lucky.
03
A slot Zipline owns
Zipline reads the source database’s replication stream from a slot it owns. It never polls and it never scans, so replication load does not grow with table size.
Connectors for this
ConnectorCategoryStatus
Postgres CDC Ships today
MySQL Database On the roadmap
SQL Server CDC Ships today
Oracle DB Database On the roadmap
CockroachDB Database On the roadmap
Different problem · one runtime

This page is orders.yaml.

orders.yaml YAML 
name: orders
connectors:
  - name: orders-db
    type: source/postgres
    host: "${POSTGRES_HOST}"
    port: "${POSTGRES_PORT}"
    database: "${POSTGRES_DATABASE}"
    user: "${POSTGRES_USER}"
    publication: "${POSTGRES_PUBLICATION}"
    slot_name: "${POSTGRES_SLOT_NAME}"
    secret_refs:
      - { field: password, ref: postgres-password }
  - name: mysql-out
    type: sink/mysql
    # required config: /connectors/mysql
routes:
  - from: orders-db
    to: [mysql-out]
zipline pipeline apply -f orders.yaml

Not every connector in this pair ships yet. The spec is what it will be · check the roadmap for dates.

All seven use cases Runtime handbook