Incident · dual write

The row committed. The event never published.

Turn database writes into a stream on Kafka, Pub/Sub, Kinesis, NATS, Pulsar, or RabbitMQ, so a service reacts to a change without polling and without a dual write.

Database
waiting
agree
Event bus
waiting
  1. idle
  2. committed
  3. publish failed
  4. diverged
  5. fixed
14:02:11 Service writes the order row. The commit succeeds.
14:02:11 The publish to Kafka times out. The request has already returned.
14:02:12 Two systems disagree about whether the order exists. Neither is wrong.
14:09:40 Fulfilment has not seen the order. Support opens a ticket for a row that is right there.
Why the usual fixes leak
Retry the publish
The process can die between the commit and the retry, and the window is exactly where the failure lives.
Outbox table
Correct, and now you own a poller, a cleanup job, and a second source of truth to keep in step.
Two-phase commit
Your database and your broker have to agree to it. Most brokers do not, and the ones that do make you pay for it on every write.

Remediation: derive the event from the commit.

  1. Your service writes the row. That is the only write it makes.
  2. Zipline reads the committed change and publishes it, so the event exists because the write did.
  3. One source feeds as many routes as you declare, each with its own state.
Kafka ships Pub/Sub roadmap Kinesis roadmap NATS roadmap Pulsar roadmap RabbitMQ roadmap
Different problem · one runtime

This page is orders.yaml.

orders.yaml YAML 
name: orders
connectors:
  - name: orders-db
    type: source/postgres
    host: "${POSTGRES_HOST}"
    port: "${POSTGRES_PORT}"
    database: "${POSTGRES_DATABASE}"
    user: "${POSTGRES_USER}"
    publication: "${POSTGRES_PUBLICATION}"
    slot_name: "${POSTGRES_SLOT_NAME}"
    secret_refs:
      - { field: password, ref: postgres-password }
  - name: kafka-out
    type: sink/kafka
    brokers: "${KAFKA_BROKERS}"
    topic_prefix: "${KAFKA_TOPIC_PREFIX}"
routes:
  - from: orders-db
    to: [kafka-out]
zipline pipeline apply -f orders.yaml
All seven use cases Runtime handbook