Connectors

Kafka

sink
Available

Publish change events to Kafka topics.

Start streaming
DirectionSink
PlacementRuntime or external
DeliveryEffectively-once (idempotent)
CategoryBus

What it does

Kafka is a sink connector. It publishes each change event to Kafka topics, deriving the topic name from a prefix or template. Because the apply is idempotent, delivery is effectively-once.

How it works

  1. 1

    Receives ordered change events from any Zipline source.

  2. 2

    Derives a topic per stream from your prefix or template.

  3. 3

    Publishes each event to Kafka at your chosen acknowledgment level.

Use cases

  • Feed change events to event-driven microservices and stream processors.
  • Bridge database changes into an existing Kafka backbone.
  • Build materialized views or caches downstream of your OLTP.

Highlights

Flexible topic routing

Route per stream with a prefix, or a template using {stream}, {origin}, {prefix}.

Secure transport

TLS and SASL (PLAIN, SCRAM-SHA-256/512) supported.

Idempotent delivery

Effectively-once, safe to retry.

Prerequisites

  • Reachable brokers
  • A topic prefix or template

Configure it

YAML
sink:
  type: kafka
  placement: runtime
  config:
    brokers: ["kafka:9092"]
    topic_prefix: cdc.orders
    acks: all

sink config

FieldTypeDefaultReqDescription
brokers list - req Broker addresses to connect to.
topic_prefix string - req * Prefix used to derive the destination topic name.
topic_template string - opt Template for the destination topic name. Tokens: {stream}, {origin}, {prefix}.
acks all|1|0 all opt Producer acknowledgment level.
tls bool false opt Connect over TLS.
sasl_mechanism PLAIN|SCRAM-SHA-256|SCRAM-SHA-512 - opt SASL authentication mechanism.
sasl_username string - opt SASL username.
sasl_password string - opt secret SASL password.
batch_size int - opt Producer batch size.
batch_timeout duration 10ms opt Max time to wait before flushing a batch.

* not required when topic_template is set.

Pairs well with

Postgres SQL Server

Learn more

Start streaming with Kafka

Follow the first-pipeline guide to wire Kafka into a running pipeline, or jump to the full config reference above.

Start streaming All connectors